Half-dead project saved! Thank you - Direct Communication

Picture (c) Przemyslaw Bielicki

In my previous post I shortly described the half-dead project in an over-waterfall company me and my team had to save three months before production. In this part I will show you how direct (instead of discrete) communication with customer or good Product Owner in general can help saving almost dead projects.

As I described earlier me and my team had a lot of problems with the systems we had to integrate - technical issues were present but they were not dominant, even taking into account the fact that the team was not experienced in .NET and T-SQL stuff.

Customer saved our project
After some time struggling with technical issues we finally were contacted by the person who knew how the product should behave. This person (let me call her Product Owner - PO) had all acceptance tests in her mind and knew exactly how to help us understand the product. PO was in fact our client - to be completely clear, the product we were delivering was either win-win or lose-lose so we had to cooperate and the success was our mutual goal.

PO was remotely available (although in the same time zone) but that was not a problem for us at all - we exchanged tons of emails and some phone calls and that was perfectly sufficient.

We were working like this for over two months and we delivered every single customer's request to the product. After the development phase our software had to pass the certification phase which, in fact, was couple-hour testing process in production. Our PO was present there and was helping other clients with testing (yes, there were other clients eventually using our product).

I could risk a claim that it was a huge success because we just found couple of minor issues that were not even concerning the software but insufficient or incomplete data in our production databases. Everybody is sure that what saved the project and what really caused it to be such success was direct contact and constant help from our PO. As I wrote in one of my previous posts, namely Customer Team Member - a way to winning together having customer as your team member is usually really good and efficient idea.

Fool's gold?
There are also (of course) problems with such attitude - the most "popular" one is the problem with actually having a customer. But even if you have a customer, how would you encourage him/her to be involved in the project (they usually aren't, especially at the beginning of the project)? It may depend on the actual contract i.e. how you define success or whether customer gain something by helping you - to know which contract best suits your current situation please refer to Peter Stevens' article: 10 Contracts for your next Agile Software Project.

It saved us - it can save You
To summarize, from my experience every time I worked with customer that was involved in the project I produced software with the highest business value (at that time). Every time I worked directly with involved customer I delivered the product that everybody treated as theirs, everybody was the owner - that's very cool feeling. This has also real business effect - satisfied customer is more likely to come back to your company. You gain credibility working together with your customers e.g. by showing them that you really work hard to meet their needs and to bring the best business value reducing costs. That usually pays off.

So, my final advice here for you would be to take some effort and time and look for a GOOD Product Owner. When you find her then start bombing her with emails, phone calls, whatever that can help your team deliver the product with highest possible business value.

What are your experiences with working directly with customers? Are they positive or negative? How would you improve thing in your project - could customer's involvement help?

Originally published on AgileSoftwareDevelopment.com

The Vendor Client Relationship (In Real World Situations)

ROTFL

Copied from {codesqueeze}.

Sysinternals (Process Explorer) saved my a**

After deploying our .NET system in production functionally speaking everything was OK. However, after couple of days we saw our processes consuming more than 80% of the CPU of some pretty powerful servers - and we were not computing any nuclear or medical killer stuff. Something was wrong - but what?

At this moment I recalled using free Sysinternals tool from Microsoft to see what's deep inside my operating system (in runtime, of course).

What really happened?
I started Process Explorer belonging to the Sysinternals suite, found my process and looked at it's runtime properties:

What I saw was the increasing number of TCP/IP connections (because used connections were not closed/returned to the pool properly). After some time spent on monitoring the process I noticed that this pattern is stable and each request caused the number of connections to database grow - until the process reached max and OS couldn't handle this process any more. Of course I fixed this issue wit one or two lines of code because it was a "little" problem (as usual) that appeared to have a big impact on the operating software.

I think wouldn't be able to discover this issue so quickly without using Sysinternals. Why? Because my colleagues were trying to investigate and fix this issue for some time before me. They didn't know Process Explorer tool :)

Direct communication as a half-dead project saver: When you start in an over-waterfall company

Picture courtesy of clagnut@flickr

At the end of year 2008 our team was given a project that was critical from the $$ point of view. We had to deliver the project by the end of March 2009 in order to avoid huge fees from our customer. The problem was that the project we got was being developed by different team in different location and the quality of the existing solution was at least questionable. It turned out that the whole old team left the company and our small commando squad had to save the day. I just have to mention that the company we all worked for was a pure-waterfall monster.

You know, agile books are fun to read, but how do you start it if you happen to be a developer in a pure waterfall company on an impossible project? I went through it and going to show what worked well for us. I will tell how we were able to get an impossible project apply the agile principles and survive.

A bit of a history
The company I used to work for sometime didn't know/use Agile methods. Everything there was pure waterfall i.e. there was a marketing that defined what it wanted (previously negotiated the contract i.e. feature list with the customer), then it passed the specification docs to product definition team that was in charge of designing the solution, finding ALL possible problems and answering ALL possible questions. Then after couple of months the technical specifications were passed to the development team that... started everything all over again. I mean that the development team usually found documents received from product definition team useless - it's not my opinion, rather my conclusion after talks with many friend-developers. Some docs like protocol specifications or conversion specs were REALLY useful but they were vast minority.

Anyway, when developers started working on the project it usually took just few classes to find some blocking issues that had not been predicted/designed in the docs. "How it's possible?" "They were supposed to find ALL possible problems and answer ALL possible questions". Well, they were supposed to but it was IMPOSSIBLE to do so. Even perfectly prepared and thoroughly checked documentation will not replace direct communication channels.

The project I'm going to describe was quite different - there was no documentation, no specification but more so there was not a single person understanding the problem. This was a real shocker for the company used to work in an "ordered" and "predictable" way.

The Project
In short we had to develop an unknown communication protocol to connect the two unknown existing systems. And the time we were given was pretty much estimated as we already knew all the guts and gears in this system. Ah - I forgot about some pretty "unimportant" detail: the test database for the test system was located around 1000 km from our desks and the VPN connection was not yet established. So, for couple of weeks we were just having a pretty "black box" environment - it was not so bad though because we had some time to learn how to compile given sources (still, this was the time when we were supposed to be developing software).

What did we have?
We had the documentation for the protocol we had to implement with examples (requests and responses). The documentation was not very useful, though because it was designed for people who knew this protocol before. This documentation was not explaining many concepts - it was assuming the reader's knowledge on quite high level - unfortunately we were GREEN.
We also had some stored procedures (in our unreachable database) that were supposed to contain business logic and they were supposed to work already - we were told that they require just small changes. Basically we had to develop a part that had to receive the requests, parse and process it and invoke already existing stored procedure with transformed parameters – we knew that we would also modify or create completely new stored procedures from scratch.

When we eventually got to know what to do we knew that we had to implement hotel availability (3 different request/responses) and booking (couple of quite complex transaction flows) messages. The availability part was supposed to be 100% ready (this way we could focus only on booking) - we just had to test it and confirm (probably do some small fixes).

To summarize: we had some germ of the C# code, fully functional stored procedures in T-SQL (MS SQL server) that required some significant rework and proprietary ASCII format specification. I also have to mention that neither of the team members was C# or T-SQL expert (I put it as delicate as possible).

Finally @Work
After some time of investigation we knew how the frontend and backend work but we still couldn't test the system because we had no database. When it was ready we established a test environment and we finally was able to integrate the whole system. We were able to start this machinery and inject some requests. We were even receiving some responses - but at this moment we had completely no idea whether these responses were correct or not.

Here is the moment we started communicating directly with our customer who started testing delivered system in test environment and giving us feedback. If you ask why we didn’t communicate with them earlier I would answer that we had nothing to show them as the test environment was unreachable for developers. If we couldn’t show them anything we weren’t able to get any feedback. That sucked…

I will leave the details of our communication process with customer for the next post.

Lessons learned so far
We took over the project in a rush and were just given a source codes from the previous team – which wasn’t the first class anyway (I mean the code). There was no unit tests, no continuous integration environment, no single product backlog but we started creating those artefacts. And you know what – this was the best decision we could make - we took as much as we could from the agile world in this project.

The main problem that was probably the root cause of all other issues was lack of communication. Before starting contacting directly with customer (who became our product owner, in fact) this was the worst part of the project because we not only didn’t know what to do but also we had no one who could verify our assumptions. Lack of communication was very visible and the deadlines were getting closer. We were really thinking that we will be late with such pace and information flow (or rather lack of it). We knew that with such level of uncertainty agile practices - especially open and effective communication - could help us saving our jobs.

In the next part I will explain how we solved our communication problems, how agile practices helped keep the project on track and keep the number of bugs close to zero as well as how the whole project finished. Stay tuned!

Feedback
Do you recall such situation in your own project(s)? What was the solution that worked for your team? I'm really curious your opinions and thoughts.

Originally published on AgileSoftwareDevelopment.com

Niezły kawałek muzyki

This post will be in Polish but don't worry it won't be about Java or technology at all. Anyway, if you want to listen to some really good music (one of the artist - Piotr Czerski - is my friend from university and some really wasting parties ;) visit this website: http://towary.art.pl.

Mam nadzieję, że Groszek się nie obrazi jeśli skopiuję post z jego bloga - nic lepszego i tak nie wymyślę na temat nowiuteńkiej płyty grupy "Towary Zastępcze" pt. "Dolne Miasto" bo o tym będzie ten wpis. Przesłuchałem tę płytę już dzisiaj dwa razy i jestem bardzo kontent.

A oto co do powiedzenia ma na temat płyty Groszek (ze wszystkim się w stu procentach zgadzam):

Na płytę składa się dwanaście zaśpiewanych wierszy, opowiadających strzępy tragicznej historii dziewczyny z Grand Hotelu. Szczególną uwagę zwraca strona internetowa towary.art.pl promująca album, na której możemy wcielić się w rolę detektywa prowadzącego śledztwo. Oglądając kolejne fragmenty video i podążając właściwym tropem, z pewnością uda nam się zdobyć odpowiedzi na trzy pytania, co pozwoli ściągnąć album w formacie mp3 za darmo z sieci. Wytrwali mogą również odnaleźć prawdziwego mordercę i dotrzeć do kolejnych bonusów.

Album dostępny jest również w bardziej namacalnej postaci. W atrakcyjnej cenie można nabyć akta sprawy, w których jednym z dowodów jest płyta. Doskonały pomysł na nietuzinkowy prezent i wsparcie twórczości artystów. Gorąco polecam!

Ja też polecam - kawał dobrej roboty i muzyki!

Java Concurrency Bugs/Programmer's Errors

Alex Miller has been posting very interesting series on Java concurrency bugs (or pitfalls programmers could fall into). Especially interesting is the last post: Inconsistent synchronization. This post is most probably inspired by the problems Alex discovered in Hibernate statistics and I am pretty proud I was able to find possible concurrency issues reading the code. Anyway, I was not able to fully and correctly solve those issues.

The conclusion for me is that I thought about "magic" solution instead of the correct one i.e. I suggested that declaring a integer field as volatile would solve the issue. In fact it would but the "correct" solution to this problem is using AtomicInteger from java.util.concurrent.atomic package.

Another conclusion is to remember to keep bombarding your Java code with FindBugs - it can find many many Java issues in your code (not only related to concurrency).

The rest of the Alex's series (so far) is here:
Java Concurrency Bugs #4: ConcurrentModificationException
Java Concurrency Bugs #3 - atomic + atomic != atomic
Java Concurrency Bugs #2 - what to synchronize on
Java Concurrency Bugs #1 - mutable statics

Enjoy you Java Concurrency Issues ;)

Fun soft: regex checker

I'm not a regular expressions expert but I use them often - as probably most of software developers. Before using them I want to be sure my regexes are written correctly and they will serve their purposes. I wrote very small command-line app for doing the check for me :) As a first input user provides the regex to check, then the user can type all texts to check against previously given regex until she types 'q' or 'exit' (for which program exits). Enjoy!

package com.bielu;

import java.io.Console;

public class RegexTester {
  public static void main(String[] args) {
    Console in = System.console();
    if (in == null) {
      System.err.println("Cannot obtain system console - exiting application");
      return;
    }

    System.out.print("Enter regex to test: ");
    String regex = in.readLine();
 
    while (true) {
      System.out.printf("Type value to match with given regex '%s': ", regex);
      String string = in.readLine();
  
      if (string.equalsIgnoreCase("q") || string.equalsIgnoreCase("exit")) {
        break;
      }
  
      if (string.matches(regex)) {
        System.out.printf("'%s' matches '%s' regex\n", string, regex);
      } else {
        System.out.printf("'%s' does NOT match '%s' regex\n", string, regex);
      }
    }
  }
}

Certified Scrum Master Training in Gdańsk

I hope you already know that a cool company that is definitely one generation ahead than their competitors (at least as far as the way they work is concerned) called Spartez organizes the first CSM Training in northern Poland, namely in Gdańsk. The training will be led by famous Henrik Kniberg from whose book my adventure with real Scrum began. It is really a PITTY I will not be there :(

But anyway - if you live somewhere in the central, eastern or northern Europe this training is definitely worth taking. I took CSM course in Zurich last week led by Angela Druckman and am very satisfied. This training is really worth it's price (or even more because I think the training is quite cheap).

Don't waste your time and sign up for this CSM session.

The power of the final keyword - updated

Picture courtesy of Okko Pyykkö@flickr

These two articles of Alex Miller:

• A final bit of advice


• JMM and final field freeze

clearly explain why using final fields in concurrent programs is extremely powerful and safe (thread-safe).

Take a look a this simple example:

public class SomeClass implements Runnable {  
    private final LinkedHashMap map;
      
    public SomeClass(LinkedHashMap copiedMap) {
        map = copiedMap;
        preProcessMap();
    }  
     
    private void preProcessMap() {
        //some non-trivial and not-so-short computations
    }

    // A read method  
    public Object get(Object key) {  
        return map.get(key);  
    }  
     
    public void run() {
        //assume that in this method only 
        //unsynchronized read operations on map may occur
        //any unsynchronized write operation is not thread-safe
    }
    // etc...  
  }

public class Main {
  public static void main(String[] args) {
      LinkedHashMap map = // map creation
      // map initialization
      new Thread(new SomeClass(map)).start();
      // etc.
  }
}

UPDATE:
I preserved the original version of the code but after some thoughts and thanks to some comments (thank you dear readers) I conclude that this example is not 100% correct. Before Thread.start() could be started the object has to be fully initialized and initialization will be completed only AFTER the SomeClass object is created. This way final keyword doesn't add any value there. On the other hand I meant this piece of code:

public class Main {
    public static void main(String[] args) {
      LinkedHashMap map = // map creation
      // map initialization
      final ExecutorService service = Executors.newFixedThreadPool(1);
      service.submit(new SomeClass(map));
      // etc.
    }
}

In such case I'm pretty sure you cannot assume that e.g. second processor (in a multi-core platform) will wait before executing newly scheduled task until the object of class SomeClass is fully initialized, right? Correct me if I'm wrong because I feel have some mess in my brain now ;)
End of UPDATE

Whatever happens in the preProcessMap() method you can be sure that LinkedHashMap will be copied and preprocessed BEFORE starting the run() method by the new thread. This way any READ (get) operation on this map in all threads will be safe and will operate on completely and correctly initialized map object.

If only you remove final keyword from LinkedHashMap declaration you cannot be sure our previous assumption. It may happen that the thread will be started BEFORE copying the map and if in run() method you process this map in any way you may get NPE.

Let me use a bottle metaphor :) For me final keyword is like the guarantee that you closed the bottle of whisky before passing it to your crazy friend who likes juggling them. If you ensure that the bottle is closed (final) before passing it to your friend you can be sure he will not spill any of the precious whisky on the floor. On the other hand if you start passing the bottle to him while trying to close it you're not sure whether it's fully closed while he starts juggling it or not. It may be closed 100% of time when you meet in your place but when you go to your other friend's party he may spill very expensive whisky and break the bottle in the same time (test and production environments metaphor this time).
Conclusion? Close the bottle before giving it to anybody else.

I will not be original and simply copy Alex Miller's advice:

Final fields are ever so important and useful. Whenever possible, strive to make your fields final. One tip for this is to enable a Save Action in Eclipse (or your IDE of choice) that automatically attempts to make fields final if possible so you can’t forget!

Why Object-Oriented languages rock?

I loved coming back from using T-SQL to C# or Java like I loved the view from the picture (view from the l'Esterel's mountains towards Cannes). Of course, SQL-like languages are very powerful and when you do some complicated computations that need many requests to the underlying relational database, this choice may be the right one. At least, as far as the performance is concerned.

If we leave performance concerns behind we still have the language itself. So, what did I miss in this language? I was thinking of enumerating things that I didn't like but after some thinking I conclude that I just missed object-orientation in general.

But what it means? Let's see... I was not able to use any design pattern (e.g. Factory, Strategy or Decorator) and I needed them very badly! Instead of using design patterns I had to add many ugly ifs.

Another thing I missed was abstraction i.e. I wasn't able to create abstract stored procedure and then implement specialized subprocedures. In T-SQL everything you can do is to add some more ifs :) This language is strongly data-oriented which is maybe good but it's not sufficient IMHO.

Yes - it's very vague but it doesn't make sense to compare stored procedures with OO constructs. T-SQL is just yet another language I've learnt and used after many years of writing Java/C++ code and I just wanted to express my feelings.

I don't know what are perspectives for such SQL languages like T-SQL or PL/SQL but I think they should evolve in an object-oriented way. It would be much easier to write stored procedures/classes in e.g. Java, test them and then deploy on the server. This would be a win-win situation. We would gain performance by having stored code invoked directly on the SQL server while we will still be able to use OO constructs and profit from unit tests, design patterns, etc.

Maybe there already are such languages but I just don't know them (most of them are procedural languages) - if you know some let me know.