Monday, December 27, 2010

Agile and the Importance of Estimating and RE-ESTIMATING

I've been developing software since 1984.

The development tools and syntax of applications have changed over the years, but one thing that has not changed is the need to try and figure out how long a project will take to complete, price it, and how you're doing once you get rolling.

Agile gives us unique benefits by breaking down a large project into small, manageable pieces.  More importantly, it allows us to start on a project with a target on providing business value soon after starting out of the gate.

Let's talk specifically about the ESTIMATING part of Agile Development.

First, a few key words;

Iteration.  An iteration is a small cycle of work.  Meetings take place, software is developed, developer work is coded, testing is done each step of the way, peer code review happens ,"user review" is completed and at the end of a small cycle (from 1 week to several months depending on the environment), you have a releasable unit of work.

Velocity. Velocity is a calculation of how much "work" is actually DONE on a project.
The reality is that for development work to be productive, it requires teams to WORK TOGETHER, not only amongst themselves and their respective "user representative", but also often with other departments.

Knowing that meetings, problems, reviews, testing, etc. are part of the natural process of Agile, your team will eventually have a Velocity at which you know they can work.

At the start of a project or with new Agile developers, the velocity may be a bit slower while they get used to the new way of doing things.  With an experienced team, the velocity can increase significantly.  There are other considerations as well such as holidays, work-life balance, company culture, etc. but you will find a "velocity per developer" for your organization in just a few iterations.

An example would be a Velocity per developer of  6 hours / day.  This would mean that if you had 6 developers you could produce 6 X 6 hours of work in a day, or 36 hours per day.

Original Estimate.  When your team first puts together the "estimates" of what the release will take to complete, you will add them up.The result is a good number to start with.

This is where the waters seem to get murky.  Some Agile purists say that you should not estimate many releases in advance.

Being in business since 1984,I have RARELY seen a project get off the ground without SOME idea of what kind of costs and time frame a project will take to complete. Some kind of estimate will take place (whether it be formal or not).  If a 4 employee company knew the new development project would take 10 years and 10 million dollars to complete, no matter how efficient it is, it won't happen.  Be real.  Everything has it's place.

Revised Estimate.  This to me is where the REAL POWER of Agile Development comes into play for a manager of a software development team or for the owner of a company.

As each developer does a small piece of work (story to some), they record the time they work on it in whatever increments work for you and the tools you have.

Remember.. Developers are people, not robots!

The recording of the time has and always will be a a necessary evil for bookkeeping and accounting types, but not the real power of developing in Agile comes from the REMAINING ESTIMATE.

Here's the math and the beauty of it.

Let's say you have a project X, with an original estimate of 1000 hours to complete over 6 months. Someone in their wisdom has decided this is the number (however obtained).  It could be an experienced Agile Team or a manager who made the decision based on marketing info or something a BA came up with, or just the wishful thinking of a business owner.  Either way, it's a start.

You have a team of 6 Developers and a User Representative or Subject Matter Expert(SME).  The User Representative is not optional in Agile or things can go very bad.

For the purpose of this exercise, let's assume you will break down the Project into 6 releases (or iterations).  Each iteration will be 1 month long.

I am currently in an environment where we have WEEKLY releases for a public facing website with several hundred thousand users.

Your cycle will change based on your company's needs and environment and the nature of the work you do.

Careful! I can warn you that weekly releases are difficult.  Where I currently am, we are on weekly releases for some very specific business reasons. I'm personally looking forward to a more relaxed schedule of a new Iteration or Release every two week to a month.  The shocking thing is that under the Waterfall method of Development, we would have planned for a release possibly every 6 months to a year.  Amazing !

At the end of two weeks, the developers have been doing work and re-estimating constantly.  If a Story (piece of work) was estimated at 5 hours, and the developer did 4 hours,they record 4 hours work.  The key here is that they RE-ESTIMATE Immediately.  If the developer thinks it will take another 3 hours to finish this task, they put a NEW Estimate of what is left to do.  This should be as realistic as possible including peer review, users testing and preproduction testing.

Some tasks may have less remaining work than expected when being worked on.  The key here is to be as realistic as possible.

Here is the beauty.  At the end of two weeks, you will see two things.  The first is what your VELOCITY is per Developer (how much productive "work" is done during a day).  My PREFERENCE is to do this in teams.  The whole idea of Agile Development is TEAM WORK.  Making 1 developer accountable for something has MANY business flaws which I can get into later (another post).  For now, let's just stick with keeping things in Teams.  At the end of a few weeks, you will know that for instance your teams' Current Velocity is X hours/day.   This lets you know as a rule how your team is doing and also let's you do the next and most important part.

As the iteration continues, you will see the REMAINING number of hours for the Iteration in comparison to the Original Estimate.  Based on the historical Velocity so far and the 'new' remaining estimate of hours left, a chart can easily be created(called a Burn Down Char) to show what the Original Estimated completion date was in comparison to the "revised" estimated completion date and can give you a new target date based on current information.

This is the cool part.  You now have several choices! 

Option 1 - Extend your final completion date
Option 2 - Add Developer Resources (usually you can see this as add X hours of Velocity /day on a chart)
Option 3 - Remove some planned work from the upcoming Release.

The most amazing thing is that you know how you are doing based on your plans, EARLY on in the development life cycle.

You are not waiting until a month before the 6 month project is over to realize you have no chance of making it in time or on budget.

There are some challenges for sure, but well worth it when you see that you produce nearly bug free code on-time and on budget.  Amazing stuff.