Earned Value, Velocity, and Predictability in Software Development

Perhaps one of the least well-understood concepts in agile project management is velocity. Sure, it’s pretty much what it sounds like – the speed at which the team is moving – but having just that high level understanding of velocity undercuts its value and power in bringing predictability to a project.

Be forewarned that this post is going to be a bit of a long and winding read. I’m going to take a brief trip back in time, to when waterfall methodology was the norm, long before we conceptually broke software projects down into cohesive units of delivered business functionality (user stories).

Of course, the need to estimate effort and track progress did not begin with the advent of agile methods. Our track record in accurate estimation of software development projects has been pretty dismal over the years – but there have been individuals and organizations who have done it quite well. And – not surprisingly – the techniques that were effective then bear striking similarities to agile concepts like story points and velocity. So, as in the spirit of standing on the shoulders of the giants who have come before us, let’s take a quick look at something called earned value.

Earned Value

Simply put, earned value is a measure of completed work. If you set out in the morning to set 100 fence posts in the ground, and by noon you have 60 of them standing, you have an earned value of 60 set fence posts. On the flip side of the coin, you have an “estimate to complete” of 40 fence posts. Simple. Straightforward.

But we all know that software development is not so simple and straightforward. Tasks are far more nebulous than setting fence posts, and vary wildly in size and complexity. So, historically, we have estimated software development work at the granularity of some sort of task (ideally, one that is small enough to conceive pretty fully when estimating), using estimated hours of effort as our measuring stick.

Let’s say we are writing a simple Set class in a brand new programming language, and have the following tasks identified:

  • Create data structure and default constructor – 2 hours
  • Implement ‘add’ method (including protection against duplicate members) – 4 hours
  • Implement ‘remove’ method – 1 hour
  • Implement ‘contains’ method – 1 hour
  • Implement ‘clone’ method – 3 hours

From this list of task level estimates, we arrive at an overall estimate of 11 hours to build the class.

Now, let’s say we roll up our sleeves, fire up the IDE, and complete the first two tasks. We now have a bare bones Set class, with an appropriate internal data structure and the ability to add objects to it. Let’s further assume that it took us 4 hours to complete that work. With this information, we know the following regarding the completed work:

  • Estimated effort: 6 hours (2 plus 4)
  • Actual effort: 4 hours
  • Earned value: 6 hours

But wait. We earned 6 hours? But we only worked 4. That’s right. What we do in earned value analysis is use an hour of estimate as a sort of currency for measuring progress. If we believe a task would take 6 hours, its value is 6 hours – and when it’s complete, we earn all that value – whether we ‘burned’ (spent) it or not. (This technique is sometimes referred to as ‘earned vs burned’ analysis).

On large projects that take place over months or years and involve many people, it is the aggregation of these simple calculations that is so valuable. For as we collect more and more data of this sort, the law of large numbers tells us that we can start to trust an emerging trend. Any single task may have been over- or under-estimated, perhaps grossly – but when hundreds of tasks are completed and we gather this data, we can be pretty sure that our efficiency – the ratio of hours burned versus hours earned – will remain pretty constant.

In order to identify and analyze trends, charts are effective tools. So project teams often use a burn-up chart like the one below, which map several important measurements over time:

  • the number of hours the team had planned to work
  • the number of hours the team actually worked
  • the number of hours the team earned (by completing tasks)

EarnedValueChart

There is a natural slope to these values when mapped over time, as you can see in the example. They start at zero, and climb steadily toward an intersection point with another line – one that represents the total scope of the project (sum of all estimated tasks) – at which point the project is complete.

In the example, I’ve shown how that top line – total estimated effort – can change over time, representing work being added to or removed from project scope.

As a project progresses, the actual and earned lines continue their emergence, onward and upward, until that magical day when the line really does intersect with the project scope line, and the project team gets to go to Disney World.

The power of a chart like this lies in its predictive capabilities. In our example, the actual effort line lags below planned effort, because the team didn’t spend as many hours on the project as we expected it to in the early going (presumably, the team was short-handed). Unless we increase the allocation of human capital to the team going forward, we’ll never make this up.

However, there is good news. The team is earning value at a higher rate than it is expending effort – which is to say, it’s beating estimates on most of its development tasks. Even more telling is the slope of the earned value line – it is increasing more rapidly than the actual effort line – meaning that this unexpected efficiency wasn’t a one time event, but seems to be pretty consistent (each “win” in terms of finishing work below schedule will increase the distance between the actual and earned lines). So while it’s pretty early to be celebrating, it does look like this team is “on pace” to finish the project below estimated effort (and, therefore, below budgeted cost) – though becase of the slow start in staffing, it may not finish ahead of schedule).

Story Points and Velocity

So, what does all this earned value stuff have to do with modern agile measurements like story points and velocity? Quite a lot, actually.

With the advent of agile, most teams no longer estimate in terms of tasks and hours. They do it in user stories and story points.

While there are many characteristics of a user story that distinguish it from a mere task, from an estimating and measurement perspective, tasks and stories are nearly indistinguishable – both are a fixed, well-defined unit of software development work, to which we can fairly reliably ascribe some level of difficulty / complexity / effort.

The unit of estimation is the other key difference. Not hours – story points. Abstract valuations of the relative size and/or complexity of a story, as compared to its peers in our project’s backlog. You’ll read over and over again in the literature that a story point is not(!) a representation of time, or of effort – merely one of scope or complexity. It bears relation to effort only through another agile measure – velocity.

Velocity is a measure, usually expressed as an average, of the number of story points a team completes (earns!) in a given sprint. If a team completes three stories in a sprint – with assigned story points of 1, 3 and 8, the team’s velocity for that sprint is 12.

So that’s how velocity is calculated. But what does it mean? Why do we bother calculating it every sprint?

Because velocity helps us predict the future, just like earned value does. In fact, in rough terms, velocity is a numerical representation of the slope of the progress lines on the chart we examined in our look at earned value.

There are actually two types of velocity – planned (or, if you prefer, estimated) velocity, and actual velocity. At the outset of a project, it’s a worthwhile exercise to make some prediction of how “fast” the team will go. And that predicted speed is likely to change over the course of the project, as the team size (typically) grows during its “ramp-up” period, and then decreases in size as the project nears completion. Using a model of planned velocity, and knowing the total number of story points in the backlog, project managers can formulate a model for answering those ultimate questions – when do we think we’ll be done, and how much is it going to cost?

Of course, any answers generated by such a model before we start measuring actual progress are likely to be wildly inaccurate. But as we begin work, and measure our actual velocity at the end of every sprint, we can see how the team is performing in comparison with our initial predictions (just as we compared earned value to plan, over time). The farther we get into a project, the more reliable the trend lines. If the team has had an actual velocity of 15 points for the last 4 sprints, it’s pretty likely to keep going at that pace (assuming we don’t add or remove any team members) – even if we initially *thought* that the team would be earning 20 points per sprint.

An agile burn-down chart, which can be generated manually, or automatically in most agile project management tools, is essentially the same animal as an earned-vs-burned effort chart – it’s a tool for helping all of the project stakeholders understand not only “where we are” on a project, but where we’re likely to be in 2, 3 or 10 sprints.

Before concluding, I’ll reiterate that story points are not in any way related to a unit of effort. You’ll see that point made – clearly and sometimes vehemently – in most of the agile literature out there. And I’m not suggesting here that it is. As I mentioned above, story points are related to effort only through velocity (whether anticipated or measured). That, however, is not the primary point of this post. My point is that in today’s software development world, as well as project management techniques used before the Great Agile Upheaval, the key to a team’s ability to predict future outcomes is best served by accurately measuring the present, and aggregating those measurements with its recent past – and capturing trends. For whether it’s the slope of the ‘earned’ line, or average measured velocity, you can’t find a better predictor of near-term future performance than near-term past performance.

Technical Knowledge is Hierarchical (with apologies to A. Maslow)

With the shortage of quality talent in software engineering continuing to worsen, there have sprung up a multitude of programs, both online and classroom-based, that claim to be able to turn a tech novice into a functioning, effective, hire-worthy software developer in as little as 6 months. Granted, these courses generally focus on a very specific technology (PHP, or Ruby on Rails, etc.). But can they really deliver? Can someone who wouldn’t know a for-loop from a Fruit Loop really emerge from a 6-month course ready to set the Rails world on fire?

I root for the underdog (Cubs fan) so I’d like to believe it can happen. And as a hiring manager in a consulting firm, I feel the pinch of the talent shortage as much as anyone. Yet, as much as it kills me to say this my answer is, in the vast majority of cases, no. It just ain’t that easy.

Shifting gears a bit, one of the things that continually amazes me about many IT managers (and, sadly, many accomplished software developers) is widespread underestimation of the ability of professionals in this field to quickly pick up new skills and concepts. Whether it’s new programming languages, new design or architectural patterns, or new communication protocols like Web sockets, far too many people in this field have an inner fear of the new. This fear, more than any real or imagined challenges in learning, is what stagnates far too many people in our field. Be ready for the challenge, my compadres – it just ain’t that hard.

See what I did there? :) Contradicted myself. How can it be both ways? As you, astute reader, have probably guessed, I’m about to use that apparent conflict to make a point.

For probably 15 years or so, I have carried around in my head a visual image representing the various skills and bits of knowledge required to be an effective practitioner of software development. It’s heavily influenced by (okay, almost a carbon copy of) a pyramidal diagram that anyone with a reasonably well-rounded education has seen at least once: Maslow’s hierarchy of needs.

In that mental model, Maslow places basic, vital human needs – food, water, elimination – at the base of the pyramid. These are fundamental – without them, we quickly expire, and won’t continue to need anything.

In a succession of layers, moving up the pyramid, we see needs that are less critical to survival, but increasingly focused on what most of us consider to be a meaningful, healthy and successful life. Things like love and friendship. Achievement and mastery. Independence. And finally, at the top, this concept he calls self-actualization – the act of becoming all you can, and were meant to, become.

What if we were to start thinking about software engineering skills and knowledge in a similar manner? Are there basic, fundamental skills that we all must have? Of course there are. Language syntax. Compilation and the ability to understand compile errors. Variables, and flow of execution, and loops and conditional statements. If you don’t “get” these things, your career in this field will suffer the same fate as a human being without food and water. Ist kaput.

CompSciHierarchyOfSkills

But once we do master those things, something wonderful and powerful begins to happen. Higher-order skills and concepts – things like object orientation, and effective relational database design – things that would have been nearly impossible to understand absent those fundamentals – are suddenly attainable. Not only attainable, but almost natural. The human brain has a wonderful way of building on one set of knowledge to form broader, more complex forms of knowledge. (This is how we’ve evolved from flint-tipped spears to intercontinental ballistic missiles over the years. The effect is real.)

And once we’ve mastered knowledge at an increased level of complexity and specialization, we’ve enabled ourselves to achieve still more. It’s a bit like starting down a big hill on a bicycle. The farther you’ve traveled, the faster you find yourself moving.

So – back to this earlier bit about learning a new programming language, and how it ain’t that hard. Think about it in the mental model of the hierarchy of skills. Once you’ve learned any programming language, learning a new language is largely something happening within the same hierarchical level – you’re not moving up the pyramid, you’re moving sideways. An if-else block is an if-else block, whether it’s in Java, Ruby, C# or Visual Basic. You don’t have to relearn what conditional logic does – you just have to discover what it looks like in the new language. Granted, there are always a few unique twists and turns in any language – new built-in features and common frameworks, fancy new operators (I’m looking at you, Ruby), etc. But these are relatively small and incremental, and quite easily learned while actively producing quality, useful code. Classroom time not required.

What, then, about the “be a programmer in 6 months” courses? Why don’t I believe in them? Because that bottom layer of the pyramid – the Fundamentals – is huge in our discipline. The amount of knowledge one needs to amass in order to be reasonably productive in this field is quite significant. The diagram I’ve included here is just a start, and in no way all-inclusive.

Courses that rush someone through will, out of necessity, spend far too little time on the fundamentals. If you’re learning Ruby on Rails in six months, the breadth of knowledge you’ll be presented with is staggering. You’re learning at every level of the pyramid. Where in this mental model would you place an ORM like ActiveRecord? Before you answer, remember that ActiveRecord presupposes knowledge of SQL, which presupposes knowledge of relational database theory, and so on, and so on…

So what happens in these courses is that a person emerges with some, but not nearly all of the fundamental skills an effective developer will need to have in the real world. I envision it as a tall, narrow slice of the pyramid. A skyscraper with a very small footprint. And that’s not a model for stability.

That’s why I don’t believe it’s possible to create an effective professional software developer of any ilk in 6 months. It just ain’t that easy.

One last thought before I wrap this one up. When I look at this diagram, and see the names of the various skills sort of suspended in space, I envision the presence of something else – something invisible, but which gives us the fabric on which all this technical stuff can be pinned. Things like passion. Intellect. Determination. Curiosity. Humility. Pride in a job well done. With all those things in place, pinning skills on the board becomes surprisingly easy. Without them, it can all fall like a house of cards.

Well, there it is. I’ve finally published that particular bit of content I’ve been carrying around in my head for so long. It’s a model that has served me well over the years, and one that has helped Redpoint identify, hire, and grow an incredibly talented and powerful team of professionals.