Agile Successes and Challenges...Part I

I changed jobs a few months ago. I moved from a company that was invested in the agile development philosophy and scrum practices to a smaller company that was just starting to build its development team from a team of one to a team of five. Of the five, I am the only one with a significant amount of experience working in a truly agile, highly successful, team-based environment. We've been trying to transition to a similar type of environment at my new job for a few months now with varying degrees of success.

I thought I would share my observations and experiences on the successes and challenges along the way. In many cases, there are multiple layers to each success and challenge that we've encountered. I will only hit on the highlights in this post.

Successes

Building a definition of done that promotes openness and collaboration

First off, let me say that our definition of done is very fluid. We have an ideal definition of done and we have our current definition of done: the ideal definition of done is where we want to be; a definition that incorporates all of those things that we'd like to adopt as our team continues to mature and evolve; our current definition is where we are at today.

Our current definition of done includes things such as story point estimation, task planning, implementation, and various stages of testing and deployments. The part I want to focus on here are the early stages. It is in the story point estimation and task planning stages that lay the foundation for a cohesive and high performing team.

Story point estimation

One of the first stages a story goes through after it's been added to the backlog is story point estimation. Our only prerequisite for pointing a story is for the story to be clearly written with “good enough” acceptance criteria. We usually try to get through a 6-8 stories during story point estimations: not so many that we're sitting in a meeting room for an hour, but enough for the product owner to be able to groom the backlog.

We have adopted planning poker for our estimation sessions; the choice for planning poker is key as it promotes brief discussions that bring to light differing points of view when there isn't consensus. These discussions are meant to be very brief: the owner of the largest estimate explains why he/she thinks the story deserves the higher point value; the owner of the smallest estimate then explains his/her reasoning for the lower point value. Taking each persons' explanation into consideration, the team then re-estimates the story until arriving at a consensus. In those cases where we've played a few rounds of planning poker on the same story and still cannot reach consensus, we table the story and come back to it later, so the estimation session doesn't grind to a halt; this is usually an indicator that the story may need to be refined further or the team needs to get more clarity on the story from the product owner. In my experience, the situation where the team cannot reach consensus is very rare.

At the end of the story point estimation, the team has a pretty good idea what is being requested and has had a few (quick) discussions that has everyone on the same page and will help move the story into the task planning stage.

Task Planning

The task planning stage comes after a story has a story point estimate associated with it. We usually try to keep these planning sessions scoped to one story at a time to avoid having too much work in progress at once. The task planning stage requires the team to come together again to discuss the story; this time at a much more granular level. The goal of this meeting is to have a road map of the individual development/coding tasks and test cases that need to be completed to meet the story's acceptance criteria.

It is crucial to keep the task planning meetings as concise as possible. Not only are you taking time away from value-adding activities (i.e. coding and testing), but the productivity of a task planning meeting is inversely proportional to the length of the meeting: the longer the meeting goes, the more fatigued/distracted people will become, so each team member taking the time to review the story and do some initial preparation prior to the meeting is crucial to get in and out of the task planning as quickly as possible while still capturing the necessary artifacts.

In our case, task planning usually starts with one person (not always the same person) discussing what he/she found in their prep work. Ideally, this person would have a good start on specific implementation items that need to be done in order for the story to be completed. In our case, this usually works best when discussed in conjunction with a simple whiteboard drawing to illustrate how the different classes and/or components work together. This initial whiteboard drawing helps to drive the rest of the meeting: specific coding tasks result from the whiteboard; other team members will usually have ideas and input to make the design better get incorporated into the development tasks; and eventually identifying the acceptance tests that will prove the story is complete.

Besides having a set of granular, clearly defined set of tasks and test cases that need to be done, the task planning also results in the entire team having a walking around knowledge of how a piece of functionality works. In the best cases, the time spent to go through the formal task planning stage pays off multiple times over; the actual coding and implementation becomes a formality and is more of an assembly line process and usually goes pretty quickly (especially when multiple team members can work on tasks in parallel) since you don't have all of the trial and error that can sometimes result when you're working in isolation (e.g. start hacking out a solution that you think will work, get about 4 hours into it before realizing a show stopper in your design, starting over, and doing this a few more times before finally arriving at code that is ready for testing).

Arguments get made about the task planning session of having the overhead of burning additional man hours since the whole team is involved as opposed to one person working on the story. However, if you weigh those additional man hours against the overhead of going down a few paths before eventually arriving at a solution, the amount of time spent working on the implementation of a story is pretty much the same if not reduced since there isn't all the churn of start-stop-reassess-the-problem-start-over cycle. Couple this with the knowledge sharing and collaboration that took place in the task planning session, and you're coming out ahead in the long run because you have a more cohesive solution that anyone on the team will be able to maintain once it has been released into production. For example, with the increased collaboration and knowledge sharing you don't have knowledge silos where Joe becomes the printer module guy and any problem that comes up has to wait until Joe gets back from vacation next week (i.e. you start to get more organization agility because now the organization is not held captive to any one person in a particular area of the application).

In my experience, the variance associated with software development gets greatly reduced when doing team-based task planning. Since the whole team had a hand in coming up with the design, the collaboration coupled with the removal of this variance often times results in the whole team being able to swarm a story and get it finished that much faster. The swarming in this case does not suffer from the overhead of bringing an additional person the coding aspect because everyone has a detailed understanding of the story, the story's history, and its associated development tasks and test cases.

More to come...

This post turned out to be a lot longer than I originally thought it would, so I'm going to turn this into a multiple parts. In future posts, I'll write about some more of our successes and challenges. In the meantime, I'd be curious to hear what others may have to say and would enjoy hearing about some of your experiences.  

Revisiting the Definition of Done

It's good to get into the habit of revisiting, and to some degree, challenging the status quo as part of the continuous improvement process promoted by lean and agile methodologies. Sometimes this can be difficult, or even scary, depending on the kinds of changes being introduced, but these are often the things that can elevate development teams to new heights of productivity and effectiveness. We went through one of these types of changes recently, so I thought I'd share some of the experiences and the results that were attained by revisiting and redefining one team's definition of done.

After months of preparation, the our development team recently split the single team consisting of six developers and two QA analysts into two separate teams consisting of three developers and one QA analyst each as an attempt to scale the development process. However, shortly after that was announced one of the testers was pulled into a separate project for various reasons, which left one of the teams without a dedicated tester. At first this situation left some in the organization concerned, including me, but after giving it some thought, I started to see this as an opportunity rather than a setback: this was the perfect opportunity to try a small experiment regarding how software is written in our shop. Rather than re-merging the two teams back into one, we continued with the two team approach. In order to succeed, the team without the dedicated tester would now have to adopt some of the practices of agile development that had been neglected for various reasons.

The Old Normal

Historically, the definition of done of a given story for the development group primarily consisted of coding and unit tests passing (sometimes, as this was often times left to the very end of the entire development cycle). Once those two items were finished, the developers moved on to the next story, and the story just coded was the QA/tester’s problem. This “definition of done” resulted in a few things (none of them good):

• a large queue of incomplete/un-shippable work as QA analyst could not keep up with the pace of the developers 
• inefficiencies if a bug was found in testing, due to context switching, as the development team had to come back to the previous story, meanwhile adding to the queue of incomplete work as the development had already started on the next story 
• code could never be promoted to our UAT environment until all the incomplete work was cleared out, which led to delays in letting the product owner and customers interact with the new features, often times only giving them a week or so to try out 40 new features and/or changes. 

For the reasons above, the entire group had been encouraged to try to get to more of a classic agile develop-test cycle rather than following the mini-waterfall that had evolved due to some of the traditional QA practices that carried over from the previous waterfall process within the organization. Due to the mini-waterfall that had evolved, it was normal to be operating within a cycle that allotted four weeks for coding and four weeks for testing in an overlapping manner; the result was coding would start on week 1, testing of the first week’s code would begin on week 2, and the last week of coding would end on week 4 with testing ending on week 5 and the developers usually writing any necessary documentation and then coming back to finish up unit tests. 

Once communicated that we were still moving forward with the two team approach, the team of three people, we’ll call them Team 1, had to rethink how they were going to code and test their own work. They had to essentially redefine their definition of done. 

The New Normal

Team 1 changed their definition of done for a story to consist of: coding, all unit tests passing, adopting a new branching strategy that called for a branch per story, merging code back into the main line within source control, deployment to the QA environment, executing all the user acceptance tests that had been identified to ensure they pass within the QA environment, and writing any documentation/wikis that may be necessary. With the new definition of done, Team 1 effectively eliminated the three problems that resulted from the previous definition of done and essentially gained an extra week of development since they were not constrained by the bottleneck that resulted from the mini-waterfall model.

The team consisting of four members, let’s call them Team 2, continued with the previous definition of done as it related to the coding and QA/testing aspects as the team dynamic between development and QA remained unchanged; if anything the QA bottleneck was eased as the QA analyst only had to worry about testing the work of three developers rather than all six. Our experiment now had our control group, Team 2, and our experimental group, Team 1; the results of Team 2 team provided a baseline to compare the results of the new practices Team 1 had adopted.

The Results

The measurement of productivity in an agile software development team is velocity, measured in terms of story points in our case. Prior to splitting up into the two teams, the group’s collective average velocity was roughly 60 points per sprint, or 240 story points per release (each production release consists of 4 sprints). Assuming the teams were split evenly, one could reasonably expect that each team’s velocity would be about even at 30 points per team per sprint. Taking into account that Team 1 would be operating with 25% less bandwidth, the expected Team 1 velocity could be calculated at 22.5 points/sprint (30 x 75%) from a pure mathematical/theoretical perspective. Story point estimates were arrived at by the consensus of both teams, so there was not any skewing of the story point data when it comes to comparing the velocity of both teams.

With Team 1 now operating in a manner that is more consistent with lean/agile concepts and practices of not building up queues of incomplete work via not moving on to the next story until the current story is considered “done” (coded, unit tested, deployed to QA, functional tested, and any documentation written), they could technically gain another full week. In our first release cycle with the two teams, the velocity for each of the teams broke down as follows:

Note: for comparison purposes, the extra week that Team 1 gained was evenly distributed across each of the four sprints

The total velocity for Team 1 and Team 2 teams was 151 story points and 176 story points respectively. Despite having 25% less bandwidth, Team 1 was only 14% less productive in terms of story points. This was also in spite of 0 points being delivered in sprint 1 which was primarily attributed to the amount and scale of changes being introduced to that team, a large portion of the first sprint being dedicated to answering support questions from production, and adapting to their new definition of done.

It is reasonable to project that the next release Team 1 could increase their velocity and come much closer in terms of velocity as Team 2 team for the overall release as they will not have to adjust to so many intrusive changes. Excluding the first sprint in which Team 1 was adapting to the many changes, they were more productive than Team 2 team; Team 1 completed 151 story points vs. 129 story points completed by Team 2 team in the last three sprints:

Team 1 was 17% more productive than Team 2 in the last three sprints

This analysis has not been meant to slight Team 2 team as both teams improved velocity compared to the adjusted average velocity coming into the release, evidenced by the table below. The small team approach proved to be very beneficial to both teams and the data lends credibility to the teams’ hypothesis that productivity would be increased by having smaller, independent teams. Comparing the projected sprint velocity to the actual velocity, the two smaller teams alone (without taking into account any of the process adjustments made by Team 1, it appears to have boosted velocity by 46%.

Team	Team Size	Projected Sprint Velocity/Productivity	Actual Sprint Velocity/Productivity	Difference	Differences Adjusted for Smaller Team Efficiencies
Team 1	3	22.5	37.75	168%	22%
Team 2	4	30	44	146%	0%

In the context of the more focused experiment, this has been a good example and proved that there is room for improvement regarding how the software development and testing process is executed. With Team 1 being forced to adapt and redefine their definition of done in order to be successful, this has been also been an example of what can be accomplished when not being afraid to take manageable risks by questioning the status quo and trying new things. 

While probably not completely 100% accurate let’s assume, for the sake of argument, that the 46% gain in productivity by Team 2 team was entirely due to the efficiencies of having a smaller team as there is not any other data to say otherwise; adjusting the difference to exclude the efficiencies of the smaller team, the data can be interpreted such that a 22% increase in productivity was achieved by adopting the new process changes despite the large amount of time/effort expended towards production support in the first sprint. Should the 46% gain not be entirely due to the smaller team aspect and perhaps a portion stemming from some other source, then this would indicate that Team 1 saw even higher gains due to the process changes introduced.

The Conclusion

This experiment in determining how large an impact the definition of done has on a software project was constrained by real world, practical business needs, so the experiment could not be conducted in a vacuum by having the two teams build and test identical features. Taking those practical constraints into consideration and making all other things equal outside the process changes introduced n order to adapt to the new environment, we were able to infer from the data in our ALM tool that there is room for improvement and perhaps an overdue re-evaluation of how software is developed and tested within our organization.

The circumstances we happened to fall into lent themselves well to such an experiment by having a control group and an experimental group. The cohesiveness of the experimental group seemed to grow with each sprint which resulted in higher velocity/productivity gains with reduced costs and a testament to what can be accomplished if the manner in which we perform our work is re-evaluated and an effort is made to continuously improve that day to day process.

In this particular case, Team 1 was forced to adapt and challenge the status quo to mold a new definition of done and how they went about getting there. The independent, two team model brought with it many transitional challenges for both teams, but additional challenges for Team 1 had to be adapted to in order to have success due to not having a dedicated QA analyst. A lot of the success Team 1 had in adapting to these circumstances while maintaining a high level of productivity, comes down to the fact that the team worked together and was open and willing to try agile best practices that took them out of their comfort zone in an effort to boost their productivity and efficiencies. As an additional, sidebar that should be noted was that Team 1 was lacking senior members (in terms of title) that Team 2 team possessed, but working as a team and the changes they adopted more than made up for what Team 1 team lacked in titles.

In general, there are most likely efficiencies that can be gained by taking the next step towards adopting some of the practices advocated by the agile community that we have been hesitant and/or afraid of for various reasons (the practices may be non-intuitive on the surface, fear of change in general, general distrust, etc.). 

The Doctor Is In...

One of the tenets of agile development is continuous integration in order help minimize the time spent trying to fix the build when changes are committed that may break it. We have had an automated build process for a few years now; our SCM engineer spent countless hours on the phone with the guys over at Urban Code to make sure we had everything setup just how we wanted.

The last step was to figure out how to notify us that a build failed. We were given the option to be notified via email or IM whenever there was a build failure. The team immediately nixed the idea of an email notification. I mean, don't we all get enough email throughout the day is it? Who wants another email filling up an already bloated inbox? So, we chose going the IM route.

Everything was going smoothly: code was being committed to source control; builds were being kicked off upon a commit; and any build that failed usually got fixed in less than an hour due to the IM notification from the build server that was sent to the whole team. A typical build history would look something like this:

Over time, the team started to ignore the IM about the failure; some people had actually stopped signing into IM just so they wouldn't be "distracted" by the failure notification. Kind of defeated the purpose of having continuous integration in the first place, don't you think?

As a result, the build was not getting fixed until either the SCM group noticed several failures in a row and emailed the team, or, even worse, until the QA portion of the team was trying to deploy a build to our test environment. People were now waiting on the team to get a good build when there were mechanisms in place to avoid this very scenario.

After experiencing one of these episodes, we would be more diligent about being more aware of the build and fixing it as soon as possible if it were to fail again...for a while; then it would be back to the same old thing. The last straw was when I noticed the build failing for four or five days straight, so I thought there had to be a better way. In the days to follow, I came across an excerpt from the book, 97 Things Every Programmer Should Know: Collective Wisdom from the Experts that had just what we needed:

"You need to give your project a voice. This can be done by email or instant messaging, informing the developers about the latest decline or improvement in numbers. But it's even more effective to embody the project in your office by using an extreme feedback device (XFD)."

After doing some additional research on extreme feedback devices, I went to check out if there was a web API from our build tool that I could leverage to check on the latest build status. Much to my delight, Urban Code included a relatively rich API that gave me exactly what I needed. The next step was to order some home automation gear from X10 and download the SDK. Before long, I was programmatically turning a light in the other room of my house on and off.

After reaquainting myself with some of the nuances of creating a Windows Service and a few .NET classes later, I was in business. The only thing left to do was figure out what I would be switching off and on. From a lot of the posts I read, red and green lava lamps seemd to be the most popular: a glowing green lamp meant all was well; a red lamp starting to bubble meant the build was broken. I figured lava lamps wouldn't go over too well with our facilities management folks, so I opted for a large, annoying siren: large so it would be tough to ignore; annoying so we would want to fix the build as soon as possible to make it turn off.

After finding a nice central location for the siren and some additional testing to make sure the radio frequency from the X10 gear was in range,  we were good to go. Needless to say, the device received a fair amount of attention at first along with feedback that randged from, "We should attach this to the ceiling," to "Can we get something that makes noise?" and just about everything in between.

The whole setup has been coined the "build doctor" (not to be confused with this one) as it makes house calls every couple of minutes to monitor the health of our builds. After word spread through the company that a flashing siren meant the build was broken, the effect has been an increased visibility into the build and an increase in accountability and attention to detail amongst the team as it relates to our build. The $60 or so to get the whole thing setup and running was a small price to pay for the short term morale boost, more transparency, and increased accountability regarding the build status. It has also led to less waiting for a good build by our testers, which was the ultimate goal.

I'd be curious to know what others have done in the way of extreme feedback devices and for any tips on trying to motivate the team to maintain a high level of awareness to the build status.

What are these "Agile Nuts" you speak of?

I’ve been in the business of writing software since 2001 where I started out writing point-of-sale web applications for an insurance company. From there I moved on to working for software development companies working on products ranging from online banking to line of business applications. For a short period of time I even dabbled as an independent consultant.

The more products and projects I was a part of, I soon came to realize that sometimes the bureaucracy associated with a company and/or product inhibited a lot of the productivity and creativity and could lead to a less effective end result, both in terms of the product and the team morale. The year was 2005, and I decided there had to be a better way.

These experiences are what drove me to seek out a better way of developing software where I would eventually stumble across this thing called agile development. The more I read about the mindset of agile development, the more I wanted to learn. My thirst for knowledge on this topic could not be quenched. Everything I read all had the same theme: focus on writing high quality software, engage the customer throughout the development process to make sure you’re writing what is desired, and empower the team to get things done. I thought this sounded too good to be true; as a software developer, this sounded exactly like the type of environment I wanted to work in, but what about the customer? What would her opinion be?

On the surface, it seemed like the customer would enjoy the process as well since she is involved in all phases of the development process rather than spending a ton of time trying to define the requirements up front and not seeing anything until many months later. I know I wouldn’t feel comfortable with that model as a customer: here the customer is investing time and money into a product with no return and nothing to show for until months, sometimes years later. The iterative approach that agile methodologies advocate gives the customer working software after each short iteration allowing the customer to quickly gauge whether the product is in line with the vision, shift in priorities based on market conditions, or, in the worst case, whether to cancel the product/project altogether. If the project is changed or even cancelled, the customer has still saved large amounts of time and money as only a few weeks have been lost instead of realizing that a change needed to be made only after the entire product was completed.

Even the most skeptical of customers can keep an eye on, and even be put at ease with, the progress of the project through the constant collaboration and feedback that is promoted through agile methodologies. Rather than the product development team and customer having an adversarial relationship, these methodologies take steps in the positive direction of customer and developers working together towards a common goal as each party are part of one cohesive team. Moving to this type of methodology seemed like a no-brainer; why isn’t the vast majority of software development occurring in this manner?

Over the last couple of years, I've played a large role in changing the way my place of employment goes about writing software. We've gone from an environment that was entrenched in the traditional waterfall methodology and have slowly moved to adopting agile methodologies. It has been a rewarding experience to see the productivity and quality gains that agile has brought to the development process and how an empowered, collaborative, cross functional team can bring about game-changing ideas and products. I'd like to share some of the experiences (good and bad) here in an effort to continue to spread the word and also as a means to selfishly solicit feedback from the community to improve our development processes moving forward.