ACCOUNTING FOR DEFECTS IN AN AGILE ENVIRONMENT

The inevitability of defects is certain. Nobody is perfect and whether we are practicing extreme programming to limit them or have practices in place to resolve them is irrelevant. We must decide how defects are handled, and every organization handles them differently. Some companies have mature #DevOps practices and look to encourage a zero-defect culture and all the benefits and challenges that brings. Others may focus on critical or high priority defects that more heavily impact their customers satisfaction. Whether your organization is the former or the latter is up to the leadership, but one challenge every enterprise discovers is how to account for those defects.

Commonly, #agile teams and enterprises struggle with understanding how to handle defects. With every organization I have consulted for, and often within many of the classes I teach, individuals, teams, and leadership alike reach out regarding best practices as it pertains to their challenges. Too often however it is more about the culture of the company than it is about what is a best practice in the industry. Are the leadership in the organization demanding accountability of every minute or hour of a team members time? Are they trying to mature their DevOps and Agile practices? Are they attempting to scale agile, whether #SAFe, #LeSS, or otherwise? We need to understand the culture to understand how to address a challenge.

It is not always a simple answer and usually requires a full understanding of the current situation and where they want to go. A solid perception of where things are and where they want to be is something many organizations, leaders, and the teams often cannot agree on. So, for the purposes of handling defects in an agile environment let us make some assumptions:

The organization

1) is maturing their DevOps practices, and

2) is trying to follow agile best practices and principles, and

3) may have cultural impediments but are seeking to overcome them.

With that said, here are my recommendations on how organizations account for defects in an agile environment.

In an agile world, teams are assigning story points to their efforts. These points are typically based on knowns and unknowns, complexity, and volume (how much is there). The equivalent of a story point typically differs a bit from team to team but most commonly, especially at scale, they equate to something close to a day of effort. However, other than for road mapping and predictability purposes these points simply provide the team a baseline of how much they can accomplish in any given sprint. They are not meant for individual or team comparisons nor for accounting for individuals time, e.g. each individual should not be expected to complete 5 points a week or 10 points a sprint as a point means different things to different people. However, this happens.

In some environments, management – either due to ignorance of the agile process or the lack or understanding of other measurable means – often focus to closely on what a team, or an individual within a team, is accomplishing in terms of points. This does not account for the fact that team members are dealing with a lot of unknowns (greenfield) in their work and their estimates of points could be significantly under or over estimated. Instead of focusing on tangible accomplishments, management is focused on story points. Instead of celebrating big deliverables, some managers may compare one person’s story points to another. This is not healthy and organizations with such challenged management should look for training.

This also creates a culture where team members will “sandbag” their estimates to make themselves look better, and the unsuspecting, ignorant agile mind will not know any better – and this is often counted on. In short, if management does not understand the agile practices and compares individuals point production to another, the team members will make their story points higher to account for this shortcoming. It makes the team member(s) look better while also encouraging less productivity.

In a mature or maturing environment managers will have been trained. They will have been required to pass some certification exam to demonstrate understanding and hopefully internalization of the practices and principles. They will have best practices in place to account for individual’s performance based on measurable and tangible deliverables and performance metrics. They will not need to use story point production, hours, or some other vanity metric to successfully account for individual performance.

Additionally, in a mature agile environment, teams demonstrate understanding of the principles and practices and know what work is expected to be delivered within a story. They do not waterfall their iterations, making each of the stories a step in the #SDLC, nor do they attempt to account for rework and defects as a story themselves. They understand Principle 7, “working software is the primary measure of success,” and that this means any defects found at the team level should be resolved before any work is considered complete or done. They have no need to account twice for a working slice of functionality, it’s built into their processes.

Immature agile teams may attempt to account for defects as additional work. While this may make sense to some, it is simply a fallacy that needs to be overcome.

When we build quality into the products we are developing, defects that are discovered are immediately fixed, no matter the size or criticality. The idea is simple: Teams create small pieces of quality products (often software) that are defect free.

When we create defects in development, and then release them to our customers, we are not following Principle #1, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Is it truly valuable if the customer is burdened with defects? No. So, teams seek to follow Principle #9, “Continuous attention to technical excellence and good design enhances agility,” but are then hampered by an organizational culture, discussed previously, that does not follow Principle #5, “Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.” 

Without the appropriate environment and support teams may be required to deliver more and more story points. So, they seek to find points wherever they can. They forget the principles and push to get more and more work out, with or without defects, and when required to fix things, write new stories for the defects and rework they just created. They “collect” double points on poorly developed work – some teams may even see defects as an opportunity to exploit rather than a challenge to overcome. This is not a recipe for success.

Some Simple Math. Defect reduction does not equal more value, but unfixed defects do equal less value. That is the simplicity of it all. If a business determines that a product has a value(V), and then as we create it we build defects(d) into the product, then the true value of the product is V-d. If we build the same product defect free, or repair/rework any defects, the value is still V, not V+d. The value does not increase as we resolve the defects we built into it. Our customers expect working products, and our success is measured as indicated by #AgileManifesto Principle #7.

So, look to build quality in. Look to include complexity (and the defects that may come with that) into your initial estimates. Focus on creating stories that are valuable pieces of a working product and include any repairs that must be completed. Do not get in the habit of creating and estimating stories for defect reduction if the defects were built by the team. It is simply double counting.

If you are required, organizationally or otherwise, to account for any defects, consider ignoring any pointing of them. Oftentimes, when we have good continuous delivery pipeline and DevOps practices in place, our teams will look to resolve team level defects early and often. Management may still request/require that all defects be logged for accounting purposes, but oftentimes do not require estimating the effort. In this case, and whenever possible, we should look to create them with zero points. This avoids the challenges mentioned previously of double-pointing work and still provides the transparency the organization may need.

The expectation with this practice is that teams are regularly resolving defects quickly, not waiting weeks or months before they do. The longer they wait, the more effort it will take to resolve the defect. The quicker we can resolve identified defects, the more likely the information that is causing the defect is in the front of our minds, making rework less challenging. In most cases quick resolution results in far less effort than delayed reactions.

There are often exceptions to the best practices, and you may discover your own. One that has commonly come up for me is the impact of integration.

As we continue to build these small things called stories, we must integrate them into a trunk or main branch of a much larger, more valuable product. As we do that there are many other tests that may need to be completed. These integration tests, user acceptance tests, exploratory and other tests may likely find defects that were not caught at the team level. This is not to say the team missed something, rather that the stories are not integrating well or that the architecture of the product may have been too tightly coupled. In any sense, there are defects that need fixing.

Again, we are in a situation where every organization handles this differently. While some may send the defect back to one or many teams responsible, others may handle it within a systems team or some other organizationally structured team. In either case it is not a fault situation but rather another one that could require accountability.

In such instances I have advised teams to either account for this defect reduction capacity via an allocation made each sprint or program increment, and/or consolidate and story point them. This allows for some accountability of downstream defect identification and reduction and provides a team or organizational objective that supports Principle #7.

Some teams struggle to build quality in due to the nature of the work or the inexperience of the team members. In this case it is important to remember that predictability is a primary measure of a successful agile team – we want to know that what teams commit to will be delivered.

In this situation, I advise newly formed teams to allocate a small portion of their capacity to defect reduction and/or rework. For example, if the team expects to deliver 50 points per two-week sprint, I might advise them to reduce their capacity by 10% or 5 points to allocate some time to their backlog of defects or those yet to come. This serves three purposes:

1) It reduces their story point capacity, so they do not overcommit and under-deliver.

2) It follows the best practices set here by not counting defect rework as additional story point completion, encouraging the teams to build quality in.

3) It provides additional capacity – should the team build better quality products with less defects – that they can use to pull additional work in later in the sprint.

The intent is to make them more predictable. As they begin to mature their processes and create fewer defects, this capacity allocation can be reduced incrementally until it no longer makes sense.

To support the agile manifesto principles and best practices of high performing teams we must identify every instance within our organization that is challenging our ability to follow them. We must look at ourselves, our culture, and our organizational structure to understand what we are doing that supports or inhibits our teams’ abilities to deliver early and often. We cannot simply look at things that present themselves on the surface as apparent and obvious but rather must understand every aspect of our value streams, flow, and continuous delivery. We must identify every impediment, look to resolve them, and relentlessly improve.

Change is coming. Building quality products that our customers want and are not burdened by is an expectation to continue to survive. Understanding how to do that starts with one simple step toward getting better: understand challenges and act on strategies and tactics to overcome them. The more often this is done, the further we shift from surviving to thriving.