Improve your software product delivery process performance using metrics (II)

This is the second article of the series. Please read the first post before this one.

During the previous article I explained the process to follow, using the simplest possible model to describe a software product delivery process, to measure and improve its performance, following a data driven improvement kata as a way to promote a continuous improvement culture .

Despite providing extremely valuable information, once we have gone through the described process for a few iterations, the limitations of such a simple model will become evident. We will need to add complexity into our model, getting closer to the real software product delivery process.

Add complexity to approach reality.

If you have a clear map of your real delivery process, you can skip this step. If you do not, the first thing I recommend is to run a value stream workshop to discover and describe your delivery process. Visualizing it as a group is a great exercise, not just for those involved in the workshop, but also for the rest of the organiza­tion. If you are not able to draw the flow of the code end to end, identifying what happens where and who is responsible for what, trying to measure the performance of the process is often pointless.

How should the model be enriched?

For our purpose, we do not need a great level of detail as outcome of the value stream workshop, just enough to create your new model. Let’s assume for now that one of the outcomes of such workshop is a drawing like the one below, which I took from an exercise for an automotive platform. Only the “green criti­cal path” is shown here. I hope you get an idea of what is needed.

Based on this diagram, you can see clearly that the delivery process can be divided in three or four stages. Try to do it in three, again, for simplicity. For this example I will divide it in four since, in automotive, the validation/verification stage is rele­vant in terms of effort, complexity, throughput and specially stability.

So our richer model can be expressed in the following way:

This new model is a better approximation to our real delivery process.

Mathematical construct and quantitative analysis

Once we have described our system using a richer model, the question is, do we need to change our mathematical construct?

As you can guess, the answer is no, we do not. We just need to enrich it. The construct used to characterize the simplified model would have been very limited if now we would have needed to modify it.

Again, Steve Smith provides in his book “Measuring continuous Delivery” what we need to extend the metrics. The overall idea is to use the same metrics and measures for the entire process (as we did with the simpler model) and for each stage (corresponding to this new model). This is essentially the case for any linear process, like the one we created. S. Smith refers to these “extensions” of the metrics and measures as indicators.

It becomes now harder to define the events to extract the data sets from since the data probably needs to be extracted from different tools. I recommend to invest time in describing these events, how the data sets are extracted from the affected tools, how to convert such data into the right units… You will also need to describe accurately which events determine the beginning and end of each stage. All this effort will allow you to iterate in the process creating more complex models.

The methodology to measu­re, process, plot and analyze the different data-sets will be analogous to the one described in the first post of this series. The only difference is that now it should be done for each stage, in addition to the overall process.

Qualitative analysis

It is easy now to realize why we tried to keep our scale simple. The number of combi­nations we need to work with increa­ses now. The scenarios for each stage are consistent with those described for the previous (simpler) model, so the qualitative analysis done for both models complement each other.

The definition of the scenarios can be personalized and adapted to your needs. Since you will also use those scenarios as a communication tool, make sure they are described in away that resonates to your workforce. Reduce their number to those that make sense to you and ignore the others.

You should execute the same qualitative analysis for each stage of the delivery process to describe in simple words the scenario that better describes the current performance of your delivery process as well as the target scenario. Please remember that the goal is to improve the performance of the overall process not to optimize any specific stage. If an improvement done in a specific stage does not have a positive impact in the overall process, you need to question the consolidation of such improvement.

Data Driven Improvement Kata

This step will require changes. Now that our system model is more complex, the methodology to improve performance of the delivery process will need to consider the company organi­zational structure as well as the cycles in which the business, product and development teams operate.

At scale, these katas cannot be structured in a single cycle, but in several ones. The book “Leading The Transformation. Applying Agile and DevOps principles at scale” by Gary Gruver and Tommy Mouser provides an idea of how to approach this continuous improvement process for larger organizations.

I suggest to consider two cycles whenever you can. In my example (very academic, to explain the process) I have consided three: business cycle, product cycle and experiments cycle.

Please check the references (at the end of the first article) to learn more about improvement kata. I will make some considerations about the above example for those of you less familiar with continuous improvement methodologies:

• The overall idea is to synchronize the three cycles.
• Business cycles are usually a year long. I would structure the business iterations in quarters.
• Define the business goals for the product and relate them both, qualitatively / qualitatively with the metrics / scenarios.
• Define the current scenario and the target scenario for the current quarter. Be concise.
• Define the most suitable cycles for your product. Bare in mind that some effort in analysis, coordination and communication with the workforce will be required on each iteration so too short cycles might bring unnecessary overhead.
• If the development teams work in weekly sprints, I would start with monthly iterations for the product level. If engineering teams work with two weeks sprints, then consider iterations of six weeks at product level.
• As described in the previous article for the simpler model, I will use the boards to explain how the iterations are defined.

The first board correspond to the business cycle, the second one to the product and the third one to the technical experiments to be performed by development teams. Click the picture to enlarge and read the advises.

Summary

In this second article, we have enriched our model introducing more complexity (four stages). We saw how the mathematical construct should be extended to characterize each stage, which allow us to get a finer grain information, increasing the accuracy of our analysis. We have seen that Throughput and Stability remain useful as key metrics, as expected.

We have also discovered that, as it happe­ned with our quantitative analysis, the qualitative one is essentially an extension of the one introduced in our previous article for our simpler model. We have modified our approach to continuous improvement adapting the data driven improvement kata to a more complex environment. The proposed example. covers, in my view, large organiza­tions. Such data driven improvement kata has been summarized in three boards, one per level: business, product and technical.

Conclusion

These two articles represent a modest attempt to summarize and structure in 5 basic steps the process that an organization should go through to improve its delivery process performance at scale using Throughput and Stability as guiding lights.

Obviously writing about this topic is easier than implementing it so I am looking forward to read about your own experience and how it contradicts, modify, complements or support the described methodology and advises.