Introduction
Nowadays the topic automated testing is becoming mainstream. Organizations and projects are investing significant effort in creating tests, using tools to automate them and plug them in their delivery chain. Combined with continuous integration tools, automate testing increases the usefulness significantly. I obviously find this trend unavoidable. Sooner or later every software organization will eventually go through it, if they have not already.
This movement is fairly new. Concepts like automate testing or continuous testing, in the context of continuous delivery, still do not have 10 years of history. We need to be careful with trends. The topic is so hot these days that the association between automated testing and quality is becoming the norm, also in Open Source.
Open Source became the winning “culture” in several industries more than five or ten years ago. Automated testing in the context of continuous delivery was not popular back then. Still, Open Source influence and adoption expanded also because of superior quality.
Principle: transparency
Transparency is about seeing what others are doing, but also about understanding. This second part is too often forgotten.
Action 1: Code review
Transparent code review, (again, see & understand) is, in my opinion, the most powerful quality assurance measure a project or organization can apply. It is the fundamental action in what some call the FLOSS development model.
It has a side effect that I really like as manager: it improves younger developers skills. It also brings many other positive side effects.
Action 2: dogfooding
A few weeks ago in a workshop with a customer, Codethink CEO Paul Sherwood was explaining this point with an example that I stopped talking about several years ago. I found it so obvious that at some point I gave up fighting for it. After listening to him, not anymore. The example was…. your organization is developing Linux based products, use Linux, not Windows.
Simple, right?
Dogfooding is another of those actions that in long term Open Source projects is frequently taken for granted but that is not the norm in commercial environment. So many projects driven by newcomers to Open Source do not pay enough attention to it.
The impact over quality of dogfooding in the mid term is impossible to calculate. Still I believe is huge.
Action 3: delivery model that maximizes the influence of early adopters
Who are early adopters? They are the developers or power users who like to consume experimental or pre-releases of your “product”. The number of those willing to report bugs is significantly bigger in relative numbers than in consumers.
Increasing the number of early adopters, reducing the hurdles they face to use your software, analyses/debug problems and report should be a key activity among those projects worried about quality assurance. Adapting your delivery process to maximize their impact, not just have a positive effect in the use cases your software was designed for, but in others, expanding the knowledge about how your software will behave in the hands of users. Like it should happen between developers and delivery engineers, the feedback loop with early adopters should be very short, so you can provide them improved pre-releases in short cycles.
Open Source has reached the current point understanding how important the role that early adopters play is.
Personal note about this third topic
Many projects pay little attention to making their frequent releases truly installable, documented, simple to debug without complicated tools or even centralized infrastructure, bug trackers simple/fast to use, treat bug reports as a valuable asset . In summary, early adopters cannot follow the pace and, when they do, they need to spend a lot of energy to be valuable.
Let’s go back to the main argument.
Conclusion
Code review, dogfooding and early adopters in transparent environments has been, I believe, the pillars that has made Open Source what it is today in terms of quality. And then, only then, automate testing, or continuous testing comes to place, in addition, not in substitution, not before, not in between…. in addition.
Are you doing Open Source? Don’t take shortcuts. Surf the “trend wave” instead of embrace it blindly. Learn first, look carefully what sustainable projects are doing.
Quality is as much as culture as it is about having a nice dashboard full of green lights. Testing => Quality is, in general, a wrong association of ideas.
And yes, test frameworks, board farms executing thousands of tests, green lights in dashboards, etc. are awesome. Probably a forth pillar in the coming future.
toscalix 