Nowadays the topic automated testing is becoming mainstream. Organizations and projects are investing significant effort in creating tests, using tools to automate them and plug them in their delivery chain. Combined with continuous integration tools, automate testing increases the usefulness significantly. I obviously find this trend unavoidable. Sooner or later every software organization will eventually go through it, if they have not already.

This movement is fairly new. Concepts like automate testing or continuous testing, in the context of continuous delivery, still do not have 10 years of history. We need to be careful with trends. The topic is so hot these days that the association between automated testing and quality is becoming the norm, also in Open Source.

Open Source became the winning “culture” in several industries more than five or ten years ago. Automated testing in the context of continuous delivery was not popular back then. Still, Open Source influence and adoption expanded also because of superior quality.

How come?

When I think about quality in Open Source, one key principle and three actions come to my mind.

Transparency is about seeing what others are doing, but also about understanding. This second part is too often forgotten.

Action 1: Code review

Transparent code review, (again, see & understand) is, in my opinion, the most powerful quality assurance measure a project or organization can apply. It is the fundamental action in what some call the FLOSS development model.

It has a side effect that I really like as manager: it improves younger developers skills. It also brings many other positive side effects.

A few weeks ago in a workshop with a customer, Codethink CEO Paul Sherwood was explaining this point with an example that I stopped talking about several years ago. I found it so obvious that at some point I gave up fighting for it. After listening to him, not anymore. The example was…. your organization is developing Linux based products, use Linux, not Windows.

Simple, right?

Dogfooding is another of those actions that in long term Open Source projects is frequently taken for granted but that is not the norm in commercial environment. So many projects driven by newcomers to Open Source do not pay enough attention to it.

The impact over quality of dogfooding in the mid term is impossible to calculate. Still I believe is huge.

Who are early adopters? They are the developers or power users who like to consume experimental or pre-releases of your “product”. The number of those willing to report bugs is significantly bigger in relative numbers than in consumers.

Increasing the number of early adopters, reducing the hurdles they face to use your software, analyse/debug problems and report should be a key activity among those projects worried about quality assurance. Adapting your delivery process to maximise their impact, not just have a positive effect in the use cases your software was designed for, but in others, expanding the knowledge about how your software will behave in the hands of users. Like it should happen between developers and delivery engineers, the feedback loop with early adopters should be very short, so you can provide them improved pre-releases in short cycles.

Open Source has reached the current point understanding how important the role that early adopters play is.

I want to make a point here before moving forward.

It seems to me that there is a new wave of Open Source projects, specially those driven by commercial organizations, that underestimate the mid term effect early adopters have on the quality of a project. I also see how the Continuous Delivery hupe, focused on the developers and delivery engineers, is leaving the early adopters behind in some cases. Specially in those Open Source projects in which the project is developed and delivered by full time dedicated engineers.

Many projects pay little attention to making their frequent releases truly installable, documented, simple to debug without complicated tools or even centralised infrastructure, bug trackers simple/fast to use, treat bug reports as a valuable asset . In summary, early adopters cannot follow the pace and, when they do, they need to spend a lot of energy to be valuable.

Let’s go back to the main argument.

Code review, dogfooding and early adopters in transparent environments has been, I believe, the pillars that has made Open Source what it is today in terms of quality. And then, only then, automate testing, or continuous testing comes to place, in addition, not in substitution, not before, not in between…. in addition.

Are you doing Open Source? Don’t take shortcuts. Surf the “trend wave” instead of embrace it blindly. Learn first, look carefully what sustainable projects are doing.

Quality is as much as culture as it is about having a nice dashboard full of green lights. Testing => Quality is, in general, a wrong association of ideas.

And yes, test frameworks, board farms executing thousands of tests, green lights in dashboards, etc. are awesome. Probably a forth pillar in the coming future.