This movement is fairly new. Concepts like automate testing or continuous testing, in the context of continuous delivery, still do not have 10 years of history. We need to be careful with trends. The topic is so hot these days that the association between automated testing and quality is becoming the norm, also in Open Source.
Open Source became the winning “culture” in several industries more than five or ten years ago. Automated testing in the context of continuous delivery was not popular back then. Still, Open Source influence and adoption expanded also because of superior quality.
When I think about quality in Open Source, one key principle and three actions come to my mind.
Action 1: Code review
Transparent code review, (again, see & understand) is, in my opinion, the most powerful quality assurance measure a project or organization can apply. It is the fundamental action in what some call the FLOSS development model.
Action 2: dogfooding
Dogfooding is another of those actions that in long term Open Source projects is frequently taken for granted but that is not the norm in commercial environment. So many projects driven by newcomers to Open Source do not pay enough attention to it.
Action 3: delivery model that maximises the influence of early adopters
Increasing the number of early adopters, reducing the hurdles they face to use your software, analyse/debug problems and report should be a key activity among those projects worried about quality assurance. Adapting your delivery process to maximise their impact, not just have a positive effect in the use cases your software was designed for, but in others, expanding the knowledge about how your software will behave in the hands of users. Like it should happen between developers and delivery engineers, the feedback loop with early adopters should be very short, so you can provide them improved pre-releases in short cycles.
Open Source has reached the current point understanding how important the role that early adopters play is.
Personal note about this third topic
It seems to me that there is a new wave of Open Source projects, specially those driven by commercial organizations, that underestimate the mid term effect early adopters have on the quality of a project. I also see how the Continuous Delivery hupe, focused on the developers and delivery engineers, is leaving the early adopters behind in some cases. Specially in those Open Source projects in which the project is developed and delivered by full time dedicated engineers.
Let’s go back to the main argument.
Are you doing Open Source? Don’t take shortcuts. Surf the “trend wave” instead of embrace it blindly. Learn first, look carefully what sustainable projects are doing.
And yes, test frameworks, board farms executing thousands of tests, green lights in dashboards, etc. are awesome. Probably a forth pillar in the coming future.