Why Codethink is a founding member of the Civil Infrastructure Platform, a Linux Foundation initiative

This blogpost was originally published on the Codethink website on Thursday March 9th.

On April 4th 2016 a new Linux Foundation initiative called the Civil Infrastructure Platform was announced. CIP aims to share efforts around building a Linux-based commodity platform for industrial grade products that need to be maintained for anything between 25 and 50 years – in some cases even longer. Codethink is one of the founding members.

Industrial grade use cases

In order to describe why this initiative is relevant let me go over the use cases that motivate companies like Siemens, Toshiba, Hitachi, and Renesas to share efforts.
During the Open Source Leadership Summit, Noriaki Fukuyasu (Linux Foundation) and myself, based on the experience of Siemens, Hitachi and Toshiba, described the development life cycle in industrial grade use cases. For example, a railway management system is as follows:
  • Analysis + design + development: 3 – 6 years
  • Customizations and extensions: 2 – 4 years
  • The certification process and other authorizations take a year.
  • Each new release or update has to go through further certifications and authorizations that take between 3 and 6 months.
  • The system is expected to work for between 25 and 50 years.
So on average, an industrial grade product might take 5 to 7 years from conception to deployment. This is coherent with our experience in other industries like automotive, where life cycles are also quite long despite the expected lifetime being shorter.
A key part of the life cycle is maintenance. Due to its length, the associated risks are high. The certification processes to introduce significant changes in any already deployed systems are painful and expensive. In addition, the capacity to simulate a production environment is, in general, limited. This is true in other cases like energy production plans, for instance.

Open Source principles in the Civil Infrastructure industry

It’s obvious that Open Source could have a dramatic impact in this industry. By sharing efforts, corporations can commoditise a significant portion of the base system focusing on differentiation factors, increasing control through transparency and the quality of that starting point over time. Collaboration with upstream will bring even higher impact benefits.
Two immediate challenges come to mind when thinking about Open Source in this industry:
  • Development of processes and practices to produce software for safety critical environments.
  • Bridging the gap between the Open Source approach for software maintenance and the approach currently taken when building large-scale platform projects. For instance, how can approaches oriented to update any specific Open Source software component to the latest upstream stable version be compatible with any typical industry SDLC?

Can you reduce the gap?

We have for years been working on transformation projects for which one of the goals has been to reduce the gap between the software our customers ship and what upstream is continuously releasing. One of the key steps is to adapt an organisation’s processes using FOSS tools. Over the years we have been a strong advocate that the closer to upstream you are, the more benefits you reap from the Open Source development model, maintenance cost reductions being one of the main advantages.

So why did we get involved in an initiative that aims to maintain a kernel for 25 years then?

The short answer would be… because we love a challenge!
Safety critical with Linux-based systems is a challenge currently being faced in the automotive industry for instance, where Codethink is a strong player. When we analysed some of the industrial-grade use cases, it called our attention not just to the magnitude of the second challenge enumerated above, related with super long term maintenance, but also the apparent conflict between the industry requirements and the referred well known Open Source practices.
Hence the main driver for an Open Source consultancy like Codethink in participating in an initiative like CIP is to learn by doing, that is, putting the Open Source development, delivery and maintenance best practices under stress in one of the toughest environments. We bring our experience in producing embedded Linux based systems and our Open Source culture, to work together with industry leaders in finding solutions to these challenges, by looking at them with FOSS eyes.

Current activities

Codethink is participating in CIP in several capacities, the most relevant being:

Kernel maintenance
The first CIP approved kernel is 4.4, an LTS kernel supported until Feb 2018. Ben Hutchings is the initial CIP kernel maintainer. Besides providing support for the reference platforms, Ben is working on several activities like backporting the security patches, such as those from the KSPP and consolidating the maintenance policies, taking those from the kernel community as reference.
Testing tooling

kernelci.org is the most successful testing project in Open Source. Its impact in the kernel community is growing, as is the number of people and companies involved. It was designed and developed as a service where the testing activities can take place in distributed board farms (labs).
Codethink has been working on making the tools easy to deploy on developer machines through a VM, so they can test kernels on directly connected boards. This first milestone of the CIP testing project is called Board At Desk – Single Developer. This activity was described at the Open Source Leadership Summit 2017 and the first beta released during ELC 2017.

Conclusion

The challenges for Open Source that Industrial-grade product development and maintenance introduce are great, especially in two aspects: safety-critical and maintenance. Codethink is working on CIP to help the industry to overcome these challenges by adding our Open Source perspective.
Learn more about the CIP project by checking the following slides and videos from the conferences in which CIP members have participated.

Moving from a traditional product/release focused delivery model to a rolling model

The past few weeks the GDP delivery team together with some key contributors, has been working on a not very visible but still important change. The GDP project has put the basis to turn GDP release based delivery model to a “rolling” one. My colleagues will provide in a coming post the technical details behind this change. I want to provide a higher level view of what is happening and why.

Some background

GDP was born as a “demo” project. The main goal was to provide a platform to show the software components for automotive that the different GENIVI Expert Groups were developing. This was done through a delivery model focused on publishing a stable and easy to consume version of the project every few months, a major release.

Strictly speaking, GDP is a derivative. It is based on poky and uses Yocto tools to “create” the Linux based platform, adding the different components developed by the GENIVI Alliance together with upstream software. For the defined purpose, the release centric model works fine, especially if you concentrate your effort is very specific areas of the software stack with a small number of dependencies on the other areas, and a limited number of contributions and environments where the system should work.

During this 2016, the GDP has grown significantly. We have more software, more contributors, more components and more target boards to take care of. Although the above model has not been not challenged yet, it was just a matter of time.

As I explained in two previous posts [1][2], the GDP is moving from a being a Demo to a Development Platform. Changing the mission means changing the goals and the target group, which implies the need to adjust the deliverable to meet the new expectations.

So, right after the 14th AMM, the Delivery Team decided to change the delivery model to better meet the new mission, providing developers the newest possible software with the an increasing quality threshold. At the same time, in order to increase the number of contributors, the GDP needs to provide a new solid platform every once in a while. That should be done trough a solid release.

What is a rolling delivery model?

The key idea behind a modern delivery release model is to ensure that the transition from one stable release to the next one takes an affordable effort. I will put an example to picture the idea.

Problem statement

Imagine an organization that publishes one release per year. Let’s assume that a particular release included 100 patches developed by employees and, during the lifetime of the release (1 year too), another 100 patches were added to the product as bug fixes and updates. At the end of the release lifetime, the product includes 200 patches that define the value the product provides to customers and users.

Either for technical or business reasons, a year later it is time to upgrade. Our organization has to create a new Linux based system with newer upstream code and they have to integrate the patches from the previous release plus the updates and bug fixes developed for the coming release.

After a simplification process done by engineers, the number of patches needed to be integrated in this newer base system is reduced to 150. The organization also wants to add to this new release another 100 patches that represent the new features they have been developing during the last year for this new version.

The delivery team now has to integrate 250 patches in the new base system, 150 of them coming from the previous release. One might think that the effort required to do this is 2.5 times the effort invested in the previous release. Maybe you think that the effort is not so high since some of the patches have been developed thinking about the new base system. There are many other considerations like this one that might affect the initial estimation. This example is obviously a simplification.

However any experienced release manager will tell you that moving patches integrated in an older system base onto a newer one (forward-porting) requires additional effort, beyond a linear relation with the number of patches. Forward-porting is the “road of hell“. Iterate this example a few times and you will understand why there are so many organizations out there that have as many people focusing on delivery as they have in development. They migrated to Linux base system keeping the traditional delivery model they had while working with closed source software.

Release based delivery model

Possible solutions

One of the paths to improve the situation is upstreamming those changes that affect generic components. Some companies also upstream their new features early in their development process, generally looking for wider testing, or after they have been released to customers, to increase adoption and reduce future maintenance effort. This is definetly a must do.

From the delivery perspective, the most popular way to tackle the problem though is reducing the release cycle, so the number of patches to forward-port in each release is smaller. The development time and the maintenance cycles are also smaller. The same applies to the complexity of the forward-porting activities. “Jumping” from one release to the next one is easier to do. Add automation of repetitive tasks to this recipe and you feel you have a win…. for some time.

The journey through the “road to hell” becomes more comfortable, but our organization is still getting burned, even in the case that our customers and ourselves can digest releasing frequently. We all know how expensive and stressful a release might become.

The most suitable option to achieve sustainability while scaling up the amount of software an organization can manage without releasing more often than your market can digest is to change your delivery model.

Rolling delivery models are a serious attempt to solve this problem, putting integration as the central element instead of the software itself.
This model is not new. Gentoo has been doing it forever, but it was Arch Linux who implemented it in a way that immediately attracted the attention of thousands of developers. Still it was a model with no hope beyond hardcore Linux developers. openSUSE brought this model to a new level by implementing a process which output was stable enough for a much wider audience, and compatible with the release of a more stable and a commercial releases. Nowadays there are other interesting examples out there that commercial organizations can learn from.

What is a rolling model?

It is still hard to define but essentially it is a process in which ideally you have one continuous integration pipeline as the one an only entry point for the software you plan to ship. Releases then become snapshots of all or part of the software already integrated after going through a specific stabilization, deployment and release process.
So ideally, if you release a portfolio, you integrate only once, reducing significantly the costs of having different engineers working on different versions of the same software and forward-porting, among other benefits.

Rolling delivery model

So a rolling delivery model is a lot more than a continuous integration chain, although that is the key point.

Please have in mind that this is an oversimplification. This description doesn’t go into detail on other key aspects like maintenance cycles, how upstreamming affects the process, strategies towards updating the released products, etc.

A transformation process that takes an organization from a release centric model to a rolling one is about doing less and doing it faster, so less people can handle more software with less pain, allowing more people to concentrate in creating value, developing new and better software instead of just shipping it.

Back to GENIVI

Moving from a release centric to a rolling model is hard work. Frequently it is easier to start all over again. Since the GDP is still a relatively small project, we can afford going through the transformation process step by step.
The first stage has been creating that single integration chain and treating GDP-ivi9, our latest release, and those that follow it, as a deliverable of what we call today Master. Ideally, no single patch will be added directly to the release branches. They should come from Master. That way, we reduce (ideally to zero) the effort of forward-porting of patches while putting in the hands of our contributors the latest software on a regular basis.
To do so, we are in the process of adapting our simple processes and CI system to the new model, GDP repository structure, the wiki contents, the task management structures, several key policies, our communication around the project…

The GDP will face a very interesting challenge since this model needs to be proven successful for a derivative. If we are able to move fast enough, it will come the time in which we will need to decide if GDP keeps being a derivative or it becomes upstream, that is, either GDP limits the delivery speed based on the Poky release cycle, or we work upstream with the Yocto project to increase our delivery speed.

That is a good problem to have, isn’t it?

If (almost) everything goes right, after adding a few needed services in GENIVI’s infrastructure and ensuring the updated software is in compliance with selected verification criteria, the same number of people will be able to manage and deliver more software. And once the new processes become more stable, automation will not just increase efficiency, it will boost the project by allowing GENIVI to achieve goals that only big organizations with large delivery teams can do. This is the kind of transformation that takes time to consolidate, but has a huge impact.

Based on my experience, I believe that if GENIVI is able to sustain this effort and keep a clear direction the next couple of years, the benefits of moving towards a rolling model will be noticeable even outside the industry.

This blog post was originally published in the GENIVI blog site on 2016/07/04. I have adapted the formatting to adapt it to Blogger. The content should be the same.

Let’s do it

Sometime we concentrate on little things that matter a lot to us but has no impact on what we are doing. I frecuently come to disscussions in my everyday live in which I put a lot of my energy but, overall are not worth it. Sometimes I have people around me telling me that there’s no point on “trying to win” those arguing but moving forward and pushing hard to accomplish the real goal. But sometimes I don’t have them.

This Akademy is being great so far, but as always happens when innovating, I’ve seen a lot of tough disscussions. Some of them matters, and some don’t. Or at least to the project as a whole or not to acomplish the global goal, which is to have a great product made by a healthy community and having a lot of people using it.

This article [1] [2] just got into my e-mail client and I want it to share it with you. Even if it’s not totally true, it reflects how much work is ahead of us.

So let’s concentrate more in what matters and a little less in other things. It is not that we are doing bad….I just want to share this feeling that we can do even better. There are tons of opportunities out there, and now we have a great product and the best community we’ve ever have.

Let’s do it.

[1] http://en.wikipedia.org/wiki/OS_market_share
[2] http://en.wikipedia.org/wiki/File:Operating_system_usage_share.svg