Logbook, a great practice for distributed teams, long term projects and products.

Logbook, diary, journal, bitacora… there many names for the same practice: write down what you do. This practice has been used among sailors and scientists for centuries, for good reasons. Nowadays there are plenty of tools to keep track of what you do but I haven’t seen anything as powerful as a team, a project or a product logbook.

Five years ago I wrote an article about my first interactions with this practice, back in the beginnings of my professional life, in the Canary Islands, Spain. The article describes the basics of any team or project logbook. You would benefit from reading it before you keep reading this article.

I would like to provide some additional insides about the diary, together a few tips and practices I have used throughout the years.

Why is the logbook useful

Keeping a logbook is useful for:

  • Engagement: you will know about what your colleagues are doing. They will know more and better about what you do. All in a contextualize manner.
  • Alignment: the logbook will reduce the need of coordination and providing status to your colleagues. It will also help in advancing divergences among team or project members. Managing dependencies better is another outcome.
  • Reporting: the logbook allow to turn the reporting activity into a pull system. Instead of providing a report to your manager, for instance, she can pick up the relevant information from the logbook. A similar thing happens when you need to provide a status report to a customer or another team. A significant part of such reports should be already in the logbook or link to it.
  • Analysis / retrospective: quarterly or yearly reviews are simpler when a logbook exists. Postmortem analysis also becomes simpler.
  • On-boarding: when a new developer joins the team for instance, the diary helps a lot. The newcomer can learn a lot about the team activity, hot topics, who is who etc. by reading it as one of her initial activities.

Who is the logbook for

Everybody would benefit from contributing to a logbook but in some situation or environment, this practices provide significant benefits:

  • Remote teams, specially those distributed across different time zones. The fewer overlapping times, the more benefit you get from the diary. Engagement and alignment become key factors here.
  • Long term projects or products: the analysis / retrospective element makes the logbook a beneficial practice.
  • Teams working on more than one project (multitasking): teams that work in more than one project benefit a lot from having a log book per project and/or a team log book. In this case last case, you might want to consider a per project tag. Reporting is another obvious argument for using a logbook.
  • SWAT teams: engagement is the main factor here. Alignment is also an obvious factor, specially in management teams.
  • Large teams: the benefit here, beside those explained already, is the reduction of the need to provide status in the different team ceremonies, maximizing discussion, retrospective or brainstorming time, for instance. The status information is already recorded on the diary. Go and read instead of repeating already documented information.

Git based vs wiki

I always recommend to use a git-based tool for the logbook. It is not just that collaboration is easier, especially for developers, but also allows to integrate the habit of writing in their workflow easily. It will also be easier to structure and visualize the information through tags. Git is specially convenient for distributed teams too, which are the ones who benefit the most from this practice in terms of alignment.

Often the diary is used by people who does not know how to use git or is not part of their day to day workflow. I have had jobs in which I did not use git on regular basis. In such cases, a wiki can be the best option. Make sure you use a wiki with conflict resolution capabilities. Otherwise, the logbook will not scale. If you use a wiki that structure pages in editable sections, that might work too.

There are tools that combine the best of both worlds, like Gitlab, Github or zim-wiki. These are my favorites.

Structure and archive

I recommend to structure the logbook per day and per user. It doesn’t matter if we are talking about a product/project diary or a team one. Other options are possible but it is simpler to write down your entries one after the other one and use the tags to open the possibility to structure and visualize the information later on in different ways.

To use the logbook as reporting tool, at some point the write permissions should be removed. From that moment on, past entries should be amended in today’s section, not modified in the corresponding date.

When should the write permission be removed?

I suggest to do it weekly by default. I usually lock it down the following Monday at lunch time, providing the possibility to those who forgot to add their entries on the Friday before or were absent for whatever reason, to add their entries.

Project or product logbooks might use the sprint as the iteration, specially in cases where those sprints are two weeks long. But in general, if you have a good number of people writing in the logbook, I prefer a week long iteration.

To archive the diary, I usually move it away from the current one. I personally prefer to avoid scrolling or information from many days where I write. So I use a different file for past entries and current ones (that week). If you are using a wiki, move the content to a different page in read-only mode.

When and what to write?

As mentioned in the original article, you should include in the logbook what is…

  • Relevant to you.
  • Relevant to your peers.
  • Relevant to those you interact with like, managers, other teams, customers, etc.

Write down all the relevant events or any relevant information as soon as you have it. Write down what has been completed, solved, blocked… State facts and ask questions instead of writing opinions. Leave the judgement for the later analysis of the logbook. Keep them out of the logbook entries unless you create a specific tag for it (#self-note could be an example).

I add an entry at least every couple hours, before taking a break, except when I invest a longer period of time in an intensive task. So I usually have four entries or more daily in my logbook. I tend to write a lot more though. Other people write less than I do and that is OK as long as each one of us write down the relevant information for ourselves first, for our colleagues and finally to others.

Include decisions and agreements, conclusions, achievements, pitfalls, mistakes, external events that have influenced your work, external references and data sources that were relevant, etc. But remember, the journal is about what you did, not about what you need or should do, what is coming or what you think. Later on, with practice, you can expand the nature of the content. Make sure you agree worth other writers on this topic. It helps to keep the logbook clean.

One special type of entry is a comment on somebody else’s entry. I recommend to use it at the beginning only to add information or additional context to somebody else’s statement, not to provide any opinion or question. This is a journal, not a forum.

Ask yourself this question, what would I like to read about what any of my colleagues or myself are doing in 3 weeks or 3 months time?

How should I write?

Be concise. One or two lines maximum per entry is the best approach. Add links to the information sources, to the tickets, messages, bug reports, patches, logs, web pages… where the information is really generated and stored. Provide context to those.

Use ways to shorten the links to those common tools you use at work on daily basis. Add tags like dates, names and any other metadata that can help to contextualize and structure the information later on.

Remember, whatever you do, make it simple so adding information to the logbook is easy and fast.

Tags

One of the key elements of the journal is the capacity to structure and visualize later on the information through tags. As mentioned, the logbook is structured by date and user (two tags), but there are others you can and should use. The more experienced the team or project is with the logbook, the more tags can be used.

Warning: agree on the tags with your colleagues up front, otherwise you loose part of the filtering and visualizing of the information capabilities later on. If there is more than one logbook at your organization, agree on the tags with the other teams and projects. Define common tags.

I recommend to start with very few tags and increment their number over time. The tags to start with might depend a little on the environment, the type of team or project. These are the ones I commonly start with:

  • date and user
  • Mood: 😀 , 🙂 and 😦 Use these tags in entries but specially by the user tag, to let others know how you feel and to be able to evaluate later on your mood in different time windows.
  • Escalation: #red #amber #green Agree with your team and/or manager the meaning and reaction to each one of these tags.
  • Absence: #sick and #vacation Use them by the user tag to tell others that you or your colleague are absent.
  • Others: #decision, #agreed and #self-note

The tags are usually located at the end of the entry. There is one exception. When you are commenting on somebody else’s entry, start with the user tag and some indent. Check the example below for this case.

Different tools might have different restriction about how to define a tag. Some have restricted keywords. I added the symbol “#” because it is a common one. For users, many tools use the “@” symbol, for instance.

Use standard emojis instead of tool specific ones. Again, make it simple. Consider emojis a tag.

Example

This is a simple example of how the journal would look like:

May

02/05/2020

@toscalix 🙂

  • Blog post finished. Referenced previous entries.
  • Could go out to do exercise for the first time in over 50 days of confinement. I will go back out again tomorrow. #decision
  • A new tag #off-topic has been created #agreed

@peter 😦

  • Blog post reviewed. Ready to publish #decision
  • Network issues again. Third time in two days #amber
  • Send any policy to @toscalix before approval for review #self-note

Conclusion

Keeping team, project or product logbooks is a very useful practice. In the short term increases engagement and alignment, in the mid terms reduces the effort in reporting. In the long terms keeping a journal improves analysis and retrospectives. on-boarding is another topic that heavily benefits from the existence of a diary.

I would like to know about your prior experience with logbooks or similar practices. If you want to try it out, feel free to contact me for questions or advice.

Why Codethink is a founding member of the Civil Infrastructure Platform, a Linux Foundation initiative

This blogpost was originally published on the Codethink website on Thursday March 9th.

On April 4th 2016 a new Linux Foundation initiative called the Civil Infrastructure Platform was announced. CIP aims to share efforts around building a Linux-based commodity platform for industrial grade products that need to be maintained for anything between 25 and 50 years – in some cases even longer. Codethink is one of the founding members.

Industrial grade use cases

In order to describe why this initiative is relevant let me go over the use cases that motivate companies like Siemens, Toshiba, Hitachi, and Renesas to share efforts.
During the Open Source Leadership Summit, Noriaki Fukuyasu (Linux Foundation) and myself, based on the experience of Siemens, Hitachi and Toshiba, described the development life cycle in industrial grade use cases. For example, a railway management system is as follows:
  • Analysis + design + development: 3 – 6 years
  • Customizations and extensions: 2 – 4 years
  • The certification process and other authorizations take a year.
  • Each new release or update has to go through further certifications and authorizations that take between 3 and 6 months.
  • The system is expected to work for between 25 and 50 years.
So on average, an industrial grade product might take 5 to 7 years from conception to deployment. This is coherent with our experience in other industries like automotive, where life cycles are also quite long despite the expected lifetime being shorter.
A key part of the life cycle is maintenance. Due to its length, the associated risks are high. The certification processes to introduce significant changes in any already deployed systems are painful and expensive. In addition, the capacity to simulate a production environment is, in general, limited. This is true in other cases like energy production plans, for instance.

Open Source principles in the Civil Infrastructure industry

It’s obvious that Open Source could have a dramatic impact in this industry. By sharing efforts, corporations can commoditise a significant portion of the base system focusing on differentiation factors, increasing control through transparency and the quality of that starting point over time. Collaboration with upstream will bring even higher impact benefits.
Two immediate challenges come to mind when thinking about Open Source in this industry:
  • Development of processes and practices to produce software for safety critical environments.
  • Bridging the gap between the Open Source approach for software maintenance and the approach currently taken when building large-scale platform projects. For instance, how can approaches oriented to update any specific Open Source software component to the latest upstream stable version be compatible with any typical industry SDLC?

Can you reduce the gap?

We have for years been working on transformation projects for which one of the goals has been to reduce the gap between the software our customers ship and what upstream is continuously releasing. One of the key steps is to adapt an organisation’s processes using FOSS tools. Over the years we have been a strong advocate that the closer to upstream you are, the more benefits you reap from the Open Source development model, maintenance cost reductions being one of the main advantages.

So why did we get involved in an initiative that aims to maintain a kernel for 25 years then?

The short answer would be… because we love a challenge!
Safety critical with Linux-based systems is a challenge currently being faced in the automotive industry for instance, where Codethink is a strong player. When we analysed some of the industrial-grade use cases, it called our attention not just to the magnitude of the second challenge enumerated above, related with super long term maintenance, but also the apparent conflict between the industry requirements and the referred well known Open Source practices.
Hence the main driver for an Open Source consultancy like Codethink in participating in an initiative like CIP is to learn by doing, that is, putting the Open Source development, delivery and maintenance best practices under stress in one of the toughest environments. We bring our experience in producing embedded Linux based systems and our Open Source culture, to work together with industry leaders in finding solutions to these challenges, by looking at them with FOSS eyes.

Current activities

Codethink is participating in CIP in several capacities, the most relevant being:

Kernel maintenance
The first CIP approved kernel is 4.4, an LTS kernel supported until Feb 2018. Ben Hutchings is the initial CIP kernel maintainer. Besides providing support for the reference platforms, Ben is working on several activities like backporting the security patches, such as those from the KSPP and consolidating the maintenance policies, taking those from the kernel community as reference.
Testing tooling

kernelci.org is the most successful testing project in Open Source. Its impact in the kernel community is growing, as is the number of people and companies involved. It was designed and developed as a service where the testing activities can take place in distributed board farms (labs).
Codethink has been working on making the tools easy to deploy on developer machines through a VM, so they can test kernels on directly connected boards. This first milestone of the CIP testing project is called Board At Desk – Single Developer. This activity was described at the Open Source Leadership Summit 2017 and the first beta released during ELC 2017.

Conclusion

The challenges for Open Source that Industrial-grade product development and maintenance introduce are great, especially in two aspects: safety-critical and maintenance. Codethink is working on CIP to help the industry to overcome these challenges by adding our Open Source perspective.
Learn more about the CIP project by checking the following slides and videos from the conferences in which CIP members have participated.

Automotive supply chain and Open Source: a personal view

Software in automotive yesterday

The automotive industry has treated software like any other component, as part of the traditional, well structured and highly controlled supply chain. Tier-1’s has been providing software to car auto-makers for some years now and both together have done what they have could to prevent consumers or downstream players in the supply chain from manipulating it, improving it, customising it nor from adapting it. It didn’t matter if the software was Open Source or not, they have treated it as if it was proprietary, promoting locked-in practices. Only very few stakeholders with the right kind of agreement could manipulate it in a very limited way. Consumers and third parties didn’t have a say.

For a software company, there has been no life outside the traditional supply chain of any auto-maker.
Automotive yesterday
From all the reasons I’ve heard since I am involved in automotive, that justifies this situation, arguments related with security together with the safety critical nature of some systems shipped in a car are among the most popular ones.
But wait, once I buy a car, I can manipulate or even change the engine, the suspensions, the tiers, the brakes… but I can’t even dream about changing the software? Eveny if it is Open Source?

Software in automotive tomorrow

In my opinion the current way software is treated by automakers, the supply chain associated to them and the current business model around software in automotive will change dramatically. And the main reason will not be the license of the software used but the fact that the increasing amount and complexity of the software shipped with any car, together with the challenges that connectivity and privacy bring, will open up the door for new new players with new business models, Open Source business models. Those new factors will also provide current stakeholders in the supply chain the opportunity to increase their services around software. New players will challenge the current controlled and centralised environment.
I believe the software supply chain will expand beyond the purchase of the car, providing inputs at every level. Initially this will take place in a semi-controlled way, specialy by dealers, but later on software will become a major point at every stage, where there will be a wide offering of security and performance improvements, deployment of new functionality, maintenance services, customizations, integration with third party services, etc., to consumers directly by ISVs, enhancing the user experience adding more choice and adaptability to their particular use cases. The role software companies today play as providers will be substitued by a partnership relation.

In the same way that we all have a relative that “fix cars”, there are many developers out there that can put their hands on the provided software of their own vehicles. There are also many companies willing to do the same for their own fleet or somebody else’s one. A new car cost a hundred times more than a mobile and its life cycle is at least five times longer. I do not think the mobile industry will be the mirror for autmotive.
I believe that the picture will be closer to the current one in the enterprise industry, even if the journey to get there is diferent. Some automakers might remain as key stakeholders when talking about software, but not like today. 
Software in automotive tomorrow

The consumers demand for more and better services and the portability needs across the different platforms and devices (cars) in order to make business with software at scale, will increase the pressure on OEMs and Tier-1s over hardware and software standarization. That pressure might become in some cases as strong as govements regulations, I think. It will be definetly stronger that in the mobile industry. I see this as a positive factor for consumers.
Software in automotive soon
Experience shows that, if you intend to control a software ecosystem, you need to become upstream and create around your software a business model that serves as a service platform for third parties (ISVs). Maybe not even then you will be able to stay in control of what the customer is consuming. In other words, to be like Apple you need to control the hardware, develop at least the software platform and create a field for third parties to make money through “your platform”, which means at least controlling the distribution and updates. 
I do not believe automakers will be able to achieve that same level of control being only a Open Source consumers in a connected world, by restricting access to your platform to anybody but those who are part of your supply chain. not even if they become Open Source contributors. And no, the Android model does not apply to a single hardware (car) vendor.

I am obviously no guru so take all this as nothing but a personal opinion from an Open Source geek. But if you think it is feasable for an automaker to achive similar levels of control with Open Source based platforms than Google or Apple has over their ecosystems in the mobile industry, I think you are at least as crazy as you claim I am.

Codethink is hiring engineers.

Codethink is steadily growing for some time now. I have been working for 18 months in the company and it is about time for me to tell you that this is an outstanding place to work if you:

  • Love Open Source and work upstream.
  • Prefer to work in a startup kind of environment than in a corporate one.
  • Love to solve complex problem for first class customers instead of living peacefully in your comfort zone.
  • Like to travel to customers and conferences once in a while.
  • Understand that being challenged every day is the best way to improve.
  • Like international and multicultural environments with a British touch.
  • Think that Manchester, UK, is not a bad place to live or to visit once in a while. Hey, I am from the Canaries and too old to party night and day when I am not working, ut that might not be your case 😉
  • Like IRC, git, vim/emacs, RSS and some other technologies that newcomers to Open Source consider… old school.
  • Are not scared of sending patches by mail.

But above all, you are willing to learn no matter how senior you are!

Codethink is an independent consultancy company based in Manchester, UK, although it has some people like myself working remotely. Our customers are also spread around the world. We are around 75-80 people now, most of them, engineers.

Most of our work is related with Linux kernel, low level system activities, compilers, delivery of linux systems, distro and SoC work.

Let me know if you could be interested in working at Codethink or send directly your CV to jobs@codethink.co.uk

Automotive, what an opportunity for KDE!

During the last year I’ve focused a significant part of my effort on driving the GENIVI Development Platform, together with some Codethink colleagues and other GENIVI professionals and community members.

What is GENIVI Development Platform?

GENIVI Development Platform (GDP) is a project and an outcome.

As a project, it can be defined as the delivery side of the GENIVI Alliance. Today is a fairly standard Open Source project, done in the open following many of the most common practices any FLOSS developer would expect.

As an outcome, GDP is a Linux base distribution (Poky) derivative, built with Yocto, that integrates the software that GENIVI community (automotive professionals) develop as Open Source software.

It still a small project but the quality of the platform and the number of people involved has grown this last year significantly.

GENIVI Alliance is a consortium of +140 companies so obviously most of the overall effort is done by paid developers. Changhyeok Bae (community member) together with Tom Pollard and Robert Marshall (Jonathan Maw before him) from Codethink Ltd, constitute the maintainers team, who are responsible for the integration, testing and release of GDP.

These guys are supported by people like myself, doing coordination, marketing, documentation, IT services, infrastructure, testing and many other key tasks which provides the project a level of robustness and scalability that any serious attempt of this nature requires nowadays.

I am interested, where can I get more?

You can find more general information about the project in the following resources:


GDP-ivi9 is the current stable version although we have moved so fast this last year in terms of the software shipped, that I recommend you to try the following:

  • If you are interested in a solid base system, try GDP 11 RC2
  • If you are interested in checking the latest UI and demo applications, try GDP 11 RC3
  • If you are interested in building from scratch you own images with the latest software, do it directly to GENIVI’s rolling release, called Master for now. You can get the latest software there. It should most ly work since we put stabilization effort on it, following the openSUSE Tumbleweed mindset in this regard.

Currently GDP supports RPi2 & 3, Minnowboard MAX and Turbot, Renesas Porter and Silk and Qualcomm Dragonboard 410c. GDP ships Qt 5.6 at the moment, since it is based in Yocto 2.1…

…which makes GDP a great target for KDE software, specially for Plasma.

 

GDP and KDE

Putting the effort on having KDE well supported in Yocto would provide the project a third life, landing on an industry that is heavily investing in Open Source with a key piece of software, with no clear competitor today in the open.It would revamp the interest of many KDE developers in porting their apps to embedded/mobile environments and would bring attention to the project from Qt professionals all over the world. Currently KDE is significantly better than anything else that is open in automotive. It would just require the effort to include it and maintain it in Yocto, which is not small, and adapting Plasma a little initially, not much.GENIVI launched a Challenge Grant Program that might help to put some funding in the equation 😉

Whatever effort done to put Plasma on Yocto (so GDP) would also be picked up by GDP’s competitor, AGL UCB (Auto Grade linux Unified Code Base), the Linux Foundation automotive group Linux (again, Yocto) based distribution. So there would be at least two players for the cost of one.

It wouldn’t surprise me if Qt companies would jump in on this effort too. In order to play in the open filed today, they need to Open Source their products, which is a big risk for most of them. Playing with KDE, which is based in the technologies they are familiar with, would be simpler for them. I bet the Qt company would be heavily interested in promoting this effort. It would help to dissipate all the pushing back from the automotive industry to the current Qt license model, GPLv3 based. And it would do it in the best possible way, by providing great ready-to-use software with no competitor. I have been one year preaching about how big the opportunity currently is for KDE, but this is not the kind of challenge that can be sustained on volunteer basis, sadly, since keeping KDE up to date in the Yocto project would require a high level of commitment from KDE as a whole.

The community probably needs first a small success story and some company/corporate push before really jumping on it, I think. The support of a couple of KDE or Qt companies would catalyze the effort.GENIVI and AGL make a significant promotion effort around the world within the automotive industry, participating in forums where KDE is unknown. Many companies that currently develop close source Qt applications for automotive would be interested in KDE which would increase our potential targets for our Corporate program.

Having KDE on GDP and AGL UCB would increase the incentive of developing new applications for many of our young developers who currently do not have automotive as a “professional target“. 

Companies like LG, Renesas, Bosch, Hartman, Intel, Jaguar Land Rover, Toyota, Visteon, Fujitsu, Mitsubishi, Volvo, among many others…. are key stakeholders of GENIVI and AGL. Isn’t it this attractive enough?A success story like the one I am proposing would be yet another example of how KDE can play a key role for the Qt ecosystem. Sadly not everybody in this ecosystem understand what a great “tool” KDE can be for them. After a hit like this one, it would be undeniable.Think about the exposure, think about where we are today… Something like this would place KDE where Unity, Android (AOSP) or GNOME are not… yet. I believe this is the kind of strategic decision that would change KDE future. But also the business perspective of those companies (specially Qt ones) who would get involved.

Let’s do it now… or somebody else will.

Update(29/10/16): to find out about the state of KDE in Yocto please read the comment to this article from Samuel Stirtzel. Thanks Samuel.