Apply agile methodologies to upstream development environments…. if you can.

 Introduction

When the Agile Manifesto became popular and based on them, agile methodologies like Scrum, XP or Kanban, upstream development was in its early stages as collaboration ecosystems of companies.
Only a few for profit organizations embraced developing upstream back then. Most of them were small and heavily influenced by FLOSS engineers vision. Free software communities were basically driven on personal basis or the very lucky ones, together with “sponsored developers”. In general, these ecosystem were not part of companies strategies.
Today, more and more companies are getting fully involved in community projects as stakeholders, not just consumers or simple contributors.
They frequently start as consumers, then, little by little they become “upstreamers”, that is, they share/publish their code with the goal to have it merged (upstream code). Not without effort, many of them become successful contributors.  After some time, some of them end up understanding that is “cheaper” to play by the project rules. In summary, they learn to become good citizens.
A subgroup of the above companies end up including these collaboration ecosystems as part of their own strategies, going from contributors to  key stakeholders. A necessary step to achieve this goal is to work upstream.
Walking this path present many challenges. One of the toughest ones is related with the differences in development methodologies used internally (mostly agile) and those used in the collaboration ecosystems.
There are two fundamental variables that, in my opinion, determine this challenge:
  1. Environment
  2. Culture

Challenges

1.- Environment

There are two dependent variables that were not taken into account (or just partially) when the agile methodologies were defined, that are relevant in upstream development:
  1. Community projects are global environments, that is, contributors are located in different “offices”, frequently in different time zones.
  2. Probably due to the original amateur condition of early contributors, together with the “distributed condition”, the development processes (so the tools) in most mature community projects, consider, manage and tolerate high levels  of latency.  “Real time” is restricted to IRC discussions and events/conferences.
These two factors has made open source what it is today. They have been “success factors”.
Agile methodologies do not embrace “distribution” environments. The widely accepted recommendation is that teams should share a physical space. It is way more than a recommendation. It is somehow a requirement.
The second case, “latency”, is considered by agile methodologies as a waste. It is not tolerated.

2.- Culture

Free Software was born as a reaction to a system that promoted corporation interests over developers, so users. The agile movement was a reaction to those methodologies that put process first, not people. Hence, it is obvious that both movements share a lot: people first
This is reflected by some when saying that FLOSS development is agile.
In my opinion, there is a big difference between what agile methodologies and what Open Source development propose in terms of principles.
Agile methodologies promotes a strong team culture. Open Source was born “based on champions”. FLOSS culture normally applies the meritocracy concept to individuals.  Open Source projects are organized around contributors, around specialists, not around teams, as we understand them in corporate environments.
This is no surprise since Agile was born in companies/corporations and Open Source was born as a viral movement, grown “by aggregation“.

The conflict

In my opinion, the more the industry embrace open source, and as result, open collaboration, the higher the conflict developers and managers will face due to the above challenges.  Companies are becoming more distributed environments and are working more and more upstream, instead of simply being consumers or occasional contributors.
In consequence, it would not surprise me if we hear more and more about  “corporate development methodologies” (a.k.a. agile) vs. “upstream development methodologies” (a.k.a. FLOSS).
Scrum, XP, Kanban -ish fans will need to face those challenges and find solutions in order to succeed in open collaboration environments. In the same way, based on the increasing influence that companies are gaining in these ecosystems, FLOSS methodologies in a few years will differ from what we knew 10 years ago.

This conflict will not be (is) about a R&D vs a product/service vision, it is not about creativity vs efficiency, it is not about micromanagement vs autonomy or teams of juniors vs specialists either. It is about methodologies applied to specific environments and its limitations. Maybe a simple update of the most successful agile methodologies will do the job…. or maybe we need to revisit some of the principles.

If you got here, maybe you want to take an extra step and answer these questions. I would appreciate it:

  1. Do you perceive this conflict as I do?
  2. Am I missing other key elements in the diagnosis?
  3. How do you think we can adapt agile methodologies so they can be adapted to FLOSS environments?
  4. I am interested in knowing how you adapt agile methodologies to overcome the above challenges. I plan to write about my experience these coming days.

Where the corporate and the upstream worlds meet…. or collide.

From the corporate world I frequently hear how hard it is to predict and track what upstream developers do. On the other side, developers that work part or full time  upstream frequently underestimate the need for communicating what they do in a way that enable others (or themselves) to provide deadlines and effort estimations. Upstream and product “time lines” and cultures often differ too much to be compatible under the same environment.
Developers that come from the “product” side of the story often refer to agile methodologies like Scrum as a good approach to solve this collision. It is unclear though the applicability of this and similar methodologies to highly distributed environments, where latency is high. Although most accept today that FLOSS development is agile, I find hard to assume that we are close to find good answers when it comes to development methodologies and upstream.
There are several different scenarios I have worked on where I have to question everything I knew up until then about this particular topic. In a couple of occasions, with a bigger or smaller community, the people I directly worked with were upstream. This allowed us to define up to a great extend the methodology we worked with.
In Linaro now this is the case for a particular project. But in general, in my Group we work upstream (in the Linux Kernel), that is, we cannot define the methodology since it comes defined by the project itself. It is not the first time I face this situation although I never did at this scale.
When you are upstream, the methodology used by the core team, together with how the project work packages and workloads are organised heavily influences the success in getting contributions from third parties. Together with other key variables, how you manage latency determines how many people can follow you and potentially contribute. In summary, the faster you move the harder it becomes for contributors to collaborate with you.
Hence agile methodologies come with a high risk: isolation. And this is obvious by just analyzing the prerequisites associated with the methodology itself.
Obviously this does not mean that scrum and other methodologies are not great ones. I am just pointing challenges associated with apply them when working in a community that you drive. You need to balance the efficiency of your team with the contributions you might potentially loose from externals.
The challenge is even bigger when you are just a participant in a community. In some way, your team goes beyond your colleagues at work. Latency is frequently too high to even consider yourself and the people that works close to you “a team” in the classical sense of the word.
If your own team is distributed, even if they are in the same time zone, then we are talking about the real deal, the master challenge for managers and developers. 
Imagine now that your team is formed by 10 people distributed in 6 different time zones in 3 different continents. Then you take one of those cool books that tries to explain how to create great software using this or that methodology and, in order for you to finish it, the writer must be a really good one or you really need to be an avid reader 🙂 
Still you need to make plans, to predict when a feature will be finished, when it will be merged, you still need to manage dependencies, expectations, budgets…. you still are tight to somebody else needs and requirements. You still need to fulfil expectations despite the fact your control over the environment is limited. You have team mates that depend on your work, that needs to know what you are doing and how…..  you are still part of a business no matter if you develop upstream or not.
How can you make compatible the upstream development processes used in the community you are contributing to with the way your company works? And how about your customers and partners, if you have direct relation with them? How can you take the good things that agile development propose and apply them to an environment with high latency, where one of your bigger challenges is to have an efficient team meeting, since people are distributed across the planet?
At Linaro, the engineering management team, together with our PMO, are taking a close look at this challenge, in order to iterate from our current system to a more efficient one, better adapted to our new reality, after our significant growth during the past 18 months. The idea is to find a place where corporate and community processes meet… without colliding. You cannot stop adapting, innovating, trying new things….. or you pay a high price.
One of the great things about Linaro is that we are a unique environment so we can innovate at various levels, not just at the technology one.

Open Sorce Forum at CebIT

For those of you attending to CebIT, let me tell you that I am giving a talk about what we do at Linaro on Monday March 16th at the Open Source Forum. It is the first time I talk about my current job in public so I am very excited about it. It will be also my first time in CebIT so double doses of excitement. And yes, I will talk a little bit about Linaro new initiative, 96boards.

Core Dump

The past months we have published a few articles about what we do at Core Development Group at Linaro. Our blog is called Core Dump. Check it out.