The concept of inventory applied to software, sometimes called catalogue, is not new. In IT/help-desk it usually refers to the software deployed in your organization. Along the history, there has been many IT Software Inventory Management tools. I first started to think about it beyond that meaning when working in deployments of Linux based desktops at scale.
The popularity that Open Source and Continuous Delivering is providing this traditionally static concept a wider scope as well as more relevance. It is still immature though, so read the article with that in mind.
1.- What is Inventory in software product development?
I like to think about the software inventory as the single source of truth of your software product so the main element for product development and delivery auditing purposes.
Isn’t that the source code?
Yes, but not only. The source code corresponding to the product that you ship (distribute) is a big part of it, but there are other important elements that should be considered part of the inventory like:
- Requirements and/or tests, logs and results.
- Technical documentation.
- Tools and pipelines configuration files.
- Packages, definitions or recipes…
- Hashes, signatures, crypto libraries
- License metadata, manifests, etc.
- Metadata associated to security checks, permissions descriptions… .
- Data associated with process performance metrics and monitoring/telemetry.
- Many more…
When defined that way, the Software Inventory is a concept relevant in every stage of the software product life cycle. When you introduce, change, produce, publish, deploy or distribute any element of your product portfolio, your software inventory should change too.
There are two interesting considerations to add.
1.- If your product is part of a supply chain, like in any Open Source upstream/downstream environment, then the software inventory concept expands and become even more relevant since it can become an essential inbound-outbound control mechanism, even at acquisition time.
2.- In critical environments, especially safety critical ones, keeping such single source of truth goes beyond a “good practice”. Integrity, traceability and reproducibility for example, can be simpler to manage with a Product Software Inventory.
When you think about this particular case, it becomes clear to me that the elements that belongs to the inventory go beyond the actual deliverables or “product sources”. It should also include those elements and tools necessary to generate them, transform them, evaluate, deploy/ship them and evaluate its purpose.
2.- Static vs dynamic concept
Considering the above, the Software Product Inventory is a living construction, so dynamic, with the capacity to be frozen at any point in time (snapshot). This might seem obvious but it implies a different approach than supply and release management has traditionally considered (deliverables).
If evaluating, adding, modifying or managing elements of the inventory requires any action that significantly increases the cycle time of any specific stage, decompose those actions, parallelize them when possible and, when there is no choice, push it right in the pipelines. Ideally, no Software Product Inventory related activity should produce any friction in the code flow.
In a Continuous delivery environment, implementing the inventory requires to take actions across the entire development and delivery processes. Here are some points to consider at key stages:
2.1.- Inbound process: stage 0 of the development process
No software or any other element can become part of the product portfolio if it is not present in the Software Inventory. It make sense to implement the Product Inventory concept as part of the inbound stage (stage 0). Following Continuous Delivery principles and practices, here are some things to avoid vs promote:
- Handovers or manual/committee-based vs code-review-like approval processes (pull vs push).
- Completeness vs walking skeleton approach.
- Management oriented (document based) vs engineering oriented (git based) tooling whenever possible.
- Reports (manual) vs evidences (automated and reproducible) as output.
- Access control vs signing and encrypting (if needed).
Avoid gate keeping activities. It is better to promote high throughput and short feedback loops than “quality gates” to improve product quality. If an evaluation is not completed, it is better to tag such piece of software as pending for a decision and letting the code flow than to have the engineers waiting for a decision of third parties.
I recognize that the concept might be too abstract to be easy to buy at first beyond the inbound and outbound (release/deploy) stages. Sadly, there is a strong tendency to pick up the concept at the inbound stage to establish early on a gate keeper, committee-based process to control the software that the developers use in the project, frequently compromising the code flow at a very early stage.
I prefer to focus on the procurement stage in the case of suppliers or how the relation is established with partners first. These are hand-over processes that heavily benefit from restructuring them, reducing the acquisition and on-boarding time and conditions.
More frequently that I would like to admit, Open Source is becoming a driver in this wrong direction, in many cases due to the proliferation of Open Source Offices in corporations that prefer to focus their initial attention in establishing specific policies for their own developers than in to changing their relations with partners and suppliers.
This is frequently due to a lack of understating of software product development at scale and what Continuous Delivery is about. In a nutshell, having their own engineers selecting the right Open Source software is prioritized over changing the relation with their existing commercial ecosystem, a more difficult but higher impact activity in many cases, according to my experience.
2.2.- Outbound process (deployment or release): last last stage of the delivery process.
The inventory accumulates all the elements required to ship/deploy the product plus all the elements required to recreate and evaluate the development and delivery process as well as the product itself, no matter if they are released or deployed. Ideally, this elements are evidence-based instead of report-based.
Like in the inbound case, each element of the Inventory should signed/encrypted as well as the overall snapshot, associated to the deployed/released product version. In case you are consuming or producing Free Software, please see Open Chain Project specification for more information about some good practices.
2.3.- Intermediate stages
As previously mentioned, the concept of Inventory is relevant at every stage of the development/delivery process. In general, it is all about generating additional/parallel outputs within the pipelines, signing and storing them in association with the related source code and binaries in a way that those evidences become “part of the product”. Using proprietary tools might break the trust chain in your process. This is something to consider carefully in safety critical environments. You will also need to consider the hardware, including dev. versions and prototypes.
A very interesting and open field is the Inventory concept in the context of safety critical certifications that traditionally have been very report-heavy-oriented. In this regard, I find the usage of system thinking very promising. Check Trustable Software, for instance.
3.- Some practical advice
3.1.- Walking skeleton vs completeness
I love the walking skeleton concept to design and implement processes in product development. It is significantly better to establish and end-to-end approach to the Inventory, where it has a light/soft/incomplete presence along the entire development/delivery cycle, than trying to implement it stage by stage following completeness, preventing you from having process-wide feedback loops.
It is not so much about doing it right as it is about moving fast in the right direction.
For instance, a frequent mistake is to concentrate most of the activities related to software license compliance on the inbound and outbound stages. Software license conformance and clearance has traditionally been perceived in many industries as a validation process performed by specialists, just like testing was done not so long ago, or a procurement action (acquisition).
Although lately more and more corporations are promoting the execution of license compliance activities at both stages (inbound and outbound), since they consume and ship more and more FOSS, they are still very report-based, specialists driven and management controlled activity.
I have witnessed enough dramatic situations to understand and promote that software license compliance is everybody’s job, just like tests or technical documentation (everything as code approach). Software license conformance and clearance, together with security, testing and technical documentation, can become the key drivers of the implementation of the Product Inventory concept. They share the same principles in this regard. The history of testing is the mirror to look at.
Decompose the software license compliance activities (conformance and clearance) and perform them across your pipelines. Start by executing simple conformance checks (REUSE) early on (inbound process, for instance). Coordinate such activities with the security team to also perform simple static code analysis. Agree with the architects or tech leaders in checking coding guidelines or other elements that can have a future impact in quality taking advantage of the Inventory concept. Add not just the software and the checks to the Inventory but also the results, logs and simple tools/scripts used.
More time consuming and intensive activities using more complex static code analysis or code scanning (licenses) tools can use the inventory as source (pull approach), instead of requesting the teams to perform such activities on their own (outside the pipelines) or establishing hand-over processes with specialists.
Be careful about how and when you include such activities in the pipelines though. Again, decompose and parallelize. And only when there is no choice, push these activities right in the delivery process. But do not break the code flow.
3.2.- Keep It Simple Stupid until it is not stupid anymore.
Here are some simple actions to start with…
At the beginning at least, use for the Inventory the same tools you are already using for the development of the product. Initially, your inventory can be nothing more than a file with a list of repos, hashes and links pointing at product elements location. This already have a value for security and software license compliance teams.
If you use a multi-repository approach pay attention to where the build tool pull the software from (definitions/recipes) to integrate the product. Make sure your initial inventory and the build tool are “in sync”. This will have a tremendous impact later on.
Export, sign and export the technical documentation living in your repositories (markdown, plantUML, .svg etc) as documents if they need to be part of the product deliverables, so you can establish simple checks to confirm their presence , integrity, etc.. This outputs should be also part of the Inventory as well.
Many of you already perform these and many other activities as part of the development and delivery process. The question is what to do with the associated metadata, the tooling used to generate them, the intermediate states required to get the output, the executed scripts, how to related them with the code and with other elements from previous stages of the process, how you guarantee their persistence, integrity, how you manage them at scale, how to store them etc..
3.3.- Act as an auditor
I always ask myself the following question: if I replicate the complete product development system providing all the product inputs for auditing purposes, how can I save time to the auditor so she does not need to understand the system itself and perform all the actions again to fully trust the output and the system itself (what/how we did, evidence based) and not myself or the workforce (who did it, report based)? Remember that you or your team will be such auditors in the future.
3.4.- Do not push the concept too early
If your stating point is very management (so reporting) driven, like for instance an environment where requirements or license information is generated and kept in .docx documents, the usage of binaries over source code is the norm, where proprietary tools are not questioned or where trust in the actual processes and outputs are based on who signs the associated reports (Officers) instead of in evidences, save your self headaches and do not try to push the Inventory concept beyond your area of direct influence. In my experience, it is worthless. You have a different battle to fight, a different and deeper problem to solve in such case.
In such cases, simply get ready in your area of influence for the future crisis to come, that will hit your product, so you get a chance to become part of the potential solution. Look for allies like the security, software license compliance and technical writing teams for instance, to expand that ring of influence. Hopefully they will see the value and the inventory how it can become a trigger to support modern and widely accepted practices within their domains across the organization.
4.- Summary
The Software Product Inventory is a high level (abstract) concept that will help you to move towards creating trustable software. Since it is part of the development process, following Continuous Delivery principles and practices to design and implement it is essential.
Some of the concepts behind this idea are probably present already in your organization but frequently using a management-driven approach, associated to the release and/or inbound stages, implemented in ways that generates a negative impact (friction) in the product code flow, so throughput and stability.
System thinking and asking questions as if you were an auditor will help you to implement simple measures first, and habits later, that will raise the quality of the product over time, even if the Product Inventory concept does not fly in your team or organization. That approach has helped me, at least.
toscalix 