GUADEC Topics 2: BuildStream Roadmap definition and management

[Agustín Benito Bethencourt <agustin benito codethink co uk>]

Hi,

Once the project management measure that target increasing the visibility of 
WIP and alignment are being put in place, it is time for me to focus some 
energy on making agreements to work together on the BuildStream roadmap. 
Please consider the following proposal as something to start implementing now 
but the real emphasis will come right after our next stable release.

## Why is it important for BuildStream to focus significant energy in the 
roadmap?

Road-mapping is the first step towards building the RIGHT thing, in the journey 
of creating the right thing right. 

When a bunch of smart people develop a product (a toolset in our case), the 
main risk is to end up creating something  nobody really wants (loves). It is 
easy to think that if you build something better or that cover a need, success 
will come. Sadly that is a naive approach to reality. Those teams or 
organizations that put emphasis in the roadmap reduce that risk.

Unlike in a commercial products, in Open Source projects the figure of a 
Product Owner makes no sense so how to take decisions around  our roadmap and 
steer contributions in the direction set by it depends up to great extend on: 
having a common vision on the tool, strong leadership around it, how well 
defined and attractive that roadmap is, together with how well we meet users 
needs, so controversy around what needs to be built remain low, among other 
things. 

## What is a Roadmap?

The roadmap is where the agreements around which kind of tool we want to build 
are reflected. This agreements comes out of ideas and requests that should be 
discussed and mature through what the agile movement calls "conversations". 
The outcome of these conversations should be tracked first, defined as a story 
that can be told to people within and outside the project. Once the roadmap is 
agreed and each action is described into user stories, it comes the time to 
prioritize those user stories that we want to build. This process is flexible 
and done through successive iterations. 

This is where the roadmap phase ends. Engineering starts by slicing these 
already prioritized stories into engineering tasks of the same size 
approximately.

A lot has been written about how to move from requests or ideas into 
engineering tasks in co-located teams. Very little though has been written 
about how to put it in practice by distributed or remote teams. So we will 
need to find our own way to manage our product roadmap by the book in terms of 
principles and adapt it to our remote and Open Source project nature when it 
comes to processes and practices. The good news is that we will not be the 
first ones doing this and there are many techniques that can be easily adapted. 
We also meet a couple of times per year which it is the right time to focus on 
the roadmap.

Initially, we will try to go as lightweight as possible, as usual, following 
the same principles of the Gitlab proposal. 

## What will we gain by having a solid roadmap?

* Roadmap discussions help to confront the different visions around the 
project. Ideally everybody will share all or part of the vision. In practice, 
that needs to be worked out regularly.

* Done it right, road-mapping strength alignment, specially in environments 
with distributed decisions making nodes, like Open Source projects.

* It reduces waste when it comes to engineering tasks since engineering tasks 
are defined based on mature and well defined needs.

* It helps to keep a clear and share high level view of the project and, what 
is important, where is it going and why. This is essential when projects scale 
up.

* A Roadmap is a living entity, that is analyzed and adapted on regular basis. 
It is the opposite of a requirements driven development.

* User stories include Acceptance criteria which leads to the creation as part 
of the engineering tasks of Acceptance Tests, reducing the Code Review effort.      
   * Acceptance tests will become an essential part of the delivery process 
overtime.

## References

* Story map: https://jpattonassociates.com/user-story-mapping/

* User story: https://en.wikipedia.org/wiki/User_story

## Proposal 

BuildStream roadmap process would have 4 main stages:

1. Brainstorming
2. User story definition.
3. (User) Story map definition.
4. From user stories to engineering tasks. 

### 1.- Brainstorming 

* Input from users and developers should take place through the mailing list 
first. 
   * Whenever somebody have an idea related with the BuildStream roadmap, 
please send a mail to the buildstream mailing list with the subject starting 
by: [Roadmap]
   * Dependencies should be considered/ explained.

* A discussion about the idea will take place in the mailing list with the 
goal to evaluate how feasible the idea is, define its impact and ensure 
completeness. 
   * At this point the goal is not to evaluate if the idea will be 
implemented, by who or with which priority.

* Once the idea is mature enough, it will be moved to the ticketing system. 
It is recommended to present the idea in the mailing list already following 
the user story template so the next step becomes easier but is not a 
requirement. 

### 2.- User Story definition 

* Once the idea is mature enough, it will be introduced in the alignment 
project as a ticket, following a template (user story template). 
   * A specific label will be added to the ticket.
   * A link to the discussion should be added to the ticket.
   * This step can be done by the presenter of the idea or by somebody else.    

* The goal is that the presenter of the idea is comfortable with it as much as 
those who might implement it and use it. An exercise of consensus should be 
driven at this point.

* Once the story map is completed, it will be introduced in the story map. 

3.- Story map definition 

* I propose to use the Story Map as a representation of our roadmap. We might 
complement this representation with others, like per release (milestone) 
feature lists. 
   * A roadmap is an alive representation so it will be constantly evolving.
   * The story map discussion is the phase where story maps are validated and 
prioritized. 
   * User stories might suffer changes as a result of evaluating them together 
with the current story map, other coming stories or under execution.
   * It is strongly recommended that Story Map discussions take place f2f.

* Additional metadata will be introduced in the ticket to reflect its state in 
the roadmap.

* Once the story map is defined and every relevant story has been considered, 
it is time to turn the high priority user stories into engineering tasks.
   * There are cases in which a user story should be developed or completed 
and cannot wait for the f2f story map discussion. In such case, the specific 
user story will be discussed remotely. 
   * A roadmap is a declaration of intentions in an Open Source context, never 
a commitment. This is why user stories are prioritized, not assigned to 
detailed timelines at this point. 

* In the story map, each user story will be linked to the ticket describing 
the user story.

### 4.- From user story to engineering task 

* Those user stories that will be immediately executed will be turned into 
engineering tasks. 
   * Each User Story will be linked to those engineering tasks that  come from 
it. 

* Engineering tasks should be designed to be done within a week approx.
   * I propose to start with this granularity, assuming we are all working in 
the open, and represent the main assignment for an engineer.
   * If there are meaningful engineering tasks that should be smaller, that is 
fine. If they are bigger, the general practice is to slice them in weekly 
tasks.

* Engineering tasks are described within a ticket, following a template, 
included in the current buildstream (project) ticketing system. 

## Additional notes and clarifications

* The above process proposal is an end goal. The transition from where we are 
will take us some time.

* Agustin volunteers to guide the process. 
The described process is a summary/simplification. I understand there are lots 
of questions to be asked. I hope I can answer most of them during GUADEC. 

* Once the proposal is solid, it will be part of our project policy 
document[1] for the record.

* I have adapted standard processes to our Open Source nature, based on 
previous experiences and literature: how to turn product ideas, feature 
requests and requirements into prioritized small enough tasks that engineers 
can execute.

* There will be exceptions and things that will require quick adaptations. We 
will need to be flexible when required.

* There will be engineering tasks that, because they are simple enough or 
obvious, might not require to go through the entire process. 
   * As rule of thumb, I would say that any task that constitute the main 
focus of anyone for some time and introduces a new functionality, improve an 
existing one or changes a existing one, should go through the process.
   * This point always generates controversy. It will require us to put in 
place general policies and adapt them over time, allowing room for flexibility.
   * There is a positive side effect of this process which is to have 
engineering tasks of about the same size, which will be the base for being 
able to forecast. Forecasting becomes essential for business reasons (budget/
costs/investment).

* The end result will be to have a new ticket space for road-mapping and 
another on for engineering tasks and bugs (the current one). Both have in 
common the mailing list.

* A side effect I have looked for with the proposal is that those engineers 
who need to participate in turning a user story into engineering tasks will 
know exactly what to focus on and when, based on a previous discussion about 
priorities. 
   * The process of creating engineering tasks should be easier than today 
because the user story will be mature enough, if the process has been followed 
correctly. This will save late discussions about what I should have been doing 
vs what I am actually doing.

* Road-mapping is the kind of activity that benefits from f2f conversations. 
But in order to be increase efficiency, some work should be done remotely.

* Those of you unfamiliar with product environments might find the above 
overwhelming. Let me tell you that is not and the benefits are so evident that 
it is becoming a trend in software development in an increasing number of 
industries.

* (User) Story Maps are developed from the user/consumer point of view which 
help developers to keep the focus on those who will use the tool.
I will show at GUADEC an initial version of our Story Map. I have shown it to 
some of you already. 

## Goal for GUADEC 

I am extremely interested in returning from GUADEC:

* Knowing if such a process will fly or not.

* If it will, your opinions, ideas and concerns about it.

* Ideally, I would like to be able to start working on it right after GUADEC 
so we can focus on the Roadmap right after our stable release is out. There is 
work that needs to be done in preparation for that moment. 

[1] https://gitlab.com/BuildStream/nosoftware/alignment/blob/master/
BuildStream_policies.md

Best Regards
-- 
Agustín Benito Bethencourt
Principal Consultant
Codethink Ltd
We respect your privacy.   See https://www.codethink.co.uk/privacy.html