3.26 retrospective

[Allan Day <aday gnome org>]

Hi all,

I personally found 3.26 to be a difficult release and thought that it might be useful to share some of my experiences, along with some thoughts on how to improve things for next time around. The issues I encountered:

1. We had two big features that landed very close to UI freeze, both of which required design iteration. This put a lot of stress on me personally and meant that other design reviews that I had planned never happened (sorry Builder).
• It's worth noting that if it wasn't for these features, the release notes would have been very sparse indeed - from that perspective I'm happy that they did land.
  
2. A design change that we had landed early in the cycle, with the intention of polishing it later, failed to resolve itself. As the issue rolled on it was often unclear what the plan was and what the deadline was for rolling back the change that had been made earlier in the cycle.
3. The release notes were seriously delayed due to 1 and there was uncertainty about screenshots because of 2. There were also various issues building the release, which made the process of taking screenshots painful and time consuming.

I'm sure that some of these issues are down to individuals. However, it does seem interesting to think about their wider causes, with a view to improving the release process. In my mind there are a number of these wider issues, including:

• Landing complex changes close to UI freeze:
• This interrupts other planned work, as people have to scramble to refine the new feature.
• If you care about UX quality, it can be stressful!
  
• It makes a bit of a mockery of the UI freeze, since features are dropped with the expectation that the freeze can be freely broken in order to refine them. This in turn creates uncertainty about whether we can expect freeze breaks to be accepted.
  
• Unrealistic expectations for what we can provide with each release, in terms of quality and feature set.
  
• Lack of awareness of, or at least uncertainty around, what our UX release blockers are.
• Uncertainty as to how late UI changes can be made.
• Lack of general involvement in the QA phase of our releases - while some of us are sweating over what we're going to put into the hands of users, a lot of people are working on other things.
• Outside of the maintainers, there isn't much awareness of the release schedule and the various deadlines that comprise it. This is particularly the case for non-developers.
  
• Lack of general awareness of the current state of master.

Below are some ideas for things we could possibly do to improve the situation. I'm sure that some of these aren't good and that you all will have better ones!

• Extend the UI freeze (or create a freeze period for major UI changes) and require that breaks get design approval.
• Either let go of the idea that each release will have an interesting feature set or figure out a way to plan releases so that they're interesting and high quality.
  
• Make it clear at which point UI freeze breaks will no longer be accepted.
• Give marketing a say in whether UI freeze breaks are accepted. (And find someone other than me to take responsibility for this!)
• Include a UI review exercise as part of the release schedule, around the UI freeze. Tie this in with identifying release blockers.
• Do more "product management" - track more issues for each release, publicise them more.
• Ensure that there are bootable VM images that can be used to review the state of master.
• Get people using nightly apps. Could Software have a way to bulk install nightly Flatpaks? Could we have some kind of QA programme that encourages people to use nightlies and report issues, particularly in the run up to a release?
• Publicise the release schedule in a way that reaches non-developer audiences: have a public calendar that anyone can subscribe to (and perhaps allow other teams to add their own deadlines), make announcements on social media, etc...
• Make sure that JHBuild isn't broken, particularly around release time.

Thoughts? Opinions? I'd be happy to invest some time in this.

Allan