Re: [BuildStream] Cache keys

[William Salmon <will salmon codethink co uk>]

oups, my examples got messed up formating, sorry, please see inline

On 08/01/2020 12:22, William Salmon via buildstream-list wrote:
> Hi All
>
> After some recent experiences I believe we still haven't got this right!
>
> Here are some thought I have had after talking to Tom Mewett and Thomas 
> Coldrick.
>
> Some time a plugin changes in a fully braking way and we need to bump 
> the version and change the cache key of every element built with or from 
> the plugin. I have come to believe this to be true for bst unless we 
> make some huge changes.
>
> But often we add a feature to a plugin, (I have just added git-lfs 
> support to the git_tag plugin)
>
> This plugin will only change its behavior if its cache key is changed.
>
> ie. The plugin only changes its behavior if you set `use-lfs: True` and 
> if you set that then its unique key will change.
>
> If we left the plugin version the same then only plugins who's behavior 
> has changed will have there cache keys changed. But users of projects 
> using the plugin but who do not have the new version will get unhelpful 
> error messages when the plugin can not validate the `use-lfs: True` 
> propriety
>
> So I must bump the plugin version so that users get useful messages but 
> I am also going to cause a huge amount of compute work as the project 
> invalidates almost all of of its cache keys.
>
> The reality is that compute is relatively cheap so this is not the end 
> of the world and is unavoidable for many plugin changes.
>
> But its still not inconsiderable.
>
> My Idea...
>
> Could we add `features` to our plugins cache keys as well as versions?
>
> Plugin authors can already do this to some extent, by having a
>
> source:
>      kind: bob
>      url: bob.com
>          features:
>              use-shinnys: True
source:
    kind: bob
    url: bob.com
    features:
        use-shinnys: True
> Then just validate for features being a dict and if any of the keys are 
> not recognized then raise a error.
>
> and then
>
> add 'use-shinnys' to the unique key only if it is used.
>
> This way if a user uses `use-shinnys: True` but they have a old plugin 
> then an error message can be generated at configure time saying that a 
> newer plugin is needed. But this is not as helpful as the version number 
> as the version number tells the user exactly which version of the plugin 
> they need.
>
> Another issue with this is that in order to deal with bugs in 
> use-shinnys we really need some thing like:
>
> source:
>      kind: bob
>      url: bob.com
>          features:
>              use-shinnys:
>                  use: True
>          version: 1
source:
    kind: bob
    url: bob.com
    features:
        use-shinnys:
            use: True
            version: 1
> This seems to be getting out of hand again.
>
> Dose anyone else have a better idea, i can see we need to do better but 
> its a tricky problem, (We would have solved it if it wasnt :) )
>
> Will
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list