[BuildStream] Cache keys



Hi All

After some recent experiences I believe we still haven't got this right!

Here are some thought I have had after talking to Tom Mewett and Thomas Coldrick.

Some time a plugin changes in a fully braking way and we need to bump the version and change the cache key of every element built with or from the plugin. I have come to believe this to be true for bst unless we make some huge changes.

But often we add a feature to a plugin, (I have just added git-lfs support to the git_tag plugin)

This plugin will only change its behavior if its cache key is changed.

ie. The plugin only changes its behavior if you set `use-lfs: True` and if you set that then its unique key will change.

If we left the plugin version the same then only plugins who's behavior has changed will have there cache keys changed. But users of projects using the plugin but who do not have the new version will get unhelpful error messages when the plugin can not validate the `use-lfs: True` propriety

So I must bump the plugin version so that users get useful messages but I am also going to cause a huge amount of compute work as the project invalidates almost all of of its cache keys.

The reality is that compute is relatively cheap so this is not the end of the world and is unavoidable for many plugin changes.

But its still not inconsiderable.

My Idea...

Could we add `features` to our plugins cache keys as well as versions?

Plugin authors can already do this to some extent, by having a

source:
        kind: bob
        url: bob.com
        features:
            use-shinnys: True

Then just validate for features being a dict and if any of the keys are not recognized then raise a error.

and then

add 'use-shinnys' to the unique key only if it is used.

This way if a user uses `use-shinnys: True` but they have a old plugin then an error message can be generated at configure time saying that a newer plugin is needed. But this is not as helpful as the version number as the version number tells the user exactly which version of the plugin they need.

Another issue with this is that in order to deal with bugs in use-shinnys we really need some thing like:

source:
        kind: bob
        url: bob.com
        features:
            use-shinnys:
                use: True
                version: 1

This seems to be getting out of hand again.

Dose anyone else have a better idea, i can see we need to do better but its a tricky problem, (We would have solved it if it wasnt :) )

Will


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]