Store buildhistory artifacts

[ndechesne]

The buildhistory feature in OE is really cool. I would like for us to find a way to get the buildhistory for:

1. Pull request. e.g. the diff between before and after the PR. There is a lot of data in there, which can be very interesting to check when we review a PR.
2. Between each nightly builds.

[koenkooi]

Do we want to store just a snapshot in each build artefact or go for a full history and pull down a buildhistory git repo, commit the changes and push those back?
It's been a few years since I attempted to have multiple concurrent runners share a buildhistory repo, it needed clear documentation of trade-offs, since the output might travel back in time, so to speak, due to some runners being slower than others.

[ndechesne]

I think what would be nice is:

1. Store the buildhistory for each branch (main for now, but stable branches later). so that we keep a history of what each 'merge' changes.
2. have the buildhistory diff generated for each PR. e.g. the delta between the 'baseline' and the PR

[lumag]

There is an issue with that: the builds for the branch might finish out-of-order, e.g. if several PRs are merged quickly. So you might end up with the buildhistory which doesn't fully represent the history of a branch.

[ndechesne]

You are right. However that should be occasional. and we could catch that during the PR CI, e.g. check that the 'base' commit has finished building, and that its buildhistory data is available, and fail the buildhistory job otherwise. maintainers could then re-trigger it.

[ricardosalveti]

Once several merges are performed together (several minor changes, or as a sequence once they all have the required approvals), that would cause a series of job failures for buildhistory, which is also a bit of a pain later to manually trigger in sequence (while other jobs could also be started based on new merges).

Are you thinking on just keeping the buildhistory folder in a shared location?

[koenkooi]

1. For PRs we can just show the difference (be it the raw diff or a processed version like buildhistory-diff) without pushing the new commit.
2. For trigger-on-commit builds, we do something similar to the PRs above: don't store the resulting commit and only show the difference
3. For nightly builds we should store the resulting commit, possibly doing git pull -Xtheirs right before the push.

This will result in having the diff available for inspection in PRs and triggered builds as well as having a full history of day-to-day changes.

And we have to decide how much we care about having a linear, correct and complete buildhistory. As @lumag points out above, it is extremely easy to have runners go out of sync which will case build failures as @ricardosalveti points out. git push failing because there's a newer commit is easy to workaround, but the options are not pushing or merging and pushing possible outdated output.

For Angstrom we decided that having the CI build only one machine for each architecture+target gave enough buildhistory coverage, the churn in all/noarch/allarch was an acceptable tradeoff. Sstate protects us a bit from churn, but it will take a non-trivial amount of work involving locking around both builds and buildhistory manipulation if we require a linear, correct and complete buildhistory. Or we build a database backend :)

[ricardosalveti]

One CI build per architecture / target should be enough to start with, not great, but as you said, can be an acceptable tradeoff.