busybox not reproducible for same commit same board build from CI without cache

[tlaurion]

Edit: false. Same commit with same build environment produces same rom. Unfortunately, the build system changed within 30 days (packages installed by apt on top of debian-11 docker, where two builds made on top of the same docker-11+same packages applied the same day produces the same rom, with hashes being different then the past) .

Conclusion: again, we need a reproducible toolstack, a docker image that would be pinned in the same commit as Heads and used under CircleCI.

---

This means we have other problems that would not be resolved by having a reproduced environment; eg: build time of some module being injected in the rom.

Diffoscope between two clean builds on the same environment are needed.

Insights given while rebuilding a same commit over CircleCI since cache expired (latest commit being 30 days ago): #1224

[tlaurion]

Something helpful would be to hack CircleCI limitations, and add a step i CircleCI builds to output each board's hashes.txt so that we can easily target what module causes the discrepancy to ease troubleshooting of reproducible builds in the future.

[tlaurion]

Something helpful would be to hack CircleCI limitations, and add a step i CircleCI builds to output each board's hashes.txt so that we can easily target what module causes the discrepancy to ease troubleshooting of reproducible builds in the future.

Lol, sorry for the noise: we already have that.

[tlaurion]

A quick comparison between hash outputs of the boards builds shows us that busybox is the culprit, consequently affecting tools.cpio, initrd and final rom image hashes.

-c1e89cebf3b2176d3b95340f336ef5335e9aee5fbbb4b217302f4c8b36e71be5  ./bin/busybox
+dd52e6cb5e0751bcb8bef82508fbd9a29fc32afc0ed66ce0049c2870ad4de705  ./bin/busybox

Renaming issue

[tlaurion]

EDIT: cause of difference is most probably linked to debian-11 docker image build stack differences here, since building from a clean github checkout produces the same rom from CircleCI.....

Again linked to #1191 #927, and most importantly voiding #936 impacts on reproducible builds: that is, controlling what is poked by configure scripts to have a reproducible outcome (rom here).

Creating a reproducible docker image with NixOS/Guix is again the lesser effort with the biggest outcome here.

[tlaurion]

Sadly, https://try.diffoscope.org/ doesn't have cbfstool and fallsback to binary comparison when provided with 2 coreboot roms. This is sad and I haven't found quickly a way to ask gently to have cbfs installed in their testing suite so users can provide ROMs to be compared online with their tool.

---

Rebuilding with CircleCI's project settings's variable CACHE_VERSION set to f2e45c4 to rebuild from clean checkout.

Comparable builds with artifacts will be

• (two days ago): https://app.circleci.com/pipelines/github/osresearch/heads/446/workflows/2ccdf262-3d1a-486d-b173-a39229ca9000
• (now) https://app.circleci.com/pipelines/github/osresearch/heads/446/workflows/c2ff73e5-df8f-4989-ac36-f373a15f34b3

Let's remember we can compare hashes.txt between builds, which states hashes for all components, some of which are packed under others. Let's inspect https://output.circle-artifacts.com/output/job/7d0bc3de-9cf5-43b7-a53a-bc06f2ef7005/artifacts/0/build/x86/x230-hotp-maximized/hashes.txt content quickly:

• bzImage is Kernel, independent. Ultimately packed under coreboot image
• modules.cpio: contains drivers not compiled in kernel. Packed under initrd.cpio.xz and ultimately packed under coreboot image
• tools.cpio: contains everything Heads tools (and /etc/config which is config at build time). Packed under initrd.cpio.xz and ultimately under coreboot image.
• heads.cpio: contained everything policies (scripts and distrib public keys and such). Packed under initrd.cpio.xz and ultimately under coreboot image.

Reusing trick under #590 (comment) permits us to download initrd.cpio.xz from artifacts and compare the binaries directly: xz -d initrd.cpio.xz ; for i in initrd.cpio modules.cpio tools.cpio heads.cpio; do cpio -i < $i; done

[tlaurion]

user@heads-tests:/tmp/osresearch1$ wget https://output.circle-artifacts.com/output/job/7e7ba054-0c2a-4224-8805-451ac17ca74c/artifacts/0/build/x86/x230-hotp-maximized/hashes.txt
user@heads-tests:/tmp/osresearch2$ wget https://output.circle-artifacts.com/output/job/04eb7afa-3cf5-4a60-b9bb-f4cd9661536a/artifacts/0/build/x86/x230-hotp-maximized/hashes.txt
user@heads-tests:~/heads$ diff /tmp/osresearch1/hashes.txt /tmp/osresearch2/hashes.txt 
1c1
< 2022-10-17 14:29:30-04:00 f2e45c4396456679284eeda9a26196ee7a3a14d1 clean
---
> 2022-10-19 12:58:13-04:00 f2e45c4396456679284eeda9a26196ee7a3a14d1 clean

Hmmmm. Ok. So building clean from CircleCI builds the same ROM today.

[tlaurion]

For safe measures, I deleted my 5.0.1 branch and created an issue in my personal github, so that when I ask github to sync against master, it triggers a build with same versioning as master.

---

Notes on versioning: heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c4.rom

• heads-x230-hotp-maximized : board name
• v0.2.0: latest tag associated with a release (github)
• 1263: number of commits since tag
• gf2e45c4: g+commit so commit here is f2e45c4
  We can ask users to go under System Settings and extract from there the commit ID of what they are using.

---

If the following build creates the same hashes as master, then we have a way to tell users to currently validate "reproducibility" of builds, asking them to fork osresearch heads project and have CircleCI follow their branch for project.

Test happening under https://app.circleci.com/pipelines/github/tlaurion/heads/1217/workflows/65c200d2-cfd7-47d3-a357-9fe89c8bf380

[tlaurion]

Interestingly enough as well, under bookwork, coreboot-utils exists, so having diffoscope installed permits to compare roms between them easily:

• sudo apt diffoscope coreboot-utils wget
• wget https://output.circle-artifacts.com/output/job/7e7ba054-0c2a-4224-8805-451ac17ca74c/artifacts/0/build/x86/x230-hotp-maximized/hashes.txt https://output.circle-artifacts.com/output/job/04eb7afa-3cf5-4a60-b9bb-f4cd9661536a/artifacts/0/build/x86/x230-hotp-maximized/hashes.txt
• wget https://output.circle-artifacts.com/output/job/7e7ba054-0c2a-4224-8805-451ac17ca74c/artifacts/0/build/x86/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c4.rom https://output.circle-artifacts.com/output/job/04eb7afa-3cf5-4a60-b9bb-f4cd9661536a/artifacts/0/build/x86/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c4.rom

And then

diffoscope on the roms downloaded returns nothing (no diff)

user@coreboot-diffoscope:~$ diffoscope heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c4.rom heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c4.rom.1 
user@coreboot-diffoscope:~$ 

And we confrm this by comparing the hash files, which shows only differences with dates of produced hashes:

user@coreboot-diffoscope:~$ diffoscope hashes.txt hashes.txt.1 
--- hashes.txt
+++ hashes.txt.1
@@ -1,8 +1,8 @@
-2022-10-17 14:29:30-04:00 f2e45c4396456679284eeda9a26196ee7a3a14d1 clean
+2022-10-19 12:58:13-04:00 f2e45c4396456679284eeda9a26196ee7a3a14d1 clean
 99dbb152674f0353bcd184572c4911fb52091bdbeceee2e5fe8c3e0adc269d94  /root/project/build/x86/x230-hotp-maximized/bzImage
 a2fe6993f933f9a99a8d2b999f400ae57f834bc6443679c45db879c1dcdf3222  /root/project/build/x86/x230-hotp-maximized/modules.cpio
 -----
 fab88a49e7c8f4455d936ad9db3bec5b0422d7e05eeb0a7503c070de732b46c4  ./lib/modules/e1000e.ko
 ae3cd021b058cba2db8124caa1b140e466a8be948eea7ed9861e43bbf769111d  ./lib/modules/ehci-hcd.ko
 0ccc01f9c82551fd34a47b8ab1e7d9df403c72006eb7a8ef3b1763d2edd3d6b5  ./lib/modules/ehci-pci.ko
 c31c91426b84b0e7a29aed13d6fc6f582803217bc574e42d1993f0fd05ad2adf  ./lib/modules/xhci-hcd.ko

[tlaurion]

Hmmm. For some reason, my branch produces ROMs appended with a 3 which is not the commit id reported by github (7 chars vs 8):

• x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c4.rom became
• x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1263-gf2e45c43.rom

Example of diffs in rom from diffoscope (where a lot more changes since offsetted by 1 char everywhere)

-00bf3d60: 3563 3400 2d54 494d 454c 4553 532d 2d4c  5c4.-TIMELESS--L
-00bf3d70: 4553 5354 494d 452d 0066 9066 9066 9090  ESSTIME-.f.f.f..
+00bf3d60: 3563 3433 0066 9090 2d54 494d 454c 4553  5c43.f..-TIMELES
+00bf3d70: 532d 2d4c 4553 5354 494d 452d 0066 9090  S--LESSTIME-.f..

https://app.circleci.com/pipelines/github/tlaurion/heads/1217/workflows/65c200d2-cfd7-47d3-a357-9fe89c8bf380/jobs/10938/artifacts

So of course, doing a diffoscope on both binaries shows differences on parts of the rom relaying that part which is considered to be the commit and included at different spaces in the firmware packing.... Will have to investigate

[tlaurion]

Still confused though....
Checking back and reviewing Makefile, the version that is part of the filename is the result of

user@heads-tests-debian-12:~/heads$ git describe --tags --dirty
v0.2.0-1263-gf2e45c4

So why that would be different output from my CircleCI instance is unknown for the moment but is impactful.

[tlaurion]

Another batch of tests, removing CACHE_VERSION for both osresearch and tlaurion CircleCI configurations (since it might be my fault since both caches were with commit version and might have caused a bug with CircleCI.

Next builds to compare:

• osresearch https://app.circleci.com/pipelines/github/osresearch/heads/446/workflows/9f8117ac-e03a-4b95-bd4f-565d0ad42aae/jobs/4805
• tlaurion https://app.circleci.com/pipelines/github/tlaurion/heads/1217/workflows/d33d91a1-0c8a-47a2-9f82-8cb44459dd82/jobs/10978

[tlaurion]

There is a difference between git reports of what is specified in Makefile to obtain git tag+commit: HEADS_GIT_VERSION := $(shell git describe --tags --dirty)

osresearch:
v0.2.0-1263-gf2e45c4

tlaurion:
v0.2.0-1263-gf2e45c4

Seems like a way to fix this would be to set the abbrev option to fixate output.
--abbrev[=<n>] use <n> digits to display object names

Github shows commit ids in an abbrev form of 7 characters. We should force the same inside of the builds for consistency and to avoid reproducibility issues on that:
HEADS_GIT_VERSION := $(shell git describe --tags --abbrev=7 --dirty)

Locally, this fixes the issue:

user@heads-tests:~/heads$ git describe --tags --abbrev=7 --dirty
v0.2.1.bis2-144-gf2e45c4-dirty
user@heads-tests:~/heads$ git describe --tags --dirty
v0.2.1.bis2-144-gf2e45c43-dirty