[BUG] compare-linux-win CI build check fails due to zephyr_version.h difference #9797
Comments
kv2019i
added
bug
|
@thesofproject/nxp Rings any bells to you? I checked multiple PRs merged last week, but I couldn't find a PR where this failure would have been seen (already pre merge). I could have missed some. |
|
@LaurentiuM1234 @marc-hb didn't we have this issue couple of months ago? Will need to check on that. |
|
@kv2019i looks like this is not only for imx builds. |
kv2019i
changed the title
This is the easiest bit to focus on first. You can downloads artefacts from every PR page, please compare these.
|
Actually, daily tests are more convenient for this: because zero PR interference. Started failing in daily 795 , 4 days ago: https://github.com/thesofproject/sof/actions/runs/12940934767
Don't wait because artefacts expire after some time. |
|
There's been a progressive rollout of a new Windows image ( Version: 20250113.1.0 -> 20250120.2.0) around the same time but it does not correlate with the failure. EDIT: it's unrelated. The problem is on the Linux side. For instance, the IMX build got the newer Windows image in 794 but 794 was still all green: As you can tell from the version numbers, minor upgrades like these happen all the time. The first step is really to download and compare |
marc-hb
changed the title
marc-hb
changed the title
|
It would seem the builds really are different now. I wonder if this is somehow related to reent cold/llext change -- the .cold section at least looks different. E.g. here for github build 796 (https://github.com/thesofproject/sof/actions/runs/12940934767#artifacts) The debug sections have been different already before, so I wonder if this just pushes out the sections to different addresses (and use of cold sections just triggers this). |
|
That will indeed impact the binary. |
|
Thanks @kv2019i . Indeed, the difference can be observed in the build log: versus A long time ago, I added magit west and git tricks to speed things up (a lot). They worked fine for a very long time but maybe some subtle git version difference or other environmental difference just broke them. I'll take a quick look. EDIT: The Linux + |
|
Around the time of daily build 795, there was a progressive rollout of Ubuntu 22 VM version 20250120.2, up from 20250105.1 This upgrade seems to correlate 100% with the loss of the git tag in git describe. The newer Ubuntu VM upgraded git from version 2.47.1 to version Git 2.48.1 Meanwhile, git on Windows did not change and stuck to https://github.com/actions/runner-images/blob/ubuntu22/20250105.1/images/ubuntu/Ubuntu2204-Readme.md |
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
More progress towards trimming down pull-request.yml Important benefit and "secret" agenda: remove the constantly failing sof-docs from daily builds, which will make them green again. Green daily builds are important to quickly spot regressions like for instance thesofproject#9797 Signed-off-by: Marc Herbert <[email protected]>
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
More progress towards trimming down pull-request.yml Important benefit and "secret" agenda: remove the constantly failing sof-docs from daily builds, which will make them green again. Green daily builds are important to quickly spot regressions like for instance thesofproject#9797 Signed-off-by: Marc Herbert <[email protected]>
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue thesofproject#9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
More progress towards trimming down pull-request.yml Important benefit and "secret" agenda: remove the constantly failing sof-docs from daily builds, which will make them green again. Green daily builds are important to quickly spot regressions like for instance thesofproject#9797 Signed-off-by: Marc Herbert <[email protected]>
More progress towards trimming down pull-request.yml Important benefit and "secret" agenda: remove the constantly failing sof-docs from daily builds, which will make them green again. Green daily builds are important to quickly spot regressions like for instance thesofproject#9797 Signed-off-by: Marc Herbert <[email protected]>
More progress towards trimming down pull-request.yml Important benefit and "secret" agenda: remove the constantly failing sof-docs from daily builds, which will make them green again. Green daily builds are important to quickly spot regressions like for instance thesofproject#9797 Signed-off-by: Marc Herbert <[email protected]>
|
Fix submitted. This should also turn the daily tests back to green. |
Fixes the git describe/tag performance hack added in commit 2328478 (".github/zephyr.yml: fix tags missing from `git -C zephyr/ describe`") which worked for an amazingly long time (1.5 year) but apparently ran its course. Git version 2.48 apparently does not like it anymore. Replace it with something slower but simpler and safer. Should fix build reproducibility issue #9797, much more details there. Also fixes commit 4bc6488 (".github/zephyr: de-hardcode the name of the zephyr remote") Signed-off-by: Marc Herbert <[email protected]>
More progress towards trimming down pull-request.yml Important benefit and "secret" agenda: remove the constantly failing sof-docs from daily builds, which will make them green again. Green daily builds are important to quickly spot regressions like for instance #9797 Signed-off-by: Marc Herbert <[email protected]>
|
After dropping sof-docs from them, daily tests are back to green! https://github.com/thesofproject/sof/actions/workflows/daily-tests.yml 
|
Status update:
Starting with daily build 795, https://github.com/thesofproject/sof/actions/runs/12940934767
the Linux +
mnft(manifest) builds started to lose git tags in thegit describeoutput that goes into their BUILD_VERSION. For instancev4.0.0-2813-g42701fdb2729became42701fdb2729. The Windows builds kept it and the Linux "zmain" (Zephyr main branch) kept it too.This correlate with a Ubuntu VM upgrade, see below.
Describe the bug
Starting today 27th Jan 2025, multiple pull-requests are showing a failure in build reproducibility between Linux and Windows builds with
error:
https://github.com/thesofproject/sof/actions/runs/12947318306/job/36114439673?pr=9794
This is shown in multiple open PRs:
To add to the mystery, I can't see the failure in any of the pull requests failed last week.
To Reproduce
Submit a pull request to SOF repository.
Reproduction Rate
100%
Expected behavior
Linux and Windows host builds should create identical binaries.
Impact
Failure seen on every CI request, can mask other errors.
The text was updated successfully, but these errors were encountered: