[Bug] CI builds for SFOS 2.2.0 are abnormally large

[Olf0]

SailfishOS VERSION (Settings → About product → Build): Coderus' SDK docker-images 2.2.0, 2.2.1, 3.0.0 & 3.0.1

HARDWARE (Settings → About product → Manufacturer & Product name): GitHub CI → Docker → Coderus' SDK-image → Scratchbox2 / MB2

FileCase VERSION (FileCase → [Top pulley] Settings → [Top pulley] About): all versions, e.g. 0.4.2-rc1

BUG DESCRIPTION

CI builds for SFOS 2.2.0 to 3.0.1 are abnormally large: > 4 MBytes, which is a magnitude larger than the usual 400 - 500 KBytes for all other CI builds.
See: https://github.com/sailfishos-applications/filecase/releases/tag/rc1%2F0.4.2

STEPS TO REPRODUCE

Trigger the ci-on-tags workflow by setting an appropriate git-tag.
Exemplary workflow (including logs) yielding the aforementioned result: https://github.com/sailfishos-applications/filecase/actions/runs/6681032070

ADDITIONAL INFORMATION

N/A

[Olf0]

@nephros, do you have any idea, why this happens?

Briefly looking at the logs did not provide me with any clue.
I have not tested the overly large binary yet (still have a [email protected] around).

[nephros]

I can only guess - likely a difference in the SDK Setup.

Either there is some link bloat happening (e.g. unnnecessary libs linked into the binary).

Could also be that the binaries are not stripped. IIRC all "modern" build setups split up generated rpms into regular binary RPM plus the -debuginfo one.

I seem to remember from waaay back that adding --enable-debug to the call to mb2 controls some of this. (IIRC enabling that caused the -debuginfo RPMs being built, thereby decreasing the size of the non-debuginfo RPM.).

On a lower level, from a quick web search it seems the RPM macro %debug_build seems to control this also. So setting that for the 2.2 builder would be the first thing I'd try.

[nephros]

Another approach:

https://doc.qt.io/qt-5/qmake-variable-reference.html#installs

Perhaps the sailfishapp qmake config sets the nostrip config option in 2.2 but does not do that in newer releases.

[EDIT]: Nope, doesn't look like it: https://github.com/sailfishos/libsailfishapp/blob/master/data/sailfishapp.prf

[Olf0]

Could also be that the binaries are not stripped. IIRC all "modern" build setups split up generated rpms into regular binary RPM plus the -debuginfo one.

I seem to remember from waaay back that adding --enable-debug to the call to mb2 controls some of this. (IIRC enabling that caused the -debuginfo RPMs being built, thereby decreasing the size of the non-debuginfo RPM.).

I can confirm all of these statements. BTW, this is also applicable to OBS' debug-flag at build.sailfishos.org.

Thank you for the brainstorming: I will extract the binaries from the ballooned RPM file some day and check if they have debug-info embedded.

P.S.: Another idea is to try if utilising Coderus' SDK-image for SFOS 2.2.1 yields a different result than the one for 2.2.0.

[Olf0]

P.S.: Another idea is to try if utilising Coderus' SDK-image for SFOS 2.2.1 yields a different result than the one for 2.2.0.

Tested with 2.2.1.18: Same result, 4436972 bytes raw / 4421693 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

[Olf0]

P.S.: Another idea is to try if utilising Coderus' SDK-image for SFOS 2.2.1 yields a different result than the one for 2.2.0.

Tested with 2.2.1.18: Same result, 4436972 bytes raw / 4421693 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

Tested with 3.0.0.8: Same result, 4437256 bytes raw / 4421989 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

[Olf0]

P.S.: Another idea is to try if utilising Coderus' SDK-image for SFOS 2.2.1 yields a different result than the one for 2.2.0.

Tested with 2.2.1.18: Same result, 4436972 bytes raw / 4421693 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

Tested with 3.0.0.8: Same result, 4437256 bytes raw / 4421989 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

Tested with 3.0.1.11: Same result, 4436976 bytes raw / 4421709 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

[Olf0]

P.S.: Another idea is to try if utilising Coderus' SDK-image for SFOS 2.2.1 yields a different result than the one for 2.2.0.

Tested with 2.2.1.18: Same result, 4436972 bytes raw / 4421693 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

Tested with 3.0.0.8: Same result, 4437256 bytes raw / 4421989 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

Tested with 3.0.1.11: Same result, 4436976 bytes raw / 4421709 Bytes gzip'ped RPM for i486 only (thus > 8 GB for i486 & armv7hl, as observed with SDK for 2.2.0).

Tested with 3.0.2.8: Now down to 541568 bytes raw / 525117 Bytes gzip'ped RPM for i486 only (thus ~ 1 GB for i486 & armv7hl, as observed with SDK for 3.1.0).

[Olf0]

Exactly the same effect of ballooned binaries when building with Coderus' SDK-images 2.2.0, 2.2.1, 3.0.0 and 3.0.1 (but 3.0.2+ is fine) is observed for FlowPlayer.

[nephros]

While I don't think it's the case with this issue, one reason for unstripped binaries being packaged binaries was OBS builder running out of disk space during stripping:

• Binaries not stripped. Karry/osmscout-sailfish#327
• https://forum.sailfishos.org/t/no-space-left-on-obs-build-sailfishos-org-during-debug-info-processing/17846

[Olf0]

Thank you for the hint, but I also do not think that GitHub-workflows run out of disk space easily.

I still have to take a closer look at the resulting, oversized RPMs rsp. their content and specifically if the contained binaries are stripped or not (i.e contain debug symbols).

This is on my ToDo list but with low priority, hence becomes moved down the list again and again.

[nephros]

I found this sailfishos/meego-rpm-config@879399a which changes behaviour of the rpm stripper script (find command within).

Also comparing the files /usr/lib/rpm/brp-strip from the Tooling packages for 3.0.1.11 and 3.0.2.8 does show these differences.

Whether or not the different calls to find can not tell ATM, but as a workaround you could do a similar call manually in/after the %install phase to ensure stripping. Or at least get meaningful output on failure.

[Olf0]

@nephros, thank you very much for having continued research on this phenomenon. I also think you have found the root cause of it, though I still do not fully comprehend what exactly is happening. Still, you opened a path to a solution which seems to be well viable even with rough / partial understanding why this occurs.

Ultimately your efforts triggered me to perform the inevitable: a thorough analysis what the effects are in detail. For a good part because I was increasingly having the impression you spend more time on this than me in my role as a repo maintainer, which I and maybe also bystanders may perceive as an imbalance, socially.

Well, on to the technical stuff:

1. Accidentally I started with FlowPlayer, not FileCase, which makes not difference, as the issue is exactly the same.
2. I unzipped the downloaded RPM-build-results_SFOS2.2.0.29+.zip archive in a directory with this name, which unpacks two RPM files, one for armv7hl and one for i486.
3. I used Midnight Comander (mc) to extract the main flowplayer executable file from each RPM file and saved it in a correspondingly created armv7hl rsp. i486 subdirectory.
4. At this point the directory structure looked looked this:

   $ cd ~/Downloads/flowplayer/0.3.2-rc5/
   $ ls -lR RPM-build-results_SFOS2.2.0.29+*
   -rw-rw-r-- 1 bsi bsi 6363910 Feb  9 02:08 RPM-build-results_SFOS2.2.0.29+.zip
   
   RPM-build-results_SFOS2.2.0.29+:
   total 6244
   drwxrwxr-x 2 bsi bsi    4096 Feb 20 04:20 armv7hl
   -rw-r--r-- 1 bsi bsi 3207369 Feb  9 00:55 flowplayer-0.3.2-rc5.armv7hl.rpm
   -rw-r--r-- 1 bsi bsi 3171545 Feb  9 00:55 flowplayer-0.3.2-rc5.i486.rpm
   drwxrwxr-x 2 bsi bsi    4096 Feb 15 20:16 i486
   
   RPM-build-results_SFOS2.2.0.29+/armv7hl:
   total 13132
   -rwxr-xr-x 1 bsi bsi 11975156 Feb  9 01:54 flowplayer
   
   RPM-build-results_SFOS2.2.0.29+/i486:
   total 11772
   -rwxr-xr-x 1 bsi bsi 12049164 Feb  9 01:55 flowplayer
   $ 
   

5. Next I took a look at the two executable files with file:

   $ cd RPM-build-results_SFOS2.2.0.29+/
   $ file armv7hl/flowplayer
   armv7hl/flowplayer: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=353b3a12adc7847baf0aa03b992c9cd4930af122, with debug_info, not stripped
   $ file i486/flowplayer
   i486/flowplayer: ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=d1a55af8bb05840fe802fba9ee414866d6a6ff9f, with debug_info, not stripped
   $ 
   

   So these executables contain debug-info and are not stripped (of all symbols).
6. GNU strip 2.34 (from GNU Binutils) fails to auto-detect the format of an armv7hl executable on an x86-64 computer, though by the help of the man page and the output of strip --info it is obvious that the appropriate option to fully strip an armv7hl executable of all symbols is: strip -v -I elf32-little -o armv7hl/flowplayer.strip-all armv7hl/flowplayer
   The same file size is yielded when no additional option or one of the equivalent options -s or --strip-all is used. For stripping an i486 executable on x86-64 the option -I elf32-little can be omitted, but it naturally does no harm when always employed (as it is still technically correct for i486 executables).
7. By looking at sailfishos/meego-rpm-config@879399a @nephros provided above, the aim is to strip only the debug information (i.e. debug symbols, plus DWARF and debugging sections) by using the option [-S|-g|-d|--strip-debug], of which -g seems to be the preferred notation by POSIX, at least for the c99 and f77 compilers.
8. Because the "verbose" option -v yields no information of interest, my second round of stripping (this time solely the debug info) was performed by strip -g -I elf32-little -o armv7hl/flowplayer.strip-debug armv7hl/flowplayer and strip -g -o i486/flowplayer.strip-debug i486/flowplayer.
9. The final result was:

   $ ls -lR armv7hl/ i486/
   armv7hl/:
   total 12688
   -rwxr-xr-x 1 bsi bsi 11975156 Feb  9 01:54 flowplayer
   -rwxrwxr-x 1 bsi bsi   451528 Feb 15 20:17 flowplayer.strip-all
   -rwxrwxr-x 1 bsi bsi   554724 Feb 24 22:54 flowplayer.strip-debug
   
   i486/:
   total 14380
   -rwxr-xr-x 1 bsi bsi 12049164 Feb  9 01:55 flowplayer
   -rwxrwxr-x 1 bsi bsi   643740 Feb 24 22:55 flowplayer.strip-all
   -rwxrwxr-x 1 bsi bsi   726676 Feb 24 22:55 flowplayer.strip-debug
   $ file */*
   armv7hl/flowplayer:             ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=353b3a12adc7847baf0aa03b992c9cd4930af122, with debug_info, not stripped
   armv7hl/flowplayer.strip-all:   ELF 32-bit LSB shared object, no machine, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=353b3a12adc7847baf0aa03b992c9cd4930af122, stripped
   armv7hl/flowplayer.strip-debug: ELF 32-bit LSB shared object, no machine, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=353b3a12adc7847baf0aa03b992c9cd4930af122, not stripped
   i486/flowplayer:                ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=d1a55af8bb05840fe802fba9ee414866d6a6ff9f, with debug_info, not stripped
   i486/flowplayer.strip-all:      ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=d1a55af8bb05840fe802fba9ee414866d6a6ff9f, stripped
   i486/flowplayer.strip-debug:    ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=d1a55af8bb05840fe802fba9ee414866d6a6ff9f, not stripped
   $ 
   

   The file sizes of the armv7hl rsp. i486 executables stripped of their debug information are almost identical to the ones produced by the Sailfish-SDK for SFOS ≥ 3.0.2.

Consequently strip -g -I elf32-little <executable> is the generic command usable; while it would be elegant not to apply it to executables produced by the Sailfish-SDK for SFOS ≥ 3.0.2, "double-stripping" makes no difference, but when using the Sailfish-SDK for SFOS ≥ 4.0.0 one has to strip aarch64 executables with -I elf64-little (on x86-64 and i486, i586, i686) or (better) not at all, because support for 64-bit target architectures was introduced by SailfishOS 4.0.0.

P.S.: Using the option -F elf32-little (i.e. set in- and output-format to the same) instead of -I elf32-little (set only input-format) does discard the architecture information for armv7hl executables, too. How to prevent this from happening still has to be researched.

[Olf0]

#45 (comment):

I found this sailfishos/meego-rpm-config@879399a …
… you could do a similar call manually in/after the %install phase to ensure stripping.

Do I understand correctly that the idea of putting an excerpt from your finding into the %install section of the spec file also alleviates the architecture mismatch, because Scratchbox2 (SB2) / MB2 / MOC / the Sailfish-SDK always compile in a VM matching the target architecture (i.e. they perform no "real" cross-compilation, correct?), hence supplying strip with an -I or -F option is not needed?

P.S.: I think I finally understood, why setting up a build environment with Scratchbox2 (SB2) / MB2 is much easier than employing "real" cross-compilation (despite being much slower): I expect many such corner-cases (as with strip) to exist, and these do not exist in a VM emulating the target architecture via qemu.

[nephros]

Do I understand correctly that the idea of putting an excerpt from your finding into the %install section of the spec file also alleviates the architecture mismatch, because Scratchbox2 (SB2) / MB2 / MOC / the Sailfish-SDK always compile in a VM matching the target architecture (i.e. they perform no "real" cross-compilation, correct?), hence supplying strip with an -I or -F option is not needed?

I have not the slightest idea, sorry.

[nephros]

For a good part because I was increasingly having the impression you spend more time on this than me in my role as a repo maintainer, which I and maybe also bystanders may perceive as an imbalance, socially.

Please do not feel pressured by anything I do - I find myself drifting from project to project at the moment, tackling small stuff here and there after something bigger somewhere else has proven frustrating - not really getting anything done for good :)

[Olf0]

I have not the slightest idea, sorry.

I will give it a try.

Please do not feel pressured by anything I do …

Sometimes a little pressure felt can be quite helpful, at least for me.
Otherwise I would have pushed this task back and back again, as I deemed it tedious and potentially resulting in nothing viable. It turned out to be not that boring and by the help of your pointer, likely a viable path arose.

I find myself drifting from project to project at the moment, tackling small stuff here and there after something bigger somewhere else has proven frustrating - not really getting anything done for good :)

Well, you are always very helpful with your expertise and actions for me. But I know this state very well; basically all my actions at GitHub are to feel successful every now and then, despite many will perceive me as successful IRL, too, but I do not perceive it that way.