deblobbify, replacing the system folder with several other components

[robertkirkman]

Potential WIP/PoC solution to:

• termux-docker images should be properly labeled with their native arch #35
• The whole system directory may be removed from the repo... #60
• Hostnames fail to resolve #63
• Missing TERM environment variable #65

Summary of changes:

• Deletes the entire system folder and replaces it with a script generate.sh that can regenerate the entire system folder.
• The root filesystem, including system folder, is regenerated using purely a Termux bootstrap .zip, and .deb files that can be built using the termux-packages build and CD system, then retrieved from the termux.dev package host.
• Adds an environment variable $TERM that is set to a value of "xterm" for the clear command to have convenient compatibility with most terminal emulators, and this has also been tested with Kitty.
• Simplifies the instances of the string "com.termux" into a single instance for slightly more convenient use with custom bootstraps.
• Fixes the existing docker.io/termux/termux-docker:aarch64 image not working on aarch64 devices that happen to have an esoteric "Cortex-A34"-type or similar processor due to the elimination of this 32-bit ARM binary blob, which Cortex-A34-type processors do not have hardware support for executing, unlike "Cortex-A72"-like processors.
• Replaces Busybox with a real build of Toybox based on the source code of AOSP 9.0.0 that has been patched to meet the requirements of termux-docker, including providing su to log in as the system user within entrypoint.sh.
• Uses mksh and bash alone to implement sh, instead of Busybox's sh.
• Uses a build of iputils based on the source code of AOSP 9.0.0 to implement /system/bin/ping and /system/bin/ping6.
• Uses the preexisting termux-packages root/dnsmasq to implement dnsmasq, which works after some adjustments to its invocations.
• Adds a third method of working around the personality() system call permissions issue, a custom build of Docker, which is not required on all ARM devices, but which might be required sometimes, and is required on my armv7l device that the --privileged and --security-opt arguments both failed to work on.
• Replaces the use of QEMU in GitHub Actions with the new official public GitHub ARM runner.
• Enables updating the termux packages within the container before publishing the ARM images.
• Deletes /system/etc/ld.config.28.txt without replacing it with anything, since it does not appear to be necessary for anything in any test so far, but it could be easily re-added if it turns out that there is something that needs it.

Tested both locally and in GitHub Actions:

• aarch64: working
• x86_64: working
• arm: working, but in some (not all) cases, the custom build of Docker with modified personality() system call permissions is required. It is unknown how to predict whether a device will require that or not, but it is suspected it might have something to do with the host kernel, which is a factor whose properties it is understood tend to vary dramatically between different ARM devices.
• i686: working

The "temporary" things like the temporary-package-storage folder and the blobs inside are only necessary for testing while this is not merged:

• bump(main/bionic-host),addpkg(main/{boringssl,iputils,mksh,toybox}-host): 9.0.0-r76 termux-packages#22906

The large amount of changes to the internal components of termux-docker could introduce unpredictable bugs or unexpected behavior that is not present with the current official termux-docker. For that reason, I hope to learn all of the current intended use cases that termux-docker is officially supposed to be used for, so that I can test my implementation with them and add additional software or fixes if needed.

TODO:

• remove the temporary-package-storage folder and all references to it (blocked by the bionic-host PR)

[twaik]

modifying the termux-packages bionic-host package is not acceptable

Nobody said it is not acceptable. I think nobody tried it.
I created the package as a solution for running android executables during regular building with toolchain to avoid using hostbuild step, but for some reason qemu does not want to run fine with aarch64 executables on amd64 host, at least in some cases.
Also it fits fine for using in termux-docker but it was not modified to use bionic-host package.

[robertkirkman]

for some reason qemu does not want to run fine with aarch64 executables on amd64 host, at least in some cases

I am not sure exactly what exact scenario you are talking about here, or what error it is you were seeing at the time, if you mean a scenario with a build.sh involving only the regular cross-compiling toolchain, then what I am about to say is probably not relevant,

But just in case knowing this helps you, there is a way to emulate an ARM processor on an x86 device that is very high in emulation accuracy and behaves almost identically to how real ARM devices behave.

When carefully configured, I have never seen qemu-system-aarch64 fail to execute ARM software on an x86 processor, as long as that same ARM software also works on real ARM devices. If I ever see frustrating errors while trying to use user-mode emulation of any kind, qemu-system-aarch64 is my go-to solution for accuracy to the exact behavior of real ARM devices, when I am in a place with only x86 devices and accessing real ARM devices is inconvenient.

Since there are very few good guides on the subject, and almost all other guides skip mentioning lots of necessary steps, leaving everything unexplained and causing most people to feel like qemu-system-aarch64 is much too tedious to consider using on a regular basis, I created this guide that covers every detail of the setup process in a very straightforward, completely explained, and future-proofed way based on the latest UEFI technology, which is also distro-agnostic to both the host and the guest, making it very clear what changes when the guest and host are mixed and matched.

[twaik]

I will just link this.
termux/termux-packages#18517 (comment)
I know it is not a simple scenario, but GIR-generated binary used for generating some XMLs for some reason hangs and CI is terminated with timeout after 6 hours so it is a no go for us. It was the very first and the last attempt (my attempt) to use cross-compiled binaries for building (to avoid using host_build step).