lighttpd: lighttpd 1.4.77

[gstrauss]

Description

lighttpd 1.4.77

Motivation and Context

modern lighttpd with bug fixes and HTTP/2 (optional)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Chore (refactoring, style fixes, git/CI config, submodule management, no code logic changes)

• Breaking change: "mod_compress" needs to be renamed "mod_deflate" for static build.
• Breaking change: the phoenix-rtos custom patch to look for precompressed files in mod_compress not longer applies, since mod_compress is no longer part of lighttpd. (lighttpd mod_compress has been subsumed by lighttpd mod_deflate.) Arbitrary content negotiation can be achieved using lighttpd mod_magnet and lua: https://wiki.lighttpd.net/AbsoLUAtion#Content-Negotiation

[Edit] These mod_compress -> mod_deflate breaking changes are less likely to affect many maintainers since https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf does not list mod_compress (or mod_deflate), and so it would have had to be added manually by the maintainer.

How Has This Been Tested?

• Already covered by automatic testing.
• New test added: (add PR link here).
• Tested by hand on: (list targets here).

Tested by lighttpd CI on many platforms, but not on phoenix-rtos. Submitting this PR will test build on phoenix-rtos.

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing linter checks and tests passed.
• My changes generate no new compilation warnings for any of the targets. (TBD)

Special treatment

• This PR needs additional PRs to work (list the PRs, preferably in merge-order).
• I will merge this PR by myself when appropriate.

[gstrauss]

Future possible changes: lighttpd 1.4.56 and later add support for alternative TLS libraries. In addition to lighttpd mod_openssl, there is now lighttpd mod_gnutls, lighttpd mod_mbedtls, lighttpd mod_wolfssl, and lighttpd mod_nss.

For resource-constrained embedded systems, lighttpd mod_mbedtls using mbedtls, or lighttpd mod_wolfssl using wolfssl, is recommended due to having a smaller footprint than openssl. The lighttpd.conf syntax to configure lighttpd TLS modules is almost identical across TLS modules, with the most important exception being TLS-library-specific spellings of ciphers. If using lighttpd TLS defaults, this is a non-issue since a custom cipher list does not need to be specified in lighttpd.conf.

[gstrauss]

lighttpd no longer ships configure, so autoreconf -fi must be run to generate configure from configure.ac.

I pushed a change to fix that in lighttpd/build.sh.

Note: It is interesting that many of the jobs in the CI matrix were successful (without attempting to lighttpd).
https://github.com/phoenix-rtos/phoenix-rtos-ports/actions/runs/12861205643

[gstrauss]

The CI fails here:

 /opt/phoenix/arm-phoenix/bin/../lib/gcc/arm-phoenix/9.5.0/../../../../arm-phoenix/bin/ld: lighttpd-plugin.o:(.rodata.load_functions+0x4c): undefined reference to `mod_evasive_plugin_init'
/opt/phoenix/arm-phoenix/bin/../lib/gcc/arm-phoenix/9.5.0/../../../../arm-phoenix/bin/ld: lighttpd-plugin.o:(.rodata.load_functions+0x6c): undefined reference to `mod_usertrack_plugin_init'

lighttpd mod_evasive and lighttpd mod_usertrack were removed in lighttpd 1.4.68, and can be replaced with lua scripts
https://wiki.lighttpd.net/ModMagnetExamples

Where is the CI getting lighttpd.conf from which it generates the static plugin list?
phoenix-rtos-ports lighttpd/build.sh contains: CONFIGFILE=$(find "${PREFIX_ROOTFS:?PREFIX_ROOTFS not set!}/etc" -name "lighttpd.conf"), but /etc/...lighttpd.conf is not present in phoenix-rtos-ports repository.

https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf
has mod_evasive and mod_usertrack commented out. That commit was May 12, 2020 and is the only commit for that file, so I don't think that is the file used by the CI here.

Related, why is that in phoenix-rtos/phoenix-rtos-project instead of part of phoenix-rtos/phoenix-rtos-ports? I would suggest improvements to that lighttpd.conf, but some improvements require a base version of lighttpd.

---

warning: comparison of integer expressions of different signedness: 'int' and 'time_t' {aka 'long long unsigned int'} [-Wsign-compare]

WTH? Why does phoenix-rtos set time_t to long long unsigned int? time_t is supposed to be a signed type. For the Y2038 problem, time_t has been changed from int to int64_t (or long long int) by any sane operating system.

[agkaminski]

warning: comparison of integer expressions of different signedness: 'int' and 'time_t' {aka 'long long unsigned int'} [-Wsign-compare]

WTH? Why does phoenix-rtos set time_t to long long unsigned int? time_t is supposed to be a signed type. For the Y2038 problem, time_t has been changed from int to int64_t (or long long int) by any sane operating system.

https://pubs.opengroup.org/onlinepubs/009695399/basedefs/time.h.html
https://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/types.h.html
"time_t and clock_t shall be integer or real-floating types."

That's actually a funny fact that the standard does not define signedness of time_t, so strictly speaking unsigned long long is a valid choice. It comes with many practical drawbacks though (lack of compatibility with everything else being one of them), so the change to the long long is already planned.

[gstrauss]

Yes, you're correct that the sign of time_t is unspecified by the standards.

It comes with many practical drawbacks though

Yes.

Given that some time functions return -1 on error induces the need to cast (time_t)-1 to avoid pedantic warnings, and makes testing with comparison operators </> prone to error if -1 is cast to a very large unsigned value.

[gstrauss]

@agkaminski Where is the CI getting lighttpd.conf from which it generates the static plugin list? That is what is causing the CI failure.

[agkaminski]

@agkaminski Where is the CI getting lighttpd.conf from which it generates the static plugin list? That is what is causing the CI failure.

Not really my piece of cake, but I see it only in the generic rootfs skel in the OS main repository:
https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf (edit - updated the link to master branch)

Regarding the warning - IMHO let it stay to further motivate time_t change ;)

[gstrauss]

Regarding the warning - IMHO let it stay to further motivate time_t change ;)

Well, there is one place in lighttpd which fails due to underflow and wraparound of unsigned (where signed time_t was incorrectly expected), but it is in a less-used feature for mod_dirlisting caching.

/github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c: In function 'mod_dirlisting_cache_check':
Warning: /github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c:1719:17: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits]
 1719 |     if (max_age < 0)
      |                 ^

[agkaminski]

Regarding the warning - IMHO let it stay to further motivate time_t change ;)

Well, there is one place in lighttpd which fails due to underflow and wraparound of unsigned (where signed time_t was incorrectly expected), but it is in a less-used feature for mod_dirlisting caching.

/github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c: In function 'mod_dirlisting_cache_check':
Warning: /github/workspace/.buildroot/phoenix-rtos-project/_build/armv7a7-imx6ull-evk/lighttpd/lighttpd-1.4.77/src/mod_dirlisting.c:1719:17: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits]
 1719 |     if (max_age < 0)
      |                 ^

Thank you for pointing out this issue and for the PR in general

[gstrauss]

As you and I pointed out, https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf is the likely origin of mod_evasive and mod_usertrack in the CI. lighttpd/build.sh greps out mod_ from the file, which includes commented-out lines. I'll submit a PR to that repo, and once accepted there, will hopefully unblock this PR.

If @Darchiv would like to switch from lighttpd mod_openssl to lighttpd mod_mbedtls for lower resource use, then the CI will run into this issue again since the lighttpd.conf is in a separate repo and is not matched in lock-step with the lighttpd build configuration in this repo.

[gstrauss]

@Darchiv lighttpd/build.sh might contain a list of lighttpd modules to build, rather than grep from lighttpd.conf in a different repository. Besides the example above for moving from mod_openssl to mod_mbedtls, if you want to provide HTTP/2 support, then mod_h2 needs to be listed.

[Darchiv]

As you and I pointed out, https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf is the likely origin of mod_evasive and mod_usertrack in the CI. lighttpd/build.sh greps out mod_ from the file, which includes commented-out lines. I'll submit a PR to that repo, and once accepted there, will hopefully unblock this PR.

If @Darchiv would like to switch from lighttpd mod_openssl to lighttpd mod_mbedtls for lower resource use, then the CI will run into this issue again since the lighttpd.conf is in a separate repo and is not matched in lock-step with the lighttpd build configuration in this repo.

The phoenix-rtos-project repository contains a reference project, which is the fastest way to start using Phoenix-RTOS. A custom project can use this repository (via forking or any other way) and override configuration stored in root-skel. Hence, if a project needs mod_mbedtls, it can use a custom lighttpd.conf and the upstream phoenix-rtos-project is not relevant anymore.

@Darchiv lighttpd/build.sh might contain a list of lighttpd modules to build, rather than grep from lighttpd.conf in a different repository. Besides the example above for moving from mod_openssl to mod_mbedtls, if you want to provide HTTP/2 support, then mod_h2 needs to be listed.

Maybe module listing can be improved, but it still has to be configurable per-project - for example by an env variable exported by the project. This can be more cumbersome and error-prone than just using an up-to-date lighttpd.conf, though.

---

Have you checked that mod_wstunnel still builds? You have removed its patch, but I assume that autogen.sh will properly include it into the build.

[Edit] These mod_compress -> mod_deflate breaking changes are less likely to affect many maintainers since https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf does not list mod_compress (or mod_deflate), and so it would have had to be added manually by the maintainer.

Employing lua just for this small mod_compress fix-up may be too much of a burden on an embedded system. Maybe there should be an (optional) way to patch mod_deflate in the same manner as mod_compress.

Just pointing out the not-so-obvious use cases we have to cover. Your PR is nevertheless valuable.

[gstrauss]

@Darchiv lighttpd/build.sh might contain a list of lighttpd modules to build, rather than grep from lighttpd.conf in a different repository. Besides the example above for moving from mod_openssl to mod_mbedtls, if you want to provide HTTP/2 support, then mod_h2 needs to be listed.

Maybe module listing can be improved, but it still has to be configurable per-project - for example by an env variable exported by the project. This can be more cumbersome and error-prone than just using an up-to-date lighttpd.conf, though.

There are many approaches to do this. lighttpd provides the mechanism plugin_static.h and uses C macros. phoenix-rtos chose to use a fragile grep pipeline with a naive 'parse' of lighttpd.conf, and results in building modules that are commented out in the reference lighttpd.conf. One alternative: phoenix-rtos could have provided a reference plugin_static.h which was simply copied into the project to be built. I'm sure there are many other valid ways. Since the way phoenix-rtos does it is with grep, please consider adding a line to better document "this is phoenix-rtos-provided convenience mechanism to generate plugin_static.h" in lighttpd/build.sh.

Please review phoenix-rtos/phoenix-rtos-project#1262, which is a prerequisite to this PR.

Have you checked that mod_wstunnel still builds? You have removed its patch, but I assume that autogen.sh will properly include it into the build.

lighttpd CI builds lighttpd on many platforms (phoenix-rtos is not part of lighttpd CI) and I have made many improvements to lighttpd since the 1.4.53 version hard-coded in the phoenix-rtos reference build. If mod_wstunnel does not build on phoenix-rtos, then I would like to fix the issue upstream, if possible. The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

[Edit] These mod_compress -> mod_deflate breaking changes are less likely to affect many maintainers since https://github.com/phoenix-rtos/phoenix-rtos-project/blob/master/_fs/root-skel/etc/lighttpd.conf does not list mod_compress (or mod_deflate), and so it would have had to be added manually by the maintainer.

Employing lua just for this small mod_compress fix-up may be too much of a burden on an embedded system. Maybe there should be an (optional) way to patch mod_deflate in the same manner as mod_compress.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos? As you said, this is a reference build. Please share the reasons why this was added and why, for a reference build, this is a required patch before you accept this PR ("Just pointing out the not-so-obvious use cases we have to cover.") The customization does not fix any bugs. It adds a feature, and does so in a very specific way by breaking/replacing the existing mod_compress caching logic. Are you planning to port that to lighttpd mod_deflate caching?

If someone developing embedded systems is using lighttpd, then as a lighttpd developer, I recommend the documentation:
https://wiki.lighttpd.net/Docs_ResourceTuning
https://wiki.lighttpd.net/Docs_Performance

If there is a need for Content Negotiation for pre-compressed files and if an embedded system has low resources, then I would recommend omitting mod_deflate and zlib and instead using lighttpd mod_magnet and lua. You'll likely find the disk, memory footprint, and CPU usage to be similar for Content-Negotiation for pre-compressed files.

If Content Negotiation for pre-compressed files must be in C on resource-constrained embedded systems, then I still recommend omitting mod_deflate, and instead writing a custom module based on lighttpd the simple mod_staticfile.c to perform the specific Content Negotation for pre-compressed files.

[gstrauss]

ping.

The phoenix-rtos project commit message for lighttpd/patches/makefile_wstunnel.patch has a reference to JIRA: DTR-460, but no direct details why it did not build.

Similar to above, the commit for lighttpd/patches/mod_compress_fix.patch references JIRA: DTR-219. Is there some place that those tickets can be viewed by someone not part of the phoenix-rtos team? Why was this added to phoenix-rtos?