Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert usermods to static libraries #4480

Draft
wants to merge 48 commits into
base: main
Choose a base branch
from

Conversation

willmmiles
Copy link
Collaborator

Redesign the usermod intergration so that usermods are implemented as PlatformIO libraries instead of headers. This permits them to call for dependencies, and eliminates the compiler flags for enabling each one, allowing the build cache to operate consistently regardless of the selected modules.

The usermod list is built using some linker magic to construct a static list in ROM memory. This eliminates the need for wasting SRAM on something well known at build time.

To use this integration:

  • Each usermod .h should be moved to a .cpp, with a static instantiation of the module class at the end
  • The module object is then passed to REGISTER_USERMOD() to add it to the compiled-in usermod list
  • Each usermod must add a 'library.json':
    • The library name must match the folder name. (This PR includes special case handling for removing the 'usermod_v2_' prefix)
    • The library.json must call for local build integration ("includeDir": "../../wled00", "libLDFMode": "chain+", and "libArchive": false)
    • The library.json can now list any library dependencies, or custom build flags that apply to only that module.
  • PlatformIO target environments now support a new 'custom_usermods' property to enable usermods, space delimited
    • Example: custom_usermods = audioreactive animartrix builds in the "audioreactive" and "animartrix" usermods

The custom_usermods property is implemented via a platformio hook script that constructs the required symlink lines and injects them in to the build environment.

Currently I have converted 'auto_save', 'audioreactive', and 'animartrix' (based on #4476) to library-type as proof-of-concept.

Redesign the usermod system so that usermods are implemented as
PlatformIO libraries instead of headers.  This permits them to call for
dependencies, and eliminates the compiler flags for enabling each one,
allowing the build cache to behave better.

The usermod list is built using some linker magic to construct a static
list in ROM memory.  This eliminates the need for wasting SRAM on
something fixed at build time.
Look for 'usermod_v2_x' as well.  This could be removed later if we
clean up the folder names.
Borrowed library definition from @netmindz's work on Aircoookie#4476.
@willmmiles willmmiles requested a review from netmindz January 11, 2025 18:48
@willmmiles willmmiles self-assigned this Jan 11, 2025
@willmmiles
Copy link
Collaborator Author

Annoyingly, PlatformIO requires locally-defined properties to be prefixed with 'custom_' ; I'd've liked it to just be 'usermods' but no such luck. :(

platformio.ini Outdated
board_build.partitions = ${esp32.default_partitions} ;; default partioning for 4MB Flash - can be overridden in build envs
# additional build flags for audioreactive - must be applied globally
AR_build_flags = -D sqrt_internal=sqrtf ;; -fsingle-precision-constant ;; forces ArduinoFFT to use float math (2x faster)
AR_lib_deps = kosme/arduinoFFT @ 2.0.1 ;; for pre-usermod-library platformio_override compatibility
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are updating to be a library, why do we still need the old AR_lib_deps info as well?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compatibility with old platformio_override files that reference it. If we remove it, git pull breaks platformIO completely until you clean all references out of your overrides.

Copy link
Collaborator

@netmindz netmindz Jan 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If folk have platform_override that is trying to use AR, is having AR_build_flags that that doesn't have the old define just going to cause confusion? They will think they have the flags, but now that's not enough to actually enable

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this would be a breaking change that will need to be staged appropriately. Unfortunately if your platformio file is outright invalid, PlatformIO in VSCode doesn't tell you what's wrong, it just blankly refuses to operate with no error message. I found it more useful to at least keep the tool operating while I was correcting my build definitions, but that's just my $0.02.

Copy link
Collaborator

@netmindz netmindz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have a bit of a play around with this over the next few days

It looks like a great start, the key things in my mind however are what extra changes we can make both in terms of code, documentation and release management to help make the transition as painless as possible

One quick question - how essential is the rename from .h to .cpp ?

usermods/audioreactive/library.json Outdated Show resolved Hide resolved
@@ -1,482 +0,0 @@
#include "wled.h"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it better to delete this file or retain but with message about the new format - What would be the best way to highlight the changes to developers of the may forks?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question! I'd figured the deletion will create an obvious merge conflict, so it's clear that something must be done. Personally I'd rather not leave a .cpp file that the build tooling will have to grovel over only for it to do nothing.

@willmmiles
Copy link
Collaborator Author

willmmiles commented Jan 11, 2025

It looks like a great start, the key things in my mind however are what extra changes we can make both in terms of code, documentation and release management to help make the transition as painless as possible

Yes. Unfortunately it looks like the docs aren't in the code proper (beyond the examples), which is a little frustrating when different versions might have different approaches. :(

One quick question - how essential is the rename from .h to .cpp ?

Some .cpp file has to instantiate the module object; and this must be in the library, or you're back to all the problems with usermod_list.cpp. You could add a .cpp file to each that just does #include <my_module_header.h>; static MyModule my_module; REGISTER_USERMOD(my_module); , but personally I'd rather see all the code together in the one file rather than a seemingly bureaucratic split. My $0.02 is that the rename is the lesser evil -- git handles merges across renames pretty well these days.

@netmindz
Copy link
Collaborator

One quick question - how essential is the rename from .h to .cpp ?

Some .cpp file has to instantiate the module object; and this must be in the library, or you're back to all the problems with usermod_list.cpp. You could add a .cpp file to each that just does #include <my_module_header.h>; static MyModule my_module; REGISTER_USERMOD(my_module); , but personally I'd rather see all the code together in the one file rather than a seemingly bureaucratic split. My $0.02 is that the rename is the lesser evil -- git handles merges across renames pretty well these days.

How about a migration script that looks for an existing .h of a usermod, moves with git and then automatically added the extra static field and call to register?

@netmindz
Copy link
Collaborator

Are you aware of the build failure when using the audioreactive usermod? @willmmiles

In file included from usermods/audioreactive/audio_reactive.cpp:2:0:
wled00/wled.h:95:26: fatal error: LITTLEFS.h: No such file or directory

@willmmiles
Copy link
Collaborator Author

One quick question - how essential is the rename from .h to .cpp ?

Some .cpp file has to instantiate the module object; and this must be in the library, or you're back to all the problems with usermod_list.cpp. You could add a .cpp file to each that just does #include <my_module_header.h>; static MyModule my_module; REGISTER_USERMOD(my_module); , but personally I'd rather see all the code together in the one file rather than a seemingly bureaucratic split. My $0.02 is that the rename is the lesser evil -- git handles merges across renames pretty well these days.

How about a migration script that looks for an existing .h of a usermod, moves with git and then automatically added the extra static field and call to register?

Not impossible. but nontrivial - the script would have to pick out the class name and any constructor arguments. We'd probably need to call on clang or something else that understands C++ syntax to be able to do a reliable job of it.

platformio.ini Outdated
@@ -424,23 +420,23 @@ build_flags = ${common.build_flags} ${esp8266.build_flags} -D WLED_RELEASE_NAME=
board = esp32dev
platform = ${esp32.platform}
platform_packages = ${esp32.platform_packages}
custom_usermods = audioreactive
build_unflags = ${common.build_unflags}
build_flags = ${common.build_flags} ${esp32.build_flags} -D WLED_RELEASE_NAME=\"ESP32\" #-D WLED_DISABLE_BROWNOUT_DET
${esp32.AR_build_flags}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way to move the build_flags used for the usermod into it's library.json?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In most cases, yes.

In the specific case of AR, the build flags aren't actually for the AR module, they're for the FFT library dependency. Unfortunately, the only way to affect some dependent library's build is to do it at the project level. PlaformIO doesn't let one library define flags for one of its dependencies. https://community.platformio.org/t/setting-flags-defines-for-building-a-librarys-dependency/36744

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, maybe there is a way, using platformio hook scripts in the library. Working on it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are certainly usermods that use more typical build_flags that affect the usermod itself, but perhaps these still need to stay in the main platformio.ini if they are actual optional flags the user chooses to set

Copy link
Collaborator Author

@willmmiles willmmiles Jan 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed! It turns out there is a way though platformio hook scripts. 8527d23

@willmmiles
Copy link
Collaborator Author

Are you aware of the build failure when using the audioreactive usermod? @willmmiles

In file included from usermods/audioreactive/audio_reactive.cpp:2:0:
wled00/wled.h:95:26: fatal error: LITTLEFS.h: No such file or directory

Yup, looking in to it - seems like some sort of case sensitivity thing.

@netmindz
Copy link
Collaborator

How about a migration script that looks for an existing .h of a usermod, moves with git and then automatically added the extra static field and call to register?

Not impossible. but nontrivial - the script would have to pick out the class name and any constructor arguments. We'd probably need to call on clang or something else that understands C++ syntax to be able to do a reliable job of it.

I'm just knocking up a quick PoC of a script. If nothing else it will help show how many of the usermods in the official AC repo do not follow proper naming convention etc

@willmmiles
Copy link
Collaborator Author

I'm just knocking up a quick PoC of a script. If nothing else it will help show how many of the usermods in the official AC repo do not follow proper naming convention etc

Yup, a breaking API change like this is a good opportunity for cleanup. Honestly I'd figured on just going through them by hand, there aren't so many it's a huge problem.

With a solution like this in hand, I was hoping to ultimately create a "core" modules folder with the modules that this team is directly supporting; and then work towards migrating anything with a "WLED_DISABLE_X" flag in to becoming a core module instead. (Yes, I recognize that will likely require some expansion of the module API to handle specialized timing and polling requirements; I think that's probably a good thing, as if we have one use case for it, there may be others.)

Further, the linker section approach demonstrated here for the static usermod list could also be used later to make static library-based FX or bus type lists, too, allowing modules to include new FX or bus types without needing any RAM to build a runtime data structure. One step at a time, though!

@netmindz
Copy link
Collaborator

I didn't want to mess up your main PR with my attempts at automatic migration, so I've popped into it's own PR for now #4481

These are the ones that don't follow any common naming

WARNING: usermods/ADS1115_v2/ADS1115_v2.h missing
WARNING: usermods/AHT10_v2/AHT10_v2.h missing
WARNING: Artemis_reciever possibly still v1 usermod
WARNING: usermods/BH1750_v2/BH1750_v2.h missing
WARNING: usermods/BME280_v2/BME280_v2.h missing
WARNING: usermods/BME68X_v2/BME68X_v2.h missing
WARNING: usermods/EXAMPLE_v2/EXAMPLE_v2.h missing
WARNING: Enclosure_with_OLED_temp_ESP07 possibly still v1 usermod
WARNING: usermods/Fix_unreachable_netservices_v2/Fix_unreachable_netservices_v2.h missing
WARNING: usermods/INA226_v2/INA226_v2.h missing
WARNING: usermods/Internal_Temperature_v2/Internal_Temperature_v2.h missing
WARNING: usermods/JSON_IR_remote/JSON_IR_remote.h missing
WARNING: usermods/LD2410_v2/LD2410_v2.h missing
WARNING: usermods/MAX17048_v2/MAX17048_v2.h missing
WARNING: usermods/MY9291/MY9291.h missing
WARNING: RelayBlinds possibly still v1 usermod
WARNING: TTGO-T-Display possibly still v1 usermod
WARNING: usermods/TetrisAI_v2/TetrisAI_v2.h missing
WARNING: Wemos_D1_mini+Wemos32_mini_shield possibly still v1 usermod
WARNING: usermods/audioreactive/audioreactive.h missing
WARNING: usermods/battery_keypad_controller/battery_keypad_controller.h missing
WARNING: usermods/mqtt_switch_v2/mqtt_switch_v2.h missing
WARNING: photoresistor_sensor_mqtt_v1 possibly still v1 usermod
WARNING: usermods/project_cars_shiftlight/project_cars_shiftlight.h missing
WARNING: usermods/rotary_encoder_change_effect/rotary_encoder_change_effect.h missing
WARNING: usermods/sensors_to_mqtt/sensors_to_mqtt.h missing
WARNING: usermods/seven_segment_display_reloaded/seven_segment_display_reloaded.h missing
WARNING: usermods/stairway_wipe_basic/stairway_wipe_basic.h missing
WARNING: usermods/word-clock-matrix/word-clock-matrix.h missing

@netmindz
Copy link
Collaborator

Fixed the naming of a few, so now just

WARNING: Artemis_reciever possibly still v1 usermod
WARNING: usermods/EXAMPLE_v2/EXAMPLE_v2.h missing
WARNING: Enclosure_with_OLED_temp_ESP07 possibly still v1 usermod
WARNING: usermods/Fix_unreachable_netservices_v2/Fix_unreachable_netservices_v2.h missing
WARNING: usermods/JSON_IR_remote/JSON_IR_remote.h missing
WARNING: RelayBlinds possibly still v1 usermod
WARNING: TTGO-T-Display possibly still v1 usermod
WARNING: Wemos_D1_mini+Wemos32_mini_shield possibly still v1 usermod
WARNING: usermods/battery_keypad_controller/battery_keypad_controller.h missing
WARNING: photoresistor_sensor_mqtt_v1 possibly still v1 usermod
WARNING: usermods/project_cars_shiftlight/project_cars_shiftlight.h missing
WARNING: usermods/rotary_encoder_change_effect/rotary_encoder_change_effect.h missing

@netmindz
Copy link
Collaborator

Are you aware of the build failure when using the audioreactive usermod? @willmmiles

In file included from usermods/audioreactive/audio_reactive.cpp:2:0:
wled00/wled.h:95:26: fatal error: LITTLEFS.h: No such file or directory

Yup, looking in to it - seems like some sort of case sensitivity thing.

I think it might be because we are missing the -D LOROL_LITTLEFS that we normally apply to esp32 builds

@netmindz
Copy link
Collaborator

Are you aware of the build failure when using the audioreactive usermod? @willmmiles

In file included from usermods/audioreactive/audio_reactive.cpp:2:0:
wled00/wled.h:95:26: fatal error: LITTLEFS.h: No such file or directory

Yup, looking in to it - seems like some sort of case sensitivity thing.

I think it might be because we are missing the -D LOROL_LITTLEFS that we normally apply to esp32 builds

Confirmed. If I hack wled.h to define LOROL_LITTLEFS the problem goes away. Do you have a wled_custom.h or something locally?

@blazoncek
Copy link
Collaborator

Any luck with the LOROL_LITTLEFS fix?

When you switch platform from 3.5.0 to 6.3.x, Lorol's LittleFS will not be needed any more.

willmmiles and others added 5 commits January 14, 2025 22:16
When processing usermods, update their include path properties before
the LDF runs, so it can see through wled.h.
Move the "all usermods" logic in to the platformio script, so the
'usermods' environment can be built in any checkout without extra setup
commands.
Comment on lines +5 to +9
all_usermods = [f for f in usermod_dir.iterdir() if f.is_dir() and f.joinpath('library.json').exists()]

if env['PIOENV'] == "usermods":
# Add all usermods
env.GetProjectConfig().set(f"env:usermods", 'custom_usermods', " ".join([f.name for f in all_usermods]))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The downside to this change is that you have no control of what you are building

For testing I was setting my local platform_override.ini to only list the mods I was trying to build.

I guess I can still do that, but just need to name my env somehting like usermods_local

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not thrilled at magic environment names, but I think it's important to be able to locally replicate what the CI system does without having to run shell commands to update local state after every git pull. I'm open to alternative suggestions!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly less need for an "all in one" build if we just let the CI handle a matrix build for usermods, then just define the one we are working on locally in our own platform_override ?

willmmiles#3

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stand by my opinion from above -- I think it's important that what the CI system does should be easily and directly reproducible by developers. I don't think it's appropriate to have to hand hack custom files in your personal workspace to replicate what the CI does; that's just asking to get it wrong.

From a practical perspective, I'd also like it to be simple to validate my local changes before pushing to the CI. If we depend on matrix builds, that becomes almost impossible. Id be stuck waiting on the CI system any time I want to have some confidence I haven't inadvertently broken the build of some random usermod.

Finally, I don't know much about GitHub CI; I'm concerned that adding dozens of extra builds per run would be costly.

@netmindz
Copy link
Collaborator

Any ideas why I'm now seeing errors for missing wled.h ?

@netmindz
Copy link
Collaborator

Any ideas why I'm now seeing errors for missing wled.h ?

Looks like the load_usermods.py doesn't have any reference to wled00, unlike the original libray.json.

No idea if this is correct, but it makes the error for away @willmmiles

diff --git a/pio-scripts/load_usermods.py b/pio-scripts/load_usermods.py
index 76359a7a..862f30e4 100644
--- a/pio-scripts/load_usermods.py
+++ b/pio-scripts/load_usermods.py
@@ -38,6 +39,7 @@ if usermods:
 cpl = env.ConfigureProjectLibBuilder
 # Our new wrapper
 def cpl_wrapper(env):
+  src_dir = proj.get("platformio", "src_dir") + "/wled00/"
   # Update usermod properties
   lib_builders = env.GetLibBuilders()  
   um_deps = [dep for dep in lib_builders if usermod_dir in Path(dep.src_dir).parents]
@@ -46,7 +48,9 @@ def cpl_wrapper(env):
     # Add include paths for all non-usermod dependencies
     for dep in other_deps:
         for dir in dep.get_include_dirs():
+            print(f"um = {um} dir = {dir}\n")
             um.env.PrependUnique(CPPPATH=dir)
+            um.env.PrependUnique(CPPPATH=src_dir)
     # Add the wled folder to the include path    

@netmindz
Copy link
Collaborator

Any ideas why I'm now seeing errors for missing wled.h ?

Looks like the load_usermods.py doesn't have any reference to wled00, unlike the original libray.json.

No idea if this is correct, but it makes the error for away @willmmiles

diff --git a/pio-scripts/load_usermods.py b/pio-scripts/load_usermods.py
index 76359a7a..862f30e4 100644
--- a/pio-scripts/load_usermods.py
+++ b/pio-scripts/load_usermods.py
@@ -38,6 +39,7 @@ if usermods:
 cpl = env.ConfigureProjectLibBuilder
 # Our new wrapper
 def cpl_wrapper(env):
+  src_dir = proj.get("platformio", "src_dir") + "/wled00/"
   # Update usermod properties
   lib_builders = env.GetLibBuilders()  
   um_deps = [dep for dep in lib_builders if usermod_dir in Path(dep.src_dir).parents]
@@ -46,7 +48,9 @@ def cpl_wrapper(env):
     # Add include paths for all non-usermod dependencies
     for dep in other_deps:
         for dir in dep.get_include_dirs():
+            print(f"um = {um} dir = {dir}\n")
             um.env.PrependUnique(CPPPATH=dir)
+            um.env.PrependUnique(CPPPATH=src_dir)
     # Add the wled folder to the include path    

Nope, I'm talking rubbish. Every time you do a build after a full clean you get the missing wled.h, however just clicking build again fixes it

@willmmiles
Copy link
Collaborator Author

Nope, I'm talking rubbish. Every time you do a build after a full clean you get the missing wled.h, however just clicking build again fixes it

Yeah, there's something wrong with building the dependency list; it's sometimes not locating the usermods early enough, so the patch-up process isn't able to configure them. I can't replicate it on my system, even in a fresh container, but the CI does it repeatably, so I was debugging in a separate branch. I'll update when I've got it sorted out.

@netmindz
Copy link
Collaborator

Nope, I'm talking rubbish. Every time you do a build after a full clean you get the missing wled.h, however just clicking build again fixes it

Yeah, there's something wrong with building the dependency list; it's sometimes not locating the usermods early enough, so the patch-up process isn't able to configure them. I can't replicate it on my system, even in a fresh container, but the CI does it repeatably, so I was debugging in a separate branch. I'll update when I've got it sorted out.

What OS and platformIO version are you on? I'm using linux and 6.1.4 and is is totally repeatable. Full clean, Build - error no wled.h, Build - OK

@willmmiles
Copy link
Collaborator Author

What OS and platformIO version are you on? I'm using linux and 6.1.4 and is is totally repeatable. Full clean, Build - error no wled.h, Build - OK

Hm, maybe that's it and it was changed in a newer PlatformIO -- library management is definitely an area of active development. I'm using whatever came down with the devcontainer; currently it's PlatformIO 6.1.17b2.

Once we've got a working solution, I'm going to see about opening a discussion on PlatformIO's own issue board about our "modules" use case and see what the devs there think about it. lib_extra_dirs would make 85% of this unnecessary; all we'd need is the "libArchive": false magic. With any luck they'll be suitably horrified by what we've had to do to work around that deprecation that they'll give some more thought on how best to handle this sort of thing.

Should now work for new *and* old versions of PlatformIO!
- Check after including wled.h
- Use WLED_DISABLE_MQTT instead of WLED_ENABLE_MQTT
A better solution for cross-module communication is required!
Define a couple pins, leave a note where the usermod list comes from.
This is now managed centrally.
@willmmiles
Copy link
Collaborator Author

Known issues in the current state:

  • Some usermods have OR dependencies on other usermods, eg "requires usermod A OR usermod B". Previously they were using the defines to perform compile-time checks for this. It's not clear how to replicate this via library.json -- platformio doesn't have a "provides" system to allow packages to export extra abstract names.
    • It's easy to make these runtime checks, though unhelpful for users
    • Maybe another 'extraScript' could help here? Run a validation and bail out?
    • Longer term, a cleanup/wider application of getUMData instead of directly calling in to other usermods might help, too
  • The "blast all build deps in to every usermod" approach is overkill, and this can weird include issues when there are conflicting downstream requirements. AsyncTCP is the current example: AWS calls for a different version than WLED, so both are being added to the build environment, and usermods sometimes being built with the "wrong" one. For this example, we can work around this by cleaning up the WLED build environment (update AWS and AsyncTCP, see Draft: AsyncWebServer queue support #4119), but I think load_usermods.py could still do a better job.
  • PlatformIO's LDF still makes mistakes on some libraries: for example, TFT_eSPI sometimes barfs on missing SPIFFS.h. This is an area of active work in PlatformIO, and newer versions seem better, but we would have to be prepared for complaints or trouble. (Note this is not a new issue with this branch, merely a consequence of actually test building usermods!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants