[web-src] Include web interface sources in distribution files; build web interface as part of autotools

[chme]

I thought, I tackle again the issue with the web interface build integration (and source distribution) into the autotools build chain ... and so far, this is the most promising solution :-)

The solution presented in this PR is a bit different from previous attempts.
It does not require special make goals to be run for the web interface. Instead the web interface is built as part of the "normal" build process:

autoreconf -i
./configure
make

The prebuilt htdocs files are deleted from git (and removed from configure.ac, Makefile.am). They are still part of the distribution tar file - but the location changed from htdocs to web-src/htdocs.

Building without the web interface by passing "--disable-webinterface" to configure is still possible.

configure.ac

• Check build requirements for building the web interface (similar to what was done in Include web interface sources in dist #1439.

  • In configure.ac a check for the presence of the npm binary is done.
  • If it is not present, another test is done whether prebuilt web interface files (htdocs) are available.
  • If prebuilt files are available, only a message is printed that the web interface will not be built and the prebuilt files will be used. Otherwise, configure fails.

• web-src/Makefile replaces the htdocs/Makefile in AC_CONFIG_FILES.

web-src/Makefile.am

Web interface build is triggered with the all-local make goal.

To support out-of-tree-builds (like it is done during make distcheck), the build checks first if source and build dirs are the same. If not the source files are copied into the build dir. And the build is done in the build dir afterwards (with installing node_modules etc.).

Installation of the web interface files is done in install-data-local.

Known Issues

• The build of the web interface as part of autotools assumes a clean build. At the moment, it will not detect if the web interface source files changed between builds. To "fix" this, it would be necessary to list all source files (instead of only the "src" folder) in WEB_FILES. All files of the web interface are now listed in Makefile,am, to mitigate the risk of adding new files without adding them in Makefile.am a make web-check goal validates that no "unknown" files exists (added to the CI action).

Testing

Some tests I did to validate this approach:

• make distcheck runs fine (on my system).
• The distribution tar file contains the web interface sources and the htdocs build.
• Building from the tar file without installed npm, will use and install the prebuilt htdocs files.
• Building on a system without npm is possible with passing disable-webinterface.

There is probably a bunch of stuff, I did not think of, that might break with these changes. Let me know if this is worth pursuing further.

---

Ref #1439 #962 #552

[chme]

mmh ... the FreeBSD npm build of the web interface is failing (see log from CI below).
I guess, this is unrelated to the changes in this PR and building the web interface with FreeBSD generally fails?
If this is true, would it be OK to disable the web interface build in the FreeBSD CI action?

  gmake[2]: Entering directory '/home/runner/work/owntone-server/owntone-server/web-src'
  /usr/local/bin/npm run build
  > [email protected] build
  > vite build --base='./'
  /home/runner/work/owntone-server/owntone-server/web-src/node_modules/rollup/dist/native.js:84
  	throw new Error(
  	      ^
  Error: Your current platform "freebsd" and architecture "x64" combination is not yet supported by the native Rollup build. Please use the WASM build "@rollup/wasm-node" instead.
  The following platform-architecture combinations are supported:
  android-arm
  android-arm64
  darwin-arm64
  darwin-x64
  linux-arm
  linux-arm (musl)
  linux-arm64
  linux-arm64 (musl)
  linux-ppc64
  linux-riscv64
  linux-s390x
  linux-x64
  linux-x64 (musl)
  win32-arm64
  win32-ia32
  win32-x64
  If this is important to you, please consider supporting Rollup to make a native build for your platform and architecture available.
      at throwUnsupportedError (/home/runner/work/owntone-server/owntone-server/web-src/node_modules/rollup/dist/native.js:84:8)
      at getPackageBase (/home/runner/work/owntone-server/owntone-server/web-src/node_modules/rollup/dist/native.js:75:3)
      at Object.<anonymous> (/home/runner/work/owntone-server/owntone-server/web-src/node_modules/rollup/dist/native.js:37:21)
      at Module._compile (node:internal/modules/cjs/loader:1565:14)
      at Object..js (node:internal/modules/cjs/loader:1708:10)
      at Module.load (node:internal/modules/cjs/loader:1318:32)
      at Function._load (node:internal/modules/cjs/loader:1128:12)
      at TracingChannel.traceSync (node:diagnostics_channel:322:14)
      at wrapModuleLoad (node:internal/modules/cjs/loader:219:24)
      at cjsLoader (node:internal/modules/esm/translators:263:5)
  Node.js v22.12.0
  gmake[2]: Leaving directory '/home/runner/work/owntone-server/owntone-server/web-src'
  gmake[2]: *** [Makefile:560: htdocs] Error 1
  gmake[1]: Leaving directory '/home/runner/work/owntone-server/owntone-server'
  gmake[1]: *** [Makefile:675: install-recursive] Error 1
  gmake: *** [Makefile:980: install] Error 2

[chme]

Giving it some thought, explicitly listing the web interface source files might not be a bad idea. They rarely change and having autotools properly detect whether a rebuild of the web interface is necessary is good to have. See 15b7e0d

Downside is, that whenever a file is added to the web interface sources, it has to be added to the list of files in Makefile.am. Forgetting to list it might not be noticed directly, as it will only effect the distribution tar file (and out-of-tree builds).

[ejurgensen]

I haven't looked at the implementation, but I think it sounds very promising from your description of it. It sounds like it will work in all normal build scenarios, and keeps the ability to distribute prebuilt htdocs. I agree that listing all the files in Makefile.am so it can catch updates is preferable.

The problem about forgetting to update is of course very real, I think that risk also exists in some cases for the C-source Makefiles (maybe the header files?). It would be nice if there was some "linter" that could check. If not, maybe we could hack together a github action that checks?

Regarding FreeBSD I just now triggered a re-run on master to see if it started failing there.

[chme]

Regarding FreeBSD I just now triggered a re-run on master to see if it started failing there.

You probably won't see it failing in the GitHub-CI action/workflow. In master the web interface is not built in the FreeBSD build. This is only happening now, because of the changes in this PR.

It would be nice if there was some "linter" that could check. If not, maybe we could hack together a github action that checks?

Yes, that makes sense. I'll look into this, maybe it can be included in the current web linter action.

[chme]

The FreeBSD issue should be fixable by updating the dependencies (especially rollup to > v4.24.1): rollup/rollup#5491.

I'll do an update and check if it is now building on FreeBSD.

I also extended the "web-lint" make goal and changed the github action to use it. Unfortunately, this required installing a bunch of dependencies ... I copied the install from the "ubuntu" workflow.
Another option would be to leave the webui action untouched and call this goal in the "ubuntu" workflow, where all dependencies are already present ...

[chme]

Rebased and squashed commits. The FreeBSD issue was solved by the recent dependency update for the web UI. The FreeBSD action now successfully builds the web UI.

I added a make web-check that checks if all files in web-src/public and web-src/src are listed in the Makefile.am file, to mitigate the risk of accidentally forgetting to add a new file to Makefile.am.

Something like chme/owntone-container@0e40273 can be used to update the Dockerfile in owntone-container.

Let me know, if something else needs to be done. At the moment, I think, this is good to be merged.

[ejurgensen]

Sounds great. I would like to take this for a test run, I will try to do that asap.

[ejurgensen]

I tried building a distribution file from this branch, unpacked it in another dir where I ran ./configure and make. Even though the htdocs dir was populated with prebuilt files, the npm build was still started? Maybe I misunderstood: "If prebuilt files are available, only a message is printed that the web interface will not be built and the prebuilt files will be used. Otherwise, configure fails."

Is it possible to use the prebuilt files on a system where npm is present? I tried with --disable-webinterface, but then it seemed the webinterface is skipped entirely (make install doesn't install it).

[ejurgensen]

Some more observations:
autoreconf -vi && ./configure --disable-webinterface && make distcheck
-> fails with:

config.status: error: cannot find input file: `web-src/Makefile.in'
make: *** [Makefile:887: distcheck] Error 1

It also leaves the build dir in a state where there's a bunch of files in owntone-28.11 that can't easily be cleaned up: git clean -x -d -f -> a bunch of permission denied due to the files being read-only. make clean didn't even try to remove them?

Note that with current master this works as expected (the configure flag doesn't impact make distcheck): autoreconf -vi && ./configure --disable-spotify && make distcheck, but in the branch it fails with:

make[2]: Entering directory '/home/espen/build/github/owchme/owntone-28.11/_build/sub'
Making all in src/inputs/librespot-c
/bin/bash: line 21: cd: src/inputs/librespot-c: No such file or directory
make[2]: *** [Makefile:670: all-recursive] Error 1
make[2]: Leaving directory '/home/espen/build/github/owchme/owntone-28.11/_build/sub'
make[1]: *** [Makefile:512: all] Error 2
make[1]: Leaving directory '/home/espen/build/github/owchme/owntone-28.11/_build/sub'
make: *** [Makefile:887: distcheck] Error 1

[chme]

I tried building a distribution file from this branch, unpacked it in another dir where I ran ./configure and make. Even though the htdocs dir was populated with prebuilt files, the npm build was still started? Maybe I misunderstood: "If prebuilt files are available, only a message is printed that the web interface will not be built and the prebuilt files will be used. Otherwise, configure fails."

Is it possible to use the prebuilt files on a system where npm is present? I tried with --disable-webinterface, but then it seemed the webinterface is skipped entirely (make install doesn't install it).

The first check is if npm is available, if yes, then the web UI is always built and prebuilt files are ignored.
My guess is, that it is preferred to build everything from source and using the prebuilt files is just a fallback.

Should there be a configure flag the will skip building the web UI if prebuilt files are available? I guess, that should be doable ...

Some more observations: autoreconf -vi && ./configure --disable-webinterface && make distcheck -> fails with:

config.status: error: cannot find input file: `web-src/Makefile.in'
make: *** [Makefile:887: distcheck] Error 1

I added commit 1e56bbd that fixes it.
There is some autotools magic at work, that I do not quite understand ... but defining DIST_SUBDIRS was copied from #1439. With having the web UI build integrated it, defining DIST_SUBDIRS is not necessary and causes the issue you noticed.

[ejurgensen]

Should there be a configure flag the will skip building the web UI if prebuilt files are available?

I think it would be best if the prebuilt files are used if present. Of course, if the source files change, rebuild. That's the way the files generated by bison/flex/gperf work. That means that it will work without modification on buildbots with limited/no internet access, and it should also make the builds deterministic.

[chme]

I think it would be best if the prebuilt files are used if present. Of course, if the source files change, rebuild. That's the way the files generated by bison/flex/gperf work.

I don't believe, that this possible. The npm build has no way of knowing, if the existing htdocs are matching the source or not. Especially as not all sources are available (node_modules with all the dependencies is missing), I think, that is the main difference to bison/flex/gperf.
If you know, how this could be achieved, I am all ears :-)

That means that it will work without modification on buildbots with limited/no internet access,

Given that the npm build does not really fit well into the autotools autoreconf/configure/make build system, I fear it is highly unlikely, that it can be done without any modifications for every usecase.

I have added a somewhat experimental commit a94bf97, that adds a --with-prebuilt-webinterface configure flag. If that flag is given to configure, then no npm build is triggered (even if npm is available) and only the prebuilt web interface is used.
This would at least give the option to skip the npm build, if there is no or only limited internet access.

[ejurgensen]

The npm build has no way of knowing, if the existing htdocs are matching the source or not

I think make just does this by comparing the target's file timestamp with the source's