Build Signal from source (and enable arm64 builds) #179

jnsgruk · 2023-10-27T16:43:16Z

This PR switches to enable build from source for Signal Desktop - the primary motivation is to enable builds for arm64. At present Signal does not provide an upstream deb for other architectures, so this is the only viable option I see.

There is a small difficulty which I've worked around, which is that the upstream package.json uses npm-run-all to run multiple targets as part of yarn generate. This was causing the scriptlets to exit early, so I've broken apart those targets into their component parts - but otherwise the process is as per their documentation.

Fixes #178

jnsgruk · 2023-10-27T17:06:25Z

@bceverly this is a start - I'm getting an odd error about not being able to prime the opt directory which I can't figure out, but I need to go make some dinner! Will play with this over the coming days!

jnsgruk · 2023-10-27T17:08:23Z

Error currently:

Executing parts lifecycle: stage cleanup                                                                                                                   
Executing action                                                                                                                                           
Executed: stage cleanup                                                                                                                                    
Executing parts lifecycle: stage desktop-file                                                                                                              
Executing action                                                                                                                                           
Executed: stage desktop-file                                                                                                                               
Executing parts lifecycle: stage gnome/sdk                                                                                                                 
Executing action                                                                                                                                           
Executed: stage gnome/sdk                                                                                                                                  
Executing parts lifecycle: prime signal-desktop                                                                                                            
Executing action                                                                                                                                           
Failed to copy '/root/stage/opt': no such file or directory.                                                                                               
Failed to execute pack in instance.                                                                                                                        
Full execution log: '/home/jon/.local/state/snapcraft/log/snapcraft-20231027-180255.545188.log'

jnsgruk · 2023-10-29T22:33:19Z

@bceverly @kenvandine could you folks give this a try? I think I've resolved the build issues and it's getting close.

lucyllewy · 2023-10-29T22:49:06Z

Unfortunately I believe we're not allowed to build from source per Signal's intervention a couple of years ago. I might be misremembering but I think I recall that Canonical had to remove the Signal snap from the store while some legal work was done behind closed doors, which included committing to using the Signal client as built upstream, as-is.

lucyllewy · 2023-10-29T22:53:52Z

I might be misremembering tho. I found this about the problem back last year: https://forum.snapcraft.io/t/what-happened-to-signal-desktop/32119/18

lucyllewy · 2023-10-29T22:56:27Z

Tagging @merlijn-sebrechts in case he can provide more memory of what the situation was and the resolution.

jnsgruk · 2023-10-30T07:29:15Z

Thanks for the heads up @lucyllewy - will wait for others to respond.

Just noticed this build has failed - was working fine last night! I'll take a look when I get a chance later today or tomorrow.

merlijn-sebrechts · 2023-10-30T10:03:18Z

I have received no additional info about last year's issue, apart from "whoops, sorry for the DMCA takedown". I don't know why it initially happened, and I don't know why they ultimately accepted the Snap package.

Their policy seems to be "no third party packages because users can't trust them".

Given that everything is open source, publicly built, and publicly logged, it should be clear that we're just compiling the signal desktop app instead of adding modifications.
Given the arch package is also building from source, is known by the Signal team, and hasn't been taken down (yet), they seem to have relaxed their stance somewhat.

I'd say to go ahead with this PR. Worst case, we just revert. Once Signal provides upstream arm64 packages, I'd like to switch back to theirs, though.

jnsgruk · 2023-10-30T10:08:46Z

Thanks @merlijn-sebrechts

One other thing I'm not quite sure about right now, is that this builds fine for me locally, but if I use `snapcraft remote-build' to use the Launchpad builders, I get a strange error in the install process:

:: [5/5] Building fresh packages...
:: error /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc: Command failed.
:: Exit code: 1
:: Command: node scripts/fetch-prebuild.js
:: Arguments:
:: Directory: /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc
:: Output:
:: downloading https://build-artifacts.signal.org/libraries/ringrtc-desktop-build-v2.33.0.tar.gz
:: node:events:491
::       throw er; // Unhandled 'error' event
::       ^
::
:: Error: getaddrinfo ENOTFOUND build-artifacts.signal.org
::     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)
:: Emitted 'error' event on ClientRequest instance at:
[30/Oct/2023:09:53:29 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8587995 "-" "-"
[30/Oct/2023:09:53:29 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8587959 "-" "-"
[30/Oct/2023:09:53:30 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8123 "-" "-"
[30/Oct/2023:09:53:31 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8123 "-" "-"
[30/Oct/2023:09:53:31 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8587962 "-" "-"
[30/Oct/2023:09:53:32 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8122 "-" "-"
[30/Oct/2023:09:53:32 +0000] "CONNECT github.com:443 HTTP/1.1" 200 3773 "-" "-"
[30/Oct/2023:09:53:44 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 157646321 "-" "-"
[30/Oct/2023:09:53:48 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 1693958 "-" "-"
[30/Oct/2023:09:54:02 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 76834043 "-" "-"
[30/Oct/2023:09:54:16 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 75487844 "-" "-"
::     at TLSSocket.socketErrorListener (node:_http_client:502:9)
::     at TLSSocket.emit (node:events:513:28)
::     at emitErrorNT (node:internal/streams/destroy:151:8)
::     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
::     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
::   errno: -3008,
::   code: 'ENOTFOUND',
::   syscall: 'getaddrinfo',
::   hostname: 'build-artifacts.signal.org'
:: }

I'm not sure why the LP builder can't see that, but I'll do some digging and update here.

lucyllewy · 2023-10-30T14:42:15Z

:: error /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc: Command failed.
:: Exit code: 1
:: Command: node scripts/fetch-prebuild.js
:: Arguments:
:: Directory: /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc
:: Output:
:: downloading https://build-artifacts.signal.org/libraries/ringrtc-desktop-build-v2.33.0.tar.gz
:: node:events:491
:: throw er; // Unhandled 'error' event
:: ^
::
:: Error: getaddrinfo ENOTFOUND build-artifacts.signal.org
:: at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)
:: Emitted 'error' event on ClientRequest instance at:

I suspect this is an issue with the http_proxy and https_proxy environment variables being killed somewhere within the scripts from upstream during the build when calling that specific subproccess.

jnsgruk · 2023-10-30T15:03:22Z

Yep, you're absolutely right -- there is an attempt to fix it here but no joy as yet. The offending file is obvious, but I can't seem to stop it from trying to fetch without a proxy, even if I pre-fetch the file. Will keep digging.

jnsgruk · 2023-10-31T15:34:23Z

Okay! I've now managed to work through the build issues on Launchpad, and have successfully launched the snap, built on Launchpad, on an amd64 machine.

The process is quite messy; there are some dependencies in the yarn install process that won't respect the proxy without patching. I've done this as part of the build, and it doesn't feel great, but it should also be more robust than trying to carry patches against the files.

@bceverly could you give this a test?

/cc @kenvandine - as promised 😉

tim-hm · 2023-10-31T17:12:49Z

Many thanks for the effort on this @jnsgruk 🙏

I've briefly tested on ubuntu-asahi (arm64) and on an arm64 vm on parallels and can confirm this is working 🙏 Are there any specific features to check?

bceverly · 2023-10-31T17:31:02Z

Thank you @jnsgruk! It's working great for me on the x13s under Mantic!!! -- Thanks, Bryan

…

On Tue, 2023-10-31 at 10:13 -0700, Tim Holmes-Mitra wrote: Many thanks for the effort on this @jnsgruk 🙏 I've briefly tested on ubuntu-asahi (arm64) and on an arm64 vm on parallels and can confirm this is working 🙏 Are there any specific features to check? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

kenvandine

I think this is good to land, and I very much appreciate the inline comments which will help greatly during the ongoing maintenance. One thing we need to consider here is the ongoing need to update various parts as signal has additional releases. It doesn't look like landing this PR will break the automated update check, and if a dependent part does need to be updated it should get caught in the test-snap-can-build action. We just need to understand we're increasing the potential need for human intervention during automated maintenance.

Thanks for your effort on this and it's a very useful reference for other similar electron snaps!

jnsgruk · 2023-11-01T08:02:35Z

Thanks! Yeh that's a good point; I've opened #181 which builds on this PR, and augments the CI to handle this automatically.

popey · 2023-11-01T12:08:23Z

Is the arm64 supported upstream?
If not, are we committing to supporting this build?
Bear in mind the relatively limited availability of Linux on (decent) arm64 devices will make supporting arm64 specific issues harder.

popey · 2023-11-04T10:56:07Z

Just chatted about this with Joe and Im going to +1 this change. Merge at your discretion.

jnsgruk · 2023-11-04T14:19:19Z

Thanks, Alan! I've also submitted a PR to the upstream repos of those node deps that need patching - hopefully we can remove that piece of complexity from the build in the future.

merlijn-sebrechts · 2023-11-05T12:42:46Z

Looks good to me. I can merge it now, but know that this won't actually change anything in the store. The signal-desktop release workflow is completely automated from GitHub using GitHub actions. We need to build the arm version here too.

So the next steps are to update the actions use remote-build as explained here: snapcrafters/gimp#244

I can provide the credentials so GH actions can do this. Do you want to do that in this PR or in a next one?

jnsgruk · 2023-11-05T15:02:35Z

Yep - I can take that on. Let's merge this and I'll do that in another PR.

jnsgruk · 2023-11-05T15:35:52Z

@merlijn-sebrechts added in #181 - which will be easier to review once this merges!

merlijn-sebrechts

LGTM!

jnsgruk mentioned this pull request Oct 27, 2023

[Enhancement]: ARM64 snap of Signal Desktop #178

Closed

jnsgruk marked this pull request as ready for review October 29, 2023 22:33

lucyllewy requested review from merlijn-sebrechts and kenvandine October 29, 2023 22:57

jnsgruk changed the title ~~WIP: enable arm64 builds~~ Build Signal from source (and enable arm64 builds) Oct 30, 2023

jnsgruk marked this pull request as draft October 30, 2023 13:07

jnsgruk marked this pull request as ready for review October 31, 2023 15:34

feat: build from source; enable arm64

2e982f0

refactor: use parts system more eloquently

384f764

kenvandine approved these changes Nov 1, 2023

View reviewed changes

jnsgruk mentioned this pull request Nov 1, 2023

ci: automatically bump node parts and use remote build #181

Merged

merlijn-sebrechts approved these changes Nov 7, 2023

View reviewed changes

merlijn-sebrechts merged commit 3697c2a into snapcrafters:candidate Nov 7, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build Signal from source (and enable arm64 builds) #179

Build Signal from source (and enable arm64 builds) #179

jnsgruk commented Oct 27, 2023 •

edited

Loading

jnsgruk commented Oct 27, 2023

jnsgruk commented Oct 27, 2023

jnsgruk commented Oct 29, 2023

lucyllewy commented Oct 29, 2023

lucyllewy commented Oct 29, 2023

lucyllewy commented Oct 29, 2023

jnsgruk commented Oct 30, 2023

merlijn-sebrechts commented Oct 30, 2023

jnsgruk commented Oct 30, 2023

lucyllewy commented Oct 30, 2023

jnsgruk commented Oct 30, 2023

jnsgruk commented Oct 31, 2023

tim-hm commented Oct 31, 2023

bceverly commented Oct 31, 2023 via email

kenvandine left a comment

jnsgruk commented Nov 1, 2023

popey commented Nov 1, 2023

popey commented Nov 4, 2023

jnsgruk commented Nov 4, 2023

merlijn-sebrechts commented Nov 5, 2023 •

edited

Loading

jnsgruk commented Nov 5, 2023

jnsgruk commented Nov 5, 2023

merlijn-sebrechts left a comment

Build Signal from source (and enable arm64 builds) #179

Build Signal from source (and enable arm64 builds) #179

Conversation

jnsgruk commented Oct 27, 2023 • edited Loading

jnsgruk commented Oct 27, 2023

jnsgruk commented Oct 27, 2023

jnsgruk commented Oct 29, 2023

lucyllewy commented Oct 29, 2023

lucyllewy commented Oct 29, 2023

lucyllewy commented Oct 29, 2023

jnsgruk commented Oct 30, 2023

merlijn-sebrechts commented Oct 30, 2023

jnsgruk commented Oct 30, 2023

lucyllewy commented Oct 30, 2023

jnsgruk commented Oct 30, 2023

jnsgruk commented Oct 31, 2023

tim-hm commented Oct 31, 2023

bceverly commented Oct 31, 2023 via email

kenvandine left a comment

Choose a reason for hiding this comment

jnsgruk commented Nov 1, 2023

popey commented Nov 1, 2023

popey commented Nov 4, 2023

jnsgruk commented Nov 4, 2023

merlijn-sebrechts commented Nov 5, 2023 • edited Loading

jnsgruk commented Nov 5, 2023

jnsgruk commented Nov 5, 2023

merlijn-sebrechts left a comment

Choose a reason for hiding this comment

jnsgruk commented Oct 27, 2023 •

edited

Loading

merlijn-sebrechts commented Nov 5, 2023 •

edited

Loading