Docs4Doge: update of gitian builder script + documentation (follow #2396) #2579

AbcSxyZ · 2021-09-19T17:37:23Z

Related to Docs4Doge initiative, based on & following PR #2396.

Following @micaelmalta work, I reused a good part of his script and edited gitian-builder.sh to fit in a single script, like it was. This is mainly an update of contrib/gitian-build.sh and his documentation doc/gitian-building.md (see it to try the script).
A lot of deleted files are coming from previous documentation images.

It's having Docker available for sure, and maybe LXC & KVM aren't so far if you're able to use/configure them (with some tests I weren't).

Pending PRs on devrandom/gitian-builder

It relies on 2 PR on devrandom/gitian-builder repository. devrandom/gitian-builder#254, already merged, and devrandom/gitian-builder#253 to get --disable-apt-cacher option.
There is some interest in the PR, I suppose it will be merged before we finish to work on this.

For now, to try the script with those features, you can replace inside gitian-build.sh the line with

git clone https://github.com/devrandom/gitian-builder

by a branch of my gitian-builder repository :

git clone https://github.com/abcsxyz/gitian-builder --branch gitian-test

Improvement in mind

It could be used like that, but some possible improvement are also possible before merging. I think I will investigate about the following from @patricklodder comment :

Make a GH Actions (since this is targeting 1.14) script outputting hashes, that triggers on changes to:

./depends

./contrib/gitian-descriptors

./contrib/gitian-builder (this set of scripts)

Any automated initialization should never touch pgp secret keys, so without those, it'd be fine to include. There's a section about external signing in the gitian doc and that should imho be the best practice anyway as this allows for keeping secrets in a protected environment while allowing a build machine to just connect freely and download things. (i.e. from a security p.o.v. there are different requirements to secure pgp secrets vs the gitian builder itself.)

+ Possibility to configure apt-cacher (@patricklodder may apt-cacher be useful for CI or something else, even personally ?) ?

P.S. : Sorry for parallel PR with #2396, I had the choice between PR @micaelmalta's repo to include my changes or create a new PR to Dogecoin Core repo. Thought it was more appropriate to do it that way.

Fix issues of Dogecoin folder getting old files and never cleaned

AbcSxyZ · 2021-09-19T19:46:46Z

External signing enabled. What would be the file to check to verify hashes of depends/, gitian-descriptors/ & gitian-build.sh with GitHub action ? But maybe could be a separate PR.

AbcSxyZ · 2021-09-19T21:47:06Z

Just added --enable-apt-cacher option to use an apt-cacher server.

contrib/gitian-build.sh

rnicoll · 2021-10-23T08:59:15Z

I think the main thing is this needs devrandom/gitian-builder#253 to be merged, and that doesn't seem to be getting attention in the original repo. I suspect because you've not really explained why it's needed, and the impact of disabling the cache.

AbcSxyZ · 2021-10-23T15:28:13Z

The maintainer should be aware of the PR#253. It supersedes a @micaelmalta PR#249 where devrandom (and other) reviewed but wanted some change, I implemented it. Another pending PR was waiting on this feature and, while pushing change on @micaelmalta fork, devrandom told me to do it upstream.

So it's noticed, they were reactive, but I've no more reply since I did the PR. Plus, modifications became slightly trickier. I was waiting on some review/tests, mainly for LXC & KVM. I will ask to see what they want to do and to clarify the situation.

If anyone can test this script with lxc & kvm, with and without apt-cacher by testing --enable-cache option. You need to use my repo using replacing the git clone command with git clone https://github.com/abcsxyz/gitian-builder --branch gitian-test. Would be also useful for PR#253.

patricklodder

The change in descriptor names from signer to signed is undocumented and incompatible with the current state of 1.21 - why is this needed?
Since so much is getting lost in the documentation, how would you feel about creating a new document, call it simple-gitian-building.md and then once it's stable and complete, we can remove the old doc later? We can, if proven to work, update links to go to the simple guide instead.

Further questions and suggestions are inside.

contrib/gitian-build.sh

AbcSxyZ · 2021-11-22T11:52:19Z

Get a response from devrandom, gitian-builder repository is going in maintenance mode because bitcoin is switching to guix build process. Did you already plan to switch to guix with 1.21 ?

patricklodder · 2021-11-26T16:50:37Z

Did you already plan to switch to guix with 1.21 ?

Not to my knowledge. I'd say we'd need to run both in parallel for a while. We can always fork gitian-builder and maintain what we need until we no longer need it.

Can we build something on the CI without removing apt-cacher? yes I think so. Question to you: what's the impact of not removing it to the success of this PR?

AbcSxyZ · 2021-11-26T17:33:22Z

Or a patch ? If it needs only this change, imo the fork doesn't worth it.

At first, would you like to keep this feature to enable gitian without apt-cacher ? Easy to remove, but may be great to facilitate signing, may be interesting if we keep it for a while.

patricklodder · 2021-11-26T17:35:52Z

At first, would you like to keep this feature to enable gitian without apt-cacher ?

Maybe we should break it up into 2 steps:

Make it work without patching anything so that the script and documentation are up-to-date
Find the smallest possible patch to enable direct building.

AbcSxyZ · 2021-11-27T17:25:16Z

The change in descriptor names from signer to signed is undocumented and incompatible with the current state of 1.21

I undo this change, I go with signer. It was not introduced by me, but it looks like it was done to make executables names consistent with gitian descriptor.

I fixed comments, using only apt-cacher. If it's possible to have tests with lxc and/or kvm please.

We can't build signed binaries now, right ? It seems the script is expecting a specific tag release on https://github.com/dogecoin/dogecoin-detached-sigs. I saw some comments where you were working on signatures.

I'm going to submit a patch for devrandom/gitian-builder repo, to disable apt-cacher.

patricklodder · 2021-11-27T22:38:22Z

If undo this change, I go with signer.

Yes, I think signer is good because that's what it has been since forever? I'd just edit the commit, I'll anyway request a squash on the first 2 commits, so may as well just rename back and --amend.

We can't build signed binaries now, right ?

We can. We're just not doing it. At least for Windows. For macOS we're doing it but with the new Apple requirements it's a bit more complicated than what we have right now.

I saw some comments where you were working on signatures.

Correct. I went into a bit of a rabbit hole to also figure out how to do the signed submissions to Apple to get rid of the security warning there. I hope to have a complete picture of what needs to get done over the next week, otherwise we should first fix Windows and then worry about macOS.

patricklodder

Did the second run focusing mostly on grammar and guide. I have 2 non-grammar things in here:

I propose to remove KVM support from this script if this is not documented
LXC_BRIDGE needs a solution

doc/gitian-building.md

contrib/gitian-build.sh

patricklodder · 2021-11-28T01:05:03Z

contrib/gitian-build.sh

-if [[ $lxc = true ]]
-then
-    export USE_LXC=1
-    export LXC_BRIDGE=br0


IMPORTANT: This should still be set for LXC build to work, recommend doing that above when processing --lxc, but do an optional setting, so that it can be overridden (on systems where I have multiple virtualization solutions, this is nearly never br0). Alternatively, document it in the guide that this must ALWAYS be set or add a parameter.

Do you want to go with something like that in option parsing :

--lxc) USE_LXC=1 export LXC_BRIDGE=${LXC_BRIDGE:-br0} ;;

Defining br0 as default value, and letting the user override it if needed ?

patricklodder · 2021-11-28T01:08:41Z

contrib/gitian-build.sh

+echo "Using ${proc} CPU and ${mem} RAM"
+
+# Control the selection of a single virtualisation software
+if [ $(($USE_LXC + $USE_DOCKER)) -ge 2 ]; then


Since KVM is not documented, could we please do an elif with -lt 1 and throw an error too for now?

Let's see your answer related to keep or remove KVM. In anticipation of the case where you want to remove it, do you want to use docker or lxc as default, or do we require from the user to specify it ?

patricklodder · 2021-11-28T01:09:24Z

contrib/gitian-build.sh

+
+# Control the selection of a single virtualisation software
+if [ $(($USE_LXC + $USE_DOCKER)) -ge 2 ]; then
+    echo "$scriptName: Specify a single virtualisation solution between Docker, LXC or KVM."


Suggested change

echo "$scriptName: Specify a single virtualisation solution between Docker, LXC or KVM."

echo "$scriptName: Specify a single virtualisation solution between Docker or LXC."

Co-authored-by: Patrick Lodder <[email protected]>

AbcSxyZ · 2021-11-28T11:07:04Z

I propose to remove KVM support from this script if this is not documented

I'm not sure what should be strategy for KVM support. You suggest removing references within the script too (your current suggestions are cleaning it for the documentation, not the script) ?

It's possibly not so far from working, I wasn't able to use LXC & KVM to test it (mysterious error where I did no investigated too long, never used this tools), but I hope it's designed to (almost) work for someone who knows it.

What I had in mind is to keep KVM support as incomplete and state it as incomplete in the expectation someone will try & fix it. It has less opportunity to happen if we remove all content related to kvm.

From Karl Fogel - Producing Open Source Software How to Run a Successful Free Software Project :

Label the areas where the documentation is known to be incomplete. By showing the readers that you are aware of its deficiencies, you align yourself with their point of view. Your empathy reassures them that they don't face a struggle to convince the project of what's important. These labels needn't represent promises to fill in the gaps by any particular date —it's equally legitimate to treat them as open requests for volunteer help

Are you sure you want to remove KVM (personally, I'm in favor of keeping it) ?

patricklodder · 2021-11-28T20:30:19Z

Are you sure you want to remove KVM (personally, I'm in favor of keeping it)?

I'm okay with keeping it but could you then please document how far you got with it? What errors did you run into?

I wasn't able to use LXC & KVM

As for LXC, the gitian-builder side definitely works because since #2501 I build it, so it's on my todo to test end-to-end with this script and we'll get that done.

AbcSxyZ · 2021-11-28T23:01:31Z

KVM

First, on debian the installation of kvm required to find manually package, I possibly used the wrong one. I used python-vm-builder 0.12.4, python-cheetah 2.4.4, ubuntu-keyring 2018.09.18.1(?).

When building the base vm for bionic, it's throwing the following error:

VMBuilder.exception.VMBuilderUserError: Invalid suite: "bionic". Valid suites are: dapper gutsy hardy intrepid jaunty karmic lucid maverick natty oneiric precise quantal raring saucy trusty utopic vivid wily xenial

I tried with trusty, modified the gitian-descriptor to fit this, but on gbuild the container do not start/can't be connected. It gets an ssh error:

--- Building for trusty amd64 ---
Stopping target if it is up
Making a new image copy
qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of qcow2)
Formatting 'target-trusty-amd64.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=13957595136 backing_file=base-trusty-amd64.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
Starting target
Checking if target is up..............................
ssh: connect to host localhost port 2223: Connection refused

LXC

I'm able to build base image, to start gbuild step. I'm having trouble with apt sources. Not able to find packages during install steps, configuration issue with apt-cacher:

abcsxyz:~$ ./gitian-build.sh  --setup --build --lxc -o l 1.14.5
pdating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Traceback (most recent call last):
        6: from ./bin/gbuild:334:in `<main>'
        5: from ./bin/gbuild:334:in `each'
        4: from ./bin/gbuild:336:in `block in <main>'
        3: from ./bin/gbuild:336:in `each'
        2: from ./bin/gbuild:341:in `block (2 levels) in <main>'
        1: from ./bin/gbuild:120:in `build_one_configuration'
./bin/gbuild:23:in `system!': failed to run on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install curl g++-aarch64-linux-gnu g++-7-aarch64-linux-gnu gcc-7-aarch64-linux-gnu binutils-aarch64-linux-gnu g++-arm-linux-gnueabihf g++-7-arm-linux-gnueabihf gcc-7-arm-linux-gnueabihf binutils-arm-linux-gnueabihf g++-7-multilib gcc-7-multilib binutils-gold git-core pkg-config autoconf libtool automake faketime bison bsdmainutils ca-certificates python3 >> var/install.log 2>&1 (RuntimeError)

install.log:

mesg: ttyname failed: Inappropriate ioctl for device                                                                                                                            
Err:1 http://10.0.2.2:3142/archive.ubuntu.com/ubuntu bionic InRelease                                                                                                           
  Could not connect to 10.0.2.2:3142 (10.0.2.2). - connect (113: No route to host)                                                                                              
Err:2 http://10.0.2.2:3142/security.ubuntu.com/ubuntu bionic-security InRelease                                                                                                 
  Unable to connect to 10.0.2.2:3142:                                                                                                                                           
Err:3 http://10.0.2.2:3142/archive.ubuntu.com/ubuntu bionic-updates InRelease                                                                                                   
  Unable to connect to 10.0.2.2:3142: 
[....]
E: Package 'curl' has no installation candidate
E: Unable to locate package g++-aarch64-linux-gnu
E: Couldn't find any package by regex 'g++-aarch64-linux-gnu'
E: Unable to locate package g++-7-aarch64-linux-gnu
E: Couldn't find any package by regex 'g++-7-aarch64-linux-gnu'
E: Unable to locate package gcc-7-aarch64-linux-gnu
E: Unable to locate package binutils-aarch64-linux-gnu
E: Unable to locate package g++-arm-linux-gnueabihf
E: Couldn't find any package by regex 'g++-arm-linux-gnueabihf'
E: Unable to locate package g++-7-arm-linux-gnueabihf
E: Couldn't find any package by regex 'g++-7-arm-linux-gnueabihf'
E: Unable to locate package gcc-7-arm-linux-gnueabihf
E: Unable to locate package binutils-arm-linux-gnueabihf
E: Package 'g++-7-multilib' has no installation candidate
E: Package 'gcc-7-multilib' has no installation candidate
E: Package 'pkg-config' has no installation candidate
E: Package 'autoconf' has no installation candidate
E: Package 'libtool' has no installation candidate
E: Package 'automake' has no installation candidate
E: Unable to locate package faketime
E: Package 'bison' has no installation candidate
E: Package 'bsdmainutils' has no installation candidate

For both LXC & KVM, it seems mainly configuration issues for me. I've the good hope that people who understand those tools are able to use the script 😅

patricklodder · 2022-05-10T14:02:12Z

I've got this to work with LXC (and with apt-cache), but gitian-builder is broken for creating non-docker ubuntu base images so I had to patch things up. I've done a PR on devrandom's repo to fix that, so that I can test this end-to-end without patching anything.

Looking at the issues we're running into, perhaps we should pin to a gitian-builder commit in the script by default. I'll propose that separately after we merge this one.

patricklodder · 2022-05-20T18:23:34Z

The PR on gitian-builder has been merged, so this should now work for both Docker and LXC base images again. Will test this over the weekend.

patricklodder

This now works as documented (per the updated documentation) with docker for me. All that is needed is a squash on the commits: would prefer to have Micael's commits squashed into one, and the followup commits into another, with an end result of a neat 2 commits.

I'll take care of the LXC documentation in a separate PR.

AbcSxyZ · 2022-05-28T12:36:11Z

Hi, can you do it yourself ? I deleted my fork, as I do not expect to work anymore on Dogecoin, just leaving my work/PRs to be reused.

patricklodder · 2022-05-28T12:43:37Z

Even better, I can do it for you.

Am sad to hear that, but thank you for everything you have done! ❤️

patricklodder · 2022-06-20T23:52:31Z

Merged with #2973! Many thanks! ❤️

micaelmalta and others added 8 commits August 21, 2021 00:07

Gitian builder

2f7fa1b

fix: clean dogecoin git folder

5beea3a

Fix issues of Dogecoin folder getting old files and never cleaned

gitian: undelete & update previous script

033d147

gitian: update description + comments

5baca2e

gitian: reorganize help menu of gitian-build.sh

5da8351

gitian-build: rewrite option explanation for --sign with gpg

7b70b2f

gitian-build: remove docker reference from setup option

a65d17a

gitian-builder: update documentation

4a3e013

AbcSxyZ mentioned this pull request Sep 19, 2021

Gitian builder #2396

Closed

gitian-builder: enable external signing

de00a2c

gitian-builder: enable apt cacher

45801ec

AbcSxyZ mentioned this pull request Sep 21, 2021

Docs4Doge Initiative - Update the Dogecoin documentation #2271

Closed

25 tasks

AbcSxyZ commented Sep 23, 2021

View reviewed changes

contrib/gitian-build.sh Show resolved Hide resolved

contrib/gitian-build.sh Show resolved Hide resolved

contrib/gitian-build.sh Outdated Show resolved Hide resolved

contrib/gitian-build.sh Show resolved Hide resolved

AbcSxyZ mentioned this pull request Oct 23, 2021

Feature: add option --disable-apt-cacher (following #249) devrandom/gitian-builder#253

Closed

patricklodder mentioned this pull request Nov 1, 2021

Open items for 1.14.5 release #2655

Closed

27 tasks

patricklodder requested changes Nov 1, 2021

View reviewed changes

patricklodder added this to the 1.14.6 milestone Nov 2, 2021

patricklodder added the enhancement label Nov 2, 2021

patricklodder changed the base branch from 1.14.5-dev to 1.14.6-dev November 8, 2021 13:19

AbcSxyZ added 2 commits November 27, 2021 13:51

gitian-build: Use apt cacher by default

50f1737

gitian-build: switch from trusty to bionic

9c83d2b

AbcSxyZ added 2 commits November 27, 2021 18:11

gitian-build: restore old documentation

965b3e1

gitian-builder: reset descriptor file from signed to signer

dcf3210

AbcSxyZ mentioned this pull request Nov 27, 2021

gitian-builder: add patches to disable apt-cacher #2723

Closed

patricklodder self-requested a review November 27, 2021 23:10

patricklodder requested changes Nov 28, 2021

View reviewed changes

AbcSxyZ and others added 6 commits November 28, 2021 11:09

gitian-build: Fix grammar

fff3e47

Co-authored-by: Patrick Lodder <[email protected]>

gitian-builder: list pgp in required dependencies + remove apache

71ab2be

Co-authored-by: Patrick Lodder <[email protected]>

gitian-builder: use console for code block

37be2a5

gitian-builder: fix console comments

8fa9366

gitian-builder: remove output of help menu

a9d3ec6

gitian-builder: use bash for code block

ffa75d9

gitian-build: add LXC_BRIDGE

de7324b

dogecoin deleted a comment from QuentinFlores Mar 6, 2022

patricklodder requested changes May 27, 2022

View reviewed changes

patricklodder mentioned this pull request May 28, 2022

Improved gitian build script #2973

Merged

patricklodder closed this Jun 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs4Doge: update of gitian builder script + documentation (follow #2396) #2579

Docs4Doge: update of gitian builder script + documentation (follow #2396) #2579

AbcSxyZ commented Sep 19, 2021 •

edited

Loading

AbcSxyZ commented Sep 19, 2021 •

edited

Loading

AbcSxyZ commented Sep 19, 2021

rnicoll commented Oct 23, 2021

AbcSxyZ commented Oct 23, 2021 •

edited

Loading

patricklodder left a comment

AbcSxyZ commented Nov 22, 2021

patricklodder commented Nov 26, 2021

AbcSxyZ commented Nov 26, 2021 •

edited

Loading

patricklodder commented Nov 26, 2021

AbcSxyZ commented Nov 27, 2021 •

edited

Loading

patricklodder commented Nov 27, 2021

patricklodder left a comment

patricklodder Nov 28, 2021

AbcSxyZ Nov 28, 2021

patricklodder Nov 28, 2021

AbcSxyZ Nov 28, 2021

patricklodder Nov 28, 2021

AbcSxyZ commented Nov 28, 2021 •

edited

Loading

patricklodder commented Nov 28, 2021

AbcSxyZ commented Nov 28, 2021 •

edited

Loading

patricklodder commented May 10, 2022

patricklodder commented May 20, 2022

patricklodder left a comment

AbcSxyZ commented May 28, 2022 •

edited

Loading

patricklodder commented May 28, 2022

patricklodder commented Jun 20, 2022

	echo "$scriptName: Specify a single virtualisation solution between Docker, LXC or KVM."
	echo "$scriptName: Specify a single virtualisation solution between Docker or LXC."

Docs4Doge: update of gitian builder script + documentation (follow #2396) #2579

Docs4Doge: update of gitian builder script + documentation (follow #2396) #2579

Conversation

AbcSxyZ commented Sep 19, 2021 • edited Loading

Pending PRs on devrandom/gitian-builder

Improvement in mind

AbcSxyZ commented Sep 19, 2021 • edited Loading

AbcSxyZ commented Sep 19, 2021

rnicoll commented Oct 23, 2021

AbcSxyZ commented Oct 23, 2021 • edited Loading

patricklodder left a comment

Choose a reason for hiding this comment

AbcSxyZ commented Nov 22, 2021

patricklodder commented Nov 26, 2021

AbcSxyZ commented Nov 26, 2021 • edited Loading

patricklodder commented Nov 26, 2021

AbcSxyZ commented Nov 27, 2021 • edited Loading

patricklodder commented Nov 27, 2021

patricklodder left a comment

Choose a reason for hiding this comment

patricklodder Nov 28, 2021

Choose a reason for hiding this comment

AbcSxyZ Nov 28, 2021

Choose a reason for hiding this comment

patricklodder Nov 28, 2021

Choose a reason for hiding this comment

AbcSxyZ Nov 28, 2021

Choose a reason for hiding this comment

patricklodder Nov 28, 2021

Choose a reason for hiding this comment

AbcSxyZ commented Nov 28, 2021 • edited Loading

patricklodder commented Nov 28, 2021

AbcSxyZ commented Nov 28, 2021 • edited Loading

KVM

LXC

patricklodder commented May 10, 2022

patricklodder commented May 20, 2022

patricklodder left a comment

Choose a reason for hiding this comment

AbcSxyZ commented May 28, 2022 • edited Loading

patricklodder commented May 28, 2022

patricklodder commented Jun 20, 2022

AbcSxyZ commented Sep 19, 2021 •

edited

Loading

AbcSxyZ commented Sep 19, 2021 •

edited

Loading

AbcSxyZ commented Oct 23, 2021 •

edited

Loading

AbcSxyZ commented Nov 26, 2021 •

edited

Loading

AbcSxyZ commented Nov 27, 2021 •

edited

Loading

AbcSxyZ commented Nov 28, 2021 •

edited

Loading

AbcSxyZ commented Nov 28, 2021 •

edited

Loading

AbcSxyZ commented May 28, 2022 •

edited

Loading