From ed5deeead5fac4bcdd2131012b1640886e26c015 Mon Sep 17 00:00:00 2001 From: Ning Shang Date: Thu, 14 Jan 2021 21:05:48 -0800 Subject: [PATCH 1/2] Differentiate ISO types: dev or release Release build: '/vagrant/build.sh' or '/vagrant/build.sh release'. Development build: '/vagrant/build.sh dev'. A release image is to be used by cold storage operators who do not need to have access to an interactive shell. For a release image to boot into a root shell successfully, an SCSI/ATA hard drive needs to be available, with a partition '/dev/sda1' used as persistent storage for the cold wallet. The partition can be formatted in ext3 or vfat for example. The device node '/dev/sda1' is required to give the operators a seamless experience when booting up the system. A few other files, including the HSM world and encrypted secrets, and compiled subzero code (in .jar and signed .sar), are not checked in to the project repository, and need to be added to their appropriate locations (please refer to README.md) before building a release image that will work out of the box. A development image is to be used by subzero developers for HSM and cold wallet development. A developer can boot into an interactive root shell without performing the above mentioned steps. The HSM software is automatically installed in the development image, but a developer needs to manually set up persistent storage for the cold wallet, put .jar and .sar files to appropriate filesystem locations, and execute them accordingly, for development and testing. --- live-usb-creator/README.md | 110 +++++++++++++----- live-usb-creator/build.sh | 19 ++- .../install_scripts/1_post_install_chroot | 5 +- .../live_scripts/install_nfast_tools_dev | 22 ++++ ...fast_tools => install_nfast_tools_release} | 9 +- 5 files changed, 132 insertions(+), 33 deletions(-) create mode 100755 live-usb-creator/live_scripts/install_nfast_tools_dev rename live-usb-creator/live_scripts/{install_nfast_tools => install_nfast_tools_release} (84%) diff --git a/live-usb-creator/README.md b/live-usb-creator/README.md index 7f64711b..37e9914d 100644 --- a/live-usb-creator/README.md +++ b/live-usb-creator/README.md @@ -10,12 +10,12 @@ Three dependencies need to be fetched out-of-band. Set the following files in place in the same directory as the `Vagrantfile`. * CodeSafe-linux64-dev-12.50.2.iso (2.6GB): supplied by the HSM vendor. -* CentOS-7-x86_64-Everything-1908.iso (10G): `curl -O http://mirror.seas.harvard.edu/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-Everything-1908.iso` -* kernel-devel-3.10.0-957.12.2.el7.x86_64.rpm (17MB): `curl -O http://archive.kernel.org/centos-vault/centos/7.6.1810/updates/x86_64/Packages/kernel-devel-3.10.0-957.12.2.el7.x86_64.rpm` +* CentOS-7-x86_64-Everything-1908.iso (10G): `curl -O https://vault.centos.org/7.7.1908/isos/x86_64/CentOS-7-x86_64-Everything-1908.iso` +* kernel-devel-3.10.0-957.12.2.el7.x86_64.rpm (17MB): `curl -L -O http://archive.kernel.org/centos-vault/centos/7.6.1810/updates/x86_64/Packages/kernel-devel-3.10.0-957.12.2.el7.x86_64.rpm` Verify the following SHA256 sums: -``` +```bash $ shasum -a 256 CodeSafe-linux64-dev-12.50.2.iso CentOS-7-x86_64-Everything-1908.iso kernel-devel-3.10.0-957.12.2.el7.x86_64.rpm 23ca2c5fc2476887926409bc69f19b772c99191b1e0cce1a3bace8d1e4488528 CodeSafe-linux64-dev-12.50.2.iso bd5e6ca18386e8a8e0b5a9e906297b5610095e375e4d02342f07f32022b13acf CentOS-7-x86_64-Everything-1908.iso @@ -28,32 +28,53 @@ This workflow uses the tool [Vagrant](https://www.vagrantup.com/) to orchestrate Build the image in a CentOS VM guest as follows: -``` -# install virtualbox using brew cask if virtualbox is not installed -brew cask install virtualbox -# install vagrant using brew cask if vagrant is not installed -brew cask install vagrant -# install vagrant-vbguest if vagrant-vbguest is not installed -vagrant plugin install vagrant-vbguest -# remove the existing centos/7 image, if one exists -vagrant box remove -f centos/7 -# install centos 7 vagrant box version 1905.1 (checksum via https://cloud.centos.org/centos/7/vagrant/x86_64/images/sha256sum.txt.asc) -vagrant box add --checksum-type sha256 --checksum de768cf0180d712a6eac1944bd78870c060999a8b6617f8f9af98ddbb9f2d271 --provider virtualbox --box-version 1905.1 centos/7 -# boot up a virtualbox VM to run the install in -vagrant up -# enter the VM -vagrant ssh -# kick off the build, from the VM (takes about 25 minutes) -/vagrant/build.sh -# will create boot.iso (1.1G) in the current directory (i.e. outside the VM) when finished -``` +* Install the toolchain: + + ```bash + # install virtualbox using brew cask if virtualbox is not installed + brew cask install virtualbox + # install vagrant using brew cask if vagrant is not installed + brew cask install vagrant + # install vagrant-vbguest if vagrant-vbguest is not installed + vagrant plugin install vagrant-vbguest + ``` + +* Build the ISO: + + ```bash + # remove the existing centos/7 image, if one exists + vagrant box remove -f centos/7 + # install centos 7 vagrant box version 1905.1 (checksum via https://cloud.centos.org/centos/7/vagrant/x86_64/images/sha256sum.txt.asc) + vagrant box add --checksum-type sha256 --checksum de768cf0180d712a6eac1944bd78870c060999a8b6617f8f9af98ddbb9f2d271 --provider virtualbox --box-version 1905.1 centos/7 + # boot up a virtualbox VM to run the install in + vagrant up + # enter the VM + vagrant ssh + # kick off the build, from the VM (takes about 5-25 minutes), to build a release image. For building a development image, use `/vagrant/build.sh dev` instead + /vagrant/build.sh + # will create boot.iso (1.1G) in the current directory (i.e. outside the VM) when finished + ``` + +* Rebuid the ISO: + + ```bash + # Logout from the VM + exit + # Halt the VM + vagrant halt + # Remove the existing centos/7 image + vagrant box remove -f centos/7 + # Clean up the git repo, preserving the artifacts obtained out-of-band (gitignored) + git clean -fd + # Following the steps in "Build the ISO" above to rebuild the ISO + ``` ## Writing image to USB drive 1. Identify device path to disk (such as `/dev/disk3`) with `diskutil list external physical` 2. Unmount all volumes on that disk with `diskutil unmountDisk /dev/disk3` 3. Identify the “raw” device path by replacing `disk` with `rdisk` in the device path. (E.g. `/dev/disk3` becomes `/dev/rdisk3`.) This speeds up writes by 3–4× in very informal testing. -3. Write ISO to that “raw” device path with `sudo dd bs=1m if=boot.iso of=/dev/rdisk5`. Remember that on MacOS you can check in on progress by sending SIGINFO with Ctrl+T. (Takes about 30 minutes in one test.) +3. Write ISO to that “raw” device path with `sudo dd bs=1m if=boot.iso of=/dev/rdisk3`. Remember that on MacOS you can check in on progress by sending SIGINFO with Ctrl+T. (Takes about 3 minutes in one test.) 4. Eject the disk with `diskutil eject /dev/disk3` 5. Remember to hold F12 during startup to get a boot menu. You’ll then select something like `UEFI: SanDisk`. @@ -73,17 +94,46 @@ In general there are 7 phases of the build process, all kicked off in sequence w One of the most time-intensive (and quiet) stages of this process is the `mksquashfs` operation (`Creating 4.0 filesystem on /var/tmp/tmp??????/images/install.img, block size 131072.`). You can get a rough check on progress by comparing `install.img` size to `install.img` size in a previous build, in a separate `vagrant ssh` session (will output some value between 0.0% and 100.0%, where 100.0% is completely done with the creation of the install image, assuming output sizes are in fact identical): -``` +```bash sudo sh -c "stat --printf='%s' /var/tmp/tmp*/images/install.img | python -c 'import sys; print \"%0.1f%%\" % (float(int(sys.stdin.read())) / 1045090304.0 * 100.0)'" ``` ## Hacking on this project + Here are the useful files you may want to tweak. VM related stuff: + * `Vagrantfile`: Provisions a VirtualBox VM in which the live system ISO will be built using the Vagrant tool. * `bootstrap.sh`: A shell script that runs at the very end of VM bringup. (Currently just mounts the CodeSafe ISO.) -* `build.sh`: Manually run by the user to kick off the ISO build. Currently includes some patches for Lorax and Kickstart files applied before running the build proper with `livemedia-creator`. +* `build.sh`: Manually run by the user to kick off the ISO build. Currently includes some patches for Lorax and Kickstart files applied before running the build proper with `livemedia-creator`. Two image types, development and release, can be built. + * Release build: `build.sh` or `build.sh release`. + + A release image is to be used by cold storage operators who do not need + to have access to an interactive shell. For a release image to boot into + a root shell successfully, an SCSI/ATA hard drive needs to be available, + with a partition '/dev/sda1' used as persistent storage for the cold + wallet. The partition can be formatted in ext3 or vfat for example. The + device node `/dev/sda1` is required to give the operators a seamless + experience when booting up the system. A few other files, including the + HSM world and encrypted secrets, and compiled subzero code (in .jar and + signed .sar), are not checked in to the project repository, and need to + be added to their appropriate locations before building a release image + that will work out of the box. + + * `data_app_subzero/` should contain `subzero-cli.jar`, + `subzero-signed.sar`, `subzero-userdata-signed`. + * `/dev/sda1` when mounted to `/hd`, should have the HSM and wallets files contained in `/hd/local` and `/hd/wallets`, respectively. + + * Development build: `build.sh dev`. + + A development image is to be used by subzero developers for HSM and cold + wallet development. A developer can boot into an interactive root shell + without performing the above mentioned steps. The HSM software is + automatically installed in the development image, but a developer needs + to manually set up persistent storage for the cold wallet, put .jar and + .sar files to appropriate filesystem locations, and execute them + accordingly, for development and testing. Install related stuff: @@ -93,6 +143,7 @@ to %packages, and added in hooks for the below. * `install_scripts/1_post_install_chroot`: This bash script is executed by anaconda (the CentOS installer) directly after install (but after `install_scripts/0_post_install_nochroot`). From the point of view of the script, the newly installed CentOS (soon to be implanted in our live ISO) is chrooted to `/`. That makes this bash script the ideal place to run Linux binaries (like useradd, systemctl, yum-config-manager, etc.) that change system state. Pretty bootloader related stuff: + * `patches/usr/share/lorax/live/`: includes changes to SYSLINUX/ISOLINUX and GRUB2 configs. * `otf_fonts/`: OTF fonts (now just SQ Market Bold) * `grub_fonts/`: PF2 versions of those fonts (PF2 is a proprietary GRUB bitmap font format) @@ -102,11 +153,18 @@ Pretty bootloader related stuff: Live system related stuff: * `live_scripts/`: All the files in this directory (except README.md) will be copied into `/usr/local/bin/` on the live system image, putting them in-$PATH. Useful for utility scripts that should be available on the live system. -* `live_scripts/install_nfast_tools`: This script is run at startup (via /usr/local/bin/startup and /etc/rc.d/init.d/livesys), and builds the nfast kernel module in /opt/nfast/driver and runs the /opt/nfast/sbin/install script. This has to be done from the live system because the install script behaves differently when no PCI card is attached to the system. The contents of this file block entering an interactive shell, thanks to `nfast_block_shell` and a bash profile addition. +* `live_scripts/install_nfast_tools_{dev|release}`: Depending on image type (development or release), this script is copied into `live_scripts/install_nfast_tools` at build time. The script `live_scripts/install_nfast_tools` is run at startup (via /usr/local/bin/startup and /etc/rc.d/init.d/livesys), and builds the nfast kernel module in /opt/nfast/driver and runs the /opt/nfast/sbin/install script. This has to be done from the live system because the install script behaves differently when no PCI card is attached to the system. The contents of this file block entering an interactive shell, thanks to `nfast_block_shell` and a bash profile addition. * `live_scripts/startup`: This script is run at startup (via /etc/rc.d/init.d/livesys). The contents of this file DO NOT block entering an interactive shell (they run async). * `live_scripts/nfast_block_shell`: Blocks until the existence of the path /.nfast-configured, which is intended to be touched after NFast Tools have been installed with `install_nfast_tools`. ## Bootloader + BIOS boot uses SYSLINUX/ISOLINUX. SYSLINUX/ISOLINUX config lives at `/usr/share/lorax/live/config_files/x86/isolinux.cfg`. EFI boot uses GRUB2. GRUB2 config template lives at `/usr/share/lorax/live/config_files/x86/grub2-efi.cfg`. + +## Troubleshooting Tips + +Normally, operators and developers log onto the LiveUSB system as `root`. For +LiveUSB image testing, you may log in as the `liveuser` user, become `root` +(with `su`), for debugging. diff --git a/live-usb-creator/build.sh b/live-usb-creator/build.sh index 8e1e3edf..c5657362 100755 --- a/live-usb-creator/build.sh +++ b/live-usb-creator/build.sh @@ -10,13 +10,30 @@ sudo cp /vagrant/patches/usr/share/lorax/live/x86.noeverything.tmpl /usr/share/l sudo cp /vagrant/patches/usr/share/lorax/live/efi.tmpl /usr/share/lorax/live/efi.tmpl ############################################################################### +# Customize live scripts based on build type: release (default) or dev +############################################################################### +rm -rf /vagrant/live_scripts/.isotype_rel /vagrant/live_scripts/.isotype_dev +if [ "$1" = "dev" ]; then + BUILD_TYPE="Dev" + cp /vagrant/live_scripts/install_nfast_tools_dev /vagrant/live_scripts/install_nfast_tools + touch /vagrant/live_scripts/.isotype_dev +elif [ "$1" = "release" ] || [ "$#" -eq 0 ]; then + BUILD_TYPE="Rel" + cp /vagrant/live_scripts/install_nfast_tools_release /vagrant/live_scripts/install_nfast_tools + touch /vagrant/live_scripts/.isotype_rel +else + echo "Error occurred. Usage: $0 [release|dev]." >&2 + exit 1 +fi + +############################################################################## # Actually build the ISO ############################################################################### # livemedia-creator refuses to run if results_dir exists sudo rm -rf /tmp/build -sudo livemedia-creator --logfile=/vagrant/livemedia-creator.log --make-iso --ks=/vagrant/rhel7-livemedia.ks --resultdir="/tmp/build" --no-virt --project="CentOS" --releasever="7.7.1908" --volid="CentOS 7 (1908) + nCipher x86_64" +sudo livemedia-creator --logfile=/vagrant/livemedia-creator.log --make-iso --ks=/vagrant/rhel7-livemedia.ks --resultdir="/tmp/build" --no-virt --project="CentOS" --releasever="7.7.1908" --volid="CentOS 7 (1908) + nCipher ($BUILD_TYPE)" ############################################################################### # Copy ISO back to host diff --git a/live-usb-creator/install_scripts/1_post_install_chroot b/live-usb-creator/install_scripts/1_post_install_chroot index 5c7e9893..89ea7c1b 100755 --- a/live-usb-creator/install_scripts/1_post_install_chroot +++ b/live-usb-creator/install_scripts/1_post_install_chroot @@ -40,5 +40,6 @@ systemctl disable kdump.service ############################################################################### # Subzero specific stuff ############################################################################### - -echo 'exec /data/app/subzero/subzero-cli.sh' >> /root/.bash_profile +if [ -f "/usr/local/bin/.isotype_rel" ]; then + echo 'exec /data/app/subzero/subzero-cli.sh' >> /root/.bash_profile +fi diff --git a/live-usb-creator/live_scripts/install_nfast_tools_dev b/live-usb-creator/live_scripts/install_nfast_tools_dev new file mode 100755 index 00000000..9fce76f8 --- /dev/null +++ b/live-usb-creator/live_scripts/install_nfast_tools_dev @@ -0,0 +1,22 @@ +#!/bin/bash + +############################################################################### +# Install NCipher for development image +############################################################################### + +# note that we have to do this after livemedia boot, because the install script behaves differently based on whether +# a PCI NShield is plugged in, and we don’t want our ISO build to depend on having a PCI Nshield plugged in + +pushd /opt/nfast/driver +./configure +make install +popd + +pushd /opt/nfast/sbin +./install +popd + +/opt/nfast/bin/config-serverstartup --enable-tcp --enable-privileged-tcp +# restart the hardserver for the changes to take effect +/opt/nfast/sbin/init.d-ncipher restart + diff --git a/live-usb-creator/live_scripts/install_nfast_tools b/live-usb-creator/live_scripts/install_nfast_tools_release similarity index 84% rename from live-usb-creator/live_scripts/install_nfast_tools rename to live-usb-creator/live_scripts/install_nfast_tools_release index e7576b4a..04715070 100755 --- a/live-usb-creator/live_scripts/install_nfast_tools +++ b/live-usb-creator/live_scripts/install_nfast_tools_release @@ -1,7 +1,7 @@ #!/bin/bash ############################################################################### -# Install NCipher +# Install NCipher for release image ############################################################################### # note that we have to do this after livemedia boot, because the install script behaves differently based on whether @@ -23,13 +23,14 @@ popd /opt/nfast/bin/nopclearfail --clear --all mkdir -p /hd +# /dev/sda1 is the harddrive used for persistent storage mount /dev/sda1 /hd # Setup symlinks for /opt/nfast/kmdata/local and /data/app/subzero/wallets mkdir -p /opt/nfast/kmdata/ -rm -rf /opt/nfast/kmdata/local/ -ln -s /hd/local /opt/nfast/kmdata/local -ln -s /hd/wallets /data/app/subzero/wallets +rm -rf /opt/nfast/kmdata/local +ln -sf /hd/local /opt/nfast/kmdata/local +ln -sf /hd/wallets /data/app/subzero/wallets nohup /opt/nfast/bin/see-stdioesock-serv --machine /data/app/subzero/subzero-signed.sar --userdata-sar /data/app/subzero/subzero-userdata-signed.sar > /hd/subzero-signed.stdout 2> /hd/subzero-signed.stderr & From 2356699a4baf8e20e842db42d892965e20d5d0d8 Mon Sep 17 00:00:00 2001 From: Ning Shang Date: Fri, 15 Jan 2021 16:50:25 -0800 Subject: [PATCH 2/2] Make live-usb-creator README more accurate and fix errors. Thank alokmenghrajani and oreparaz for their feedback. --- live-usb-creator/README.md | 47 +++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/live-usb-creator/README.md b/live-usb-creator/README.md index 37e9914d..f0548f28 100644 --- a/live-usb-creator/README.md +++ b/live-usb-creator/README.md @@ -71,11 +71,11 @@ Build the image in a CentOS VM guest as follows: ## Writing image to USB drive -1. Identify device path to disk (such as `/dev/disk3`) with `diskutil list external physical` -2. Unmount all volumes on that disk with `diskutil unmountDisk /dev/disk3` +1. Identify device path to disk (such as `/dev/disk3`) with `diskutil list external physical`. We will call it `/dev/diskN` in the rest of the document. +2. Unmount all volumes on that disk with `diskutil unmountDisk /dev/diskN` 3. Identify the “raw” device path by replacing `disk` with `rdisk` in the device path. (E.g. `/dev/disk3` becomes `/dev/rdisk3`.) This speeds up writes by 3–4× in very informal testing. -3. Write ISO to that “raw” device path with `sudo dd bs=1m if=boot.iso of=/dev/rdisk3`. Remember that on MacOS you can check in on progress by sending SIGINFO with Ctrl+T. (Takes about 3 minutes in one test.) -4. Eject the disk with `diskutil eject /dev/disk3` +3. Write ISO to that “raw” device path with `sudo dd bs=1m if=boot.iso of=/dev/rdiskN`. Remember that on MacOS you can check in on progress by sending SIGINFO with Ctrl+T. (Takes about 3 minutes in one test.) +4. Eject the disk with `diskutil eject /dev/diskN` 5. Remember to hold F12 during startup to get a boot menu. You’ll then select something like `UEFI: SanDisk`. ## Build process @@ -111,29 +111,38 @@ VM related stuff: A release image is to be used by cold storage operators who do not need to have access to an interactive shell. For a release image to boot into - a root shell successfully, an SCSI/ATA hard drive needs to be available, - with a partition '/dev/sda1' used as persistent storage for the cold + a root shell successfully, + * An SCSI/ATA hard drive needs to be available, with a partition + '/dev/sda1' used as persistent storage for the cold wallet. The partition can be formatted in ext3 or vfat for example. The device node `/dev/sda1` is required to give the operators a seamless - experience when booting up the system. A few other files, including the - HSM world and encrypted secrets, and compiled subzero code (in .jar and - signed .sar), are not checked in to the project repository, and need to - be added to their appropriate locations before building a release image - that will work out of the box. - - * `data_app_subzero/` should contain `subzero-cli.jar`, - `subzero-signed.sar`, `subzero-userdata-signed`. - * `/dev/sda1` when mounted to `/hd`, should have the HSM and wallets files contained in `/hd/local` and `/hd/wallets`, respectively. + experience when booting up the system. + * A provisioned HSM needs to be operational on the machine. + * A few other files, including the HSM world and encrypted secrets, and + the compiled subzero code (in .jar and signed .sar), are not checked in + to the project repository, and need to be added to the appropriate + locations in the subzero repository's `live-usb-creator` directory before + building a release image that will work out of the box. Specifically, + + * `data_app_subzero/` should contain `subzero-cli.jar`, + `subzero-signed.sar`, `subzero-userdata-signed`. + * `/dev/sda1` when mounted to `/hd`, should have the HSM and wallets + files contained in `/hd/local` and `/hd/wallets`, respectively. * Development build: `build.sh dev`. A development image is to be used by subzero developers for HSM and cold wallet development. A developer can boot into an interactive root shell without performing the above mentioned steps. The HSM software is - automatically installed in the development image, but a developer needs - to manually set up persistent storage for the cold wallet, put .jar and - .sar files to appropriate filesystem locations, and execute them - accordingly, for development and testing. + automatically installed in the development image to allow a subzero + developer to set up a customizable testing environment with a minimal + hardware requirement. A developer can manually provision the HSM, set up + persistent storage for the cold wallet and HSM files, execute the .jar + and .sar files for development and testing. + + Note that right now an interactive shell for the `liveuser` user is + available for a debugging purpose, in both the `dev` and `release` images. + The default booting behavior is different between these two image flavors. Install related stuff: