From d3b956bc867d46721e2efae471154d22054bb75e Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 19:12:27 -0500 Subject: [PATCH 01/90] Add requirements.txt --- requirements.txt | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 requirements.txt diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 000000000..80cd1c5cd --- /dev/null +++ b/requirements.txt @@ -0,0 +1,4 @@ +jsonpickle +matplotlib +numpy +pexpect From 71bf2eacaa0ae13685105bd6c6d6ff05ae0d7a3f Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 19:13:08 -0500 Subject: [PATCH 02/90] Draft documentation for upcoming changes --- docs/contributing/contrib.md | 21 ++++++- docs/tutorial/tutorial.md | 117 ++++++++++++++++++----------------- 2 files changed, 78 insertions(+), 60 deletions(-) diff --git a/docs/contributing/contrib.md b/docs/contributing/contrib.md index e5f8ca82a..9f6477185 100644 --- a/docs/contributing/contrib.md +++ b/docs/contributing/contrib.md @@ -1,19 +1,36 @@ ## PaSh on Docker: A Pocket Guide -This is a pocket guide for running PaSh in a docker container. +This is a pocket guide for running PaSh in a Docker container. ### Loading image -The following will pull and launch the latest image (tag: `pash:latest`) using the name `pash-play` +#### From Docker Hub + +The following will pull the latest Docker image (tag: `pash:latest`), +and start a new container using the name `pash-play`. ```sh docker pull binpash/pash docker run --name pash-play -it pash ``` + This will use `pash-play` for the container (to be able to start/stop it later) and `-it` runs it interactively (subsuming the next command). To restart after you exit, run `docker start -i pash-play` (flag `-i` starts it interactively.) + +#### From source code + +If you want to use or develop PaSh for a specific commit, this will +build the latest Docker image. + +```sh +git clone git@github.com:binpash/pash.git +cd pash/scripts +docker build -t "pash:latest" . +``` + + ### Customizing image To be used for continuous integration and testing, this image has been configured to have _read-only_ access to the repo through a different user. diff --git a/docs/tutorial/tutorial.md b/docs/tutorial/tutorial.md index 7280863dc..698cc1624 100644 --- a/docs/tutorial/tutorial.md +++ b/docs/tutorial/tutorial.md @@ -49,92 +49,93 @@ The new parallel script has POSIX constructs added to explicitly guide paralleli ## Installation -#### Natively on Linux +Regardless of your target platform, PaSh requires the `PASH_TOP` +environment variable to point to a PaSh installation on disk. The +installation method impacts the value of this variable. -_Ubuntu, Fedora, Debian, Arch:_ -Use `up`: -```sh -curl up.binpa.sh | sh -``` +### Unix-likes -_Other distros_: -Use the system's package manager to install the following packages (some of which might already be installed in your system): +Download your preferred system package (all architectures). -``` -git libtool m4 curl automake pkg-config libffi-dev python python3 python3-pip wamerican-insane bc bsdmainutils python3-testresources python3-setuptools locales locales-all wget netcat-openbsd -``` +| Package | Install command | +|-----------------------------------|-------------------------------| +| [Red Hat (`pash.rpm`)](#link-tbd) | `yum localinstall ./pash.rpm` | +| [Debian (`pash.deb`)](#link-tbd) | `apt install ./pash.deb` | -Then clone the PaSh repository and run `setup-pash.sh` as follows: -```sh -git clone git@github.com:binpash/pash.git -./pash/scripts/setup-pash.sh -``` +Run `pa.sh` to verify installation of the following paths. -As noted at the end of `setup-pash.sh`, make sure you set `PASH_TOP` pointing to the absolute path of the directory `pa.sh` resides (you can optionally place that in your `PATH`). +| Path | Purpose | +|------------------|-----------------------------------------| +| `/usr/bin/pa.sh` | symlink to `/opt/pash/pa.sh` | +| `/usr/lib/pash` | source code (as seen in the repository) | +| `/var/log/pash` | package build logs | -_Installation Flags_: +If something goes wrong, please submit your build logs with a bug +report. -PaSh offers the following installation flags: -* `-p/--prepare` that installs all the required dependencies if your system is supported -* `-o/--opt-agg` that downloads g++-10 and builds an optimized version of some aggregators (not currently implemented in the main PaSh workflow) -To use these flags, see the following installation procedure: -```sh +### Source +[FPM]: https://fpm.readthedocs.io/en/latest/cli-reference.html#general-options + +Source deployments start with a clone. + +``` git clone git@github.com:binpash/pash.git -cd pash/ && bash scripts/install.sh -p --opti-agg ``` -This will install all the dependencies for PaSh on your machine (sudo is used) and the g++-10 toolchain, but also build the respective aggregators. - -#### Docker +You can either use the source code to create a Docker image, or deploy +PaSh on the current host. For Docker instructions, see the +[pash-on-docker +guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide). -PaSh on Docker is useful when native installation is not an option -- for example, to allow development on Windows or OS X. -First [make sure you install Docker](https://docs.docker.com/get-docker/) on your machine. -Note that, depending on the configuration, PaSh on Docker may or may not be able to exploit all available hardware resources. -There are two main options for installing PaSh via Docker. +For native deployment, install [FPM][] to prepare native packages. For +a Debian-based distribution (as an example), you can deploy PaSh on +the current host for some label `X`. -_Major Releases:_ -The easiest is to `pull` the docker image [from Docker Hub](https://hub.docker.com/r/binpash/pash): -```sh -docker pull binpash/pash ``` -We refresh this image (as well as other images) on every major release. +./pash/scripts/package/build-packages.sh X +sudo apt install ./pash/scripts/package/output/pash-X.deb +``` -[//]: # "TODO(@nvasilakis, @dkarnikis): Need to automate this per release." +RHEL is no different. -_Latest Commit:_ -To built the latest Docker container from scratch, run `docker build` in scripts: -```sh -git clone git@github.com:binpash/pash.git -cd pash/scripts -docker build -t "pash:latest" . ``` -This will build a fresh Docker image using the latest commit---recommended for development. - -In all the above cases, launching the container is done via: -```sh -docker run --name pash-play -it pash +./pash/scripts/package/build-packages.sh X +sudo yum localinstall ./pash/scripts/package/output/pash-X.rpm ``` -PaSh can be found in the container's `/opt/pash` directory, so run `cd pash; git pull` to fetch the latest updates. -More information in the [pash-on-docker guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide). -#### Windows using WSL +[`build-packages.sh`](../../scripts/package/build-packages.sh) defines +supported systems and distribution-specific modifications. + + +### Docker +[Docker]: https://docs.docker.com/get-docker/ + +In the context of a [Docker][] container, `PASH_TOP` is `/opt/pash`. The +same directory holds is a copy of the Git repository. Use `git -C +/opt/pash pull` to update the code in a container. PaSh may not be +able to exploit all available hardware resources when used in a +container. + +For instructions, see the [pash-on-docker +guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide). + +[//]: # "TODO(@nvasilakis, @dkarnikis): Need to automate this per release." + + +### Windows using WSL To run PaSh on windows without Docker, install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10). A short tutorial is included in the [contributing](../contributing/contrib.md) guide. -## Running Scripts -All scripts in this guide assume that `$PASH_TOP` is set to the top directory of the PaSh codebase (e.g., `/opt/pash` on docker) -**To run scripts in this section of the tutorial, make sure you are in the `intro` directory of the `evaluation`:** -```sh -cd $PASH_TOP/evaluation/intro -``` +## Running Scripts + +To use this section, run `cd $PASH_TOP/evaluation/intro` first. -> In the following examples, you can avoid including `$PASH_TOP` before `pa.sh` by adding `PASH_TOP` in your `PATH`, which amounts to adding an `export PATH=$PATH:$PASH_TOP` in your shell configuration file. #### Intro: Hello, Parallel World! From 5911c01dcf0496b1df548d1690dec8e0abc3b6f8 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 19:15:45 -0500 Subject: [PATCH 03/90] Add `scripts/distro.sh` I see some distribution inference rules that seemed to pop up several times. The script seems to be useful in several places, but I have not deduplicated the code yet. That would involve changes that are out of scope. --- scripts/distro.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 scripts/distro.sh diff --git a/scripts/distro.sh b/scripts/distro.sh new file mode 100644 index 000000000..fb3741765 --- /dev/null +++ b/scripts/distro.sh @@ -0,0 +1,13 @@ +#! /usr/bin/env bash +# Infer Unix-like distribution +# +# TODO: This logic appears verbatim in a few places. +# Deduplicate by having other scripts call this one. + +if type lsb_release >/dev/null 2>&1 ; then + distro=$(lsb_release -i -s) +elif [ -e /etc/os-release ] ; then + distro=$(awk -F= '$1 == "ID" {print $2}' /etc/os-release) +fi + +printf '%s\n' "$distro" | LC_ALL=C tr '[:upper:]' '[:lower:]' From b91351bf3041e3e7ee38ccec439edf6c248e7116 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 19:27:21 -0500 Subject: [PATCH 04/90] Help user with setup at entry point Deployment would be easier if `pa.sh` helped users with `PASH_TOP` and a default `~/.pash_init`. I saw scripts to modify the user's rcfiles (which is not ideal, imo). With process substitution, the process for setting `PASH_TOP` is a little nicer. $ cd "$(find ~ -type f pa.sh | xargs -n 1 dirname | head -n 1)" $ ./pa.sh # Add this to your shell configuration. export PASH_TOP='/home/sage/prj/pash' export PATH="$PASH_TOP:$PATH" # If your shell supports process substitution (bash, zsh, etc.) # run `. <(./pa.sh)` to use PaSh in this shell. The key benefit for deployment is that you can simply instruct a user to actually start using PaSh and follow its instructions, as opposed to adding installation steps. Since this is Bash, I use `return` to prevent accidental execution of further instructions when `pa.sh` is sourced. --- pa.sh | 27 ++++++++++++++++++++++++++- scripts/pash-init.sh | 12 ++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 scripts/pash-init.sh diff --git a/pa.sh b/pa.sh index ba619f5e3..cfe3fa884 100755 --- a/pa.sh +++ b/pa.sh @@ -1,6 +1,31 @@ #!/usr/bin/env bash -export PASH_TOP=${PASH_TOP:-${BASH_SOURCE%/*}} +# Help out with shell configuration +if [ -z "$PASH_TOP" ]; then + export PASH_TOP="$(readlink -f $(dirname "${BASH_SOURCE[0]}"))" + export PATH="$PASH_TOP:$PATH" + printf '# Add this to your shell configuration.\n\n' + printf " export PASH_TOP='%s'\n" "$PASH_TOP" + printf ' export PATH="$PASH_TOP:$PATH"\n\n' + printf '# If your shell supports process substitution (bash, zsh, etc.)\n' + printf '# run `. <(%s)` to use PaSh in this shell.\n' "$0" + exit 0 +fi + +# Halt if script was sourced. The above is only a setup aid that +# should not interfere with the script's normal operation. +return 2>/dev/null + + +rcfile=~/.pash-init +if [ ! -f "$rcfile" ]; then + printf '# Include system-level rules\n' > "$rcfile" + printf ". '%s/scripts/pash-init.sh'\n" "$PASH_TOP" >> "$rcfile" + chmod +x "$rcfile" + printf 'Created %s\n' "$rcfile" +fi + + export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib/" # point to the local downloaded folders export PYTHONPATH=${PASH_TOP}/python_pkgs/ diff --git a/scripts/pash-init.sh b/scripts/pash-init.sh new file mode 100644 index 000000000..711ceb870 --- /dev/null +++ b/scripts/pash-init.sh @@ -0,0 +1,12 @@ +#! /usr/bin/env bash +# Source this in any new ~/pash-init.sh + +export distro="$(/usr/local/pash/scripts/distro.sh)" + +# Adapt to Docker +if [ -f /.dockerenv ]; then + export PASH_TOP=/opt/pash + export LC_ALL='en_US.UTF-8' + export LANG='en_US.UTF-8' + export LANGUAGE='en_US.UTF-8' +fi From 3a8d9136ddc9295a97b45b7efb54b82a109fc242 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 22:36:58 -0500 Subject: [PATCH 05/90] Draft package build system --- scripts/package/.gitignore | 2 + scripts/package/after-install.sh | 44 +++++++++++++++ scripts/package/repl.sh | 31 ++++++++++ scripts/package/tools/build | 97 ++++++++++++++++++++++++++++++++ scripts/package/tools/help | 20 +++++++ scripts/package/tools/start | 13 +++++ 6 files changed, 207 insertions(+) create mode 100644 scripts/package/.gitignore create mode 100755 scripts/package/after-install.sh create mode 100755 scripts/package/repl.sh create mode 100755 scripts/package/tools/build create mode 100644 scripts/package/tools/help create mode 100755 scripts/package/tools/start diff --git a/scripts/package/.gitignore b/scripts/package/.gitignore new file mode 100644 index 000000000..2626ba200 --- /dev/null +++ b/scripts/package/.gitignore @@ -0,0 +1,2 @@ +fpm +output diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh new file mode 100755 index 000000000..5a354faa9 --- /dev/null +++ b/scripts/package/after-install.sh @@ -0,0 +1,44 @@ +#! /usr/bin/env bash +# Called by system package manager to request on-site setup on host. + +set -euo pipefail + +pashd=/usr/lib/pash +logd=/var/log/pash + +ln -s "$pashd/pa.sh" /usr/bin/pa.sh + +say() { + printf 'pash: after-install: ' + printf $@ + printf '\n' +} + +say_task() { + say '%s (log: %s/%s)' "$1" "$logd" "$2" +} + +mkdir -p "$logd" +cd "$pashd/compiler/parser" + +case $(bash "$pashd/scripts/distro.sh") in + freebsd*) + cp Makefile Makefile.backup + gsed -i 's/ make/ gmake/g' Makefile + + say_task "compile libdash (FreeBSD)" libdash.log + gmake libdash &> "$logd/libdash.log" + + say_task "compile runtime (FreeBSD)" runtime.log + gmake -C ../../runtime/ &> "$logd/runtime.log" + ;; + *) + say_task "compile libdash" libdash.log + make libdash &>"$logd/libdash.log" + + say_task "compile runtime" runtime.log + make -C ../../runtime/ &>"$logd/runtime.log" + ;; +esac + +wait diff --git a/scripts/package/repl.sh b/scripts/package/repl.sh new file mode 100755 index 000000000..e005360a7 --- /dev/null +++ b/scripts/package/repl.sh @@ -0,0 +1,31 @@ +#! /usr/bin/env bash +# Launch REPL for building PaSh system packages. + +# We need FPM's source code to build a key Docker image. +if [ ! -d fpm ]; then + git clone --depth 1 git@github.com:jordansissel/fpm.git +fi + +# Make a Docker image that knows about building packages. +if ! docker image inspect fpm 2>&1 >/dev/null; then + make -C fpm docker-release-everything +fi + +cd "$(dirname "${BASH_SOURCE[0]}")" # Help limit filesystem writes to this directory. +mkdir -p output + +main() { + docker run \ + --entrypoint /bin/bash \ + --interactive \ + --tty \ + --rm \ + --user "$(id -u):$(id -g)" \ + --volume "$PASH_TOP/scripts/package/output:/out" \ + --volume "$PASH_TOP/scripts/package/tools:/tools" \ + --volume "$PASH_TOP:/src" \ + fpm --rcfile /tools/start -i +} + +# Only enter if script was not sourced +(return 2>/dev/null) || main "$@" diff --git a/scripts/package/tools/build b/scripts/package/tools/build new file mode 100755 index 000000000..48de2a76b --- /dev/null +++ b/scripts/package/tools/build @@ -0,0 +1,97 @@ +#! /usr/bin/env bash + +# Builds a specific package in the context of FPM's Docker container. +# https://fpm.readthedocs.io/ + +# Enforce preconditions +set -ex +version="$1" +install_dir=/usr/lib/pash +set +x -u; : "$version" +shift # for remaining $@ + + +print_dotfpm() { + output_format="$1" + output_file="$2" + + # I/O rules, with Docker container paths + flag --chdir /src + flag --input-type dir + flag --output-type "$output_format" + flag --package "$output_file" + flag --log info + flag --verbose + + # Discovery information + flag --name pash + flag --description "Parallelize POSIX shell scripts" + flag --license mit + flag --architecture all + flag --url 'https://github.com/binpash/pash' + flag --version "$version" + flag --epoch "$(date +%s)" + + # System-level dependencies + flag --depends automake + flag --depends bash + flag --depends bc + flag --depends bsdmainutils + flag --depends curl + flag --depends git + flag --depends g++-10 + flag --depends libffi-dev + flag --depends libtool + flag --depends locales + flag --depends locales-all + flag --depends m4 + flag --depends netcat-openbsd + flag --depends pkg-config + flag --depends python # Redundant? + flag --depends python3 + flag --depends python3-pip + flag --depends python3-setuptools + flag --depends python3-testresources + flag --depends wamerican-insane + flag --depends wget + + # Handle requirements.txt with `python3 -m pip` + flag --python-bin /bin/python3 + flag --python-obey-requirements-txt + flag --python-internal-pip + + # Tell PaSh to set itself up on a target system. + flag --after-install "$PASH_TOP/scripts/package/after-install.sh" + + # Map contents from here to target system. + for f in LICENSE pa.sh annotations compiler docs evaluation runtime scripts; do + printf "%s=%s\n" "$f" "$install_dir/$f" + done +} + +flag() { + if [ "$#" -gt 1 ]; then + printf "%s '%s'\n" "$1" "$2" + else + printf "%s\n" "$1" + fi +} + +trap 'rm -f .fpm' EXIT + +for output_format in "$@"; do + output_file="/out/pash-${version}" + + # Map formats to extensions, when they differ. + case "$output_format" in + *) + output_file="${output_file}.${output_format}" + ;; + esac + + print_dotfpm "$output_format" "$output_file" > .fpm + set +e + /fpm/bin/fpm && chmod 440 "$output_file" + set -e + printf '\n' +done diff --git a/scripts/package/tools/help b/scripts/package/tools/help new file mode 100644 index 000000000..2a224e1c4 --- /dev/null +++ b/scripts/package/tools/help @@ -0,0 +1,20 @@ +This is a shell for FPM's Docker container. + + https://fpm.readthedocs.io/ + +/src - Mounted to host's $PASH_TOP +/tools - Mounted to host's $PASH_TOP/scripts/package/tools +/out - Mounted to host's $PASH_TOP/scripts/package/output + +Write a new package with mode 440 using this command. + + build VERSION [OUTPUT_FORMAT ...] + +With regards to FPM's CLI: + + - VERSION is the value of --version. + - OUTPUT_FORMAT is some value of -t/--output-type + +Ownership of result goes to host user and group, not the container's. + +Run `build_help` to repeat this message. diff --git a/scripts/package/tools/start b/scripts/package/tools/start new file mode 100755 index 000000000..693f565df --- /dev/null +++ b/scripts/package/tools/start @@ -0,0 +1,13 @@ +#! /usr/bin/env bash +. ~/.bashrc + +build_help() { + cat /tools/help +} + +export PASH_TOP="/src" +export PS1='pkg> ' +export PATH="/tools:$PATH" +export -f build_help + +build_help From 07b3676617c1bba24af176793833b5d0b782629e Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 22:46:50 -0500 Subject: [PATCH 06/90] Use .pash_init, not .pash-init --- pa.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pa.sh b/pa.sh index cfe3fa884..f7110cf83 100755 --- a/pa.sh +++ b/pa.sh @@ -17,7 +17,7 @@ fi return 2>/dev/null -rcfile=~/.pash-init +rcfile=~/.pash_init if [ ! -f "$rcfile" ]; then printf '# Include system-level rules\n' > "$rcfile" printf ". '%s/scripts/pash-init.sh'\n" "$PASH_TOP" >> "$rcfile" From cec354748c56e7ba185e3346dc265845d5ae3f86 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 22:50:10 -0500 Subject: [PATCH 07/90] Allow shell options for container front end --- scripts/package/{repl.sh => shell.sh} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename scripts/package/{repl.sh => shell.sh} (95%) diff --git a/scripts/package/repl.sh b/scripts/package/shell.sh similarity index 95% rename from scripts/package/repl.sh rename to scripts/package/shell.sh index e005360a7..325056bae 100755 --- a/scripts/package/repl.sh +++ b/scripts/package/shell.sh @@ -24,7 +24,7 @@ main() { --volume "$PASH_TOP/scripts/package/output:/out" \ --volume "$PASH_TOP/scripts/package/tools:/tools" \ --volume "$PASH_TOP:/src" \ - fpm --rcfile /tools/start -i + fpm --rcfile /tools/start "$@" } # Only enter if script was not sourced From 102b61642da8eabc568c4ca9edc9d97e11c6ad45 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 23:15:54 -0500 Subject: [PATCH 08/90] Remove some after-install.sh errors regarding printf and ln --- scripts/package/after-install.sh | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh index 5a354faa9..6f05c8d6d 100755 --- a/scripts/package/after-install.sh +++ b/scripts/package/after-install.sh @@ -6,20 +6,13 @@ set -euo pipefail pashd=/usr/lib/pash logd=/var/log/pash -ln -s "$pashd/pa.sh" /usr/bin/pa.sh - -say() { - printf 'pash: after-install: ' - printf $@ - printf '\n' -} - say_task() { - say '%s (log: %s/%s)' "$1" "$logd" "$2" + printf 'pash: %s (log: %s/%s)\n' "$1" "$logd" "$2" } -mkdir -p "$logd" +mkdir -vp "$logd" cd "$pashd/compiler/parser" +ln -fs "$pashd/pa.sh" /usr/bin/pa.sh case $(bash "$pashd/scripts/distro.sh") in freebsd*) From 30c3e44d96935386db8c8d911a881f3b6bbe7a01 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 23:16:33 -0500 Subject: [PATCH 09/90] Exclude '*.git*' and clobber built packages --- scripts/package/tools/build | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 48de2a76b..26e8559ef 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -22,6 +22,7 @@ print_dotfpm() { flag --package "$output_file" flag --log info flag --verbose + flag --force # Discovery information flag --name pash @@ -64,9 +65,8 @@ print_dotfpm() { flag --after-install "$PASH_TOP/scripts/package/after-install.sh" # Map contents from here to target system. - for f in LICENSE pa.sh annotations compiler docs evaluation runtime scripts; do - printf "%s=%s\n" "$f" "$install_dir/$f" - done + flag --exclude '*.git*' + printf ".=%s\n" "$install_dir" } flag() { From 170bd2df9a8a355408a5482a1a7a28935566eba9 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 23:16:57 -0500 Subject: [PATCH 10/90] Mark distro.sh executable --- scripts/distro.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 scripts/distro.sh diff --git a/scripts/distro.sh b/scripts/distro.sh old mode 100644 new mode 100755 From 47a4a94ca8fa23c2ba009e8984b13cedede6b43a Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 23:57:17 -0500 Subject: [PATCH 11/90] Improve config preview --- pa.sh | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/pa.sh b/pa.sh index f7110cf83..89c38c7af 100755 --- a/pa.sh +++ b/pa.sh @@ -2,13 +2,20 @@ # Help out with shell configuration if [ -z "$PASH_TOP" ]; then - export PASH_TOP="$(readlink -f $(dirname "${BASH_SOURCE[0]}"))" + me="$(readlink -f "${BASH_SOURCE[0]}")" + export PASH_TOP="$(dirname "$me")" export PATH="$PASH_TOP:$PATH" printf '# Add this to your shell configuration.\n\n' printf " export PASH_TOP='%s'\n" "$PASH_TOP" printf ' export PATH="$PASH_TOP:$PATH"\n\n' - printf '# If your shell supports process substitution (bash, zsh, etc.)\n' - printf '# run `. <(%s)` to use PaSh in this shell.\n' "$0" + printf '# If your shell supports process substitution (bash, zsh, etc.),\n' + + preview="$(command -v pa.sh)" + if [ $? -eq 0 ]; then + preview='pa.sh' + fi + + printf '# then run `. <(%s)` to use this config in your current shell.\n' "$preview" exit 0 fi From 24abb5de4af53a628b5e18c9bdc6eb69075ac5cc Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 23:57:40 -0500 Subject: [PATCH 12/90] Rephrase Docker volumes --- scripts/package/shell.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/package/shell.sh b/scripts/package/shell.sh index 325056bae..35fd41e92 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/shell.sh @@ -11,8 +11,10 @@ if ! docker image inspect fpm 2>&1 >/dev/null; then make -C fpm docker-release-everything fi -cd "$(dirname "${BASH_SOURCE[0]}")" # Help limit filesystem writes to this directory. -mkdir -p output +here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" +PASH_TOP=$(readlink -f "$here/../..") +mkdir -p "$here/output" +echo $here main() { docker run \ @@ -21,8 +23,8 @@ main() { --tty \ --rm \ --user "$(id -u):$(id -g)" \ - --volume "$PASH_TOP/scripts/package/output:/out" \ - --volume "$PASH_TOP/scripts/package/tools:/tools" \ + --volume "$here/output:/out" \ + --volume "$here/tools:/tools" \ --volume "$PASH_TOP:/src" \ fpm --rcfile /tools/start "$@" } From 39ab2f8775fb9fd9b1b700479d77e0cacb7723d6 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Tue, 1 Mar 2022 23:57:52 -0500 Subject: [PATCH 13/90] Exclude package build output from future packages --- scripts/package/tools/build | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 26e8559ef..020ffbc29 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -66,6 +66,7 @@ print_dotfpm() { # Map contents from here to target system. flag --exclude '*.git*' + flag --exclude '*package/output*' printf ".=%s\n" "$install_dir" } From 91851e4d26472e315fe2d33f40e9d71ba792d695 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 00:07:54 -0500 Subject: [PATCH 14/90] Use PASH_TOP in distro.sh --- scripts/pash-init.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/pash-init.sh b/scripts/pash-init.sh index 711ceb870..21f0e7fa3 100644 --- a/scripts/pash-init.sh +++ b/scripts/pash-init.sh @@ -1,7 +1,8 @@ #! /usr/bin/env bash # Source this in any new ~/pash-init.sh -export distro="$(/usr/local/pash/scripts/distro.sh)" +set -u +export distro="$("$PASH_TOP/scripts/distro.sh")" # Adapt to Docker if [ -f /.dockerenv ]; then From 7ff2ae0bf2cd9a621092d613c1a98f0481a84d45 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 00:08:05 -0500 Subject: [PATCH 15/90] Exclude python cache files in package builds --- scripts/package/tools/build | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 020ffbc29..adb42e014 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -66,6 +66,8 @@ print_dotfpm() { # Map contents from here to target system. flag --exclude '*.git*' + flag --exclude '*.pyc' + flag --exclude '__pycache__' flag --exclude '*package/output*' printf ".=%s\n" "$install_dir" } From 73f04a2a06eed8961419463b3a6a8bb326192a63 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 00:09:01 -0500 Subject: [PATCH 16/90] Remove --epoch due to incorrect Debian package specifier --- scripts/package/tools/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index adb42e014..7595f8bf8 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -31,7 +31,7 @@ print_dotfpm() { flag --architecture all flag --url 'https://github.com/binpash/pash' flag --version "$version" - flag --epoch "$(date +%s)" + # System-level dependencies flag --depends automake From 58be73e3b0be8a7adb803e3a2750da2087e46265 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 12:39:27 -0500 Subject: [PATCH 17/90] Stop trying to be clever with previews --- pa.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/pa.sh b/pa.sh index 89c38c7af..60dbd4145 100755 --- a/pa.sh +++ b/pa.sh @@ -9,13 +9,7 @@ if [ -z "$PASH_TOP" ]; then printf " export PASH_TOP='%s'\n" "$PASH_TOP" printf ' export PATH="$PASH_TOP:$PATH"\n\n' printf '# If your shell supports process substitution (bash, zsh, etc.),\n' - - preview="$(command -v pa.sh)" - if [ $? -eq 0 ]; then - preview='pa.sh' - fi - - printf '# then run `. <(%s)` to use this config in your current shell.\n' "$preview" + printf '# then run `. <(%s)` to use this config in your current shell.\n' "$0" exit 0 fi From 0892ad8ffb145f4ee6287c9ba0c2ffd239dc0c7d Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 12:40:25 -0500 Subject: [PATCH 18/90] Remove unhelpful compilation code Some code only calls `make` with no target specified. No `autogen.sh`, `configure`, etc. Maybe that's fine for the runtime, but it's not enough for the `libdash` derivative. I wonder if there's a different script that I haven't seen yet that does this better. --- scripts/package/after-install.sh | 22 ---------------------- 1 file changed, 22 deletions(-) diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh index 6f05c8d6d..28004a0a1 100755 --- a/scripts/package/after-install.sh +++ b/scripts/package/after-install.sh @@ -13,25 +13,3 @@ say_task() { mkdir -vp "$logd" cd "$pashd/compiler/parser" ln -fs "$pashd/pa.sh" /usr/bin/pa.sh - -case $(bash "$pashd/scripts/distro.sh") in - freebsd*) - cp Makefile Makefile.backup - gsed -i 's/ make/ gmake/g' Makefile - - say_task "compile libdash (FreeBSD)" libdash.log - gmake libdash &> "$logd/libdash.log" - - say_task "compile runtime (FreeBSD)" runtime.log - gmake -C ../../runtime/ &> "$logd/runtime.log" - ;; - *) - say_task "compile libdash" libdash.log - make libdash &>"$logd/libdash.log" - - say_task "compile runtime" runtime.log - make -C ../../runtime/ &>"$logd/runtime.log" - ;; -esac - -wait From 3191b89618f3b85c95727a208e2c8c3bf8c47798 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 21:07:50 -0500 Subject: [PATCH 19/90] Compile libdash --- scripts/package/after-install.sh | 5 +++++ scripts/package/tools/build | 1 + 2 files changed, 6 insertions(+) diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh index 28004a0a1..c66880c64 100755 --- a/scripts/package/after-install.sh +++ b/scripts/package/after-install.sh @@ -5,6 +5,7 @@ set -euo pipefail pashd=/usr/lib/pash logd=/var/log/pash +export PASH_TOP="$pashd" say_task() { printf 'pash: %s (log: %s/%s)\n' "$1" "$logd" "$2" @@ -13,3 +14,7 @@ say_task() { mkdir -vp "$logd" cd "$pashd/compiler/parser" ln -fs "$pashd/pa.sh" /usr/bin/pa.sh + +cd "$pashd/compiler/parser" +git clone https://github.com/angelhof/libdash/ +make libdash diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 7595f8bf8..b0b71b148 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -68,6 +68,7 @@ print_dotfpm() { flag --exclude '*.git*' flag --exclude '*.pyc' flag --exclude '__pycache__' + flag --exclude '*compiler/parser/libdash*' flag --exclude '*package/output*' printf ".=%s\n" "$install_dir" } From 70de48c1aff47ec52de2d54bf13fcb526867ca24 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 21:48:22 -0500 Subject: [PATCH 20/90] Draft remainder of C compilation --- scripts/package/after-install.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh index c66880c64..784206707 100755 --- a/scripts/package/after-install.sh +++ b/scripts/package/after-install.sh @@ -15,6 +15,10 @@ mkdir -vp "$logd" cd "$pashd/compiler/parser" ln -fs "$pashd/pa.sh" /usr/bin/pa.sh -cd "$pashd/compiler/parser" +rm -rf libdash git clone https://github.com/angelhof/libdash/ make libdash + +cd "$pashd/runtime" +make clean +make From 848c5b0c34653a93e1702ccead335395b3f8a28d Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 23:04:16 -0500 Subject: [PATCH 21/90] Remove extra echo --- scripts/package/shell.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/package/shell.sh b/scripts/package/shell.sh index 35fd41e92..34e774d9f 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/shell.sh @@ -14,7 +14,6 @@ fi here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" PASH_TOP=$(readlink -f "$here/../..") mkdir -p "$here/output" -echo $here main() { docker run \ From 87d18350605c626cb598933d3690654ef13b3bc5 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 23:58:08 -0500 Subject: [PATCH 22/90] Add Python install step --- .gitignore | 1 + scripts/install-python.sh | 6 ++++++ scripts/package/after-install.sh | 10 ++++++++-- 3 files changed, 15 insertions(+), 2 deletions(-) create mode 100755 scripts/install-python.sh diff --git a/.gitignore b/.gitignore index e3c9c8857..ec46da1ff 100644 --- a/.gitignore +++ b/.gitignore @@ -31,6 +31,7 @@ revision.tex node_modules python_pkgs +python_pkgs_root # local directory w/ related work bib.to diff --git a/scripts/install-python.sh b/scripts/install-python.sh new file mode 100755 index 000000000..c75c6f951 --- /dev/null +++ b/scripts/install-python.sh @@ -0,0 +1,6 @@ +#! /usr/bin/env bash + +set -euo pipefail +PYTHON_PKG_DIR="$PASH_TOP/python_pkg_root" +python3 -m pip install --root "$PYTHON_PKG_DIR" --ignore-installed -r "$PASH_TOP/requirements.txt" +ln -frs "$(find "$PYTHON_PKG_DIR" -type d -name site-packages)" "$PASH_TOP/python_pkgs" diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh index 784206707..aafdb0116 100755 --- a/scripts/package/after-install.sh +++ b/scripts/package/after-install.sh @@ -12,13 +12,19 @@ say_task() { } mkdir -vp "$logd" + +say_task "python dependencies" "install-python.log" +"$pashd/scripts/install-python.sh" 2>&1 | tee "$logd/install-python.log" + cd "$pashd/compiler/parser" ln -fs "$pashd/pa.sh" /usr/bin/pa.sh +say_task "libdash" "libdash.log" rm -rf libdash git clone https://github.com/angelhof/libdash/ -make libdash +make libdash 2>&1 | tee "$logd/libdash.log" +say_task "runtime" "runtime.log" cd "$pashd/runtime" make clean -make +make 2>&1 | tee "$logd/runtime.log" From 16e27f016b726840e25904b394bc3d9e6f279edd Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 2 Mar 2022 23:58:46 -0500 Subject: [PATCH 23/90] Fix typo in .gitignore --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index ec46da1ff..ede1ae4d5 100644 --- a/.gitignore +++ b/.gitignore @@ -31,7 +31,7 @@ revision.tex node_modules python_pkgs -python_pkgs_root +python_pkg_root # local directory w/ related work bib.to From c79c77840825b75996ff2cc5f798ce36883ede32 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 3 Mar 2022 22:37:20 -0500 Subject: [PATCH 24/90] Add pacman support --- scripts/package/tools/build | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index b0b71b148..0542e0c84 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -56,6 +56,12 @@ print_dotfpm() { flag --depends wamerican-insane flag --depends wget + # [arch-ext]: FPM seems to think every tar binary supports --zstd + # at time of writing, so fall back to gzip on Arch. + # + # Update this with other [arch-ext] commented lines. + flag --pacman-compression gz + # Handle requirements.txt with `python3 -m pip` flag --python-bin /bin/python3 flag --python-obey-requirements-txt @@ -88,6 +94,11 @@ for output_format in "$@"; do # Map formats to extensions, when they differ. case "$output_format" in + pacman) + # [arch-ext]: Update this file extension when the other + # [arch-ext]-commented line changes. + output_file="${output_file}.tar.gz" + ;; *) output_file="${output_file}.${output_format}" ;; From 05ba2088704c3f94d81296680361994e63ae65ea Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 3 Mar 2022 22:41:41 -0500 Subject: [PATCH 25/90] Expand exclude rules --- scripts/package/tools/build | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 0542e0c84..7c1e11604 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -73,9 +73,11 @@ print_dotfpm() { # Map contents from here to target system. flag --exclude '*.git*' flag --exclude '*.pyc' - flag --exclude '__pycache__' + flag --exclude '*__pycache__*' flag --exclude '*compiler/parser/libdash*' + flag --exclude '*python_pkg*' flag --exclude '*package/output*' + flag --exclude '*package/fpm*' printf ".=%s\n" "$install_dir" } From c7930d1596c1e81c736c762fa4aafc440dec3ca3 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 13:27:44 -0500 Subject: [PATCH 26/90] Compute package file names differently Distribution-specifics were split between two spots in scripts/package/tools/build. Reason being the file name was computed separately from the flags. - Add functions for distribution specifics. - Default to 'now' as a version (helpful for quick iteration) --- scripts/package/tools/build | 138 +++++++++++++++++++++++------------- 1 file changed, 89 insertions(+), 49 deletions(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 7c1e11604..cda184e72 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -1,25 +1,22 @@ #! /usr/bin/env bash -# Builds a specific package in the context of FPM's Docker container. +# Build a specific package in the context of FPM's Docker container. # https://fpm.readthedocs.io/ -# Enforce preconditions -set -ex -version="$1" +set -eo pipefail +version="${1:-now}" install_dir=/usr/lib/pash -set +x -u; : "$version" -shift # for remaining $@ +if [ -n "$1" ]; then shift; fi +set -u print_dotfpm() { output_format="$1" - output_file="$2" - + # I/O rules, with Docker container paths flag --chdir /src flag --input-type dir flag --output-type "$output_format" - flag --package "$output_file" flag --log info flag --verbose flag --force @@ -32,35 +29,35 @@ print_dotfpm() { flag --url 'https://github.com/binpash/pash' flag --version "$version" - - # System-level dependencies + # Common system-level dependencies. Keep here only what every + # distribution can find in their respective package repositories. flag --depends automake flag --depends bash flag --depends bc flag --depends bsdmainutils flag --depends curl flag --depends git - flag --depends g++-10 - flag --depends libffi-dev flag --depends libtool - flag --depends locales - flag --depends locales-all flag --depends m4 - flag --depends netcat-openbsd flag --depends pkg-config - flag --depends python # Redundant? flag --depends python3 - flag --depends python3-pip - flag --depends python3-setuptools - flag --depends python3-testresources - flag --depends wamerican-insane flag --depends wget - # [arch-ext]: FPM seems to think every tar binary supports --zstd - # at time of writing, so fall back to gzip on Arch. - # - # Update this with other [arch-ext] commented lines. - flag --pacman-compression gz + # Distribution specifics + case "$output_format" in + pacman) + print_dotfpm_pacman + ;; + deb) + print_dotfpm_deb + ;; + rpm) + print_dotfpm_rpm + ;; + *) + print_dotfpm_other + ;; + esac # Handle requirements.txt with `python3 -m pip` flag --python-bin /bin/python3 @@ -71,6 +68,10 @@ print_dotfpm() { flag --after-install "$PASH_TOP/scripts/package/after-install.sh" # Map contents from here to target system. + # + # TODO: Consider include-only approach. This will pick up any + # junk not expressly excluded. Maybe couple to Git's tracked + # files. flag --exclude '*.git*' flag --exclude '*.pyc' flag --exclude '*__pycache__*' @@ -78,37 +79,76 @@ print_dotfpm() { flag --exclude '*python_pkg*' flag --exclude '*package/output*' flag --exclude '*package/fpm*' + printf ".=%s\n" "$install_dir" } + +# +# These functions add distribution-specific flags. +# + +print_dotfpm_pacman() { + # FPM seems to think every tar binary supports --zstd + # at time of writing, so fall back to gzip on Arch. + print_dotfpm_output_path .tar.gz + flag --pacman-compression gz + flag --depends openbsd-netcat +} + + +print_dotfpm_rpm() { + print_dotfpm_output_path .rpm +} + + +print_dotfpm_deb() { + print_dotfpm_output_path .deb + + flag --depends libffi-dev + flag --depends locales + flag --depends locales-all + flag --depends netcat-openbsd + flag --depends python3-pip + flag --depends python3-setuptools + flag --depends python3-testresources + flag --depends wamerican-insane +} + + +print_dotfpm_other() { + print_dotfpm_output_path "$output_format" +} + + +# +# The rest defines how the above works and actually does something. +# + +print_dotfpm_output_path() { + output_file="${output_file_prefix}${1}" + flag --package "$output_file" +} + flag() { if [ "$#" -gt 1 ]; then printf "%s '%s'\n" "$1" "$2" else printf "%s\n" "$1" - fi + fi } -trap 'rm -f .fpm' EXIT - -for output_format in "$@"; do - output_file="/out/pash-${version}" - - # Map formats to extensions, when they differ. - case "$output_format" in - pacman) - # [arch-ext]: Update this file extension when the other - # [arch-ext]-commented line changes. - output_file="${output_file}.tar.gz" - ;; - *) - output_file="${output_file}.${output_format}" - ;; - esac +main() { + trap 'rm -f .fpm' EXIT + + for output_format in "$@"; do + output_file_prefix="/out/pash-${version}" + print_dotfpm "$output_format" > .fpm + set +e + /fpm/bin/fpm && chmod 440 "$output_file" + set -e + printf '\n' + done +} - print_dotfpm "$output_format" "$output_file" > .fpm - set +e - /fpm/bin/fpm && chmod 440 "$output_file" - set -e - printf '\n' -done +main "$@" From 41b46339bb5c902802e6833fdc5b29bccf5b8b3d Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 13:59:02 -0500 Subject: [PATCH 27/90] Clone fpm to the same place --- scripts/package/shell.sh | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/package/shell.sh b/scripts/package/shell.sh index 34e774d9f..e4b29ce4c 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/shell.sh @@ -1,9 +1,12 @@ #! /usr/bin/env bash # Launch REPL for building PaSh system packages. +here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" +PASH_TOP=$(readlink -f "$here/../..") + # We need FPM's source code to build a key Docker image. -if [ ! -d fpm ]; then - git clone --depth 1 git@github.com:jordansissel/fpm.git +if [ ! -d "$here/fpm" ]; then + git clone --depth 1 git@github.com:jordansissel/fpm.git "$here/fpm" fi # Make a Docker image that knows about building packages. @@ -11,8 +14,6 @@ if ! docker image inspect fpm 2>&1 >/dev/null; then make -C fpm docker-release-everything fi -here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" -PASH_TOP=$(readlink -f "$here/../..") mkdir -p "$here/output" main() { From 9137a175d8181dfd8168e08c426ce83ad5c97c48 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 15:13:34 -0500 Subject: [PATCH 28/90] Use ensurepip in post-install --- scripts/package/after-install.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh index aafdb0116..cd7f414dc 100755 --- a/scripts/package/after-install.sh +++ b/scripts/package/after-install.sh @@ -14,6 +14,11 @@ say_task() { mkdir -vp "$logd" say_task "python dependencies" "install-python.log" + +if ! command -v pip 2>&1 >/dev/null; then + /bin/python3 -m ensurepip --upgrade +fi + "$pashd/scripts/install-python.sh" 2>&1 | tee "$logd/install-python.log" cd "$pashd/compiler/parser" From f322c414578bad13a4ff862107d59d4e9b5c61db Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 15:13:47 -0500 Subject: [PATCH 29/90] Warn against system package manager use in build step --- scripts/package/tools/build | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index cda184e72..fe5b9bab7 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -2,6 +2,8 @@ # Build a specific package in the context of FPM's Docker container. # https://fpm.readthedocs.io/ +# +# Do not use any system package manager here. set -eo pipefail version="${1:-now}" From e19bc2bde8e88be5b9716c0851b6728919d594d1 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 15:14:14 -0500 Subject: [PATCH 30/90] Add FreeBSD and Python packages --- scripts/package/tools/build | 139 ++++++++++++++++++++++++++++-------- 1 file changed, 108 insertions(+), 31 deletions(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index fe5b9bab7..6b0647440 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -31,49 +31,36 @@ print_dotfpm() { flag --url 'https://github.com/binpash/pash' flag --version "$version" - # Common system-level dependencies. Keep here only what every - # distribution can find in their respective package repositories. - flag --depends automake - flag --depends bash - flag --depends bc - flag --depends bsdmainutils - flag --depends curl - flag --depends git - flag --depends libtool - flag --depends m4 - flag --depends pkg-config - flag --depends python3 - flag --depends wget - - # Distribution specifics + # Target information case "$output_format" in + deb) + print_dotfpm_deb + ;; + freebsd) + print_dotfpm_freebsd + ;; pacman) print_dotfpm_pacman ;; - deb) - print_dotfpm_deb + python) + print_dotfpm_python ;; rpm) print_dotfpm_rpm ;; *) - print_dotfpm_other + printf "Unsupported format: '%s'" "output_format" 1>&2 + exit 1 ;; esac - # Handle requirements.txt with `python3 -m pip` - flag --python-bin /bin/python3 - flag --python-obey-requirements-txt - flag --python-internal-pip - # Tell PaSh to set itself up on a target system. flag --after-install "$PASH_TOP/scripts/package/after-install.sh" # Map contents from here to target system. # # TODO: Consider include-only approach. This will pick up any - # junk not expressly excluded. Maybe couple to Git's tracked - # files. + # junk not expressly excluded. Maybe bind to Git's tracked files? flag --exclude '*.git*' flag --exclude '*.pyc' flag --exclude '*__pycache__*' @@ -87,7 +74,7 @@ print_dotfpm() { # -# These functions add distribution-specific flags. +# Target-specific flags # print_dotfpm_pacman() { @@ -95,43 +82,133 @@ print_dotfpm_pacman() { # at time of writing, so fall back to gzip on Arch. print_dotfpm_output_path .tar.gz flag --pacman-compression gz + + flag --depends autoconf + flag --depends automake + flag --depends bash + flag --depends bc + flag --depends curl + flag --depends curl + flag --depends git + flag --depends inetutils + flag --depends libffi + flag --depends libtool + flag --depends libtool + flag --depends m4 + flag --depends make flag --depends openbsd-netcat + flag --depends pkg-config + flag --depends python + flag --depends python-pip + flag --depends sudo + flag --depends wget } print_dotfpm_rpm() { print_dotfpm_output_path .rpm + + flag --depends autoconf + flag --depends automake + flag --depends bash + flag --depends bc + flag --depends curl + flag --depends diffutils + flag --depends gcc + flag --depends gcc-c++ + flag --depends git + flag --depends glibc-langpack-en + flag --depends hostname + flag --depends libjpeg-devel + flag --depends libtool + flag --depends libtool + flag --depends m4 + flag --depends make + flag --depends nc + flag --depends pip + flag --depends pkg-config + flag --depends procps + flag --depends python + flag --depends python-devel + flag --depends python3 + flag --depends python3-pip + flag --depends python3-setuptools + flag --depends python3-testresources + flag --depends wget + flag --depends zlib-devel } print_dotfpm_deb() { print_dotfpm_output_path .deb + flag --depends autoconf + flag --depends automake + flag --depends bash + flag --depends bc + flag --depends bsdmainutils + flag --depends curl + flag --depends git flag --depends libffi-dev + flag --depends libtool flag --depends locales flag --depends locales-all + flag --depends m4 flag --depends netcat-openbsd + flag --depends pkg-config flag --depends python3-pip flag --depends python3-setuptools flag --depends python3-testresources flag --depends wamerican-insane + flag --depends wget +} + + +print_dotfpm_python() { + # TODO: Is this extension correct? + print_dotfpm_output_path .tar + + # Handle requirements.txt with `python3 -m pip` + # That is, the 'internal' pip. + flag --python-bin /bin/python3 + flag --python-obey-requirements-txt + flag --python-internal-pip } -print_dotfpm_other() { - print_dotfpm_output_path "$output_format" +print_dotfpm_freebsd() { + # TODO: Is this extension correct? + print_dotfpm_output_path .tar + + flag --depends autoconf + flag --depends automake + flag --depends bash + flag --depends bc + flag --depends curl + flag --depends gcc + flag --depends git + flag --depends gmake + flag --depends gsed + flag --depends libffi + flag --depends libtool + flag --depends m4 + flag --depends pkg-config + flag --depends py38-pip + flag --depends wget } -# -# The rest defines how the above works and actually does something. -# print_dotfpm_output_path() { output_file="${output_file_prefix}${1}" flag --package "$output_file" } + +# +# The rest defines how the above works. +# + flag() { if [ "$#" -gt 1 ]; then printf "%s '%s'\n" "$1" "$2" From 13ecd37e1036ed906d1155be7c2d6d9635428b7f Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 15:15:13 -0500 Subject: [PATCH 31/90] Announce .fpm deletion --- scripts/package/tools/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 6b0647440..b5d5abc3b 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -218,7 +218,7 @@ flag() { } main() { - trap 'rm -f .fpm' EXIT + trap 'rm -vf .fpm' EXIT for output_format in "$@"; do output_file_prefix="/out/pash-${version}" From 8b65338781a710ca5c6749282d7ec498cd617842 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 19:20:43 -0500 Subject: [PATCH 32/90] Pull out code for merging later --- scripts/install-python.sh | 6 ------ scripts/package/after-install.sh | 35 -------------------------------- 2 files changed, 41 deletions(-) delete mode 100755 scripts/install-python.sh delete mode 100755 scripts/package/after-install.sh diff --git a/scripts/install-python.sh b/scripts/install-python.sh deleted file mode 100755 index c75c6f951..000000000 --- a/scripts/install-python.sh +++ /dev/null @@ -1,6 +0,0 @@ -#! /usr/bin/env bash - -set -euo pipefail -PYTHON_PKG_DIR="$PASH_TOP/python_pkg_root" -python3 -m pip install --root "$PYTHON_PKG_DIR" --ignore-installed -r "$PASH_TOP/requirements.txt" -ln -frs "$(find "$PYTHON_PKG_DIR" -type d -name site-packages)" "$PASH_TOP/python_pkgs" diff --git a/scripts/package/after-install.sh b/scripts/package/after-install.sh deleted file mode 100755 index cd7f414dc..000000000 --- a/scripts/package/after-install.sh +++ /dev/null @@ -1,35 +0,0 @@ -#! /usr/bin/env bash -# Called by system package manager to request on-site setup on host. - -set -euo pipefail - -pashd=/usr/lib/pash -logd=/var/log/pash -export PASH_TOP="$pashd" - -say_task() { - printf 'pash: %s (log: %s/%s)\n' "$1" "$logd" "$2" -} - -mkdir -vp "$logd" - -say_task "python dependencies" "install-python.log" - -if ! command -v pip 2>&1 >/dev/null; then - /bin/python3 -m ensurepip --upgrade -fi - -"$pashd/scripts/install-python.sh" 2>&1 | tee "$logd/install-python.log" - -cd "$pashd/compiler/parser" -ln -fs "$pashd/pa.sh" /usr/bin/pa.sh - -say_task "libdash" "libdash.log" -rm -rf libdash -git clone https://github.com/angelhof/libdash/ -make libdash 2>&1 | tee "$logd/libdash.log" - -say_task "runtime" "runtime.log" -cd "$pashd/runtime" -make clean -make 2>&1 | tee "$logd/runtime.log" From 964db36a77680c0f6e17658c3ce8623b4cbc9230 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 19:21:23 -0500 Subject: [PATCH 33/90] Draft package hooks --- scripts/package/fire-hook.sh | 19 +++++++ scripts/package/hooks/after-install/hook | 67 ++++++++++++++++++++++++ scripts/package/hooks/before-test/deb | 2 + scripts/package/hooks/before-test/hook | 2 + scripts/package/hooks/before-test/pacman | 2 + scripts/package/hooks/before-test/rpm | 2 + scripts/package/tools/build | 10 +++- 7 files changed, 102 insertions(+), 2 deletions(-) create mode 100755 scripts/package/fire-hook.sh create mode 100755 scripts/package/hooks/after-install/hook create mode 100755 scripts/package/hooks/before-test/deb create mode 100755 scripts/package/hooks/before-test/hook create mode 100755 scripts/package/hooks/before-test/pacman create mode 100755 scripts/package/hooks/before-test/rpm diff --git a/scripts/package/fire-hook.sh b/scripts/package/fire-hook.sh new file mode 100755 index 000000000..007aab403 --- /dev/null +++ b/scripts/package/fire-hook.sh @@ -0,0 +1,19 @@ +#! /usr/bin/env bash + +# Fire hook script in response to an event. Hooks are format-specific, +# so Debian packages can respond differently than Arch packages, for +# example. + +set -eu +event_name="$1" +output_format="$2" + +here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" +event="$here/hooks/${event_name}" + +if [ -f "$event/hook" ]; then + # Pass directory name so the script don't need to figure out where + # it is. -f implies the script does not have to be Bash, but use + # of non-executables must raise an error. + "$event/hook" "$event" "$output_format" +fi diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook new file mode 100755 index 000000000..07999d88e --- /dev/null +++ b/scripts/package/hooks/after-install/hook @@ -0,0 +1,67 @@ +#! /usr/bin/env bash + +# This script is called by a system package manager to request on-site +# setup on host. +# +# 'any.sh' means 'use for any target', so do not use +# distribution-specific knowledge here. + +set -euo pipefail + +PASH_LOG=/var/log/pash +PASH_TOP=/usr/lib/pash + +post_install_current_target() { + this_directory="$1" + output_format="$2" + recipe="${this_directory}/${output_format}" + + # Give the user a place to look when things go wrong. + log_file="$PASH_LOG/post-install.log" + mkdir -vp "$PASH_LOG" + printf "pash: post-install log: '%s'\n" "$log_file" 1>&2 + + # Put pa.sh in PATH, replacing any existing link. -f helps if a + # distribution leaves the link after uninstallation, but adds the + # risk of breaking a working link. + ln -fs "$PASH_TOP/pa.sh" /usr/bin/pa.sh + + export PASH_TOP + ( + post_install_python_project + post_install_libdash + post_install_runtime + [ -f "$recipe" ] && "$recipe" # -f, because I want non-executables to raise an error + ) > "$log_file" +} + + +post_install_python_project() { + # Guarentee pip availability. + # + # TODO: Guarentee parity between Python binary used here, and the + # one used for FPM Python output. + /bin/python3 -m ensurepip --upgrade + + # PaSh is not organized for use as a Python package, so link + # site-packages to a location the scripts understand. + local PYTHON_PKG_DIR="$PASH_TOP/python_pkg_root" + python3 -m pip install --root "$PYTHON_PKG_DIR" --ignore-installed -r "$PASH_TOP/requirements.txt" + ln -frs "$(find "$PYTHON_PKG_DIR" -type d -name site-packages)" "$PASH_TOP/python_pkgs" +} + + +post_install_libdash() { + cd "$PASH_TOP/compiler/parser" + rm -rf libdash # might have survived uninstallation, and I don't trust the make cache. + git clone https://github.com/angelhof/libdash/ + make libdash +} + +post_install_runtime() { + cd "$PASH_TOP/runtime" + make clean + make +} + +return 2>/dev/null || post_install_current_target "$@" diff --git a/scripts/package/hooks/before-test/deb b/scripts/package/hooks/before-test/deb new file mode 100755 index 000000000..c526352d3 --- /dev/null +++ b/scripts/package/hooks/before-test/deb @@ -0,0 +1,2 @@ +#! /usr/bin/env sh +sudo apt install -y "/package/pash-$1.deb" diff --git a/scripts/package/hooks/before-test/hook b/scripts/package/hooks/before-test/hook new file mode 100755 index 000000000..2caa7c149 --- /dev/null +++ b/scripts/package/hooks/before-test/hook @@ -0,0 +1,2 @@ +#! /usr/bin/env sh +"$1/$2" diff --git a/scripts/package/hooks/before-test/pacman b/scripts/package/hooks/before-test/pacman new file mode 100755 index 000000000..7146ed103 --- /dev/null +++ b/scripts/package/hooks/before-test/pacman @@ -0,0 +1,2 @@ +#! /usr/bin/env sh +sudo pacman -Uy "/package/pash-$1.tar.gz" diff --git a/scripts/package/hooks/before-test/rpm b/scripts/package/hooks/before-test/rpm new file mode 100755 index 000000000..2f3202161 --- /dev/null +++ b/scripts/package/hooks/before-test/rpm @@ -0,0 +1,2 @@ +#! /usr/bin/env sh +sudo yum localinstall -y "/package/pash-$1.deb" diff --git a/scripts/package/tools/build b/scripts/package/tools/build index b5d5abc3b..996b6a303 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -54,8 +54,14 @@ print_dotfpm() { ;; esac - # Tell PaSh to set itself up on a target system. - flag --after-install "$PASH_TOP/scripts/package/after-install.sh" + # Schedule post-install hook with format-specifier. + after_install="$(mktemp)" + ( + printf '#! /usr/bin/env\n' + printf "\$PASH_TOP/scripts/package/fire-hook.sh 'after-install' '%s'\n" "$output_format" + ) >"$after_install" + flag --after-install "$after_install" + # Map contents from here to target system. # From 65dbf1e996db1a0cbbb2cf2fbfd8aaa6d2621e7c Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 19:29:50 -0500 Subject: [PATCH 34/90] Pass version information to format hook --- scripts/package/fire-hook.sh | 7 +++---- scripts/package/hooks/before-test/hook | 2 +- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/scripts/package/fire-hook.sh b/scripts/package/fire-hook.sh index 007aab403..8368391e9 100755 --- a/scripts/package/fire-hook.sh +++ b/scripts/package/fire-hook.sh @@ -5,15 +5,14 @@ # example. set -eu -event_name="$1" -output_format="$2" +event_name="$1"; shift here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" event="$here/hooks/${event_name}" if [ -f "$event/hook" ]; then - # Pass directory name so the script don't need to figure out where + # Pass directory path so the script don't need to figure out where # it is. -f implies the script does not have to be Bash, but use # of non-executables must raise an error. - "$event/hook" "$event" "$output_format" + "$event/hook" "$event" "$@" fi diff --git a/scripts/package/hooks/before-test/hook b/scripts/package/hooks/before-test/hook index 2caa7c149..f649b9452 100755 --- a/scripts/package/hooks/before-test/hook +++ b/scripts/package/hooks/before-test/hook @@ -1,2 +1,2 @@ #! /usr/bin/env sh -"$1/$2" +"$1/$2" "$3" From 29d671f227a53908ff6d9e1a1aaa7e779c535e41 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 19:30:06 -0500 Subject: [PATCH 35/90] Avoid sudo in pacman before-test hook --- scripts/package/hooks/before-test/pacman | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/package/hooks/before-test/pacman b/scripts/package/hooks/before-test/pacman index 7146ed103..fc8af8ff8 100755 --- a/scripts/package/hooks/before-test/pacman +++ b/scripts/package/hooks/before-test/pacman @@ -1,2 +1,4 @@ #! /usr/bin/env sh -sudo pacman -Uy "/package/pash-$1.tar.gz" +set -eu +pacman -Sy +pacman -U "/package/output/pash-$1.tar.gz" From c2472a3a5878dc504c036490f8393736005d7faf Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 19:30:18 -0500 Subject: [PATCH 36/90] Add deploy.sh --- scripts/package/deploy.sh | 60 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100755 scripts/package/deploy.sh diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh new file mode 100755 index 000000000..c84b5cdb6 --- /dev/null +++ b/scripts/package/deploy.sh @@ -0,0 +1,60 @@ +#! /usr/bin/env bash + +# Build, install, and use PaSh in a new Docker container. +# +# deploy.sh IMAGE VERSION FORMAT +# +# Exit code matches that from running `./tests/$3/main.sh` in the +# container. 0 is a sign of PaSh's functionality in the other +# environment. +# +# Scripts in ./tests may use /package in the container to store +# information on the host, but printing should suffice. +# +# $1 - IMAGE name from Docker Hub to use +# $2 - VERSION of PaSh to install in a container +# $3 - FPM package FORMAT for the named PaSh version + +set -euo pipefail + +me="$(readlink -f "${BASH_SOURCE[0]}")" +mydirname="$(dirname "$me")" +myname="$(basename "$me")" + +if [ -f /.dockerenv ]; then + set +x + # Pass control to vertification script. + target_version="$1" + output_format="$2" + + # Keep this here, so tests don't need to keep setting the variable. + export PASH_TOP=/usr/lib/pash + + "$mydirname/fire-hook.sh" "before-test" "$output_format" "$target_version" + + # Basic example must always work, no matter where we're going. + [ -d "$PASH_TOP" ] + test_script="${PASH_TOP}/evaluation/intro/hello-world.sh" + time pa.sh "$test_script" + time bash "$test_script" +else + image="$1" + target_version="$2" + output_format="$3" + + # Perform a one-off build for the package. + "$mydirname/shell.sh" -ic "build '$target_version' '$output_format'" + + # It's important to use Docker hub here because we want to verify + # PaSh's behavior against other people's work, namely official OS + # images. + docker pull "$image" + + # Invoke self in the container. + docker run \ + --interactive \ + --tty \ + --volume "${mydirname}:/package" \ + "$image" \ + "/package/${myname}" "$target_version" "$output_format" +fi From ffaabf7cd550a7302dd79dcf4b64044a71955fbc Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 20:48:38 -0500 Subject: [PATCH 37/90] Use python3 in PATH in after-install --- scripts/package/hooks/after-install/hook | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index 07999d88e..e800d0914 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -27,21 +27,21 @@ post_install_current_target() { ln -fs "$PASH_TOP/pa.sh" /usr/bin/pa.sh export PASH_TOP + ( post_install_python_project post_install_libdash post_install_runtime [ -f "$recipe" ] && "$recipe" # -f, because I want non-executables to raise an error - ) > "$log_file" + ) | tee "$log_file" } post_install_python_project() { # Guarentee pip availability. # - # TODO: Guarentee parity between Python binary used here, and the - # one used for FPM Python output. - /bin/python3 -m ensurepip --upgrade + # TODO: Use same Python binary in FPM Python output. + python3 -m ensurepip --upgrade # PaSh is not organized for use as a Python package, so link # site-packages to a location the scripts understand. From 14c7e924baf0c6949bd1abea32aed1f70d18489a Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 20:48:57 -0500 Subject: [PATCH 38/90] Avoid sudo in apt test --- scripts/package/hooks/before-test/deb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/package/hooks/before-test/deb b/scripts/package/hooks/before-test/deb index c526352d3..746ebb8e9 100755 --- a/scripts/package/hooks/before-test/deb +++ b/scripts/package/hooks/before-test/deb @@ -1,2 +1,3 @@ #! /usr/bin/env sh -sudo apt install -y "/package/pash-$1.deb" +apt update +apt install -y "/package/output/pash-$1.deb" From acafa81b6c8dc520c4b1b8bfb211d6fa1d5da10c Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 20:49:21 -0500 Subject: [PATCH 39/90] Fix after-install flow --- scripts/package/tools/build | 34 +++++++++++++++++++++------------- scripts/pash-init.sh | 1 - 2 files changed, 21 insertions(+), 14 deletions(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 996b6a303..059ad6333 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -13,7 +13,8 @@ set -u print_dotfpm() { - output_format="$1" + local output_format="$1" + local after_install="$2" # I/O rules, with Docker container paths flag --chdir /src @@ -31,6 +32,9 @@ print_dotfpm() { flag --url 'https://github.com/binpash/pash' flag --version "$version" + # Pass control to PaSh after installation + flag --after-install "$after_install" + # Target information case "$output_format" in deb) @@ -54,15 +58,6 @@ print_dotfpm() { ;; esac - # Schedule post-install hook with format-specifier. - after_install="$(mktemp)" - ( - printf '#! /usr/bin/env\n' - printf "\$PASH_TOP/scripts/package/fire-hook.sh 'after-install' '%s'\n" "$output_format" - ) >"$after_install" - flag --after-install "$after_install" - - # Map contents from here to target system. # # TODO: Consider include-only approach. This will pick up any @@ -210,6 +205,14 @@ print_dotfpm_output_path() { flag --package "$output_file" } +# Post-installation steps depend on the target. Rather than ask one +# script to entertain every platform, dispatch to a specialized +# script. +print_post_install_script() { + local output_format="$1" + printf '#! /usr/bin/env bash\n' + printf "\$PASH_TOP/scripts/package/fire-hook.sh 'after-install' '%s'\n" "$output_format" +} # # The rest defines how the above works. @@ -223,14 +226,19 @@ flag() { fi } -main() { - trap 'rm -vf .fpm' EXIT +main() { for output_format in "$@"; do + after_install="$(mktemp)" + print_post_install_script "$output_format" > "$after_install" + cat "$after_install" output_file_prefix="/out/pash-${version}" - print_dotfpm "$output_format" > .fpm + + print_dotfpm "$output_format" "$after_install" > .fpm + set +e /fpm/bin/fpm && chmod 440 "$output_file" + rm -vf "$after_install" .fpm set -e printf '\n' done diff --git a/scripts/pash-init.sh b/scripts/pash-init.sh index 21f0e7fa3..213e3a9ad 100644 --- a/scripts/pash-init.sh +++ b/scripts/pash-init.sh @@ -6,7 +6,6 @@ export distro="$("$PASH_TOP/scripts/distro.sh")" # Adapt to Docker if [ -f /.dockerenv ]; then - export PASH_TOP=/opt/pash export LC_ALL='en_US.UTF-8' export LANG='en_US.UTF-8' export LANGUAGE='en_US.UTF-8' From de2e77c0e66b03036a5d3973110a1c22c9afe40a Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 20:50:02 -0500 Subject: [PATCH 40/90] Replace extra curl dependency with gcc on Arch --- scripts/package/tools/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 059ad6333..1e25d8ba7 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -89,7 +89,7 @@ print_dotfpm_pacman() { flag --depends bash flag --depends bc flag --depends curl - flag --depends curl + flag --depends gcc flag --depends git flag --depends inetutils flag --depends libffi From 65900d464def53841ab547715fdfc2d6863dae35 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:16:17 -0500 Subject: [PATCH 41/90] Use old Python package hack --- scripts/package/hooks/after-install/hook | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index e800d0914..68940e421 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -38,16 +38,15 @@ post_install_current_target() { post_install_python_project() { - # Guarentee pip availability. - # - # TODO: Use same Python binary in FPM Python output. - python3 -m ensurepip --upgrade - - # PaSh is not organized for use as a Python package, so link - # site-packages to a location the scripts understand. - local PYTHON_PKG_DIR="$PASH_TOP/python_pkg_root" + # PaSh is not organized for use as a Python package. + # This destructive operation compensates for that. + local PYTHON_PKG_DIR="$PASH_TOP/python_pkg" + rm -rf "$PYTHON_PKG_DIR" + mkdir "$PYTHON_PKG_DIR" python3 -m pip install --root "$PYTHON_PKG_DIR" --ignore-installed -r "$PASH_TOP/requirements.txt" - ln -frs "$(find "$PYTHON_PKG_DIR" -type d -name site-packages)" "$PASH_TOP/python_pkgs" + local pkg_path="$(find "$PYTHON_PKG_DIR" \( -name "site-packages" -or -name "dist-packages" \) -type d)" + cd "$pkg_path" + mv -v * "${PYTHON_PKG_DIR}/" } From afbd1f78dd849d1bb49fc1cc57dd40950372e887 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:16:27 -0500 Subject: [PATCH 42/90] Simplify comment about -f --- scripts/package/hooks/after-install/hook | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index 68940e421..74195c029 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -32,7 +32,7 @@ post_install_current_target() { post_install_python_project post_install_libdash post_install_runtime - [ -f "$recipe" ] && "$recipe" # -f, because I want non-executables to raise an error + [ -f "$recipe" ] && "$recipe" # -f, so non-executables raise an error ) | tee "$log_file" } From d61657408f67d4a7347a47ac3b356ebb6a0d2819 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:16:38 -0500 Subject: [PATCH 43/90] Avoid sudo in yum test --- scripts/package/hooks/before-test/rpm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/hooks/before-test/rpm b/scripts/package/hooks/before-test/rpm index 2f3202161..ea08c5176 100755 --- a/scripts/package/hooks/before-test/rpm +++ b/scripts/package/hooks/before-test/rpm @@ -1,2 +1,2 @@ #! /usr/bin/env sh -sudo yum localinstall -y "/package/pash-$1.deb" +yum localinstall -y "/package/pash-$1.rpm" From e2f83156863bcf63e24fc5a1852458bb57e23c48 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:16:50 -0500 Subject: [PATCH 44/90] Use known PASH_TOP in post-install context --- scripts/package/tools/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 1e25d8ba7..d7c9cb8fa 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -211,7 +211,7 @@ print_dotfpm_output_path() { print_post_install_script() { local output_format="$1" printf '#! /usr/bin/env bash\n' - printf "\$PASH_TOP/scripts/package/fire-hook.sh 'after-install' '%s'\n" "$output_format" + printf "/usr/lib/pash/scripts/package/fire-hook.sh 'after-install' '%s'\n" "$output_format" } # From 60c950ce0b4c7d786781bb69868f8d1756b64f82 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:31:37 -0500 Subject: [PATCH 45/90] Progogate exit codes differently in package/tools/build --- scripts/package/hooks/after-install/hook | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index 74195c029..8b32433be 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -29,10 +29,13 @@ post_install_current_target() { export PASH_TOP ( + set -e post_install_python_project post_install_libdash post_install_runtime - [ -f "$recipe" ] && "$recipe" # -f, so non-executables raise an error + if [ -f "$recipe" ]; then + "$recipe" # -f, so non-executables raise an error + fi ) | tee "$log_file" } @@ -63,4 +66,5 @@ post_install_runtime() { make } -return 2>/dev/null || post_install_current_target "$@" + +post_install_current_target "$@" From 836bc6ade7d83775662dc1e763f253c53056bd39 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:59:21 -0500 Subject: [PATCH 46/90] Show timing lines --- scripts/package/deploy.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index c84b5cdb6..ad508b7e8 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -35,8 +35,10 @@ if [ -f /.dockerenv ]; then # Basic example must always work, no matter where we're going. [ -d "$PASH_TOP" ] test_script="${PASH_TOP}/evaluation/intro/hello-world.sh" + set -x time pa.sh "$test_script" time bash "$test_script" + set +x else image="$1" target_version="$2" From fe585e1429ff05d53b6c53ce987eb849b3da24a1 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:59:35 -0500 Subject: [PATCH 47/90] Remove deploy.sh container after use --- scripts/package/deploy.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index ad508b7e8..9decfa5e1 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -56,6 +56,7 @@ else docker run \ --interactive \ --tty \ + --rm \ --volume "${mydirname}:/package" \ "$image" \ "/package/${myname}" "$target_version" "$output_format" From 0ae726ab1288f9b7a5e5cf42c5cfb06ef494b408 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 21:59:54 -0500 Subject: [PATCH 48/90] Fix typo in PYTHON_PKG_DIR --- scripts/package/hooks/after-install/hook | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index 8b32433be..ae6a62772 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -43,7 +43,7 @@ post_install_current_target() { post_install_python_project() { # PaSh is not organized for use as a Python package. # This destructive operation compensates for that. - local PYTHON_PKG_DIR="$PASH_TOP/python_pkg" + local PYTHON_PKG_DIR="$PASH_TOP/python_pkgs" rm -rf "$PYTHON_PKG_DIR" mkdir "$PYTHON_PKG_DIR" python3 -m pip install --root "$PYTHON_PKG_DIR" --ignore-installed -r "$PASH_TOP/requirements.txt" From 366adefc208f17727d6f4e4340e85bc004a9222c Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 22:00:31 -0500 Subject: [PATCH 49/90] Add g++-10 for .deb --- scripts/package/tools/build | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index d7c9cb8fa..4d81b801a 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -149,6 +149,7 @@ print_dotfpm_deb() { flag --depends bc flag --depends bsdmainutils flag --depends curl + flag --depends g++-10 flag --depends git flag --depends libffi-dev flag --depends libtool From a37cb4bb169040c303f08b391615a0e01d93910f Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 22:08:02 -0500 Subject: [PATCH 50/90] Add iterate.sh --- scripts/package/iterate.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 scripts/package/iterate.sh diff --git a/scripts/package/iterate.sh b/scripts/package/iterate.sh new file mode 100755 index 000000000..5d7207266 --- /dev/null +++ b/scripts/package/iterate.sh @@ -0,0 +1,6 @@ +#! /usr/bin/env bash +set -u +./deploy.sh "$1" 0.0.1 "$2" +read -p "Repeat? [Y/n]" -n 1 -r +echo +[[ $REPLY =~ ^[Yy]$ ]] && "$0" From 0029aec71d737d3dbb2661ba280518aadc4a9bb9 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 22:08:16 -0500 Subject: [PATCH 51/90] Add gcc10-optimize.sh --- scripts/gcc10-optimize.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 scripts/gcc10-optimize.sh diff --git a/scripts/gcc10-optimize.sh b/scripts/gcc10-optimize.sh new file mode 100755 index 000000000..26376997e --- /dev/null +++ b/scripts/gcc10-optimize.sh @@ -0,0 +1,6 @@ +#! /usr/bin/env sh +apt-get install software-properties-common -y +add-apt-repository ppa:ubuntu-toolchain-r/test -y +apt-get install g++-10 +update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 100 +update-alternatives --set g++ /usr/bin/g++-10 From 187c675d92ea87135b1695cfe5f577a8e6d61fc8 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 22:42:04 -0500 Subject: [PATCH 52/90] Use sudo for rpm again --- scripts/package/hooks/before-test/rpm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/hooks/before-test/rpm b/scripts/package/hooks/before-test/rpm index ea08c5176..1304c41a2 100755 --- a/scripts/package/hooks/before-test/rpm +++ b/scripts/package/hooks/before-test/rpm @@ -1,2 +1,2 @@ #! /usr/bin/env sh -yum localinstall -y "/package/pash-$1.rpm" +sudo yum localinstall "/package/output/pash-$1.rpm" From 7a13a373bb53ca8d67e944a938d591a96a09d327 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 22:43:00 -0500 Subject: [PATCH 53/90] Revert tutorial until public links are available --- docs/tutorial/tutorial.md | 117 +++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 59 deletions(-) diff --git a/docs/tutorial/tutorial.md b/docs/tutorial/tutorial.md index 698cc1624..7280863dc 100644 --- a/docs/tutorial/tutorial.md +++ b/docs/tutorial/tutorial.md @@ -49,93 +49,92 @@ The new parallel script has POSIX constructs added to explicitly guide paralleli ## Installation -Regardless of your target platform, PaSh requires the `PASH_TOP` -environment variable to point to a PaSh installation on disk. The -installation method impacts the value of this variable. +#### Natively on Linux +_Ubuntu, Fedora, Debian, Arch:_ +Use `up`: -### Unix-likes - -Download your preferred system package (all architectures). - -| Package | Install command | -|-----------------------------------|-------------------------------| -| [Red Hat (`pash.rpm`)](#link-tbd) | `yum localinstall ./pash.rpm` | -| [Debian (`pash.deb`)](#link-tbd) | `apt install ./pash.deb` | - - -Run `pa.sh` to verify installation of the following paths. - -| Path | Purpose | -|------------------|-----------------------------------------| -| `/usr/bin/pa.sh` | symlink to `/opt/pash/pa.sh` | -| `/usr/lib/pash` | source code (as seen in the repository) | -| `/var/log/pash` | package build logs | - -If something goes wrong, please submit your build logs with a bug -report. +```sh +curl up.binpa.sh | sh +``` +_Other distros_: +Use the system's package manager to install the following packages (some of which might already be installed in your system): -### Source -[FPM]: https://fpm.readthedocs.io/en/latest/cli-reference.html#general-options +``` +git libtool m4 curl automake pkg-config libffi-dev python python3 python3-pip wamerican-insane bc bsdmainutils python3-testresources python3-setuptools locales locales-all wget netcat-openbsd +``` -Source deployments start with a clone. +Then clone the PaSh repository and run `setup-pash.sh` as follows: -``` +```sh git clone git@github.com:binpash/pash.git +./pash/scripts/setup-pash.sh ``` -You can either use the source code to create a Docker image, or deploy -PaSh on the current host. For Docker instructions, see the -[pash-on-docker -guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide). +As noted at the end of `setup-pash.sh`, make sure you set `PASH_TOP` pointing to the absolute path of the directory `pa.sh` resides (you can optionally place that in your `PATH`). -For native deployment, install [FPM][] to prepare native packages. For -a Debian-based distribution (as an example), you can deploy PaSh on -the current host for some label `X`. +_Installation Flags_: -``` -./pash/scripts/package/build-packages.sh X -sudo apt install ./pash/scripts/package/output/pash-X.deb -``` +PaSh offers the following installation flags: +* `-p/--prepare` that installs all the required dependencies if your system is supported +* `-o/--opt-agg` that downloads g++-10 and builds an optimized version of some aggregators (not currently implemented in the main PaSh workflow) -RHEL is no different. - -``` -./pash/scripts/package/build-packages.sh X -sudo yum localinstall ./pash/scripts/package/output/pash-X.rpm +To use these flags, see the following installation procedure: +```sh +git clone git@github.com:binpash/pash.git +cd pash/ && bash scripts/install.sh -p --opti-agg ``` -[`build-packages.sh`](../../scripts/package/build-packages.sh) defines -supported systems and distribution-specific modifications. +This will install all the dependencies for PaSh on your machine (sudo is used) and the g++-10 toolchain, but also build the respective aggregators. +#### Docker -### Docker -[Docker]: https://docs.docker.com/get-docker/ +PaSh on Docker is useful when native installation is not an option -- for example, to allow development on Windows or OS X. +First [make sure you install Docker](https://docs.docker.com/get-docker/) on your machine. +Note that, depending on the configuration, PaSh on Docker may or may not be able to exploit all available hardware resources. +There are two main options for installing PaSh via Docker. -In the context of a [Docker][] container, `PASH_TOP` is `/opt/pash`. The -same directory holds is a copy of the Git repository. Use `git -C -/opt/pash pull` to update the code in a container. PaSh may not be -able to exploit all available hardware resources when used in a -container. - -For instructions, see the [pash-on-docker -guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide). +_Major Releases:_ +The easiest is to `pull` the docker image [from Docker Hub](https://hub.docker.com/r/binpash/pash): +```sh +docker pull binpash/pash +``` +We refresh this image (as well as other images) on every major release. [//]: # "TODO(@nvasilakis, @dkarnikis): Need to automate this per release." +_Latest Commit:_ +To built the latest Docker container from scratch, run `docker build` in scripts: +```sh +git clone git@github.com:binpash/pash.git +cd pash/scripts +docker build -t "pash:latest" . +``` +This will build a fresh Docker image using the latest commit---recommended for development. + +In all the above cases, launching the container is done via: +```sh +docker run --name pash-play -it pash +``` +PaSh can be found in the container's `/opt/pash` directory, so run `cd pash; git pull` to fetch the latest updates. +More information in the [pash-on-docker guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide). -### Windows using WSL +#### Windows using WSL To run PaSh on windows without Docker, install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10). A short tutorial is included in the [contributing](../contributing/contrib.md) guide. - - ## Running Scripts -To use this section, run `cd $PASH_TOP/evaluation/intro` first. +All scripts in this guide assume that `$PASH_TOP` is set to the top directory of the PaSh codebase (e.g., `/opt/pash` on docker) + +**To run scripts in this section of the tutorial, make sure you are in the `intro` directory of the `evaluation`:** +```sh +cd $PASH_TOP/evaluation/intro +``` +> In the following examples, you can avoid including `$PASH_TOP` before `pa.sh` by adding `PASH_TOP` in your `PATH`, which amounts to adding an `export PATH=$PATH:$PASH_TOP` in your shell configuration file. #### Intro: Hello, Parallel World! From 9bb8b8bb06ab23fb87b616cc03a4b5e6316eaf53 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 23:43:18 -0500 Subject: [PATCH 54/90] Add packaging.md --- docs/tooling/packaging.md | 51 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/tooling/packaging.md diff --git a/docs/tooling/packaging.md b/docs/tooling/packaging.md new file mode 100644 index 000000000..466ee4fcc --- /dev/null +++ b/docs/tooling/packaging.md @@ -0,0 +1,51 @@ +# Packaging PaSh + +`scripts/package` packages PaSh into various formats. It is a suitable +replacement for deployments depending on `install.sh`, +`setup-pash.sh`, `up.sh`, `distro-deps.sh`, and/or `pkg.sh` + + + +## Building Packages + +``` + shell.sh +``` + +Enter a Docker container for building packages. Run without arguments +to enter a REPL and print usage information for the `build` command. + +Since `shell.sh` is just a Bash, you can pass it commands. For +example, `./shell.sh -ic 'build 0.0.1 deb'` launches an interactive +shell that builds a Debian package using version `0.0.1`. + + +## Testing Packages + +``` + deploy.sh IMAGE VERSION FORMAT +``` + +`deploy.sh` runs `./shell.sh -ic 'build VERSION FORMAT'`, then +installs PaSh in a Docker container based on a `IMAGE` from DockerHub. + +Here's an example that builds a `pacman`/`.tar.gz` package for +deployment in Arch, a `.deb` package for Debian and Ubuntu, and an RPM +package for Fedora. + +``` +$ version=0.0.1 +$ PATH="$PASH_TOP/scripts/package:$PATH" +$ deploy.sh archlinux $version pacman +$ deploy.sh debian $version deb +$ deploy.sh ubuntu $version deb +$ deploy.sh fedora $version rpm +``` + +## Developing Packages + +``` + iterate.sh IMAGE FORMAT +``` + +Run `deploy.sh IMAGE 0.0.1 FORMAT`, then prompt to repeat. From 0c5dbb603babe4b52d267e1d3ca9b265ae6bc86e Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 23:43:55 -0500 Subject: [PATCH 55/90] Revert .gitignore: python_pkg_root --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index ede1ae4d5..e3c9c8857 100644 --- a/.gitignore +++ b/.gitignore @@ -31,7 +31,6 @@ revision.tex node_modules python_pkgs -python_pkg_root # local directory w/ related work bib.to From 5d5bc970fce03686e6964d803e8f81cb08422847 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 23:52:37 -0500 Subject: [PATCH 56/90] Remove duplicate libtool dependency --- scripts/package/tools/build | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/package/tools/build b/scripts/package/tools/build index 4d81b801a..c2f08e30b 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -94,7 +94,6 @@ print_dotfpm_pacman() { flag --depends inetutils flag --depends libffi flag --depends libtool - flag --depends libtool flag --depends m4 flag --depends make flag --depends openbsd-netcat From 27d5b80b51d2f1e2c7b92a6bb51b7d9c0e69b972 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Fri, 4 Mar 2022 23:57:51 -0500 Subject: [PATCH 57/90] Update header comment in after-install/hook --- scripts/package/hooks/after-install/hook | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index ae6a62772..b4d947a99 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -1,10 +1,7 @@ #! /usr/bin/env bash # This script is called by a system package manager to request on-site -# setup on host. -# -# 'any.sh' means 'use for any target', so do not use -# distribution-specific knowledge here. +# setup on a host, after the package itself is installed. set -euo pipefail From 344ab342cef659ed671540fb001705624e67632b Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Sat, 5 Mar 2022 00:00:32 -0500 Subject: [PATCH 58/90] Add comment to before-test hook --- scripts/package/hooks/before-test/hook | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/package/hooks/before-test/hook b/scripts/package/hooks/before-test/hook index f649b9452..625b533f1 100755 --- a/scripts/package/hooks/before-test/hook +++ b/scripts/package/hooks/before-test/hook @@ -1,2 +1,8 @@ #! /usr/bin/env sh +# $1 - Path to containing directory +# $2 - Output format, as used in FPM +# $3 - Package version +# +# Dispatch to script, such that the script can assume it runs for a +# specific package format. "$1/$2" "$3" From e910fa982b383dce9298897078e70ec5a8087b1e Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 7 Mar 2022 15:37:33 -0500 Subject: [PATCH 59/90] Combine distro.sh and gcc10-optimize.sh into utils.sh --- scripts/distro.sh | 13 ------------- scripts/gcc10-optimize.sh | 6 ------ scripts/pash-init.sh | 4 +++- scripts/utils.sh | 21 +++++++++++++++++++++ 4 files changed, 24 insertions(+), 20 deletions(-) delete mode 100755 scripts/distro.sh delete mode 100755 scripts/gcc10-optimize.sh diff --git a/scripts/distro.sh b/scripts/distro.sh deleted file mode 100755 index fb3741765..000000000 --- a/scripts/distro.sh +++ /dev/null @@ -1,13 +0,0 @@ -#! /usr/bin/env bash -# Infer Unix-like distribution -# -# TODO: This logic appears verbatim in a few places. -# Deduplicate by having other scripts call this one. - -if type lsb_release >/dev/null 2>&1 ; then - distro=$(lsb_release -i -s) -elif [ -e /etc/os-release ] ; then - distro=$(awk -F= '$1 == "ID" {print $2}' /etc/os-release) -fi - -printf '%s\n' "$distro" | LC_ALL=C tr '[:upper:]' '[:lower:]' diff --git a/scripts/gcc10-optimize.sh b/scripts/gcc10-optimize.sh deleted file mode 100755 index 26376997e..000000000 --- a/scripts/gcc10-optimize.sh +++ /dev/null @@ -1,6 +0,0 @@ -#! /usr/bin/env sh -apt-get install software-properties-common -y -add-apt-repository ppa:ubuntu-toolchain-r/test -y -apt-get install g++-10 -update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 100 -update-alternatives --set g++ /usr/bin/g++-10 diff --git a/scripts/pash-init.sh b/scripts/pash-init.sh index 213e3a9ad..5c2aa0809 100644 --- a/scripts/pash-init.sh +++ b/scripts/pash-init.sh @@ -2,7 +2,9 @@ # Source this in any new ~/pash-init.sh set -u -export distro="$("$PASH_TOP/scripts/distro.sh")" +. "$PASH_TOP/scripts/utils.sh" + +export distro="$(infer_unix_like_distro)" # Adapt to Docker if [ -f /.dockerenv ]; then diff --git a/scripts/utils.sh b/scripts/utils.sh index 57e99002d..57e6ff21c 100755 --- a/scripts/utils.sh +++ b/scripts/utils.sh @@ -43,3 +43,24 @@ append_nl_if_not(){ fi } + +# TODO: This logic appears verbatim in a few places. +# Deduplicate by having other scripts call this one. +infer_unix_like_distro() { + if type lsb_release >/dev/null 2>&1 ; then + distro=$(lsb_release -i -s) + elif [ -e /etc/os-release ] ; then + distro=$(awk -F= '$1 == "ID" {print $2}' /etc/os-release) + fi + + printf '%s\n' "$distro" | LC_ALL=C tr '[:upper:]' '[:lower:]' +} + + +use_alternative_debian_gpp10() { + apt-get install software-properties-common -y + add-apt-repository ppa:ubuntu-toolchain-r/test -y + apt-get install g++-10 + update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 100 + update-alternatives --set g++ /usr/bin/g++-10 +} From f93a1269e0c39157081b7c497ffec847874621ea Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 7 Mar 2022 15:40:51 -0500 Subject: [PATCH 60/90] Revert contrib.md --- docs/contributing/contrib.md | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/docs/contributing/contrib.md b/docs/contributing/contrib.md index 9f6477185..e5f8ca82a 100644 --- a/docs/contributing/contrib.md +++ b/docs/contributing/contrib.md @@ -1,36 +1,19 @@ ## PaSh on Docker: A Pocket Guide -This is a pocket guide for running PaSh in a Docker container. +This is a pocket guide for running PaSh in a docker container. ### Loading image -#### From Docker Hub - -The following will pull the latest Docker image (tag: `pash:latest`), -and start a new container using the name `pash-play`. +The following will pull and launch the latest image (tag: `pash:latest`) using the name `pash-play` ```sh docker pull binpash/pash docker run --name pash-play -it pash ``` - This will use `pash-play` for the container (to be able to start/stop it later) and `-it` runs it interactively (subsuming the next command). To restart after you exit, run `docker start -i pash-play` (flag `-i` starts it interactively.) - -#### From source code - -If you want to use or develop PaSh for a specific commit, this will -build the latest Docker image. - -```sh -git clone git@github.com:binpash/pash.git -cd pash/scripts -docker build -t "pash:latest" . -``` - - ### Customizing image To be used for continuous integration and testing, this image has been configured to have _read-only_ access to the repo through a different user. From 2e5f9dd1102d2633395466fea9c0573e758087d3 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 7 Mar 2022 15:49:36 -0500 Subject: [PATCH 61/90] Revert pa.sh --- pa.sh | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/pa.sh b/pa.sh index 60dbd4145..ba619f5e3 100755 --- a/pa.sh +++ b/pa.sh @@ -1,32 +1,6 @@ #!/usr/bin/env bash -# Help out with shell configuration -if [ -z "$PASH_TOP" ]; then - me="$(readlink -f "${BASH_SOURCE[0]}")" - export PASH_TOP="$(dirname "$me")" - export PATH="$PASH_TOP:$PATH" - printf '# Add this to your shell configuration.\n\n' - printf " export PASH_TOP='%s'\n" "$PASH_TOP" - printf ' export PATH="$PASH_TOP:$PATH"\n\n' - printf '# If your shell supports process substitution (bash, zsh, etc.),\n' - printf '# then run `. <(%s)` to use this config in your current shell.\n' "$0" - exit 0 -fi - -# Halt if script was sourced. The above is only a setup aid that -# should not interfere with the script's normal operation. -return 2>/dev/null - - -rcfile=~/.pash_init -if [ ! -f "$rcfile" ]; then - printf '# Include system-level rules\n' > "$rcfile" - printf ". '%s/scripts/pash-init.sh'\n" "$PASH_TOP" >> "$rcfile" - chmod +x "$rcfile" - printf 'Created %s\n' "$rcfile" -fi - - +export PASH_TOP=${PASH_TOP:-${BASH_SOURCE%/*}} export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib/" # point to the local downloaded folders export PYTHONPATH=${PASH_TOP}/python_pkgs/ From ee36b6bccd2e07c6f4554ba0d69fc7f03ba5ec2e Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 7 Mar 2022 19:38:07 -0500 Subject: [PATCH 62/90] Stop destructive Python module movement --- pa.sh | 10 ++++++++-- scripts/package/hooks/after-install/hook | 13 ++++--------- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/pa.sh b/pa.sh index ba619f5e3..13c8c4474 100755 --- a/pa.sh +++ b/pa.sh @@ -2,8 +2,14 @@ export PASH_TOP=${PASH_TOP:-${BASH_SOURCE%/*}} export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib/" -# point to the local downloaded folders -export PYTHONPATH=${PASH_TOP}/python_pkgs/ + +# Until PaSh is set up in a way that naturally integrates with +# Python's system-level paths, prefer local package installations. +export PYTHONPATH="$( + find "$PASH_TOP" \ + -type d \( -name site-packages -or -name dist-packages \) \ + -printf '%p:' +)$PASH_TOP/python_pkgs" ## Save the umask to first create some files and then revert it old_umask=$(umask) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index b4d947a99..e697fb331 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -38,15 +38,10 @@ post_install_current_target() { post_install_python_project() { - # PaSh is not organized for use as a Python package. - # This destructive operation compensates for that. - local PYTHON_PKG_DIR="$PASH_TOP/python_pkgs" - rm -rf "$PYTHON_PKG_DIR" - mkdir "$PYTHON_PKG_DIR" - python3 -m pip install --root "$PYTHON_PKG_DIR" --ignore-installed -r "$PASH_TOP/requirements.txt" - local pkg_path="$(find "$PYTHON_PKG_DIR" \( -name "site-packages" -or -name "dist-packages" \) -type d)" - cd "$pkg_path" - mv -v * "${PYTHON_PKG_DIR}/" + python3 -m pip install \ + --ignore-installed \ + --requirement "$PASH_TOP/requirements.txt" \ + --root "$PASH_TOP/python_pkgs" } From 0d820efe34326eab8d1af45423be3d1e2f1a1b0b Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 7 Mar 2022 19:38:22 -0500 Subject: [PATCH 63/90] Allow sourcing of after-install hook for debugging --- scripts/package/hooks/after-install/hook | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index e697fb331..45acf3964 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -59,4 +59,4 @@ post_install_runtime() { } -post_install_current_target "$@" +(return 2>/dev/null) || post_install_current_target "$@" From a3121b19b1b38f5a479c4ea9df21e6a7c91225a8 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 7 Mar 2022 19:38:38 -0500 Subject: [PATCH 64/90] Make test package deployments non-interactive --- scripts/package/hooks/before-test/deb | 4 ++++ scripts/package/hooks/before-test/pacman | 2 +- scripts/package/hooks/before-test/rpm | 4 ++-- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/scripts/package/hooks/before-test/deb b/scripts/package/hooks/before-test/deb index 746ebb8e9..ffadcb972 100755 --- a/scripts/package/hooks/before-test/deb +++ b/scripts/package/hooks/before-test/deb @@ -1,3 +1,7 @@ #! /usr/bin/env sh apt update + +# Suppress mid-installation timezone prompt +DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata + apt install -y "/package/output/pash-$1.deb" diff --git a/scripts/package/hooks/before-test/pacman b/scripts/package/hooks/before-test/pacman index fc8af8ff8..7c67a0860 100755 --- a/scripts/package/hooks/before-test/pacman +++ b/scripts/package/hooks/before-test/pacman @@ -1,4 +1,4 @@ #! /usr/bin/env sh set -eu pacman -Sy -pacman -U "/package/output/pash-$1.tar.gz" +LC_ALL=en_US.UTF-8 pacman --noconfirm -U "/package/output/pash-$1.tar.gz" diff --git a/scripts/package/hooks/before-test/rpm b/scripts/package/hooks/before-test/rpm index 1304c41a2..45820f904 100755 --- a/scripts/package/hooks/before-test/rpm +++ b/scripts/package/hooks/before-test/rpm @@ -1,2 +1,2 @@ -#! /usr/bin/env sh -sudo yum localinstall "/package/output/pash-$1.rpm" +#! /usr/bin/env bash +sudo yum localinstall -y "/package/output/pash-${1}.rpm" From 6ff4e01f318350a83397b22643305fe90c8fadf0 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 10 Mar 2022 15:16:45 -0500 Subject: [PATCH 65/90] Ask reader to enable Docker experimental features for packaging --- docs/tooling/packaging.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/tooling/packaging.md b/docs/tooling/packaging.md index 466ee4fcc..546e62646 100644 --- a/docs/tooling/packaging.md +++ b/docs/tooling/packaging.md @@ -2,18 +2,22 @@ `scripts/package` packages PaSh into various formats. It is a suitable replacement for deployments depending on `install.sh`, -`setup-pash.sh`, `up.sh`, `distro-deps.sh`, and/or `pkg.sh` +`setup-pash.sh`, `up.sh`, `distro-deps.sh`, and/or `pkg.sh`. ## Building Packages +[eef]: https://github.com/docker/cli/blob/master/experimental/README.md ``` shell.sh ``` -Enter a Docker container for building packages. Run without arguments -to enter a REPL and print usage information for the `build` command. +To use this feature, [enable experimental features][eef] in Docker. + +`shell.sh` enters a Docker container for building packages. Run +without arguments to enter a REPL and print usage information for the +`build` command. Since `shell.sh` is just a Bash, you can pass it commands. For example, `./shell.sh -ic 'build 0.0.1 deb'` launches an interactive From 42f85dc86f3791dd1e19793605e78d08fb04431b Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 15 Jun 2022 13:04:32 -0400 Subject: [PATCH 66/90] Remove pash-init.sh --- scripts/pash-init.sh | 14 -------------- 1 file changed, 14 deletions(-) delete mode 100644 scripts/pash-init.sh diff --git a/scripts/pash-init.sh b/scripts/pash-init.sh deleted file mode 100644 index 5c2aa0809..000000000 --- a/scripts/pash-init.sh +++ /dev/null @@ -1,14 +0,0 @@ -#! /usr/bin/env bash -# Source this in any new ~/pash-init.sh - -set -u -. "$PASH_TOP/scripts/utils.sh" - -export distro="$(infer_unix_like_distro)" - -# Adapt to Docker -if [ -f /.dockerenv ]; then - export LC_ALL='en_US.UTF-8' - export LANG='en_US.UTF-8' - export LANGUAGE='en_US.UTF-8' -fi From 0875402a36a35f9266edb74d524c50d0645cd386 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 23 Jun 2022 15:12:12 -0400 Subject: [PATCH 67/90] Remove dead code --- scripts/utils.sh | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/scripts/utils.sh b/scripts/utils.sh index c2738a3d9..faba1af2b 100755 --- a/scripts/utils.sh +++ b/scripts/utils.sh @@ -44,25 +44,6 @@ append_nl_if_not(){ } -# Begin merge from https://github.com/zyrolasting/pash/pull/2 Note -# from @zyrolasting: This code appeared while usage() was being -# authored, causing a merge conflict. I am only merging -# infer_unix_like_distro and use_alternative_debian_gpp10 -# to fix the conflict. -# -# TODO: This logic appears verbatim in a few places. -# Deduplicate by having other scripts call this one. -infer_unix_like_distro() { - if type lsb_release >/dev/null 2>&1 ; then - distro=$(lsb_release -i -s) - elif [ -e /etc/os-release ] ; then - distro=$(awk -F= '$1 == "ID" {print $2}' /etc/os-release) - fi - - printf '%s\n' "$distro" | LC_ALL=C tr '[:upper:]' '[:lower:]' -} - - use_alternative_debian_gpp10() { apt-get install software-properties-common -y add-apt-repository ppa:ubuntu-toolchain-r/test -y @@ -70,7 +51,6 @@ use_alternative_debian_gpp10() { update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 100 update-alternatives --set g++ /usr/bin/g++-10 } -# end merge from https://github.com/zyrolasting/pash/pull/2 install_deps_source_setup() { From d180d2fe4d7a484e3d4b593bcb12dcb4a8a8ba62 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 23 Jun 2022 16:21:03 -0400 Subject: [PATCH 68/90] Add package.yaml workflow --- .github/workflows/package.yaml | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 .github/workflows/package.yaml diff --git a/.github/workflows/package.yaml b/.github/workflows/package.yaml new file mode 100644 index 000000000..83a3acf1c --- /dev/null +++ b/.github/workflows/package.yaml @@ -0,0 +1,27 @@ +name: Packaging +on: + pull_request: + branches: + - main + - future + +jobs: + package-pash: + runs-on: ubuntu-latest + if: github.event.pull_request.draft == false + steps: + - name: Checkout code + uses: actions/checkout@v2 + with: + ref: ${{ github.event.pull_request.head.sha }} + - name: Make .deb + run: bash scripts/package/deploy.sh ubuntu "$GITHUB_HEAD_REF" deb + - name: Make .rpm + run: bash scripts/package/deploy.sh fedora "$GITHUB_HEAD_REF" rpm + - name: Make .tar.gz (Pacman) + run: bash scripts/package/deploy.sh archlinux "$GITHUB_HEAD_REF" pacman + - name: Upload Packages as Artifacts + uses: actions/upload-artifact@v3 + with: + name: Packages + path: scripts/package/output From 677cf540a709f46188144d3a1b75745f883b0e59 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 30 Jun 2022 14:42:07 -0400 Subject: [PATCH 69/90] Patch PASH_TOP after installation I've been told that fixing `pa.sh`'s fallback value for `PASH_TOP` would cause performance penalties. This patch updates `PASH_TOP` in `pa.sh` as a post-installation step for packages Verified with existing Docker images, but do not assume 100% compatibility with all package managers. Unsure if there are special rules out there that prevent an installer writing to a file extracted from a package. --- scripts/package/hooks/after-install/hook | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/package/hooks/after-install/hook b/scripts/package/hooks/after-install/hook index 45acf3964..963faa3d0 100755 --- a/scripts/package/hooks/after-install/hook +++ b/scripts/package/hooks/after-install/hook @@ -8,6 +8,10 @@ set -euo pipefail PASH_LOG=/var/log/pash PASH_TOP=/usr/lib/pash +# We don't want the entry point to set its own environment variable, +# because that would entail checks that cannot be optimized. +sed -i "s@^export[[:space:]]\{1,\}PASH_TOP=.*@export PASH_TOP='$PASH_TOP'@" "${PASH_TOP}/pa.sh" + post_install_current_target() { this_directory="$1" output_format="$2" From 20179f1870fd7b4d400d7a7855f5a40f5699a23d Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Wed, 15 Jun 2022 13:04:32 -0400 Subject: [PATCH 70/90] Prepare for GitHub Actions --- .github/workflows/package-build.yaml | 73 ++++++++++++++++++++ .github/workflows/package-gui.yaml | 35 ++++++++++ .github/workflows/package-pull-requests.yaml | 17 +++++ scripts/package/deploy.sh | 4 +- scripts/package/shell.sh | 18 ++++- scripts/package/tools/build | 2 +- scripts/package/tools/help | 5 +- scripts/package/tools/start | 11 +-- 8 files changed, 146 insertions(+), 19 deletions(-) create mode 100644 .github/workflows/package-build.yaml create mode 100644 .github/workflows/package-gui.yaml create mode 100644 .github/workflows/package-pull-requests.yaml diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml new file mode 100644 index 000000000..53cea8feb --- /dev/null +++ b/.github/workflows/package-build.yaml @@ -0,0 +1,73 @@ +# This is a reusable workflow, as defined by +# https://docs.github.com/en/actions/using-workflows/reusing-workflows +# +# Maintain this separately as the "meat" of the packaging work. Other +# files of form packaging-*.yaml call this one from a different +# perspective. +# +# To handle the documented limitations of reusable workflows, make +# sure the near the /!\ comments match your intended repository and +# user. + +name: Package for publication +on: + workflow_call: + inputs: + fallback_version: + type: string + required: true + git_ref: + type: string + required: true + deb_version: + type: string + required: false + rpm_version: + type: string + required: false + pacman_tgz_version: + type: string + required: false +defaults: + run: + shell: bash +jobs: + package-pash: + runs-on: ubuntu-latest + steps: + - name: Login to Docker Hub + uses: docker/login-action@v1 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + - name: Clone PaSh + uses: actions/checkout@v2 + with: + ref: ${{ inputs.git_ref }} + - name: Clone FPM + uses: actions/checkout@v2 + with: + repository: jordansissel/fpm + token: ${{ secrets.GH_PAT }} + path: scripts/package/fpm + # I generate a script because: + # 1. It holds job-specific information constant across package builds. + # 2. Job steps do not preserve Bash functions because they each get their own shell. + - name: Generate build script + run: | + cat >ci-build.sh <<'EOF' + ls + ./scripts/package/deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3" + EOF + chmod +x ci-build.sh + - name: Make .deb + run: ./ci-build.sh ubuntu '${{ inputs.deb_version }}' deb + - name: Make .rpm + run: ./ci-build.sh fedora '${{ inputs.rpm_version }}' rpm + - name: Make .tar.gz (Pacman) + run: ./ci-build.sh archlinux '${{ inputs.pacman_tgz_version }}' pacman + - name: Upload Packages as Artifacts + uses: actions/upload-artifact@v3 + with: + name: Packages + path: package/output diff --git a/.github/workflows/package-gui.yaml b/.github/workflows/package-gui.yaml new file mode 100644 index 000000000..49ef5950c --- /dev/null +++ b/.github/workflows/package-gui.yaml @@ -0,0 +1,35 @@ +# For use in GitHub's Actions GUI +# https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/ +name: On-demand Packaging +on: + workflow_dispatch: + inputs: + fallback_version: + # TODO: The comment in ./package-pull-requests.yaml + # impacts whether this is good for the user to read. + description: 'What fallback version string can I use in any package format? Use 0.0.0 for throwaway prototypes. Compatibility not guarenteed for all package formats.' + default: '0.0.0' + required: true + git_ref: + description: 'Git reference, like a branch name or commit hash' + default: 'main' + required: true + deb_version: + description: "What version string should I use in .deb format packages, if not the fallback?" + required: false + rpm_version: + description: '.rpm?' + required: false + pacman_tgz_version: + description: "pacman's .tar.gz?" + required: false +jobs: + build-packages: + uses: zyrolasting/pash/.github/workflows/package-build.yaml@main + with: + fallback_version: ${{ inputs.fallback_version }} + deb_version: ${{ inputs.deb_version }} + rpm_version: ${{ inputs.rpm_version }} + pacman_tgz_version: ${{ inputs.pacman_tgz_version }} + git_ref: ${{ inputs.git_ref }} + secrets: inherit diff --git a/.github/workflows/package-pull-requests.yaml b/.github/workflows/package-pull-requests.yaml new file mode 100644 index 000000000..cb8d96e3d --- /dev/null +++ b/.github/workflows/package-pull-requests.yaml @@ -0,0 +1,17 @@ +name: Package Public Pull Requests (always 0.0.0) +on: + pull_request: + branches: + - main + - future +jobs: + build-packages: + if: ${{ github.event.pull_request.draft == false }} + uses: zyrolasting/pash/.github/workflows/package-build.yaml@main + with: + # TODO: consider using version files in the repository + # 0.0.0 has been tested to work in Fedora, Ubuntu, Debian, and Arch, + # which does not imply compatibility with all package formats. + fallback_version: '0.0.0' + git_ref: ${{ github.event.pull_request.head.sha }} + secrets: inherit diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index 9decfa5e1..578581f95 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -45,7 +45,7 @@ else output_format="$3" # Perform a one-off build for the package. - "$mydirname/shell.sh" -ic "build '$target_version' '$output_format'" + "$mydirname/shell.sh" "$target_version" "$output_format" # It's important to use Docker hub here because we want to verify # PaSh's behavior against other people's work, namely official OS @@ -54,8 +54,6 @@ else # Invoke self in the container. docker run \ - --interactive \ - --tty \ --rm \ --volume "${mydirname}:/package" \ "$image" \ diff --git a/scripts/package/shell.sh b/scripts/package/shell.sh index e4b29ce4c..a9104ba75 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/shell.sh @@ -5,6 +5,7 @@ here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" PASH_TOP=$(readlink -f "$here/../..") # We need FPM's source code to build a key Docker image. +printf "COMPARE\n >$here\n >%s\n " "$(pwd)" if [ ! -d "$here/fpm" ]; then git clone --depth 1 git@github.com:jordansissel/fpm.git "$here/fpm" fi @@ -17,16 +18,27 @@ fi mkdir -p "$here/output" main() { + local docker_interactive_flags='' + local entrypoint_interactive_flags='' + + if [[ $- == *i* ]]; then + set -x + docker_interactive_flags='--interactive --tty' + entrypoint_interactive_flags='--rcfile' + set +x + fi + docker run \ --entrypoint /bin/bash \ - --interactive \ - --tty \ --rm \ --user "$(id -u):$(id -g)" \ --volume "$here/output:/out" \ --volume "$here/tools:/tools" \ --volume "$PASH_TOP:/src" \ - fpm --rcfile /tools/start "$@" + $docker_interactive_flags \ + fpm \ + $entrypoint_interactive_flags \ + /tools/start "$@" } # Only enter if script was not sourced diff --git a/scripts/package/tools/build b/scripts/package/tools/build index c2f08e30b..1f4f7c532 100755 --- a/scripts/package/tools/build +++ b/scripts/package/tools/build @@ -53,7 +53,7 @@ print_dotfpm() { print_dotfpm_rpm ;; *) - printf "Unsupported format: '%s'" "output_format" 1>&2 + printf "Unsupported format: '%s'\n" "$output_format" 1>&2 exit 1 ;; esac diff --git a/scripts/package/tools/help b/scripts/package/tools/help index 2a224e1c4..62df6a49f 100644 --- a/scripts/package/tools/help +++ b/scripts/package/tools/help @@ -15,6 +15,5 @@ With regards to FPM's CLI: - VERSION is the value of --version. - OUTPUT_FORMAT is some value of -t/--output-type -Ownership of result goes to host user and group, not the container's. - -Run `build_help` to repeat this message. +Ownership of output files goes to host user and group, not the +container's. diff --git a/scripts/package/tools/start b/scripts/package/tools/start index 693f565df..72a992c21 100755 --- a/scripts/package/tools/start +++ b/scripts/package/tools/start @@ -1,13 +1,6 @@ #! /usr/bin/env bash -. ~/.bashrc - -build_help() { - cat /tools/help -} - export PASH_TOP="/src" -export PS1='pkg> ' export PATH="/tools:$PATH" -export -f build_help -build_help +# Non-interactive shells are used to build packages, namely via CI. +[[ $- == *i* ]] || build "$@" From eb629de1ef3a4ad0aed219b55268d4b306b9a9a5 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 11 Jul 2022 15:30:46 -0400 Subject: [PATCH 71/90] Try all branches for gui packaging workflow --- .github/workflows/package-gui.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/package-gui.yaml b/.github/workflows/package-gui.yaml index 49ef5950c..fa88c62e8 100644 --- a/.github/workflows/package-gui.yaml +++ b/.github/workflows/package-gui.yaml @@ -3,6 +3,8 @@ name: On-demand Packaging on: workflow_dispatch: + branches: + - '*' inputs: fallback_version: # TODO: The comment in ./package-pull-requests.yaml From bd8a9dd9cc20caa8c7fd1f5f24ca7a4bd62ff8d0 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 11 Jul 2022 16:03:45 -0400 Subject: [PATCH 72/90] Use different wd --- .github/workflows/package-build.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 53cea8feb..a10a08b44 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -56,8 +56,8 @@ jobs: - name: Generate build script run: | cat >ci-build.sh <<'EOF' - ls - ./scripts/package/deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3" + cd scripts/package + ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3" EOF chmod +x ci-build.sh - name: Make .deb From e7966b32efe8e92c3b41ed6da869d15d6955aab3 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 11 Jul 2022 16:06:23 -0400 Subject: [PATCH 73/90] Use consistent working directory path for make --- scripts/package/shell.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/package/shell.sh b/scripts/package/shell.sh index a9104ba75..261dca48d 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/shell.sh @@ -5,14 +5,13 @@ here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" PASH_TOP=$(readlink -f "$here/../..") # We need FPM's source code to build a key Docker image. -printf "COMPARE\n >$here\n >%s\n " "$(pwd)" if [ ! -d "$here/fpm" ]; then git clone --depth 1 git@github.com:jordansissel/fpm.git "$here/fpm" fi # Make a Docker image that knows about building packages. if ! docker image inspect fpm 2>&1 >/dev/null; then - make -C fpm docker-release-everything + make -C "$here/fpm" docker-release-everything fi mkdir -p "$here/output" From e3c482dae69cca8655b7328776242ceb46e0eea6 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 11 Jul 2022 16:13:48 -0400 Subject: [PATCH 74/90] Try removing --squash from fpm Makefile --- scripts/package/shell.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/package/shell.sh b/scripts/package/shell.sh index 261dca48d..ff4f87fce 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/shell.sh @@ -7,6 +7,7 @@ PASH_TOP=$(readlink -f "$here/../..") # We need FPM's source code to build a key Docker image. if [ ! -d "$here/fpm" ]; then git clone --depth 1 git@github.com:jordansissel/fpm.git "$here/fpm" + sed -i 's/--squash//g' "$here/fpm/Makefile" fi # Make a Docker image that knows about building packages. From 6c241f1d8360c780ae5337c8ce090c338ecf772f Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 25 Jul 2022 12:39:08 -0400 Subject: [PATCH 75/90] Upgrade docker login to v2 and don't logout at end of job --- .github/workflows/package-build.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index a10a08b44..c1fc038d3 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -36,10 +36,11 @@ jobs: runs-on: ubuntu-latest steps: - name: Login to Docker Hub - uses: docker/login-action@v1 + uses: docker/login-action@v2 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} + logout: false - name: Clone PaSh uses: actions/checkout@v2 with: From 02019b4a6280e78c25cc4eff91ec75bac0569096 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 25 Jul 2022 12:47:07 -0400 Subject: [PATCH 76/90] Try manual login in deploy.sh --- scripts/package/deploy.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index 578581f95..5efcc9113 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -47,6 +47,11 @@ else # Perform a one-off build for the package. "$mydirname/shell.sh" "$target_version" "$output_format" + if [ -n "$DOCKERHUB_PASSWORD" ]; then + echo "$DOCKERHUB_PASSWORD" | \ + docker login -u "$DOCKERHUB_USERNAME" --password-stdin + fi + # It's important to use Docker hub here because we want to verify # PaSh's behavior against other people's work, namely official OS # images. From c1c4683a86de15b6b31cba24b49a689aa7ede668 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 25 Jul 2022 12:52:18 -0400 Subject: [PATCH 77/90] Delete trailing whitespace --- scripts/package/deploy.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index 5efcc9113..f9b16c5c8 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -31,7 +31,7 @@ if [ -f /.dockerenv ]; then export PASH_TOP=/usr/lib/pash "$mydirname/fire-hook.sh" "before-test" "$output_format" "$target_version" - + # Basic example must always work, no matter where we're going. [ -d "$PASH_TOP" ] test_script="${PASH_TOP}/evaluation/intro/hello-world.sh" From d042c5f2b6e16c9800891248cffbf33512f626c1 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Mon, 25 Jul 2022 13:00:39 -0400 Subject: [PATCH 78/90] Alternative workflow --- .github/workflows/package-build.yaml | 28 ++++++++-------------------- scripts/package/deploy.sh | 5 ----- 2 files changed, 8 insertions(+), 25 deletions(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index c1fc038d3..0ba0ea3dd 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -35,12 +35,6 @@ jobs: package-pash: runs-on: ubuntu-latest steps: - - name: Login to Docker Hub - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - logout: false - name: Clone PaSh uses: actions/checkout@v2 with: @@ -51,22 +45,16 @@ jobs: repository: jordansissel/fpm token: ${{ secrets.GH_PAT }} path: scripts/package/fpm - # I generate a script because: - # 1. It holds job-specific information constant across package builds. - # 2. Job steps do not preserve Bash functions because they each get their own shell. - - name: Generate build script + - name: Build packages run: | - cat >ci-build.sh <<'EOF' cd scripts/package - ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3" - EOF - chmod +x ci-build.sh - - name: Make .deb - run: ./ci-build.sh ubuntu '${{ inputs.deb_version }}' deb - - name: Make .rpm - run: ./ci-build.sh fedora '${{ inputs.rpm_version }}' rpm - - name: Make .tar.gz (Pacman) - run: ./ci-build.sh archlinux '${{ inputs.pacman_tgz_version }}' pacman + build() { + ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3"; + } + echo "$DOCKERHUB_TOKEN | docker login -u "$DOCKERHUB_TOKEN" --password-stdin; + build ubuntu '${{ inputs.deb_version }}' deb; + build fedora '${{ inputs.rpm_version }}' rpm; + build archlinux '${{ inputs.pacman_tgz_version }}' pacman; - name: Upload Packages as Artifacts uses: actions/upload-artifact@v3 with: diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index f9b16c5c8..17b4f4b8a 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -47,11 +47,6 @@ else # Perform a one-off build for the package. "$mydirname/shell.sh" "$target_version" "$output_format" - if [ -n "$DOCKERHUB_PASSWORD" ]; then - echo "$DOCKERHUB_PASSWORD" | \ - docker login -u "$DOCKERHUB_USERNAME" --password-stdin - fi - # It's important to use Docker hub here because we want to verify # PaSh's behavior against other people's work, namely official OS # images. From 267122920f34a564b080748c85d54c9faa5a64b1 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 15:57:16 -0400 Subject: [PATCH 79/90] Use the right workflow --- .github/workflows/package-gui.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/package-gui.yaml b/.github/workflows/package-gui.yaml index fa88c62e8..c07f6f08b 100644 --- a/.github/workflows/package-gui.yaml +++ b/.github/workflows/package-gui.yaml @@ -27,7 +27,7 @@ on: required: false jobs: build-packages: - uses: zyrolasting/pash/.github/workflows/package-build.yaml@main + uses: zyrolasting/pash/.github/workflows/package-build.yaml@fpm with: fallback_version: ${{ inputs.fallback_version }} deb_version: ${{ inputs.deb_version }} From ef6e7281309be3a18070f6697505fd40e3b219ba Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 21:08:13 -0400 Subject: [PATCH 80/90] Fix docker command in package-build.yaml --- .github/workflows/package-build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 0ba0ea3dd..227c89ea2 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -51,7 +51,7 @@ jobs: build() { ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3"; } - echo "$DOCKERHUB_TOKEN | docker login -u "$DOCKERHUB_TOKEN" --password-stdin; + echo "$DOCKERHUB_TOKEN" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin; build ubuntu '${{ inputs.deb_version }}' deb; build fedora '${{ inputs.rpm_version }}' rpm; build archlinux '${{ inputs.pacman_tgz_version }}' pacman; From bdf6367635d3adb5e67ca1737098909eb59c421a Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 21:11:46 -0400 Subject: [PATCH 81/90] Add --username --- .github/workflows/package-build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 227c89ea2..939429322 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -51,7 +51,7 @@ jobs: build() { ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3"; } - echo "$DOCKERHUB_TOKEN" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin; + echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin; build ubuntu '${{ inputs.deb_version }}' deb; build fedora '${{ inputs.rpm_version }}' rpm; build archlinux '${{ inputs.pacman_tgz_version }}' pacman; From ec7a45cc501ef2c57d61bb5ae7b4c36c510fe2e4 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 21:14:36 -0400 Subject: [PATCH 82/90] Circulate docker credentials --- .github/workflows/package-build.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 939429322..c9fdec6e0 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -51,6 +51,8 @@ jobs: build() { ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3"; } + DOCKERHUB_TOKEN='${{ secrets.DOCKERHUB_TOKEN }}' + DOCKERHUB_USERNAME='${{ secrets.DOCKERHUB_USERNAME }}' echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin; build ubuntu '${{ inputs.deb_version }}' deb; build fedora '${{ inputs.rpm_version }}' rpm; From 4511bd33299f1e43c47412e5949b4df7b7e63c58 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 21:18:39 -0400 Subject: [PATCH 83/90] Remove --squash --- .github/workflows/package-build.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index c9fdec6e0..46ad6b024 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -47,10 +47,11 @@ jobs: path: scripts/package/fpm - name: Build packages run: | - cd scripts/package build() { ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3"; } + cd scripts/package + sed -i 's/--squash//g' ./fpm/Makefile DOCKERHUB_TOKEN='${{ secrets.DOCKERHUB_TOKEN }}' DOCKERHUB_USERNAME='${{ secrets.DOCKERHUB_USERNAME }}' echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin; From 1931a48c342b06952e6786b03afa1fd9061f38fb Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 21:31:44 -0400 Subject: [PATCH 84/90] Fix yaml --- .github/workflows/package-build.yaml | 2 +- .github/workflows/package-gui.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 46ad6b024..4659ac98c 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -62,4 +62,4 @@ jobs: uses: actions/upload-artifact@v3 with: name: Packages - path: package/output + path: scripts/package/output diff --git a/.github/workflows/package-gui.yaml b/.github/workflows/package-gui.yaml index c07f6f08b..bfd44a26f 100644 --- a/.github/workflows/package-gui.yaml +++ b/.github/workflows/package-gui.yaml @@ -9,8 +9,8 @@ on: fallback_version: # TODO: The comment in ./package-pull-requests.yaml # impacts whether this is good for the user to read. - description: 'What fallback version string can I use in any package format? Use 0.0.0 for throwaway prototypes. Compatibility not guarenteed for all package formats.' - default: '0.0.0' + description: 'What fallback version string can I use for some package format? Compatibility not guarenteed for all package formats.' + default: '0.0.1' required: true git_ref: description: 'Git reference, like a branch name or commit hash' From 1de0316a0c80bb55913122fee9a47b6915dcf0d5 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 21:35:07 -0400 Subject: [PATCH 85/90] Remove excess quotes --- .github/workflows/package-build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 4659ac98c..797abe583 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -48,7 +48,7 @@ jobs: - name: Build packages run: | build() { - ./deploy.sh "$1" "${2:-'${{ inputs.fallback_version }}'}" "$3"; + ./deploy.sh "$1" "${2:-${{ inputs.fallback_version }}}" "$3"; } cd scripts/package sed -i 's/--squash//g' ./fpm/Makefile From 5131b5981c76070815fbdfdd6a9a3a872c009057 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 22:42:26 -0400 Subject: [PATCH 86/90] Prep for shipment --- .github/workflows/package-build.yaml | 1 - docs/tooling/packaging.md | 35 ++++++++++++++++---------- scripts/package/{shell.sh => build.sh} | 1 + scripts/package/deploy.sh | 2 +- scripts/package/tools/help | 19 -------------- 5 files changed, 24 insertions(+), 34 deletions(-) rename scripts/package/{shell.sh => build.sh} (96%) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 797abe583..4330735b5 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -51,7 +51,6 @@ jobs: ./deploy.sh "$1" "${2:-${{ inputs.fallback_version }}}" "$3"; } cd scripts/package - sed -i 's/--squash//g' ./fpm/Makefile DOCKERHUB_TOKEN='${{ secrets.DOCKERHUB_TOKEN }}' DOCKERHUB_USERNAME='${{ secrets.DOCKERHUB_USERNAME }}' echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin; diff --git a/docs/tooling/packaging.md b/docs/tooling/packaging.md index 546e62646..559581767 100644 --- a/docs/tooling/packaging.md +++ b/docs/tooling/packaging.md @@ -10,18 +10,17 @@ replacement for deployments depending on `install.sh`, [eef]: https://github.com/docker/cli/blob/master/experimental/README.md ``` - shell.sh +build.sh VERSION [OUTPUT_FORMAT ...] ``` -To use this feature, [enable experimental features][eef] in Docker. +Creates new package files using the repository's current content, each +with mode 440. All output files appear in `scripts/package/output`. -`shell.sh` enters a Docker container for building packages. Run -without arguments to enter a REPL and print usage information for the -`build` command. +Command-line arguments correspond to those found in [FPM's +CLI](https://fpm.readthedocs.io/en/v1.14.2/cli-reference.html): -Since `shell.sh` is just a Bash, you can pass it commands. For -example, `./shell.sh -ic 'build 0.0.1 deb'` launches an interactive -shell that builds a Debian package using version `0.0.1`. + - `VERSION` is the value of `--version`. + - `OUTPUT_FORMAT` is some value for `--output-type` ## Testing Packages @@ -30,11 +29,11 @@ shell that builds a Debian package using version `0.0.1`. deploy.sh IMAGE VERSION FORMAT ``` -`deploy.sh` runs `./shell.sh -ic 'build VERSION FORMAT'`, then -installs PaSh in a Docker container based on a `IMAGE` from DockerHub. +`deploy.sh` runs `build.sh VERSION FORMAT`, then installs PaSh in a +Docker container based on a `IMAGE` from DockerHub. -Here's an example that builds a `pacman`/`.tar.gz` package for -deployment in Arch, a `.deb` package for Debian and Ubuntu, and an RPM +In this example, `deploy.sh` installs a `pacman`/`.tar.gz` package +deploys in Arch, a `.deb` package for Debian and Ubuntu, and an RPM package for Fedora. ``` @@ -52,4 +51,14 @@ $ deploy.sh fedora $version rpm iterate.sh IMAGE FORMAT ``` -Run `deploy.sh IMAGE 0.0.1 FORMAT`, then prompt to repeat. +Runs `deploy.sh IMAGE 0.0.1 FORMAT`, then prompts to repeat. + + +## GitHub Actions + +The source code repository defines workflows for GitHub actions. The +workflows for packaging all use `deploy.sh` to build and run PaSh's +"Hello, World" example. + +**On-Demand Packaging** is the only workflow that can be directly +used in the GitHub GUI. diff --git a/scripts/package/shell.sh b/scripts/package/build.sh similarity index 96% rename from scripts/package/shell.sh rename to scripts/package/build.sh index ff4f87fce..089980d2f 100755 --- a/scripts/package/shell.sh +++ b/scripts/package/build.sh @@ -12,6 +12,7 @@ fi # Make a Docker image that knows about building packages. if ! docker image inspect fpm 2>&1 >/dev/null; then + sed -i 's/--squash//g' "$here/fpm/Makefile" make -C "$here/fpm" docker-release-everything fi diff --git a/scripts/package/deploy.sh b/scripts/package/deploy.sh index 17b4f4b8a..40bd7f559 100755 --- a/scripts/package/deploy.sh +++ b/scripts/package/deploy.sh @@ -45,7 +45,7 @@ else output_format="$3" # Perform a one-off build for the package. - "$mydirname/shell.sh" "$target_version" "$output_format" + "$mydirname/build.sh" "$target_version" "$output_format" # It's important to use Docker hub here because we want to verify # PaSh's behavior against other people's work, namely official OS diff --git a/scripts/package/tools/help b/scripts/package/tools/help index 62df6a49f..e69de29bb 100644 --- a/scripts/package/tools/help +++ b/scripts/package/tools/help @@ -1,19 +0,0 @@ -This is a shell for FPM's Docker container. - - https://fpm.readthedocs.io/ - -/src - Mounted to host's $PASH_TOP -/tools - Mounted to host's $PASH_TOP/scripts/package/tools -/out - Mounted to host's $PASH_TOP/scripts/package/output - -Write a new package with mode 440 using this command. - - build VERSION [OUTPUT_FORMAT ...] - -With regards to FPM's CLI: - - - VERSION is the value of --version. - - OUTPUT_FORMAT is some value of -t/--output-type - -Ownership of output files goes to host user and group, not the -container's. From 41fb1a955c7f8114658fc07f8ee66ad3cda85081 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 22:48:08 -0400 Subject: [PATCH 87/90] Update packaging.md Add GitHub Actions paragraph content --- docs/tooling/packaging.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/tooling/packaging.md b/docs/tooling/packaging.md index 559581767..1b2e3b37c 100644 --- a/docs/tooling/packaging.md +++ b/docs/tooling/packaging.md @@ -1,11 +1,10 @@ # Packaging PaSh -`scripts/package` packages PaSh into various formats. It is a suitable +`scripts/package` packages PaSh into various formats. The software inside is a suitable replacement for deployments depending on `install.sh`, `setup-pash.sh`, `up.sh`, `distro-deps.sh`, and/or `pkg.sh`. - ## Building Packages [eef]: https://github.com/docker/cli/blob/master/experimental/README.md @@ -58,7 +57,11 @@ Runs `deploy.sh IMAGE 0.0.1 FORMAT`, then prompts to repeat. The source code repository defines workflows for GitHub actions. The workflows for packaging all use `deploy.sh` to build and run PaSh's -"Hello, World" example. +"Hello, World" example. One workflow responds to new pull requests. **On-Demand Packaging** is the only workflow that can be directly -used in the GitHub GUI. +used in the GitHub GUI. You can find it under the **Actions** tab. + +![image](https://user-images.githubusercontent.com/1312121/181671950-89ec5f57-5b9f-4fdb-90a2-6d1099257ad1.png) + +Click "Run Workflow" and fill out the form. The form defines what commit of PaSh gets packaged, and each package format may have its own version string. From 0ba9784d49bdfb9cf78ebe0da822324e5bed027b Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 22:52:00 -0400 Subject: [PATCH 88/90] Add warning --- docs/tooling/packaging.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/tooling/packaging.md b/docs/tooling/packaging.md index 1b2e3b37c..a6d26d906 100644 --- a/docs/tooling/packaging.md +++ b/docs/tooling/packaging.md @@ -15,6 +15,10 @@ build.sh VERSION [OUTPUT_FORMAT ...] Creates new package files using the repository's current content, each with mode 440. All output files appear in `scripts/package/output`. +**Clean up junk files and new test results!** The scripts use a +reasonable ignore list, but otherwise defaults to including files +within the PaSh repository. + Command-line arguments correspond to those found in [FPM's CLI](https://fpm.readthedocs.io/en/v1.14.2/cli-reference.html): From e30d33a928d48a5b3e1af027ea19b842dc2dd717 Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 22:58:41 -0400 Subject: [PATCH 89/90] Reword docs --- docs/tooling/packaging.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/tooling/packaging.md b/docs/tooling/packaging.md index a6d26d906..50023b0db 100644 --- a/docs/tooling/packaging.md +++ b/docs/tooling/packaging.md @@ -32,12 +32,16 @@ CLI](https://fpm.readthedocs.io/en/v1.14.2/cli-reference.html): deploy.sh IMAGE VERSION FORMAT ``` -`deploy.sh` runs `build.sh VERSION FORMAT`, then installs PaSh in a -Docker container based on a `IMAGE` from DockerHub. +Performs a test deployment of PaSh, as follows: -In this example, `deploy.sh` installs a `pacman`/`.tar.gz` package -deploys in Arch, a `.deb` package for Debian and Ubuntu, and an RPM -package for Fedora. +1. Run `build.sh VERSION FORMAT` +2. Start Docker container based on `docker pull IMAGE` +3. Install the package from Step 1 in the container from Step 2. +4. Run the newly-installed `pa.sh` against a bundled `evaluation/intro/hello-world.sh` example. + +Here, `deploy.sh` installs a `pacman`/`.tar.gz` package deploys in +Arch, a `.deb` package for Debian and Ubuntu, and an RPM package for +Fedora. ``` $ version=0.0.1 From f61e80f135f57c26a185f23a20b41f545afd07ab Mon Sep 17 00:00:00 2001 From: Sage Gerard Date: Thu, 28 Jul 2022 23:04:52 -0400 Subject: [PATCH 90/90] Use sed for both clone points --- .github/workflows/package-build.yaml | 1 + scripts/package/build.sh | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/package-build.yaml b/.github/workflows/package-build.yaml index 4330735b5..72d4d0107 100644 --- a/.github/workflows/package-build.yaml +++ b/.github/workflows/package-build.yaml @@ -51,6 +51,7 @@ jobs: ./deploy.sh "$1" "${2:-${{ inputs.fallback_version }}}" "$3"; } cd scripts/package + sed -i 's/--squash//g' fpm/Makefile DOCKERHUB_TOKEN='${{ secrets.DOCKERHUB_TOKEN }}' DOCKERHUB_USERNAME='${{ secrets.DOCKERHUB_USERNAME }}' echo "$DOCKERHUB_TOKEN" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin; diff --git a/scripts/package/build.sh b/scripts/package/build.sh index 089980d2f..ff4f87fce 100755 --- a/scripts/package/build.sh +++ b/scripts/package/build.sh @@ -12,7 +12,6 @@ fi # Make a Docker image that knows about building packages. if ! docker image inspect fpm 2>&1 >/dev/null; then - sed -i 's/--squash//g' "$here/fpm/Makefile" make -C "$here/fpm" docker-release-everything fi