Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add lock-files script #17

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
203 changes: 203 additions & 0 deletions lock-files.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
#!/usr/bin/env bash
#
# Create and manipulate the lock files.
#
# We use lock files `Cargo-recent.lock` and `Cargo-minimal.lock` to pin
# dependencies and check specific versions in CI.
#
# Shellcheck can't search dynamic paths
# shellcheck source=/dev/null

set -u

main() {
set_globals
assert_cmds
handle_command_line_args "$@"
}

set_globals() {
# The directory of this script.
script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# Used below by `say`.
script=${0##*/}

# Used below by `verbose_say`.
flag_verbose=false

# Sourcing this file requires `flag_verbose` to be set.
. "$script_dir/stdlib.sh"

# Environment sanity checks
assert_nz "$HOME" "\$HOME is undefined"
assert_nz "$0" "\$0 is undefined"

# Make all cargo invocations verbose.
export CARGO_TERM_VERBOSE=true

recent="Cargo-recent.lock"
minimal="Cargo-minimal.lock"

msrv="1.63.0"
}

handle_command_line_args() {
local _no_args=false
local _help=false
local _create=false
local _msrv=""
local _update=false

if [ $# -eq 0 ]; then
_no_args=true
fi

local _arg
for _arg in "$@"; do
case "${_arg%%=*}" in
create )
_create=true
;;

update )
_update=true
;;

--msrv )
if is_value_arg "$_arg" "--msrv"; then
_msrv="$(get_value_arg "$_arg")"
else
say_err "the --msrv option requires a toolchain version argument"
print_help
exit 1
fi

;;

-h | --help )
_help=true
;;

--verbose)
# verbose is a global flag
flag_verbose=true
;;

*)
echo "Unknown argument '$_arg', displaying usage:"
echo "${_arg%%=*}"
_help=true
;;

esac

done

if [ "$_create" = true ]; then
if [ -z "$_msrv" ]; then
msrv="$_msrv"
fi

verbose_say "Creating lock files, MSRV: $msrv"
create
fi

if [ "$_update" = true ]; then
update
verbose_say "Your git index will now be dirty if lock file update is required"
fi

if [ "$_help" = true ]; then
print_help
exit 0
fi

if [ "$_no_args" = true ]; then
verbose_say "no option supplied, defaulting to update"
update
fi
}

is_value_arg() {
local _arg="$1"
local _name="$2"

echo "$_arg" | grep -q -- "$_name="
return $?
}

get_value_arg() {
local _arg="$1"

echo "$_arg" | cut -f2 -d=
}

# Creates the minimal and recent lock files.
#
# If this function fails you may want to be lazy and just duplicate `Cargo.lock`
# as the minimal and recent lock files.
create() {
# Attempt to create a minimal lock file.
rm --force Cargo.lock > /dev/null
cargo +nightly check --all-features -Z minimal-versions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 65a74a1:

It seems like this script needs this line to fail in order to be useful. But I believe this fails on every single one of our repos becaue our minimal dependencies themselves don't specify their minimal dependencies correctly.

Though I guess I haven't tried it for some of the new leaf crates..

Copy link
Member Author

@tcharding tcharding Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this script needs this line to fail in order to be useful.

What do you mean by this? My main idea behind this script was that if this succeeds then win and one doesn't have to have the -Z command in shell history or in mind. And if it this fails the script documents my lazy 'just duplicate recent'. But since the script lives in an obscure repo is this even useful, I don't know?

I used -Z minimal-versions recently and it worked, to my surprise (maybe in bech32, but not sure).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used -Z minimal-versions recently and it worked, to my surprise (maybe in bech32, but not sure).

Ooo, nice.

I think the MSRV update to 1.63 solved a lot of our dependency compatibility issues.

And if it this fails the script documents my lazy 'just duplicate recent'

I don't think it's too useful as documentation because it's inside a script in a fairly obscure repo.

I also think we should move away from "duplicate the recent lockfile" to "just don't have a minimal lockfile". This will clearly signal which repos still need somebody to do the hard work of running -Z minimal then patching everything up.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, if the script were to try using -Zminimal, and on failure, output a warning and not produce the lockfile, that'd be pretty cool.

I guess we also need to check that the "recent" lockfile produced by cargo update works with MSRV. Again, in my experience it almost never does. And we need to hard fail if neither lockfile can be produced.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good ideas, will re-spin.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the script doesn't fail correctly in workspace because cargo check in a workspace does not exit with non-zero if the last crate builds successfully, I couldn't find a way to make it do so.

I also think we should move away from "duplicate the recent lockfile" to "just don't have a minimal lockfile".

Leaving this for another day.

need_ok "failed to build with -Z minimial-versions, you might have to use a recent lock file for minimal"

# We only want to create the minimal lock file if we can build with current MSRV.
cargo "+$msrv" --locked check --all-features
need_ok "failed to build with minimal lock file and MSRV $_msrv"
cp Cargo.lock "$minimal"

# If that worked we can create a recent lock file.
cargo update
cp Cargo.lock "$recent"
}

# Updates the minimal and recent lock files.
update() {
for file in "$minimal" "$recent"; do
cp --force "$file" Cargo.lock
cargo check
cp --force Cargo.lock "$file"
done
}

assert_cmds() {
need_cmd cat
need_cmd cp
need_cmd cargo
}

say() {
echo "$script: $1"
}

say_err() {
say "$1" >&2
}

verbose_say() {
if [ "$flag_verbose" = true ]; then
say "$1"
fi
}

print_help() {
cat <<EOF
Usage: $script [OPITON] [COMMAND]

COMMAND:

create Create new minimal and recent lock files.
update Update the minimal and recent lock files (default).

OPTION:

--msrv=version The Rust toolchain version to use as MSRV.
--verbose Enable verbose output.
--help, -h Display this help information.
EOF
}

# Main script
main "$@"
56 changes: 56 additions & 0 deletions stdlib.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# rust-bitcoin stdlib for Bash (of sorts).
#
# Much of the code here was originally stolen from the old `rustup.sh` script
# https://github.com/rust-lang-deprecated/rustup.sh/blob/master/rustup.sh
#
# If you have never read the comments at the top of that file, consider it, they are gold.
#
# No shebang, this file should not be executed.
# shellcheck disable=SC2148
#
# Disable because `flag_verbose` is referenced but not assigned, however we check it is non-zero.
# shellcheck disable=SC2154

set -u

err() {
echo "$1" >&2
exit 1
}

need_cmd() {
if ! command -v "$1" > /dev/null 2>&1
then err "need '$1' (command not found)"
fi
}

need_ok() {
if [ $? != 0 ]; then err "$1"; fi
}

assert_nz() {
if [ -z "$1" ]; then err "assert_nz $2"; fi
}

# Run a command that should never fail. If the command fails execution
# will immediately terminate with an error showing the failing
# command.
ensure() {
"$@"
need_ok "command failed: $*"
}

# This is just for indicating that commands' results are being
# intentionally ignored. Usually, because it's being executed
# as part of error handling.
ignore() {
run "$@"
}

# Assert that we have a nightly Rust toolchain installed.
need_nightly() {
cargo_ver=$(cargo --version)
if echo "$cargo_ver" | grep -q -v nightly; then
err "Need a nightly compiler; have $(cargo --version) (use RUSTUP_TOOLCHAIN=+nightly cmd)"
fi
}
105 changes: 105 additions & 0 deletions template.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
#
# Bash shell script template.
#
# Shellcheck can't search dynamic paths
# shellcheck source=/dev/null

# Note we don't use `set -x` because error handling is done manually.
# If you use pipes you may want to use `set -euxo pipefail` instead.
set -u

main() {
set_globals
assert_cmds
handle_command_line_args "$@"
}

set_globals() {
# The git repository where script is run from.
# repo_dir=$(git rev-parse --show-toplevel)

# The directory of this script.
script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# Used below by `say`.
script=${0##*/}

# Used below by `verbose_say`.
flag_verbose=false

# Sourcing this file requires `flag_verbose` to be set.
. "$script_dir/stdlib.sh"

# Environment sanity checks
assert_nz "$HOME" "\$HOME is undefined"
assert_nz "$0" "\$0 is undefined"

# Make all cargo invocations verbose.
export CARGO_TERM_VERBOSE=true
}

handle_command_line_args() {
local _help=false

local _arg
for _arg in "$@"; do
case "${_arg%%=*}" in
-h | --help )
_help=true
;;

--verbose)
# verbose is a global flag
flag_verbose=true
;;

*)
echo "Unknown argument '$_arg', displaying usage:"
echo "${_arg%%=*}"
_help=true
;;

esac

done

if [ "$_help" = true ]; then
print_help
exit 0
fi

verbose_say "Enabled verbose output"
}

assert_cmds() {
need_cmd cat
}

say() {
echo "$script: $1"
}

say_err() {
say "$1" >&2
}

verbose_say() {
if [ "$flag_verbose" = true ]; then
say "$1"
fi
}

print_help() {
cat <<EOF
Usage: $script [--verbose]

Options:

--verbose, Enable verbose output
--help, -h Display usage information
EOF
}

# Main script
main "$@"