Skip to content

Commit

Permalink
fix: better array parsing, small testing framework, asdf list fix (#561)
Browse files Browse the repository at this point in the history
Fixes the `docker.sh` array parsing to properly handle an array not
being set. Extracts aforementioned logic to be in a generic `yaml.sh`
library in hopes it can be more usable in the future, since moving to Go
isn't happening yet.

As part of that, I felt we needed some tests here. So, I wrote a _very_
simple testing framework to call `*_test.sh` scripts, much like Go. If
the text returns a non-zero exit code, it is considered failed. Added
some tests for the new logic.

I will add some documentation for this in the PR shortly, but I wanted to
get it out for comments (since it is simple and described here).

Fixed an issue where newer asdf versions seem to put `*` in front of
versions as well as a space, so our original `^` regex no longer
works.
  • Loading branch information
jaredallard authored Jun 16, 2023
1 parent 327de32 commit b96f4c0
Show file tree
Hide file tree
Showing 8 changed files with 127 additions and 15 deletions.
9 changes: 9 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,15 @@ This project uses devbase, which exposes the following build tooling: [devbase/d

<!-- <</Stencil::Block>> -->

### Testing Bash Scripts

This project currently has a very simple bash testing framework that will likely evolve to use
[bats](https://github.com/bats-core/bats-core) in the future. For now, you can add a test anywhere
as long as it is suffixed with `_test.sh` and it will be run when you run `make test`.

A test is called with `bash` and should return a non-zero exit code if the test fails. For an example
of a test, see [shell/lib/yaml_test.sh](shell/lib/yaml_test.sh).

### Replacing a Remote Version of the a Package with Local Version

_This is only applicable if this repository exposes a public package_.
Expand Down
5 changes: 0 additions & 5 deletions bootstrap.lock

This file was deleted.

16 changes: 16 additions & 0 deletions scripts/bash-test-runner.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
# Runs all files with _test.sh at the end of the filename
set -euo pipefail

# Find all files with _test.sh at the end of the filename
# and run them
mapfile -t test_files < <(find . -name "*_test.sh" | sort | grep -v "./node_modules")

for test_file in "${test_files[@]}"; do
echo "Running tests in $test_file"
# shellcheck disable=SC1090
bash "$test_file" || {
echo "Tests failed in '$test_file'"
exit 1
}
done
5 changes: 5 additions & 0 deletions scripts/test.override.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Calls extra test functions
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
exec "${DIR}/bash-test-runner.sh"
25 changes: 16 additions & 9 deletions shell/ci/release/docker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,21 @@ source "${LIB_DIR}/buildx.sh"
# shellcheck source=../../lib/logging.sh
source "${LIB_DIR}/logging.sh"

# shellcheck source=../../lib/yaml.sh
source "${LIB_DIR}/yaml.sh"

if [[ ! -f $MANIFEST ]]; then
error "Manifest file '$MANIFEST' required for building Docker images"
fatal "See https://github.com/getoutreach/devbase#building-docker-images for details"
fi

# get_image_field returns a field from the manifest of the image
# get_image_field is a helper to return a field from the manifest
# for a given image. It will return an empty string if the field
# is not set.
get_image_field() {
local name="$1"
local image="$1"
local field="$2"
# shellcheck disable=SC1087 # this is a yq/jq filter that shellcheck thinks is bash
yq -r ".[\"$name\"]$field[]" "$MANIFEST"
yaml_get_array "$(yaml_construct_object_filter "$image" "$field")" "$MANIFEST"
}

# build_and_push_image builds and pushes a docker image to
Expand All @@ -48,7 +52,7 @@ build_and_push_image() {
# - linux/amd64
#
# See buildkit docs: https://github.com/docker/buildx#building-multi-platform-images
mapfile -t platforms < <(get_image_field "$image" '.platforms')
mapfile -t platforms < <(get_image_field "$image" 'platforms')
if [[ -z $platforms ]] || [[ $platforms == "null" ]]; then
platforms=("linux/arm64" "linux/amd64")
fi
Expand All @@ -60,7 +64,7 @@ build_and_push_image() {
#
# See docker docs:
# https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information
mapfile -t secrets < <(get_image_field "$image" '.secrets')
mapfile -t secrets < <(get_image_field "$image" 'secrets')
if [[ -z $secrets ]] || [[ $secrets == "null" ]]; then
secrets=("id=npmtoken,env=NPM_TOKEN")
fi
Expand All @@ -73,7 +77,7 @@ build_and_push_image() {
# as the repository. If this is not the main image (appName), we'll
# append the appName to the repository to keep the images isolated
# to this repository.
local remote_image_name=$(get_image_field "$image" '.pushTo')
local remote_image_name=$(get_image_field "$image" 'pushTo')
if [[ -z $remote_image_name ]] || [[ $remote_image_name == "null" ]]; then
local remote_image_name="$imageRegistry/$image"

Expand Down Expand Up @@ -128,8 +132,10 @@ build_and_push_image() {
docker buildx build "${args[@]}" -t "$image" --load "$buildContext"
)

info "🔐 Scanning docker image for vulnerabilities"
"${TWIST_SCAN_DIR}/twist-scan.sh" "$image" || echo "Warning: Failed to scan image" >&2
if [[ $CI == "true" ]]; then
info "🔐 Scanning docker image for vulnerabilities"
"${TWIST_SCAN_DIR}/twist-scan.sh" "$image" || echo "Warning: Failed to scan image" >&2
fi

if [[ -n $CIRCLE_TAG ]]; then
echo "🔨 Building and Pushing Docker Image (production)"
Expand All @@ -142,6 +148,7 @@ build_and_push_image() {
fi
}

# Build and (on tags: push) all images in the manifest
mapfile -t images < <(yq -r 'keys[]' "$MANIFEST")
for image in "${images[@]}"; do
build_and_push_image "$image"
Expand Down
2 changes: 1 addition & 1 deletion shell/lib/asdf.sh
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ asdf_devbase_ensure() {
asdf_plugin_install "$plugin" || echo "Warning: Failed to install language '$name', may fail to invoke things using that language"

# Install the version if it doesn't already exist
if ! asdf list "$plugin" | grep -qE "^$version$"; then
if ! asdf list "$plugin" | grep -qE "$version$"; then
# Install the language, retrying w/ AMD64 emulation if on macOS or just retrying on failure once.
asdf install "$plugin" "$version" || asdf_install_retry "$plugin" "$version"
fi
Expand Down
32 changes: 32 additions & 0 deletions shell/lib/yaml.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env bash
# yaml is a general purpose bash yaml parsing library

# yaml_get_array returns a newline separated list of values
# from a yaml array. If a value is not set, it will return
# an empty string.
#
# $1 yq filter
# $2 yaml file
yaml_get_array() {
local filter="$1"
local file="$2"
yq -r "$filter | .[]?" "$file"
}

# yaml_construct_object_filter creates a yq filter for all
# arguments passed to access a field on an object. For example,
# if you have a yaml file with the following contents:
#
# foo:
# bar:
# baz: 1
#
# yaml_construct_object_filter foo bar baz will return
# .["foo"]["bar"]["baz"]
yaml_construct_object_filter() {
local filter="."
for arg in "$@"; do
filter+="[\"$arg\"]"
done
echo "$filter"
}
48 changes: 48 additions & 0 deletions shell/lib/yaml_test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Tests the shell/lib/yaml.sh library
#
# Not needed for tests:
# - SC2155: Declare and assign separately to avoid masking return values.
# shellcheck disable=SC2155

# No -e because we want to handle errors to make them
# obvious.
set -uo pipefail

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"

# shellcheck source=../lib/yaml.sh
source "${DIR}/../lib/yaml.sh"

test_yaml_get_array() {
# should be able to get values from an array
local yaml_file=$(mktemp)
{
echo "set:"
echo " - bar"
echo " - baz"
} >"$yaml_file"

echo "Should be able to get values from an array"
local got=$(yaml_get_array ".set" "$yaml_file")
local expected=$({
echo "bar"
echo "baz"
})
if [[ $got != "$expected" ]]; then
echo "Expected '$expected', got '$got'"
exit 1
fi

echo "Should not error if the field is not set"
got=$(yaml_get_array ".not_set" "$yaml_file" 2>&1)
if [[ $got != "" ]]; then
echo "Expected empty value for unset field, got: $got" >&2
exit 1
fi

rm -f "$yaml_file"
return 0
}

test_yaml_get_array

0 comments on commit b96f4c0

Please sign in to comment.