-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
VAULT-31181: Add
pipeline tool to Vault (#28536)
As the Vault pipeline and release processes evolve over time, so too must the tooling that drives them. Historically we've utilized a combination of CI features and shell scripts that are wrapped into make targets to drive our CI. While this approach has worked, it requires careful consideration of what features to use (bash in CI almost never matches bash in developer machines, etc.) and often requires a deep understanding of several CLI tools (jq, etc). `make` itself also has limitations in user experience, e.g. passing flags. As we're all in on Github Actions as our pipeline coordinator, continuing to utilize and build CLI tools to perform our pipeline tasks makes sense. This PR adds a new CLI tool called `pipeline` which we can use to build new isolated tasks that we can string together in Github Actions. We intend to use this utility as the interface for future release automation work, see VAULT-27514. For the first task in this new `pipeline` tool, I've chosen to build two small sub-commands: * `pipeline releases list-versions` - Allows us to list Vault versions between a range. The range is configurable either by setting `--upper` and/or `--lower` bounds, or by using the `--nminus` to set the N-X to go back from the current branches version. As CE and ENT do not have version parity we also consider the `--edition`, as well as none-to-many `--skip` flags to exclude specific versions. * `pipeline generate enos-dynamic-config` - Which creates dynamic enos configuration based on the branch and the current list of release versions. It takes largely the same flags as the `release list-versions` command, however it also expects a `--dir` for the enos directory and a `--file` where the dynamic configuration will be written. This allows us to dynamically update and feed the latest versions into our sampling algorithm to get coverage over all supported prior versions. We then integrate these new tools into the pipeline itself and cache the dynamic config on a weekly basis. We also cache the pipeline tool itself as it will likely become a repository for pipeline specific tooling. The caching strategy for the `pipeline` tool itself will make most workflows that require it super fast. Signed-off-by: Ryan Cragun <[email protected]>
- Loading branch information
1 parent
afd023e
commit ce58852
Showing
33 changed files
with
2,024 additions
and
108 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Create dynamic pipeline configuration | ||
| description: Create dynamic test configuration by restoring existing valid config or creating new config | ||
|
|
||
| inputs: | ||
| github-token: | ||
| description: An elevated Github token to access private HashiCorp modules. | ||
| vault-edition: | ||
| description: The vault edition to use when generating the dynamic config | ||
| vault-version: | ||
| description: The vault version to use when generating the dynamic config | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: dyn-cfg-metadata | ||
| id: dyn-cfg-metadata | ||
| shell: bash | ||
| run: | | ||
| # We're using a weekly cache key here so that we only regenerate the configuration on a | ||
| # weekly basis. If/when Github decides to purge our tiny config file cache we'll also | ||
| # recreate it as necessary. | ||
| # | ||
| # Uses GITHUB_ENV instead of GITHUB_OUTPUT because composite actions are broken, | ||
| # see: https://github.com/actions/cache/issues/803#issuecomment-1793565071 | ||
| { | ||
| echo "DYNAMIC_CONFIG_KEY=$(date +%Y-%m-%U)" | ||
| echo "DYNAMIC_CONFIG_PATH=enos/enos-dynamic-config.hcl" | ||
| } | tee -a "$GITHUB_ENV" | ||
| - name: Try to restore dynamic config from cache | ||
| id: dyn-cfg-cache | ||
| uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2 | ||
| with: | ||
| path: ${{ env.DYNAMIC_CONFIG_PATH }} | ||
| key: dyn-cfg-${{ env.DYNAMIC_CONFIG_KEY }} | ||
| - if: steps.dyn-cfg-cache.outputs.cache-hit != 'true' | ||
| id: dyn-cfg-set-up-pipeline | ||
| # If we can't restore it from config then set up pipeline and generate it | ||
| name: Set up the pipeline tool | ||
| uses: ./.github/actions/set-up-pipeline | ||
| with: | ||
| github-token: ${{ inputs.github-token }} | ||
| - if: steps.dyn-cfg-cache.outputs.cache-hit != 'true' | ||
| id: dyn-cfg-generate | ||
| name: Create dynamic configuration | ||
| shell: bash | ||
| run: | | ||
| # Make sure that any branch specific dynamic config has been generated | ||
| pipeline generate enos-dynamic-config -d ./enos -f enos-dynamic-config.hcl -v ${{ inputs.vault-version }} -e ${{ inputs.vault-edition }} -n 3 --log debug |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| --- | ||
| name: Install the pipeline tool | ||
| description: Install the pipeline tool | ||
|
|
||
| inputs: | ||
| github-token: | ||
| description: An elevated Github token to access private HashiCorp modules. | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - uses: ./.github/actions/set-up-go | ||
| with: | ||
| github-token: ${{ inputs.github-token }} | ||
| no-restore: true # Don't download vault's modules for pipeline | ||
| - name: pipeline-metadata | ||
| id: pipeline-metadata | ||
| shell: bash | ||
| # Uses GITHUB_ENV instead of GITHUB_OUTPUT because composite actions are broken, | ||
| # see: https://github.com/actions/cache/issues/803#issuecomment-1793565071 | ||
| run: | | ||
| gobin=$(go env GOBIN) | ||
| if [[ -z "$gobin" ]]; then | ||
| gobin="$(go env GOPATH)/bin" | ||
| fi | ||
| { | ||
| echo "PIPELINE_HASH=$(git ls-tree HEAD tools/pipeline --object-only)" | ||
| echo "PIPELINE_PATH=$gobin/pipeline" | ||
| } | tee -a "$GITHUB_ENV" | ||
| - name: Try to restore pipeline from cache | ||
| id: pipeline-cache | ||
| uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2 | ||
| with: | ||
| path: ${{ env.PIPELINE_PATH }} | ||
| key: pipeline-${{ env.PIPELINE_HASH }} | ||
| - if: steps.pipeline-cache.outputs.cache-hit != 'true' | ||
| id: pipeline-build | ||
| name: Build pipeline | ||
| shell: bash | ||
| env: | ||
| GOPRIVATE: github.com/hashicorp/* | ||
| run: | | ||
| git config --global url."https://${{ inputs.github-token }}@github.com".insteadOf https://github.com | ||
| make tools-pipeline |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: BUSL-1.1 | ||
|
|
||
| # Code generated by pipeline generate enos-dynamic-config DO NOT EDIT. | ||
|
|
||
| # This file is overwritten in CI as it contains branch specific and sometimes ever-changing values. | ||
| # It's checked in here so that enos samples and scenarios can be performed, just be aware that this | ||
| # might change out from under you. | ||
|
|
||
| globals { | ||
| sample_attributes = { | ||
| aws_region = ["us-east-1", "us-west-2"] | ||
| distro_version_amzn = ["2023"] | ||
| distro_version_leap = ["15.6"] | ||
| distro_version_rhel = ["8.10", "9.4"] | ||
| distro_version_sles = ["15.6"] | ||
| distro_version_ubuntu = ["20.04", "24.04"] | ||
| upgrade_initial_version = ["1.16.1", "1.16.2", "1.16.3", "1.17.0-rc1", "1.17.0", "1.17.1", "1.17.2", "1.17.3", "1.17.4", "1.17.5", "1.17.6", "1.18.0-rc1", "1.18.0"] | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.