build: Parallelize 'run-tests'

Because nextest does not support running doctests, we are parallelizing the run-tests.sh run in buildkite to run nextest in one job and doctests in another. This means spending a little more on CPU, but shortening the total wall clock duration of a build (plus gaining the features provided by nextest). Change-Id: I8211a10ba41179d2b0ed1d888588863011df5afc Reviewed-on: https://gerrit.readyset.name/c/readyset/+/8547 Tested-by: Buildkite CI Reviewed-by: Michael Zink <[email protected]>
readysettech · Jan 3, 2025 · 57c4997 · 57c4997
1 parent 65e7cb7
commit 57c4997
Show file tree

Hide file tree

Showing 2 changed files with 25 additions and 5 deletions.
diff --git a/.buildkite/pipeline.public-common.yml b/.buildkite/pipeline.public-common.yml
@@ -78,6 +78,7 @@ steps:
 
   - label: ':rust: Run tests'
     key: rust-tests
+    parallelism: 2
     commands:
       - '[ -d public ] && cd public'
       - '.buildkite/run-tests.sh'
@@ -88,6 +89,7 @@ steps:
       - docker-compose#v3.7.0:
           run: app
           env:
+          - BUILDKITE_PARALLEL_JOB
           - SCCACHE_BUCKET=readysettech-build-sccache-us-east-2
           - SCCACHE_REGION=us-east-2
           - CARGO_INCREMENTAL=0

diff --git a/.buildkite/run-tests.sh b/.buildkite/run-tests.sh
@@ -8,10 +8,28 @@ upload_artifacts() {
     exit 1
 }
 
-echo "+++ :rust: Run tests"
 export DISABLE_TELEMETRY=true
 export PROPTEST_MAX_SHRINK_TIME=1800000
-cargo --locked nextest run --workspace --features failure_injection --exclude readyset-clustertest \
-    --exclude benchmarks --ignore-default-filter \
-    && cargo --locked test --workspace --features failure_injection --exclude readyset-clustertest \
-    --exclude benchmarks --doc || upload_artifacts
+
+# If we aren't actually running run-tests.sh as a parallel job, just follow the
+# "job 0" path.
+: "${BUILDKITE_PARALLEL_JOB:=0}"
+
+if [[ "$BUILDKITE_PARALLEL_JOB" == "0" ]]; then
+    # Run nextest
+    echo "+++ :rust: Run tests (nextest)"
+    cargo --locked nextest run --workspace --features failure_injection \
+        --exclude readyset-clustertest \
+        --exclude benchmarks --ignore-default-filter \
+        || upload_artifacts
+elif [[ "$BUILDKITE_PARALLEL_JOB" == "1" ]]; then
+    # Run doctests, because at this time nextest does not support doctests
+    echo "+++ :rust: Run tests (doctest)"
+    cargo --locked test --workspace --features failure_injection \
+        --exclude readyset-clustertest \
+        --exclude benchmarks --doc \
+        || upload_artifacts
+else
+    echo "No command defined for this parallel job."
+    exit 1
+fi