Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel job execution #75

Merged
merged 5 commits into from
Jan 15, 2025
Merged

Parallel job execution #75

merged 5 commits into from
Jan 15, 2025

Conversation

rahearn
Copy link
Contributor

@rahearn rahearn commented Jan 14, 2025

🎫 Addresses issue: no issue

🛠 Summary of changes

  • update concurrent gitlab-runner global setting to 10 to allow a single manager to retrieve up to 10 concurrent job
  • update default instances value to 2, to have a little more redundancy, while keeping the sandbox config to 1 for resource use

👀 Screenshots and Evidence

nice speedup (though some of that on the fastest build is eliminating one of the sequential build jobs)

Screenshot 2025-01-14 at 5 29 50 PM

@rahearn rahearn self-assigned this Jan 14, 2025
@rahearn rahearn marked this pull request as ready for review January 14, 2025 22:25
@rahearn rahearn requested a review from a team January 14, 2025 22:25
@@ -81,6 +81,7 @@ if pgrep 'gitlab-runner' > /dev/null ; then
else
echo "Registering GitLab Runner with name $RUNNER_NAME"
if gitlab-runner register; then
sed -e 's/concurrent = 1$/concurrent = 10/' -i.bak .gitlab-runner/config.toml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question] I'm wondering if this can be configured through the env like with the other settings? e.g., PREPARE_EXEC

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question (nb)] how did you land on 10?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Um, yes, this script will have access to the env, I'll make this configurable now.

10 just seemed like a good balance between parallelism and not running out of memory quota

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure you can. I was trying to find the variable--the ones we use come from gitlab-runner register --help and concurrency doesn't seem to be listed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I misunderstood.

No, this is the only way to set the concurrent setting, but gitlab-runner creates this file when you call register and reloads it when it changes. What I was going to do was update the concurrent = 10 to be concurrent = $OUR_ENV_SETTING so that we can still change this per-customer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to you, why this doesn't have a setting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really! Other than the counter argument that it does have a setting, just not via ENV vars. We could probably switch out all of our ENV var settings and just bundle a .toml file instead. If we have to add any more sed commands then we'll probably want to just go ahead and move everything to the file.

@rahearn rahearn requested a review from zjrgov January 15, 2025 15:31
@rahearn rahearn merged commit ba4b185 into main Jan 15, 2025
2 checks passed
@rahearn rahearn deleted the mutliple-manager-instances branch January 15, 2025 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants