-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel job execution #75
Conversation
runner-manager/.profile
Outdated
@@ -81,6 +81,7 @@ if pgrep 'gitlab-runner' > /dev/null ; then | |||
else | |||
echo "Registering GitLab Runner with name $RUNNER_NAME" | |||
if gitlab-runner register; then | |||
sed -e 's/concurrent = 1$/concurrent = 10/' -i.bak .gitlab-runner/config.toml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[question] I'm wondering if this can be configured through the env like with the other settings? e.g., PREPARE_EXEC
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[question (nb)] how did you land on 10?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Um, yes, this script will have access to the env, I'll make this configurable now.
10 just seemed like a good balance between parallelism and not running out of memory quota
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure you can. I was trying to find the variable--the ones we use come from gitlab-runner register --help
and concurrency doesn't seem to be listed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, I misunderstood.
No, this is the only way to set the concurrent
setting, but gitlab-runner creates this file when you call register and reloads it when it changes. What I was going to do was update the concurrent = 10
to be concurrent = $OUR_ENV_SETTING
so that we can still change this per-customer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense to you, why this doesn't have a setting?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really! Other than the counter argument that it does have a setting, just not via ENV vars. We could probably switch out all of our ENV var settings and just bundle a .toml file instead. If we have to add any more sed
commands then we'll probably want to just go ahead and move everything to the file.
🎫 Addresses issue: no issue
🛠 Summary of changes
concurrent
gitlab-runner global setting to 10 to allow a single manager to retrieve up to 10 concurrent job👀 Screenshots and Evidence
nice speedup (though some of that on the fastest build is eliminating one of the sequential build jobs)