Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: auto-set GitHub runner make job count #9690

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

iamamyth
Copy link

@iamamyth iamamyth commented Jan 9, 2025

Dynamically determine an appropriate number of make jobs for the GitHub runner container, based upon the number of available CPU cores and RAM, using the rule that each job requires a dedicated core and 2.25GiB of RAM.

@selsta
Copy link
Collaborator

selsta commented Jan 9, 2025

Seems this does not compile with Debian and Ubuntu CI jobs

@iamamyth
Copy link
Author

iamamyth commented Jan 9, 2025

This PR follows from the discussion in #9510. Based on testing with GitHub macOS runners, I determined the suitable memory requirement to be 2.25GiB per runner, rather than the 2.75GiB hypothesized in the thread: With three jobs (current config) on a 3 CPU + 7GiB machine, a full build takes roughly 8m 24s, versus 11m 10s with two jobs. Due to noisy neighbors, there's considerable performance variance (observed 40s difference between two identical runs), but the results suggest 2.25GiB suffices.

Should the need arise, this PR allows custom memory requirements to be set per OS.

Build output confirms this PR results in job counts identical to the current fixed configuration (by design, the dynamically determined count appears in the build output because it's sourced from an environment variable).

@iamamyth iamamyth force-pushed the gh-auto-jobcount-latest branch from 9a20099 to 8dc82cf Compare January 9, 2025 03:12
@iamamyth
Copy link
Author

iamamyth commented Jan 9, 2025

@selsta Build should now work. A typo snuck in when preparing a rebased version of this branch and I couldn't fix it due to GitHub instability (also the reason my prior comment follows the PR by an hour).

Dynamically determine an appropriate number of make jobs for the GitHub
runner container, based upon the number of available CPU cores and RAM,
using the rule that each job requires a dedicated core and 2.25GiB of RAM.
@iamamyth iamamyth force-pushed the gh-auto-jobcount-latest branch from 8dc82cf to 07774e6 Compare January 9, 2025 03:56
@plowsof
Copy link
Contributor

plowsof commented Jan 14, 2025

Looks good so far although my basic arithmetic skills are horrendous (thank you for the 9/4 comment which i seen later, i have just about grasped that 2.25 = 9 quarters)

"But where's the math.floor?" i asked myself. TIL Bash does floor rounding by division automatically 1

The only suggestion i have would be returning the ram in bytes at the source rather than dealing with 3 different units.

powershell can return value in bytes directly "TotalPhysicalMemory"2 and vmstat has --unit B.

untested: powershell -Command "(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory"

@iamamyth
Copy link
Author

If it were trivial to achieve, I would have initially used a common unit (probably KiB, because that's what /proc/meminfo reports, as bytes are unreasonably small). I'd rather not have anything to do with powershell, as that introduces a second major set of OS tools. I could change vmstat to use KiB, but I'm not sure exactly what that would accomplish: It's fairly obvious the command reports MiB, and the unit conversions between KiB, MiB, and GiB, aren't particularly complex (aside from the appearance of k, M, etc, in the commands, the denominators also pretty clearly indicate the scales, and relations between them).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants