Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize creation of OpenStack runners #270

Merged
merged 8 commits into from
Apr 29, 2024

Conversation

yhaliaw
Copy link
Collaborator

@yhaliaw yhaliaw commented Apr 25, 2024

Overview

Parallelize the creation of new OpenStack server with multiprocessing.Pool.
The charm should spawn a process per new OpenStack runner created (upwards of 10 processes).

Rationale

This will greatly reduce the reconcile time period, as runner creation is done in parallel. In theory, we should see a maximum 10-fold reduction in reconcile time.

Checklist

@yhaliaw yhaliaw changed the base branch from main to feat/openstack-integration April 25, 2024 11:27
@yhaliaw yhaliaw marked this pull request as ready for review April 29, 2024 05:08
@yhaliaw yhaliaw requested a review from a team as a code owner April 29, 2024 05:08
)
for _ in range(delta)
]
with Pool(processes=min(delta, 10)) as pool:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what happens if there are more runners that are required than the number of processes spawned by the pool?

@yhaliaw yhaliaw merged commit f388b85 into feat/openstack-integration Apr 29, 2024
3 of 10 checks passed
@yhaliaw yhaliaw deleted the feat/openstack-parallel-spawn branch April 29, 2024 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants