Add research about running operations in target-matching environments #221
Conversation
| - for getting version, manipulating changelog, etc. during release syncing | ||
| - affected jobs: `pull_from_upstream`, `propose_downstream` | ||
| - run in `long-running` workers | ||
| - anything else? |
There was a problem hiding this comment.
let me know if you know about some other use-cases I should consider here
lbarcziova
force-pushed
the
target-matching-env
branch
from
636ee58 to
f52af6a
Compare
| - build separate images for each Fedora version | ||
| - use Celery for task routing; tasks requiring specific environments would be routed accordingly | ||
| - by default, tasks would run on the "main" worker (e.g. currently F41-based). | ||
| - tasks requiring target-matching environments would be refactored (e.g. release syncing for |
There was a problem hiding this comment.
I am wondering if it could be easier and useful, since we will have workers for every needed Fedora and CentOS, to run the process_messages task in the worker with the latest release and then just route any other job based on the target? I am not sure if it is better to distribute more the tasks on the workers, since we have them. Or not.
There was a problem hiding this comment.
you mean e.g. using the build/test-target-overrides? It is a good point to think about, I will note it here, I see issues only with things like storing to the DB (with the current implementation I think that would mean each target would be in a dedicated group model)
There was a problem hiding this comment.
I am not sure I understand what you mean by build/test-target-overrides.
I am suggesting to route every job, except process_messages, and not only some of them to the "target" worker. We can choose the target from the target key (in copr_build, test, koji jobs ecc.) and other times using the dist_git_branches key (for propose_downstream as an example).
I see it easier, probably, to balance resources for the workers in this way; like increasing resources for the worker associated with the latest fedora (that I image it is the most used). And maybe decrease resources for the worker associated with the oldest fedora we have... or something like that. But I don't have much arguments for it, just wondering if it makes sense...
I am not sure if this will change something on the database group model, I can't see it.
There was a problem hiding this comment.
I am not sure I understand what you mean by build/test-target-overrides.
I meant utilising current build/test_target_override implementation (setting these to the one particular target that will be processed in the task) for doing the routing which you describe later. And I mentioned DB group model because if we process each of the targets in a dedicated task, we need to somehow think about when/how the group model is created. But I get your point about balancing out the utilisation of the workers, so will definitely note that!
| - with switching to F41 based image, we have unblocked users needing the latest RPM macros | ||
| - once there is more user demand for any of the use cases, we can consider the options above, depending on | ||
| the particular use-case | ||
| - the container-based solution with dedicated workers looks like the best fit for running the whole tasks to me |
There was a problem hiding this comment.
Personally, I agree with this.
lbarcziova
force-pushed
the
target-matching-env
branch
from
f52af6a to
b31f38d
Compare
Fixes #219