Add research about running operations in target-matching environments

Fixes #219

research/target-matching-env/index.md

@@ -0,0 +1,70 @@
+---
+title: Running operations in target-matching environments
+authors: lbarczio
+---
+For some of the operations run in our service, it would be beneficial
+to run them in target-matching environment, e.g. parsing the specfile placed 
+in F41 branch should happen on F41. Currently, we base our images
+on Fedora 41.
+
+## Use cases
+- user-configured commands (`actions` in config)
+  - affected jobs: `copr_build`, `propose_downstream`, `pull_from_upstream`
+  - run in on demand pods in `sandbox` (dedicated Openshift project)
+- building SRPM
+  - affected jobs: 
+    - `upstream_koji_build` -  in `long-running` workers
+    - `copr_build` - in Copr, we just pass the required env as an argument for the API call
+- specfile parsing
+  - for getting version, manipulating changelog, etc. during release syncing
+  - affected jobs: `pull_from_upstream`, `propose_downstream` 
+  - run in `long-running` workers
+- anything else?
+
+
+## Possible solutions
+### container-based solutions
+#### dedicated workers for each Fedora version
+- for building SRPM, running release syncing (=> specfile parsing)
+- build separate images for each Fedora version
+- use Celery for task routing; tasks requiring specific environments would be routed accordingly
+- by default, tasks would run on the "main" worker (e.g. currently F41-based).
+- tasks requiring target-matching environments would be refactored (e.g. release syncing for 
+all branches (currently one task) will need to be split)
+- potential waste of resources
+- shouldn't introduce any delays
+
+#### sandcastle adjustments
+- for running `actions`
+- build separate images for each Fedora version
+- adjust the code running sandcastle to dynamically pass the image reference
+- for commands not specified by users, e.g. `rpmbuild`, `fedpkg`, we could also utilise
+sandcastle, without the need of deploying the pod in different project, but this would lead
+to slower execution (overhead of scheduling the pods)
+
+### Copr resalloc
+- set up a custom resalloc instance, most of the tooling would need to be implemented
+- communication via SSH 
+- relying on external infra, might be slow
+
+### Testing Farm reservations
+- [docs](https://docs.testing-farm.io/Testing%20Farm/0.1/cli.html#reserve)
+- `testing-farm reserve` to get a chosen OS
+- communication via SSH
+- there are already users using it in CI for provisioning systems for Ansible Molecule
+- not yet a proper API, a proper supported use case in plan
+- in the future a target machine could be obtained in <30s
+- still relying on external infra
+- could be considered for running actions
+
+### mock
+- [mock in a container](https://rpm-software-management.github.io/mock/#mock-inside-podman-fedora-toolbox-or-docker-container)
+requires privileged mode which is not possible in our current OpenShift instance
+
+## Next steps
+- with switching to F41 based image, we have unblocked users needing the latest RPM macros
+- once there is more user demand for any of the use cases, we can consider the options above, depending on 
+the particular use-case
+  - the container-based solution with dedicated workers looks like the best fit for running the whole tasks to me
+  - for the `actions`, the `sandcastle` adjustments could be enough, 
+  but for the future I could see using TF reservations as a complete replacement