pip-compile executing on every branch, every time

[sichapman]

What would you like help with?

Other

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab

Please tell us more about your question or problem

I assume this is intended behaviour but posting on the off chance it's not, or there's a way to configure around it.

The pip-compile manager executes pip-compile against every single pre-existing renovate/ branch whenever that repo is scanned. It's basically doing constant shadow lock file maintenance on every branch. As we have lots of Python apps this is adding lots of time onto our loops.

In my opinion, it should only execute when the branch has just been created or rebased. While it's theoretically nice to keep the transitive dependencies up to date if a branch ends up hanging around for a little while, it's almost certainly going to get rebased due to falling behind or having merge conflict anyway, and if it does get merged with a gone-slightly-out-of-date requirements.txt, that will soon get tidied up by the "proper" lock file maintenance. As it currently stands, it just adds extra time onto every single scan of the repo.

Logs (if relevant)

No response

[RahulGautamSingh]

With repositoryCache-enabled this should not happen as often, as the branch gets skipped if the upgrades are same. But, if the base branch changes frequently then even that won't help as much.

Only way I can think of reducing this is by doing rebaseWhen=behind-base-branch.

[sichapman]

I never even noticed repositoryCache (is there a list of recommend settings anywhere?), yes that has improved things alot, and should be useful overnight when branches can pile up without anyone around to merge.

Forcing behind-base-branch does not reduce the amount of pip-compile executions though. I don't mind p-c running if a rebase has actually occurred, since it's essentially a brand new commit. But running when nothing has changed seems wrong.

[rarkins]

I haven't looked in detail at the implementation but it's possible that Renovate is running pip-compile every run to see if the updates are current, i.e. it's not because the branch needs rebasing.

We have had a similar challenge with lockFileMaintenance on other managers. If we re-ran the command every run, it's quite inefficient. On the other hand, it does get out of date sometimes.

Maybe we could use a very simple approach to all of these where we don't don't try again unless/until the commit is X days old. The challenge here would be that if the results are stable, then once the branch reaches X days old then we'd go back to re-running pip-compile every run unless we intentionally force pushed the same commit every X days to prevent it (which is also somewhat undesirable, but maybe better).