Multiple Verifications running for a Stage

[jessesuen]

Description

Stages seem to be allowing multiple Verifications to be run at the same time. Each coincided with a promotion.

$ k get ar
NAME                                     STATUS    AGE
dev.01j02dz4tbp75sf7zmvgkw3d3j.9e5f6ab   Failed    16h
dev.01j02hycta7xsjz7xczamptk6c.9d167fb   Failed    14h
dev.01j03w1h1njwvj432nbh8m7xvk.a72c52c   Running   158m
dev.01j03wxxntp3a3xm5pbpdzzzkd.bd13218   Running   142m
dev.01j03z1dzstgnx5tw85ty3apxr.97465d1   Running   105m

I feel that we should only be allowing one at a time to happen.

We should also decide how Promotion should affect currently running Verifications. e.g. should it:

• wait for current verification to complete, then immediately launch a new one against the freight?
• terminate in-flight verification (since it's ambiguous what exactly was verified)?
• block the promotion until verification is completed?

Steps to Reproduce

Promote something else while a verification is already running.

Version

v0.7.0

[krancour]

Possibly a duplicate of #2128?

[hiddeco]

block the promotion until verification is completed?

I feel this would (again) steer more towards verification being an element of promotion, which based on the further discussion in #2128 does not appear to be true. In addition, it hypothetically opens a gap to e.g. quickly issue re-verifications to block a promotion from ever starting.

Given this, I am inclined to say we should wait for a running verification to complete (and maybe terminate it if there is new current Freight), before triggering a new verification.

[krancour]

which based on the further discussion in #2128...

fwiw, my thoughts there were in no way definitive/final.

Let's hash this out f2f when time permits.

[krancour]

Close enough to a duplicate of #2128 that I am closing this one to keep discussion all in one place.