[Bug] Recording archive notifications don't always appear

[andrewazores]

Current Behavior

When an automated rule is created that has an archival period, the archived recording upload notifications don't always appear as expected. The deletion notifications for maintaining the correct number of archived copies do seem reliable.

{
  "name": "test",
  "description": "",
  "matchExpression": "target.alias.startsWith('quarkus-cryostat-agent') && !target.connectUrl.startsWith('http')",
  "eventSpecifier": "template=Continuous,type=TARGET",
  "archivalPeriodSeconds": 10,
  "initialDelaySeconds": 0,
  "preservedArchives": 2,
  "maxAgeSeconds": 0,
  "maxSizeBytes": 10485760
}

Expected Behavior

When the archived copies count has reached the limit, there should always be a matching pair of uploaded/deleted notifications - an upload of a new recording, and a deletion of the one oldest recording.

Steps To Reproduce

$ ./smoketest.bash -t quarkus-cryostat-agent
$ # wait for deployment, then visit https://localhost:8443 and create the automated rule as given above

Environment

Anything else?

No response

[andrewazores]

Maybe related to cryostatio/cryostat-web#1286

Probably related to at least one of:

• fix(rules): automatic refire with backoff #732
• feat(async): long-running async job API #698
• fix(rules): add blocking annotation for rules error #526
• [Bug] Recording archive failure notification title mismatch #761