DEVPROD-13428: Reset Amboy database in e2e tests when mutations enqueue jobs

[minnakt]

DEVPROD-13428

Description

Why does the test fail?

The tests in host_buttons.ts fail because the host is sometimes in a decommissioned state. You cannot reprovision or restart Jasper for a decommissioned host.

But the host i-0d0ae8b83366d22be is defined as "running" in the local test data. That shouldn't happen.

True, it seems like this should be impossible because the database should reset after every test that executes a mutation.

What is happening then?

In all of the test failures linked in the corresponding JIRA ticket, the test that runs directly before is host_update_status.ts. This test calls a mutation that creates a job in Amboy called EnqueueTerminateHostJob.

I think the test was flaky due to a race condition. The two patterns would be:

• Sometimes it would succeed because the Amboy job ran first and marked the host as decommissioned. Then we cleared the database and the host was reset correctly.
• Sometimes it would fail because we cleared the database first. Then the Amboy job got picked up and ran a second later, marking the host as decommissioned.

I think that clearing the Amboy database will help avoid future occurrences of similar bugs. By removing newly enqueued jobs from the Amboy database, we can ensure that the data will not change after we reset the Evergreen database.

[sophstad]

Looks great + nice writeup. I probably would've just renamed the test to zzz_host_update_status.ts so it would run last 😉

[minnakt]

merging with codegen failing because the fix would be requesting a field that doesn’t exist on the backend yet, requiring coordinated deploys, so just not updating to keep it safe