Rerun System Tests on failure #524
Conversation
We've seen that from time to time System Tests workflow fails due to problems with connectivity to Electrum or due to some other randomly occurring issues (like the problems with installing dependencies). We now add a job to the System Tests workflow which is started only if one of the frevious jobs failed. The job will trigger a Rerun workflow, which will then trigger the System Tests again. The reason why we need to rerun via a separate workflow is because reruns cannot be called on the workflows/jobs which are still running.
michalinacienciala
force-pushed
the
retry-system-tests
branch
from
c3ee2c7 to
2c2b77b
Compare
We've been implementing workflow retries as a solution to some non-deterministic failures of the 'System Tests' workflow that are not indicative of the problems with the tested functionalities. The solution that we have built so far could lead to constant rerty loop in case of consisting failures. Now we're adding a retry limit - we will not automatically execute more than 3 attempts of the workflow.
|
IMHO we should hold with adding general job reruns. We should try to fix all the problems we're experiencing on installations and test execution. |
This work is captured in #531 and is on our sprint board. |
|
We've decided to close this PR in order to not introduce a solution that could make us accustomed to the environment configuration issues we're currently facing (described in #531). |
We've seen that from time to time System Tests workflow fails due to problems with connectivity to Electrum or due to some other randomly occurring issues (like the problems with installing dependencies). We now add a job to the System Tests workflow which is started only if one of the previous jobs failed. The job will trigger a Rerun workflow, which will then trigger the System Tests again. The reason why we need to rerun via a separate workflow is because reruns cannot be called on the workflows/jobs which are still running.
To prevent from getting stuck in the constant retries loop we've added a retry limit - we will not automatically execute more than 3 attempts of the workflow.
Inspired by solution presented here: https://gist.github.com/philip-gai/e3c02e68a32e8964fa3df2167b15cbff.
TODO:
workflowandread:orgscopes