Refine how repeated tests are tracked

[mokagio]

Recently we had to take on some tech debt and enable test repetition in our unit tests to cope with the noise introduced by a few flaky tests.

If a test fails, xcodebuild will retry it up to 3 times. This is usually enough to compensate for the flakiness and obtain the successful test necessary for the whole test suite to be considered successful.

At the Test Analytics level, however, the build appears as a failure. See purple annotation in the screenshot below:

(Apologies for the arrow noise in the screenshot. I annotated it to service two issue in one go, the other one being #52.)

What I think is going on is that the test failures are collected and reported, correctly, but the library does not account for retries.

In a sense, I like how failures are always reported because it doesn't let flaky tests off the hook.
However, this paints a somewhat inconsistent picture between the success graph of the pipeline which is all green and that of the analytics which is all red.

It would be great if Test Analytics could detect automatic retries like the one Xcode allows for.

One idea could to track all the tests so that the reliability ratio is accurate at the Test Analytics end, but use the Xcode test report to determine whether a test run was ultimately successful or not.

I could buy into a radical "if a test failed, even just once, the build failed" approach. However, I fear that would be confusing for users who are not aware of test repetition.

[iampatbrown]

@mokagio I'm on board with what you're saying.

One idea could to track all the tests so that the reliability ratio is accurate at the Test Analytics end, but use the Xcode test report to determine whether a test run was ultimately successful or not.

There has been discussion around this and I believe it's planned for the future.

I could buy into a radical "if a test failed, even just once, the build failed" approach. However, I fear that would be confusing for users who are not aware of test repetition.

I believe some flexibility is important here. People need the information that helps them, which might be different for each person. I think the main challenge is supporting multiple platforms that have different vocabulary/approaches when it comes to testing.

I'm not sure how much progress we can make on this one right now, but, it will definitely be addressed at some point :)

[kelly-scalapay]

Having the same issue with test-collector-javascript