Flapping tests

[AlexDaniel]

When running the tests in a loop like this:

while PERL6LIB=lib/ prove --exec=perl6 t/*.t; do :; done

Sometimes I get this output:

t/00-meta.t ............ ok   
t/01-basics.t .......... ok   
t/02-async.t ........... ok   
t/03-expiry.t .......... 1/12 # Failed test 'pausing between gets evicts the older entries'
# at t/03-expiry.t line 90
# expected: 'ABCDBA'
#      got: 'ABCDCBA'
# Looks like you failed 1 test of 12
t/03-expiry.t .......... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/12 subtests 
t/04-exceptions.t ...... ok   
t/05-monitoring.t ...... ok     
t/06-slurpies.t ........ ok   
t/07-async-producer.t .. ok   
t/08-refreshing.t ...... ok     

Test Summary Report
-------------------
t/03-expiry.t        (Wstat: 256 Tests: 12 Failed: 1)
  Failed test:  12
  Non-zero exit status: 1

[robertlemmen]

this test checks actual temporal behavior of cache eviction, and is inherently racy and problematic. I'll try and make it more robust by pacing it differently.

In any case, I do not think that these test failures are a sign of regressions in rakudo, they are simply a terrible test. Not sure how to make it better though...

Out of interest: were you running these on a very slow or heavily loaded system? I can't get them to fail even with extra load on mine...

[AlexDaniel]

In any case, I do not think that these test failures are a sign of regressions in rakudo

That's correct, I never said they were :) The reason this ticket is linked from rakudo/rakudo#2047 is because this module showed up (because of flapping tests) when we were hunting for regressions.

Out of interest: were you running these on a very slow or heavily loaded system? I can't get them to fail even with extra load on mine...

Yes, toaster runs tests of a bunch of modules simultaneously, so that's heavy loaded for sure. But I was also able to reproduce it locally in a loop (took quite a bit, but it happened).

[robertlemmen]

right, I had another look but can't find a good way to make these tests truly stable in the presence of a loaded system. I have doubled the margins, given that they previously only failed for you sometimes, I guess that makes it quite unlikely now.

I'll close the issue for now, but if anyone sees these flapping again then please let me know and I'll try to tune them further.