Verification experiments should become sophisticated tests

[ali-ramadhan]

Following the discussion on testing infrastructure from a long time ago (PR #139) it would be good to convert any verification tests that result from #346 into an actual end-to-end test of the model. Not sure how the comparison with MITgcm would be implemented but maybe the others are good enough.

test/verification/ might be a good directory for them?

They can be run separately (separate pipeline) so they don't have to run quickly and can take a while so they can be comprehensive.

[glwagner]

Will we have GPU resources to run these verification tests? They might take some time (~1 day) to run.

[christophernhill]

Typically you rerun from a saved spun up state etc.... and check for departure from saved trajectory. So then the run is short. We should have 700 ish 32GB V100 by fall though. We got the last piece sorted to make that happen this afternoon. We will have to share, but it should help.

…

On Thu, Aug 8, 2019 at 21:30 Gregory L. Wagner ***@***.***> wrote: Will we have GPU resources to run these tests? They might take some time (~1 day) to run. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#347>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA27DYFT5YXWTBIDXLPSDFTQDTCD5ANCNFSM4IKPWH5A> .

[ali-ramadhan]

If rerunning from a saved state is the way to go then they might even fit within the old tests although might still be nice to split tests into a fast "unit test" suite and a slow "comprehensive" test suite.

I guess a really obscure/esoteric bug could cause a verification experiment to change behavior at an early/transient stage so that the verification fails (but not if it's started from a pickup file). But in practice running from a pickup and checking that everything matches should catch almost any bug, and is much more practical.

Sounds like they would be regression tests but checking against strict verification tests.

Only issue would be that we might not want to save the pickup data in git (for 256^3 verification tests it could be hundreds of MiB) but I'm sure we could figure something out. Some verifications may have to be run at high resolution. Git LFS is an option but haven't heard great things about it...

[ali-ramadhan]

This was demonstrated in #381 so I think we can close this issue.

We can discuss whether we should make a regression test for each verification experiment in the future.