End-to-end test workflow

[charles-turner-1]

Is your feature request related to a problem? Please describe.

The end to end build test workflow is currently problematic.

1. End to end build tests (as implemented on 199 cordex xfails #304) work when run directly, on Gadi, ie

$ ssh gadi 
[ct1163@gadi-login-07 ~]$ qsub /g/data/xp65/admin/access-nri-intake-catalog/bin/test_end_to_end.sh
130528194.gadi-pbs
[ct1163@gadi-login-07 ~]$ cat test_end_to_end.sh.o130528194 
...

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========== 108 passed, 1 xfailed, 57 warnings in 896.94s (0:14:56) ============

======================================================================================
                  Resource Usage on 2024-12-12 13:42:25:
   Job Id:             130528194.gadi-pbs
   Project:            iq82
   Exit Status:        0
   Service Units:      8.10
   NCPUs Requested:    12                     NCPUs Used: 12              
                                           CPU Time Used: 00:37:50        
   Memory Requested:   64.0GB                Memory Used: 64.0GB          
   Walltime requested: 00:30:00            Walltime Used: 00:15:11        
   JobFS requested:    100.0MB                JobFS used: 3.13MB          
======================================================================================

2. Running the test from the workflow results in a failure status - despite exit status 0 being returned from the PBS job.

Additional things to note

• The workflow currently runs using @rbeucher's Gadi account, using the version of access-nri-intake-catalog contained within conda/analysis3-24.12, against the tests in /g/data/xp65/admin/access-nri-intake-catalog, from whatever branch is specified by the user in the workflow. This is fraught with potential errors, including but probably not limited to:
  • Git branch mismatch - If you run the workflow from branch x, but /g/data/xp65/admin/access-nri-intake-catalog is checked out to branch y, then the workflow will run against the tests contained in branch y, not x.
  • Package version mismatch: If the tests in branch y at /g/data/xp65/admin/access-nri-intake-catalog refer to or test changes introduced to the package since the released packaged installed in conda/analysis3-24.12, the tests will fail.

Describe the feature you'd like

The workflow should checkout the head commit of the branch it is triggered from, locally install that package version, and then run the end to end tests against that, before reporting back to the workflow.

This is currently not easily achievable without creating a bunch of build artefacts (virtual environments, PBS output, etc) in @rbeucher's user area. Creating a service user to handle this would be ideal.

Describe alternatives you've considered

N/A.