Remove double process execution monitoring layers

[fmigneault]

Context & Feature

At the moment, processes are executed via monitoring the WPS-1 endpoint provided by pywps with an owslib.wps.WebProcessingService. This pywps also monitors the sub-job execution, creating unnecessary layers of job monitoring, harder debugging, and complicated logging.

A single celery job monitor should be implemented, directly monitoring the real job execution of the process (cwltool). Also, because celery only monitors the status/result of the WPS Execution sent to pywps which resides under the same app as the API, the cwltool operations are actually executed by a thread-worker of the API (weaver manager) instead of a celery worker. This is simply not the desired behavior, cannot be scaled, and doesn't use full worker/queue capabilities.

The full stack of process/monitor is as follows:

1. [EMS] POST Weaver /job, submits job to db
2. Celery worker picks job and calls WPSExecute on PyWPS and monitors status until success/failure
3. PyWPS receives the job (Weaver manager API side) and creates its interal workers to run it
4. PyWPS handler runs job (cwltool or remote wps)

In the case of EMS-workflow, this whole stack is repeated for the underlying ADES called the same way for each step.

Considerations

We need to consider what to do about validation of WPS I/O. This was the reason the pywps layer was originally employed.

• Since executed cwltool or remote WPS service called already do this kind of validation on their side by validating the received inputs against the package/process definition, we could simply report the invalid input received by them when the ADES/EMS attempts to execute it. We would only need to return this error from the job execution, but no preemptive validation of I/O would be done.
• other approach???

The WPS-1/2 endpoint with pywps should be preserved for the sole purpose of providing back-compatibility with WPS-1/2 by redirecting the job submission just as the WPS-3 REST-API does, using following correspondance (see also #126):

• GetCapabilities => GET /processes
• DescribeProcess => GET /processes/{id}
• Job Execute => POST /processes/{id}/jobs
• Job GetStatus (n/a) use GET /processes/{id}/jobs/{id} instead

With celery>=4.3, the Task Result has result_extended option which allows to store additional metadata about received inputs/function-name/etc. This should be enabled to have even better tracking of executed/pending tasks.

Helpful References

• example celery config with docker-compose: https://stackoverflow.com/a/45908901/5936364
• celery configs: https://docs.celeryproject.org/en/4.4.2/userguide/configuration.html#configuration-directives