[Feature] Even further job logging improvements

[fmigneault]

Context

PR #80 provided some improvements for logging execution percentage of the job.
Still, when the underlying job process is running a CWL package, any logging/output is lost (especially via a docker execution) since printed lines are not added to the WPS execution log.

To solve this issue, we can add a stdout capture of the CWL process that will act as a hook to capture and extend the WPS log with the app's logs. With this we get as fine grained details as printed out by the app.
https://www.commonwl.org/user_guide/05-stdout/index.html

Only remaining issue lies with current job monitoring using an internal WPS (see #21). Because of this, the full log is only retrieved at the end of the job execution, instead of gradually as they are being printed out.

weaver/weaver/wps_restapi/processes/processes.py

Lines 199 to 203 in d2992ab

	if execution.isSucceded():
	job.status = map_status(STATUS_SUCCEEDED)
	job.status_message = "Job succeeded{}.".format(msg_progress)
	wps_package.retrieve_package_job_log(execution, job)
	job.save_log(logger=task_logger)

This removes any opportunity to provide job 'progression' reporting by the app, which is very inconvenient for long running applications that provide gradual updates.

Unless using very dirty hacks, there is no way to directly pass the job uuid to the sub-process (celery task), so that logs can be appended gradually, because it itself relies on the owslib WPSExecute request sent to PyWPS Service.

Investigate

All above 'progressive' logging assume that it is possible to retrieve the log asynchronously while a docker application is running.
This needs to be investigated further as it could be not available until CWL completes the subprocess.
edit: quick test using tail -f std[err|out].log shell call show that the file is correctly updated while the docker app is running locally. We should only need to retrieve them and append them to the job logs. This can be done directly at WpsPackage level. For remote process we can't do more than what is provided on the status monitoring route (ie: what is already done since PR #132).

Steps

• add the 'base' CWL stdout hook to at least retrieve the full log at the end
• implement Remove double process execution monitoring layers #21
• use async routine that gradually appends the new log entries to the provided job full log to get gradual updates of the job execution (will require managing some concurrent job read/saves)

Relates to DAC-535