Add functionality for other backend services (besides SLURM)

[joecummings]

No description provided.

[lichtefeld]

@joecummings Do you have a specific backend service that should be my next task to implement after the 5.0 upgrade is complete?

[joecummings]

I was thinking of expanding to include cloud services like https://pegasus.isi.edu/documentation/user-guide/execution-environments.html#cloud-aws-google-jetstream for robustness. I rate this as a medium-low priority because I don't believe these services are widely used within the projects with which I'm familiar.

[lichtefeld]

@marjorief Does it make sense for this wrapper to also support the 'Donut' cluster that is available for ISI at large? (If so I'll break that feature request off into its own issue as ideally its closer to how the SAGA cluster is supported rather than a cloud provider.)

@joecummings Supporting either AWS EC2 or Google Cloud Platform is not as plug-and-play as using the SAGA cluster is. In order to configure either of those remote platforms, the instance needs to be setup as an HTCondor worker that can communicate with the HTConder master node. Given this overhead, a good first step may be to support the addition of custom sites to the internal sites catalog which the wrapper generates but leave the other overhead of configuring a job to run on that site to the workflow creator. It does however look like supporting AWS Batch via Pegasus is an easier initial configuration as this service configures the compute nodes used automatically based on the parameters provided.

If supporting these remote compute environments is a priority (or becomes one) I think any service option could be accomplished as a medium-sized task.