Using files mounted on EFS as input files in AWS Batch

[jameswagner]

Hello, I'd like to use files that are on an EFS mount as input files but when these are passed in, I get a file not found error. It seems these are looking for files on the workflow container, and the aws.batch.volumes configuration option puts the mount in the process container.

I thought this was maybe a problem solved with xpack-amzn plugin, but after adding an evaluation license key this continues to persist. Is this something that is not supported even with the plugin?

[pditommaso]

Please reach Seqera support via https://support.seqera.io/

[odoublewen]

I'm think I'm trying to do more or less the same thing. I have input data for my workflow that is already on a network storage appliance, hosted by AWS. We can mount the volume into our Batch EC2 instances. But when we run the workflow with Batch executor, Nextflow of course still thinks it has to stage the data from the mount (on the head node) to s3, and from s3 back to the Batch instance. Is there a way to get Nextflow to "realize" that certain local paths will exist on the Batch instance, and thus avoid the unnecessary network traffic?

I see aws.batch.volumes in the aws config scope and it almost seems like the intent of this config is to enable what is described above.

Hi @jameswagner did you ever figure this out?