v0.6.5

miniwdl run

• Add example download plugins (API may still change)
• Set docker LogConfig for resource detection container so it works as expected with non-default configuration e.g. ECS (#340 @kislyuk)

Standing limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)

Please vote on these and/or report other interoperability problems via Issues!

WDL

• Statically prevent workflow from calling itself
• Make size() work on null inputs, per WDL spec
• lark-parser 0.8.1

v0.6.4

miniwdl run

• Fix Docker failure with lengthy call names (#327 @kislyuk)
• Use POSIX ERE for sub() (#321 @DavyCats cf. openwdl/wdl#243)
• macOS compatibility improvements
  • default --runtime-cpu-max and --runtime-memory-max to limits of Docker for Mac's virtual machine
  • miniwdl run_self_test failure displays a hint on overriding TMPDIR
  • remaining issues tracked in #145

miniwdl check

• Whitelist additional task runtime keys (#323 @claymcleod)

WDL

• Improve type inference for Map literals (#256 @nh13)
• Improve type inference for read_json() (#320 @illusional)

v0.6.3

miniwdl run

• New interface to "plug in" support for additional input file URI schemes such as s3://, with security credentials. Example here but the interface may still change.
• The output environments and JSON now refer to Files via the symbolic links created in the output_links subdirectory, which in turn refer to the original (hard-linked) files. This means all File outputs exhibit paths within that directory, making them easier to postprocess.
• Input JSON keys for subworkflow calls may now (but still needn't) include the subworkflow name as a namespace following the call name, as expected by Cromwell (#193 @DavyCats)
• Understand ~ as home directory in command-line file input (@kislyuk)
• Remove query string from local filename of WDL imported by URI (#312 @kislyuk)

Standing limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)

Please vote on these and/or report other interoperability problems via Issues!

WDL

• Forbid any call's name from colliding with that of the containing workflow.

v0.6.2

Getting Started

miniwdl run

• Accept HTTP[S] URIs for the -i input JSON file
• Fix log file handle leak which would eventually cause 'Too many open files' error (#301)
• Improved error reporting in file URI download step
• Document guidelines for optimal performance and reliability

Standing limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)

Please vote on these and/or report other interoperability problems via Issues!

miniwdl check

• Prevent StringCoercion and OptionalCoercion warnings on some typical interpolation use cases
• Improve UnusedImport detection

WDL

• Permit "comments" in input JSON (ignore keys starting with #)
• Allow URI-downloaded WDL to use .. updir relative imports

v0.6.1

miniwdl localize

This new tool receives WDL and a Cromwell-style input JSON for it, which may reference Files by URIs. It downloads such URIs to a local directory, and generates equivalent JSON with the local filenames.

Although miniwdl run can download URI inputs without this intermediate step, this tool can be helpful to avoid repeating downloads across multiple runs/attempts.

WDL

• Permit "~{sep=' ' array}" interpolations where array item type may be optional (null values are written as empty strings)
• Fix negation/infix operator precedence in WDL boolean expressions
• Forbid identifiers starting with _, per WDL spec

miniwdl run

• Permit relative imports from WDL files downloaded from URI
  • It's now possible to run many workflows from the internet without manually downloading anything in preparation, e.g. miniwdl run https://raw.githubusercontent.com/my/repo/master/workflow.wdl input1=https://s3.amazonaws.com/mybucket/file.txt
• --runtime-defaults accepts a JSON object or filename with default values for all task runtime sections
• --max-runtime-{cpu,memory} renamed to --runtime-{cpu,memory}-max for consistency

Standing limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)

Please vote on these and/or report other interoperability problems via Issues!

Housekeeping

• Sped up Travis CI builds by (i) splitting the suite across multiple jobs and (ii) caching slow-evolving apt/PyPI dependencies in reusable docker image layers.

v0.6.0

miniwdl run

• --dir behavior revised to be more predictable: always run in a new timestamp-named subdirectory of the specified directory, unless the latter ends with .
• Automatically download input files specified as public http[s]/ftp/gs URIs
• Honor runtime.maxRetries to retry task command or docker failures
• output_links feature expanded to apply to all nested task/workflow runs, and include Files embedded in structs and other composite output values
• Manipulate ownership/permissions to improve security and user experience:
  • as run completes, chown everything in run directory to invoking user and primary group
  • run task commands with membership in invoking user's primary group, ensuring access to working directory even if they've dropped privileges (a good docker security practice)
  • --as-me to force all task commands to run as invoking user (more secure, but blocks commands that assume root e.g. apt-get)
• Revert to always running command in bash (as required by WDL spec) rather than container $SHELL

miniwdl check

• Suppress UnnecessaryQuantifier for optional File? task outputs

v0.5.2

miniwdl run

• generates a "rerun" script in the run directory, which can be sourced from the shell to re-run with the same inputs (#149)
• use docker image's $SHELL to interpret task command, instead of hard-coding bash
• log messages have been revised and reformatted to facilitate programmatic consumption (#248)
• probable fix for a sporadic hang under erratic host performance, such as when out of memory (#263)
• based on runtime.memory, create host memory reservations to avoid overbooking parallel memory-intensive tasks (#250)
  • memory reservations affect task scheduling, but aren't enforced limits (unlike runtime.cpu, which creates both reservations and limits)
• shares thread pools to respect task concurrency limits across sub-workflow calls (#203)

Known limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)
• Scheduler ignores runtime.disks and runtime.maxTries (#251)

Please vote on these and/or report other interoperability problems via Issues!

v0.5.1

miniwdl run

• support optional File? outputs from tasks (#255, openwdl/wdl#310)
• display the available inputs/outputs in source order
• fix duplication in workflow.log file

Known limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)
• Scheduler considers each tasks's runtime.cpu (default 1) versus the available host processors, but ignores runtime.{memory,disks} (#250)
• Scheduler may exceed the -@ parallelism limit argument when sub-workflows are called, potentially up to # host processors (#203).

Please vote on these and/or report other interoperability problems via Issues!

miniwdl cromwell

• Cromwell 47

WDL

• permit unbound, optional declarations in task/workflow bodies (outside of inputs) as a way of creating a null value for later use (#253)
• WDL.Type.Any always coerces to/from optional types
• preserve Value.expr references when value is coerced

v0.5.0 parallel workflows on local host

miniwdl run

The built-in runner can now parallelize tasks and sub-workflows on the local host, using Docker Swarm locally to schedule containers (which it'll enable automatically). Start with miniwdl run_self_test to quickly test the installation viability.

Known limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)
• Scheduler considers each tasks's runtime.cpu (default 1) versus the available host processors, but ignores runtime.{memory,disks} (#250)
• Scheduler may exceed the -@ parallelism limit argument when sub-workflows are called, potentially up to # host processors (#203).

Please vote on these and/or report other interoperability problems via Issues!

Other changes:

• Add :latest tag to runtime.docker if no other tag present (#232 @jdidion)
• Mount all input files into one in-container directory (except when filenames would collide) to improve interoperability using hts index files

miniwdl cromwell

• Cromwell 46.1

miniwdl check

• Suppress UnusedImport warning when imported struct definitions are used (#236 @DavyCats)

WDL

• Fix str() on Apply AST nodes for certain functions (#242 @DavyCats)

v0.4.1 maintenance

miniwdl run

• reduce likelihood of CPython RecursionError due to deep-copying of WDL values (#239 @nh13)

Standing limitations:

• Calls run sequentially (#202, #203)
• Task input files are mounted read-only; commands to rename or remove them will fail (#210)
• Task output files must reside in or under the initial working directory (#214)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)

Please vote on these and/or report other interoperability problems via Issues!

miniwdl cromwell

• accept relative file paths in input JSON file, interpreting them relative to current working directory (#230 @prihoda)
• Cromwell 45.1

WDL

• recognize version development and use the most up-to-date grammar available (#233 @jdidion)