v0.8.3

miniwdl run

• Task call caching (experimental): cache entries are automatically invalidated if any referenced input/output files were removed or modified

v0.8.2

miniwdl run

File URI download improvements:

• Allow tasks' as well as workflows' File inputs to declare default URI e.g. File an_input = "https://..."
  • Use this feature with caution, as it's not explicitly allowed (nor disallowed) by the WDL specification; setting URIs in JSON inputs is more portable
• Prevent run failures caused by multiple concurrent attempts to download the same URI
• Enable download of public S3 objects even if operative IAM policy prevents access to the containing bucket
• Workflow calls are cached based on original input URIs instead of temporary/cached local copy filename

miniwdl check

• Add warning for unbound declarations outside of task/workflow input{} sections in WDL 1.0 documents

v0.8.1

miniwdl run

• Polish CLI with parameter_meta and defaults for input help, ANSI colors, and clarified error messages (@dpark01 @slzarate)
• Fix bugs handling input/output files with unusual characters in their names (#409 #410 #415)
• --error-json reports errors in parsing WDL source code or validating inputs as well as execution (#384 @MarkAZhang)
• Call caching: refined task digest algorithm to ignore WDL comments and most whitespace outside of command script
  • Effectively invalidates existing cache entries (no action required; fine to delete them)
  • More such invalidations are likely to come, as the feature continues maturing
• Legacy note: The out/ folder of output file links found in each run folder was called output_links/ prior to v0.8.0. Currently a symlink output_links/ -> out/ is generated as well, but this will be discontinued in the future.

WDL

• Improved type unification algorithm for compound types (#406)

meta

• Relax requirements.txt version constraints (#413 @TMiguelT)
• Add flake8 linter (#407 @kislyuk)

v0.8.0

miniwdl run

• File inputs provided as s3:// URIs can be downloaded automatically without an extra plugin.
  • opt-in configuration to detect host's AWS credentials
  • otherwise an EC2 instance profile can be assumed, or public S3 URIs are always accessible
  • adds to existing support for public gs:// and web URIs.
• The generated folder of symlinks to output files is renamed from output_links/ to out/ for brevity.
  • For now an output_links/→out/ symlink appears as well; this backwards-compatibility feature will be removed in the future.
• Experimental call caching can be enabled.
  • This feature will continue maturing in subsequent versions, probably invalidating existing cached results.

miniwdl check

• On syntax or typechecking error, suggest adding WDL version declaration if absent.

WDL package

• Each node in the abstract syntax tree returned by WDL.load() now has a parent attribute, via which one can navigate from any node back up to the root WDL.Document.

miniwdl cromwell

• Retired per #351

v0.7.5

miniwdl run

• improve JSON input validation error messages (#389)
• -i input.json accepts YAML as well as JSON (non-standard feature)
• speed up task directory postprocessing on network file systems
• expanded documentation

WDL package

• development library functions suffix() quote() squote() (openwdl/wdl#362 #379 @patmagee)
• development grammar removes certain string interpolation options and adds sep() function (openwdl/wdl#229 #381 @illusional)
• fix WDL.Value.rewrite_files() on compound values that use the same File value multiple times
• size() honors unnecessary 'B' unit for bytes

miniwdl cromwell

Deprecation notice: miniwdl cromwell will be retired in the near future (#351)

v0.7.4

miniwdl run

• Terminal logging displays a bottom "status bar" with the elapsed time, tasks finished/ready/running, and their CPU and memory reservations. Black terminal background suggested for optimal experience.
• For use with log collectors, --log-json writes logs in JSON instead of the default format, affecting both the standard error stream and log files left in the run directory.

miniwdl check

• New MissingVersion warning on documents that don't begin by declaring the WDL language version (and therefore default to the trending-obsolete draft-2)

WDL package

• WDL.Document.wdl_version exposes the declared language version, if any

miniwdl cromwell

Deprecation notice: miniwdl cromwell will probably be retired soon, subject to feedback on #351

v0.7.3

miniwdl check

• Individual lint warnings can be suppressed by a WDL comment containing the string !WarningName on the same or following line (#183)
• --no-suppress displays all lint warnings, even suppressed ones
• --strict causes non-zero exit status if any lint warning is displayed (#315)

miniwdl run

• _LAST symbolic link: created alongside each default auto-provisioned, timestamp-named run directory, to make it easier to refer to the most recent run from the command line.
• config option [task_runtime] memory_limit_multiplier to set hard limits on container memory usage based on runtime.memory (which had been used as a scheduling reservation only)
• fix sporadic race conditions when multiple miniwdl processes run concurrently on the same host/swarm (#368 #374)

WDL package

• Add WDL.Document.{source_lines,source_comments} to represent comments in the original source code
• Tighten validation and error messaging for import namespaces (#370)

miniwdl cromwell

Deprecation notice: miniwdl cromwell will probably be retired soon, subject to feedback on #351

v0.7.2

miniwdl run

• make output_links/ symlinks relative instead of absolute (when the target is within the run directory)
• flock the top-level workflow.log file (task.log if running a task directly) to help other processes determine if run is still underway
• task parallelism scalability improvements from stress-testing

WDL package

• Adds helper Value.rewrite_files() and Value.rewrite_env_files()

miniwdl cromwell

Deprecation notice: miniwdl cromwell may be retired in the medium-term future; see and comment on #351

v0.7.1

miniwdl run

• Workflow-level plugin coroutine interface
• Fix errant behavior with multiple installed task plugins
• New examples of plugins customizing the runner, logging task CPU/memory usage and uploading outputs to cloud storage. (Full plugin API documentation is still needed.)
• Ensure creation of empty output_links/ folder in run directory, even if task outputs no files
• Config option to generate output_links/ with hardlinks instead of symbolic links ([file_io] output_hardlinks)
• Config option to delete task container working directory after completion ([task_runtime] delete_work)
• Ensure graceful abort on receipt of SIGPIPE

WDL package

• read_json() infers WDL types for nested lists/objects, so that WDL accessor expressions can navigate them (#320 #331 @illusional)
  • Add WDL.Type.unify() supporting this
• Prevent confusion between names beginning with runtime, meta, and parameter_meta and actual keywords (#363 @DavyCats)
• bump lark-parser version

miniwdl localize

Following last month's deprecation notice, this tool has been repurposed to provide a way to "prime" the URI file download cache based on given workflow inputs and/or specified URIs, before actually running the workflow. Doing so can prevent download effort duplication if it's planned to launch multiple concurrent workflows using common reference data URIs.

miniwdl cromwell

Deprecation notice: miniwdl cromwell may be retired in the medium-term future; see and comment on #351

v0.7.0

miniwdl run

• New configuration loader sourcing from command-line arguments, environment variables, and/or .cfg file.
• URI input file downloads can be cached for reuse in multiple runs. (Opt-in via configuration)
• Rigorize error logging and reporting, including error.json left in each failed run directory (#346)
• Store a copy of the original WDL source files in each top-level run directory (#292)
• Task runner plugins are able to manipulate inputs/outputs, commands, etc. (example). Workflow-level and other plugin extension points still to come.
• File URI download plugin interface revised (breaking changes) to fit with the new plugin scheme (example)

miniwdl cromwell

Deprecation notice: miniwdl cromwell may be retired in the medium-term future; see and comment on #351

miniwdl localize

Deprecation notice: miniwdl localize will probably be retired shortly, superseded by the new URI cache feature.