Code

handle CWL Directory type#474

Code

Merged

fmigneault merged 104 commits intomastercrim-ca/weaver:masterfrom dir-typecrim-ca/weaver:dir-typeCopy head branch name to clipboardNov 23, 2022

+5,262-1,132

Commits

Commits on Oct 15, 2022

• first impl to handle CWL Directory type (relates to #466)

  fmigneault

  committed
  25f47e3

  Copy full SHA for 25f47e3

• fix lint

  fmigneault

  committed
  220bedd

  Copy full SHA for 220bedd

• improve typings

  fmigneault

  committed
  164f240

  Copy full SHA for 164f240

• ignore duplicate S3 bucket creation during tests

  fmigneault

  committed
  fdc8bb8

  Copy full SHA for fdc8bb8

• [wip] obtain remote dir listing for CWL Directory input

  fmigneault

  committed
  03dcb1a

  Copy full SHA for 03dcb1a

• replace dir-listing by dir-fetching method + obtain option kwargs by scheme for fetch-file/dir

  fmigneault

  committed
  92cdcad

  Copy full SHA for 92cdcad

• fix linting issues with pylint=2.15.4

  fmigneault

  committed
  4d8e630

  Copy full SHA for 4d8e630

Commits on Oct 17, 2022

• handle repr_json format datetime + listing S3 bucket contents

  fmigneault

  committed
  f76e3db

  Copy full SHA for f76e3db

Commits on Oct 18, 2022

• [wip] download pool for dir listing + impl handlers by schemes

  fmigneault

  committed
  e8c3ae7

  Copy full SHA for e8c3ae7

• impl dir-type handling per ref scheme

  fmigneault

  committed
  dd8d386

  Copy full SHA for dd8d386

• improve typings + reuse CWL IO def class + tests for dir resolution

  fmigneault

  committed
  d0da33f

  Copy full SHA for d0da33f

Commits on Oct 19, 2022

• fix parameters passed to fetch_file

  fmigneault

  committed
  3e4afc2

  Copy full SHA for 3e4afc2

Commits on Oct 21, 2022

• add multiple dir listing test cases parsing different HTML representations

  fmigneault

  committed
  a441b73

  Copy full SHA for a441b73

• more tests for dir filter listing + refactor s3 dir listing/download together

  fmigneault

  committed
  316eeb1

  Copy full SHA for 316eeb1

• local directory handling with different output methods + more tests for dir handling

  fmigneault

  committed
  77f6627

  Copy full SHA for 77f6627

Commits on Oct 23, 2022

• more test conditions and operations to run dir listing

  fmigneault

  committed
  dc2b9cb

  Copy full SHA for dc2b9cb

• fixing tests and filtering dirs

  fmigneault

  committed
  eda654f

  Copy full SHA for eda654f

• fix more test cases

  fmigneault

  committed
  b5e1728

  Copy full SHA for b5e1728

• fix requirements-dev dependencies for pylint

  fmigneault

  committed
  b2c4c04

  Copy full SHA for b2c4c04

• fix lint + invalid CWLIODefinition class use

  fmigneault

  committed
  606fd2c

  Copy full SHA for 606fd2c

• Merge branch 'master' into dir-type

  fmigneault

  authored
  3892cee

  Copy full SHA for 3892cee

• fix lint & download file path ref

  fmigneault

  committed
  38999f4

  Copy full SHA for 38999f4

Commits on Oct 24, 2022

• fix warning BeautifulSoup using LXML parser

  fmigneault

  committed
  9955812

  Copy full SHA for 9955812

• replace dataclass by namedtuple + fix convert tests

  fmigneault

  committed
  f2c2c48

  Copy full SHA for f2c2c48

• working unit tests for fetch_directory

  fmigneault

  committed
  7111565

  Copy full SHA for 7111565

• revert namedtuple to allow mutable definitions

  fmigneault

  committed
  0d544ae

  Copy full SHA for 0d544ae

• fixes for CWL I/O definition parsing

  fmigneault

  committed
  a52cb0e

  Copy full SHA for a52cb0e

Commits on Oct 25, 2022

• fix unit tests

  fmigneault

  committed
  2c8d48f

  Copy full SHA for 2c8d48f

• minor text edits

  fmigneault

  committed
  95c83a5

  Copy full SHA for 95c83a5

• add missing docstring for CWLIODefinition.symbols

  fmigneault

  committed
  dca32ac

  Copy full SHA for dca32ac

• define PACKAGE_FILE_TYPE and PACAKGE_DIRECTORY_TYPE definitions

  fmigneault

  committed
  8916a1a

  Copy full SHA for 8916a1a

• fix docf lint

  fmigneault

  committed
  5a5c938

  Copy full SHA for 5a5c938

• fix typing definitions

  fmigneault

  committed
  6382391

  Copy full SHA for 6382391

• fix imports order

  fmigneault

  committed
  1a6915b

  Copy full SHA for 1a6915b

• fix dir listing functional tests

  fmigneault

  committed
  0223fa6

  Copy full SHA for 0223fa6

• fix imports lint

  fmigneault

  committed
  59da430

  Copy full SHA for 59da430

• rename FormatDefault oneOf[mediaType, mimeType] without 'deploy' mention since also used in execute inputs

  fmigneault

  committed
  895eee2

  Copy full SHA for 895eee2

• fix typos

  fmigneault

  committed
  7907adc

  Copy full SHA for 7907adc

• Merge branch 'master' into dir-type

  fmigneault

  authored
  0ef87f7

  Copy full SHA for 0ef87f7

Commits on Oct 26, 2022

• Merge branch 'master' into dir-type

  fmigneault

  authored
  6bdf26f

  Copy full SHA for 6bdf26f

• support aws s3 access-point and virtual-host style URIs

  fmigneault

  committed
  62931a4

  Copy full SHA for 62931a4

• Merge branch 'dir-type' of https://github.com/crim-ca/weaver into dir-type

  fmigneault

  committed
  cbcd079

  Copy full SHA for cbcd079

• Merge branch 'master' into dir-type

  fmigneault

  authored
  b53c4aa

  Copy full SHA for b53c4aa

• fix parsing of S3 references from HTTP URIs

  fmigneault

  committed
  387dc2e

  Copy full SHA for 387dc2e

Commits on Oct 27, 2022

• Merge branch 'master' into dir-type

  fmigneault

  authored
  c33af14

  Copy full SHA for c33af14

• Merge branch 'master' into dir-type

  fmigneault

  authored
  e85f7b1

  Copy full SHA for e85f7b1

Commits on Oct 31, 2022

• prepare process with dir as output

  fmigneault

  committed
  fb9537d

  Copy full SHA for fb9537d

• update docs with dir-type references

  fmigneault

  committed
  11edb9b

  Copy full SHA for 11edb9b

Commits on Nov 1, 2022

• add docs sample dir listing

  fmigneault

  committed
  48f0486

  Copy full SHA for 48f0486

• Merge branch 'master' into dir-type

  fmigneault

  authored
  cede281

  Copy full SHA for cede281

• Update CHANGES.rst

  Show description for b4aaec7
  fmigneault

  authored
  b4aaec7

  Copy full SHA for b4aaec7

• fix docs

  fmigneault

  committed
  b5d54da

  Copy full SHA for b5d54da

Commits on Nov 3, 2022

• update S3-related docs for dir-type

  fmigneault

  committed
  acb8324

  Copy full SHA for acb8324

• add more tests for S3 ref validation from http conversion

  fmigneault

  committed
  75e4107

  Copy full SHA for 75e4107

• more S3 reference formats validation, tests and documentation updates

  fmigneault

  committed
  eeeb6f3

  Copy full SHA for eeeb6f3

Commits on Nov 4, 2022

• more tests for AWS S3 parsing

  fmigneault

  committed
  9c3d760

  Copy full SHA for 9c3d760

• add test for paring S3 request configs

  fmigneault

  committed
  9222119

  Copy full SHA for 9222119

• fix py36 backport literal type arguments

  fmigneault

  committed
  354e1ae

  Copy full SHA for 354e1ae

Commits on Nov 7, 2022

• fix test for s3 ref resolution from http

  fmigneault

  committed
  5a030c5

  Copy full SHA for 5a030c5

• fix test mock:// scheme resolution with schema validators dynamically created from pattern regex

  fmigneault

  committed
  9d4bd11

  Copy full SHA for 9d4bd11

Commits on Nov 8, 2022

• add test for deploy/describe process with dir as output + support CWL InlineJavascriptRequirement

  fmigneault

  committed
  0276b6b

  Copy full SHA for 0276b6b

• [wip] add test with workflow using directory type

  fmigneault

  committed
  dddafbe

  Copy full SHA for dddafbe

• fix lint

  fmigneault

  committed
  db95dc7

  Copy full SHA for db95dc7

Commits on Nov 9, 2022

• setup directory storage utility + more robust storage locations using respective output IDs

  fmigneault

  committed
  071cc0d

  Copy full SHA for 071cc0d

• [wip] workflow step dir working - chaining/collect prev output to fix

  fmigneault

  committed
  f2e61b2

  Copy full SHA for f2e61b2

• fix lint

  fmigneault

  committed
  bafc70f

  Copy full SHA for bafc70f

• transfer all output storage setup to method

  fmigneault

  committed
  cf07ac4

  Copy full SHA for cf07ac4

• Merge branch 'master' into dir-type

  fmigneault

  authored
  1ee4223

  Copy full SHA for 1ee4223

• fix tests output location checks with updated nested output-id location

  fmigneault

  committed
  8a9d5b4

  Copy full SHA for 8a9d5b4

• fix test aws s3 ref validation

  fmigneault

  committed
  8ba5b7c

  Copy full SHA for 8ba5b7c

• patch type hints

  fmigneault

  committed
  f318e36

  Copy full SHA for f318e36

• remove tmp file added by mistake

  fmigneault

  committed
  3b11f86

  Copy full SHA for 3b11f86

• make better use of parametrized tests + adjust name of parsing/convert CWL utils

  fmigneault

  committed
  2f5dd83

  Copy full SHA for 2f5dd83

Commits on Nov 10, 2022

• fix file/dir path resolution for directory storage managing its contained files

  fmigneault

  committed
  638403c

  Copy full SHA for 638403c

• consider directory type in expected output pattern check to collect workflow step outputs

  fmigneault

  committed
  2faae69

  Copy full SHA for 2faae69

Commits on Nov 11, 2022

• functional test with workflow dir listing chaining between steps

  fmigneault

  committed
  1240852

  Copy full SHA for 1240852

• doc updates

  fmigneault

  committed
  a733047

  Copy full SHA for a733047

Commits on Nov 12, 2022

• patch partially invalid URL regex

  fmigneault

  committed
  330dfe7

  Copy full SHA for 330dfe7

• fix schema tests

  fmigneault

  committed
  513a405

  Copy full SHA for 513a405

• fix tests

  fmigneault

  committed
  c422c91

  Copy full SHA for c422c91

• add explicit error about unsupported CLI upload directory (download result supported)

  fmigneault

  committed
  79640c2

  Copy full SHA for 79640c2

• fix tests

  fmigneault

  committed
  69aae9f

  Copy full SHA for 69aae9f

• fix imports lint

  fmigneault

  committed
  f9360cb

  Copy full SHA for f9360cb

Commits on Nov 14, 2022

• more debug for workflow tests

  fmigneault

  committed
  9976b4d

  Copy full SHA for 9976b4d

• adjust workflow test considering possible distinct permissions during listing

  fmigneault

  committed
  6ef83b3

  Copy full SHA for 6ef83b3

• backport fix for python 3.6

  fmigneault

  committed
  550ef1a

  Copy full SHA for 550ef1a

Commits on Nov 15, 2022

• patch other misuse of re.Pattern in Python 3.6

  fmigneault

  committed
  10af4ff

  Copy full SHA for 10af4ff

• ensure all lint checkers are executed in test pipeline

  fmigneault

  committed
  8b8f2f6

  Copy full SHA for 8b8f2f6

Commits on Nov 16, 2022

• test dir output with S3 storage as WPS outputs

  fmigneault

  committed
  6eba738

  Copy full SHA for 6eba738

• Merge branch 'master' into dir-type

  fmigneault

  authored
  133d39b

  Copy full SHA for 133d39b

Commits on Nov 17, 2022

• Merge branch 'master' into dir-type

  fmigneault

  authored
  258948d

  Copy full SHA for 258948d

• force pywps create aws s3 bucket on test setup + validate aws s3 bucket/region info

  fmigneault

  committed
  23f0359

  Copy full SHA for 23f0359

Commits on Nov 18, 2022

• test s3 bucket ouputs - setup bucket to avoid not existing error after pywps setup

  fmigneault

  committed
  ca8b0f3

  Copy full SHA for ca8b0f3

• Merge branch 'master' into dir-type

  fmigneault

  authored
  4e2fc45

  Copy full SHA for 4e2fc45

• add missing definitions for InlineJavascriptRequirement

  fmigneault

  committed
  1b0e14c

  Copy full SHA for 1b0e14c

• fix invalid InlineJavascriptRequirement schemas

  fmigneault

  committed
  e3760f9

  Copy full SHA for e3760f9

Commits on Nov 22, 2022

• fix cwl conflicting/invalid schema definitions

  fmigneault

  committed
  bad4cf9

  Copy full SHA for bad4cf9

• revert unused util for separate PR

  fmigneault

  committed
  0d06349

  Copy full SHA for 0d06349

• add more tests for AWS S3 region/bucket validation

  fmigneault

  committed
  3acf954

  Copy full SHA for 3acf954

• improved bucket regex to handle more cases specified by AWS S3 naming rules

  fmigneault

  committed
  80a7344

  Copy full SHA for 80a7344

• add missing test for AWS S3 outposts endpoint resolution

  fmigneault

  committed
  624bdcc

  Copy full SHA for 624bdcc

• check unhandled file fetch scheme + ignore explicit raising OSError (intended early stop of problem cases)

  fmigneault

  committed
  cac0fd7

  Copy full SHA for cac0fd7

Commits on Nov 23, 2022

• add missing test cases for fetch_directory from JSON response and other invalid cases

  fmigneault

  committed
  e76a593

  Copy full SHA for e76a593

• ignore coverage backport code

  fmigneault

  committed
  2a6d7de

  Copy full SHA for 2a6d7de