Code

Genie import DAG prototype#64

Code

Open

callachennault wants to merge 77 commits intocBioPortal:mastercBioPortal/datahub-study-curation-tools:masterfrom callachennault:aws-protocallachennault/datahub-study-curation-tools:aws-protoCopy head branch name to clipboard

+621-285

Commits

Commits on Sep 16, 2024

• Move imports, create params, pass env vars to tasks

  callachennault

  committed
  886e633

  Copy full SHA for 886e633

Commits on Sep 30, 2024

• Add executor config, dockerfile, and requirements

  callachennault

  committed
  2975eab

  Copy full SHA for 2975eab

• rename executor config

  callachennault

  committed
  0ff67e5

  Copy full SHA for 0ff67e5

• fix executor import

  callachennault

  committed
  9f391f6

  Copy full SHA for 9f391f6

• fix executor config import again

  callachennault

  committed
  2bdfb3b

  Copy full SHA for 2bdfb3b

• add sys path append before import

  callachennault

  committed
  7c72872

  Copy full SHA for 7c72872

Commits on Oct 2, 2024

• Make data and import dag, split bash operations into scripts

  callachennault

  committed
  5729010

  Copy full SHA for 5729010

• Fix params title

  callachennault

  committed
  244c729

  Copy full SHA for 244c729

• Fix dep chain

  callachennault

  committed
  07ab75a

  Copy full SHA for 07ab75a

• Change trigger rule for pull_data_from_synapse

  callachennault

  committed
  828f58a

  Copy full SHA for 828f58a

• comment out git push for now

  callachennault

  committed
  2d359e2

  Copy full SHA for 2d359e2

• remove handle_error function

  callachennault

  committed
  29bc3ab

  Copy full SHA for 29bc3ab

• Fix param name

  callachennault

  committed
  71d6a84

  Copy full SHA for 71d6a84

• remove handle_error function again

  callachennault

  committed
  3310744

  Copy full SHA for 3310744

• Add handle_error back in

  callachennault

  committed
  f400020

  Copy full SHA for f400020

• remove double curly brackets

  callachennault

  committed
  6db3691

  Copy full SHA for 6db3691

• add handle_error back to synapse script

  callachennault

  committed
  0cef4f4

  Copy full SHA for 0cef4f4

• add test ssh task

  callachennault

  committed
  f9be41c

  Copy full SHA for f9be41c

Commits on Oct 4, 2024

• Add import_genie_dag, rename old dag to genie_ssh_dag

  callachennault

  committed
  f8095d6

  Copy full SHA for f8095d6

• rename tasks

  callachennault

  committed
  20d104f

  Copy full SHA for 20d104f

• add double brackets to params

  callachennault

  committed
  8175e92

  Copy full SHA for 8175e92

• remove param to test

  callachennault

  committed
  0aaea0d

  Copy full SHA for 0aaea0d

• fix command arg

  callachennault

  committed
  40515b9

  Copy full SHA for 40515b9

• change default value for importer param and fix logic bug

  callachennault

  committed
  ba119ac

  Copy full SHA for ba119ac

Commits on Oct 7, 2024

• Add arg for passing scripts directory

  Show description for 46046dc
  averyniceday

  authored
  46046dc

  Copy full SHA for 46046dc

• Update import_genie_dag.py

  averyniceday

  authored
  f61ed73

  Copy full SHA for f61ed73

• Merge pull request #1 from averyniceday/patch-1

  Show description for 7cc0321
  callachennault

  authored
  7cc0321

  Copy full SHA for 7cc0321

• Update import_genie_dag.py

  averyniceday

  authored
  57d1bd9

  Copy full SHA for 57d1bd9

• Merge pull request #2 from averyniceday/patch-1

  Show description for 82383aa
  averyniceday

  authored
  82383aa

  Copy full SHA for 82383aa

• Update import_genie_dag.py

  averyniceday

  authored
  62e251f

  Copy full SHA for 62e251f

• Update import_genie_dag.py

  averyniceday

  authored
  e4ea5b5

  Copy full SHA for e4ea5b5

• Update import_genie_dag.py

  averyniceday

  authored
  9875ec0

  Copy full SHA for 9875ec0

• Update import_genie_dag.py

  averyniceday

  authored
  2d8f236

  Copy full SHA for 2d8f236

• Update import_genie_dag.py

  averyniceday

  authored
  3282997

  Copy full SHA for 3282997

• Update import_genie_dag.py

  averyniceday

  authored
  d0d1fe0

  Copy full SHA for d0d1fe0

• Update import_genie_dag.py

  averyniceday

  authored
  2de1e22

  Copy full SHA for 2de1e22

• Update import_genie_dag.py

  averyniceday

  authored
  7307cde

  Copy full SHA for 7307cde

• Update import_genie_dag.py

  averyniceday

  authored
  90b821e

  Copy full SHA for 90b821e

• Update import_genie_dag.py

  averyniceday

  authored
  3236e80

  Copy full SHA for 3236e80

• Update import_genie_dag.py

  averyniceday

  authored
  7ffaf48

  Copy full SHA for 7ffaf48

• Update import_genie_dag.py

  averyniceday

  authored
  ba03aac

  Copy full SHA for ba03aac

• Update import_genie_dag.py

  averyniceday

  authored
  a5fcc25

  Copy full SHA for a5fcc25

• Update import_genie_dag.py

  averyniceday

  authored
  0b6b5cf

  Copy full SHA for 0b6b5cf

• Update import_genie_dag.py

  averyniceday

  authored
  a1cd6ea

  Copy full SHA for a1cd6ea

• Update import_genie_dag.py

  averyniceday

  authored
  5c13bed

  Copy full SHA for 5c13bed

Commits on Oct 10, 2024

• Update import_genie_dag.py

  averyniceday

  authored
  1a316f6

  Copy full SHA for 1a316f6

Commits on Oct 21, 2024

• Update import_genie_dag.py

  averyniceday

  authored
  5975481

  Copy full SHA for 5975481

• Update import_genie_dag.py

  averyniceday

  authored
  224683f

  Copy full SHA for 224683f

Commits on Oct 22, 2024

• Update import_genie_dag.py

  averyniceday

  authored
  1dfb454

  Copy full SHA for 1dfb454

• Update import_genie_dag.py

  averyniceday

  authored
  d621bec

  Copy full SHA for d621bec

Commits on Oct 28, 2024

• add genie repo to study path

  callachennault

  committed
  3acace1

  Copy full SHA for 3acace1

• move folder contents instead of folder

  callachennault

  committed
  a674cb4

  Copy full SHA for a674cb4

• cp files instead of mv

  callachennault

  committed
  eebce75

  Copy full SHA for eebce75

• print git error output

  callachennault

  committed
  b61a52e

  Copy full SHA for b61a52e

• print cwd

  callachennault

  committed
  281e137

  Copy full SHA for 281e137

• print cwd

  callachennault

  committed
  8051e8a

  Copy full SHA for 8051e8a

• hardcode path for testing

  callachennault

  committed
  504f08a

  Copy full SHA for 504f08a

• try to run git status

  callachennault

  committed
  3871153

  Copy full SHA for 3871153

Commits on Oct 29, 2024

• Make git_push a bash task

  callachennault

  committed
  4f7d8c4

  Copy full SHA for 4f7d8c4

• set up git user after clone

  callachennault

  committed
  7dfc4ad

  Copy full SHA for 7dfc4ad

• move git commit user setup to git push script

  callachennault

  committed
  47ef2de

  Copy full SHA for 47ef2de

• remove python git push task, add git fetch, change push_to_repo to boolean

  callachennault

  committed
  691bd21

  Copy full SHA for 691bd21

• add fetch genie script

  callachennault

  committed
  719f5f7

  Copy full SHA for 719f5f7

Commits on Oct 30, 2024

• use python task decorator, update fetch script and git push

  callachennault

  committed
  d3efb5f

  Copy full SHA for d3efb5f

• instantiate decorated tasks

  callachennault

  committed
  d5e1cd8

  Copy full SHA for d5e1cd8

• add git fetch to task chain

  callachennault

  committed
  9b14023

  Copy full SHA for 9b14023

• Try fixing python script import

  callachennault

  committed
  aceaa86

  Copy full SHA for aceaa86

• Try short circuit operator

  callachennault

  committed
  97fb5ec

  Copy full SHA for 97fb5ec

• remove native obj flag

  callachennault

  committed
  3200754

  Copy full SHA for 3200754

• update triggger rules

  callachennault

  committed
  f2f3334

  Copy full SHA for f2f3334

• Clean up dag, add clean_genie_repo task

  callachennault

  committed
  d35e7ee

  Copy full SHA for d35e7ee

Commits on Nov 27, 2024

• Clean up genie import dag

  callachennault

  committed
  19502eb

  Copy full SHA for 19502eb

Commits on Dec 3, 2024

• Update start_date, add max_active_runs, remove unused default env vars

  callachennault

  committed
  db751af

  Copy full SHA for db751af

Commits on Dec 5, 2024

• remove genie ssh dag

  callachennault

  committed
  d265ca5

  Copy full SHA for d265ca5

Commits on Jan 7, 2025

• Add import scripts path as arg to clone_db_wrapper

  callachennault

  committed
  97e5dff

  Copy full SHA for 97e5dff

Commits on Jan 10, 2025

• Set schedule_interval to None

  callachennault

  committed
  4fdb015

  Copy full SHA for 4fdb015

Commits on Jan 15, 2025

• Cleanup dag, change email notif, add import_status task and watcher task

  callachennault

  committed
  3a83012

  Copy full SHA for 3a83012