Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor github workflows to DAG #307

Draft
wants to merge 117 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
5e18011
refactor github workflows
pawloch00 Jan 2, 2025
d41a1d6
split to smaller workflows
pawloch00 Jan 2, 2025
539d855
refactor
pawloch00 Jan 2, 2025
6028bcb
fix
pawloch00 Jan 2, 2025
fd13a92
fix cluster delete
pawloch00 Jan 2, 2025
048f57e
fix job name
pawloch00 Jan 2, 2025
2eeadca
set ubuntu version to fixed one
pawloch00 Jan 2, 2025
21af685
pass secretes to integration tests
pawloch00 Jan 2, 2025
f618ee4
fix cluster create
pawloch00 Jan 2, 2025
50d9729
fix
pawloch00 Jan 2, 2025
7c108df
pass creds to all workflows
pawloch00 Jan 2, 2025
51a64a6
update deletion
pawloch00 Jan 2, 2025
63c8e8b
fix
pawloch00 Jan 2, 2025
80d553b
change to string
pawloch00 Jan 3, 2025
25b536b
fix
pawloch00 Jan 3, 2025
fe583c7
test
pawloch00 Jan 3, 2025
4fcfd26
fix
pawloch00 Jan 3, 2025
f0c3d75
fix
pawloch00 Jan 3, 2025
7b4503f
fix
pawloch00 Jan 3, 2025
9d70213
fix
pawloch00 Jan 3, 2025
0f8b1d1
fix
pawloch00 Jan 3, 2025
537ef2e
fix
pawloch00 Jan 3, 2025
a061b4f
fix
pawloch00 Jan 3, 2025
66efe3f
fix
pawloch00 Jan 3, 2025
1a58ff0
fix zone
pawloch00 Jan 3, 2025
ca516e0
fix zone
pawloch00 Jan 3, 2025
556de1e
add run-integration-tests
pawloch00 Jan 3, 2025
3df12f7
fix network name env
pawloch00 Jan 3, 2025
fbbb1ba
fix
pawloch00 Jan 3, 2025
3d4a71c
fix cluster creation
pawloch00 Jan 7, 2025
415d84d
remove --private from cluster
pawloch00 Jan 7, 2025
8a09832
fix
pawloch00 Jan 7, 2025
4094698
fix tests
pawloch00 Jan 7, 2025
59d79ce
add docker auth
pawloch00 Jan 7, 2025
47478ff
fix cluster create
pawloch00 Jan 7, 2025
de9aaf0
fix expect install
pawloch00 Jan 7, 2025
33ac104
add private cluster test
pawloch00 Jan 7, 2025
ff2acc4
delete private cluster
pawloch00 Jan 7, 2025
5dc9d71
delete private cluster
pawloch00 Jan 7, 2025
e412dae
fix name
pawloch00 Jan 7, 2025
6da6800
fix
pawloch00 Jan 7, 2025
72f3ce8
add cluster authentication
pawloch00 Jan 7, 2025
cb44c14
add run-integration-tests
pawloch00 Jan 7, 2025
2af9a58
cluster private runs on same machine
pawloch00 Jan 7, 2025
1932e66
rename
pawloch00 Jan 7, 2025
8f9567e
remove comment
pawloch00 Jan 7, 2025
8f8f0ef
rm private
pawloch00 Jan 7, 2025
5a6c971
rm private
pawloch00 Jan 7, 2025
291d199
remove comment
pawloch00 Jan 7, 2025
cec7ece
fix concurency in deletion
pawloch00 Jan 7, 2025
1d55abe
fix
pawloch00 Jan 7, 2025
18de08f
fix formatting
pawloch00 Jan 7, 2025
3a1ff0d
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 8, 2025
d203127
fix
pawloch00 Jan 8, 2025
8b6aebd
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 15, 2025
d232010
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 17, 2025
ed6925b
change year
pawloch00 Jan 17, 2025
42d18c1
Merge branch 'ppawl-worfklows-cleanup' of https://github.com/AI-Hyper…
pawloch00 Jan 17, 2025
68ec37a
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 20, 2025
23c3a35
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 20, 2025
c3b16c1
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 20, 2025
67141a4
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 20, 2025
14e0f87
add cache
pawloch00 Jan 21, 2025
5b40522
Merge branch 'ppawl-worfklows-cleanup' of https://github.com/AI-Hyper…
pawloch00 Jan 21, 2025
8cf93b2
fix
pawloch00 Jan 21, 2025
8583f79
fix
pawloch00 Jan 21, 2025
6c7a406
create cache
pawloch00 Jan 21, 2025
83598ee
save to cache
pawloch00 Jan 21, 2025
dad8060
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 21, 2025
5572c00
fix
pawloch00 Jan 21, 2025
0c6b33f
Merge branch 'ppawl-worfklows-cleanup' of https://github.com/AI-Hyper…
pawloch00 Jan 21, 2025
978e5cd
fix
pawloch00 Jan 21, 2025
c84a382
use prefix
pawloch00 Jan 21, 2025
62ec63e
fix
pawloch00 Jan 21, 2025
699b804
fix
pawloch00 Jan 21, 2025
a45a0ad
fix
pawloch00 Jan 21, 2025
751246f
use cached deps
pawloch00 Jan 21, 2025
3738a24
add caching to linting
pawloch00 Jan 21, 2025
a82dd62
parallelize pytype
pawloch00 Jan 21, 2025
e844a39
fix
pawloch00 Jan 21, 2025
d60e163
fix
pawloch00 Jan 21, 2025
b6de3dc
add python packages directory
pawloch00 Jan 21, 2025
26845d9
fix
pawloch00 Jan 21, 2025
5a797b1
fix order
pawloch00 Jan 21, 2025
8e55531
add caching for tools binaries
pawloch00 Jan 21, 2025
6e33bd3
fix
pawloch00 Jan 21, 2025
66f2a4e
fix
pawloch00 Jan 21, 2025
088ee22
fix
pawloch00 Jan 21, 2025
ab8cdd4
install more deps
pawloch00 Jan 21, 2025
8d9ab57
use existing cache
pawloch00 Jan 21, 2025
4bf84fa
debug
pawloch00 Jan 21, 2025
32293ea
update path for linting
pawloch00 Jan 21, 2025
688f4b5
fix
pawloch00 Jan 21, 2025
ceb3acd
add debug output
pawloch00 Jan 21, 2025
7eede4b
fix
pawloch00 Jan 22, 2025
c05edb2
test
pawloch00 Jan 22, 2025
f569c72
fix
pawloch00 Jan 22, 2025
0e54cb1
fix
pawloch00 Jan 22, 2025
7375c03
fix
pawloch00 Jan 22, 2025
b129966
fix
pawloch00 Jan 22, 2025
81087f9
fix caching
pawloch00 Jan 22, 2025
136be15
add pip cache to cached paths
pawloch00 Jan 22, 2025
3523f17
fix
pawloch00 Jan 22, 2025
2520226
fix
pawloch00 Jan 22, 2025
c041d5c
fix
pawloch00 Jan 22, 2025
56f7f60
install deps only if cache not found
pawloch00 Jan 22, 2025
adc72d8
fix
pawloch00 Jan 22, 2025
818210f
dont install lint deps
pawloch00 Jan 22, 2025
3d3789e
fix
pawloch00 Jan 22, 2025
f3f7729
build cache for 3.11
pawloch00 Jan 22, 2025
281f29d
set n of jobs in pytype
pawloch00 Jan 22, 2025
aca1924
Merge branch 'main' into ppawl-worfklows-cleanup
pawloch00 Jan 23, 2025
2ea23e6
add option to rebuild cache
pawloch00 Jan 23, 2025
58ffa89
rm empty
pawloch00 Jan 23, 2025
003445d
change key cache
pawloch00 Jan 23, 2025
bb9de7c
change name of cache
pawloch00 Jan 23, 2025
d6694b6
restore to previous version
pawloch00 Jan 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
Signed-off-by: Piotr Pawłowski <[email protected]>
pawloch00 committed Jan 7, 2025
commit 1d55abe9fe903609b115ccff0d255e6a1dc8fc83
24 changes: 0 additions & 24 deletions .github/workflows/build_tests.yaml
Original file line number Diff line number Diff line change
@@ -61,28 +61,16 @@ jobs:
linter:
uses: ./.github/workflows/lint_and_format.yml
needs: [set-variables]
concurrency: # We support one build or nightly test to run at a time currently.
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
run-unit-tests:
concurrency: # We support one build or nightly test to run at a time currently.
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
uses: ./.github/workflows/unit_tests.yaml
needs: [linter, set-variables]
run-integration-tests:
concurrency:
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
uses: ./.github/workflows/integration_tests.yaml
secrets: inherit
needs: [run-unit-tests, set-variables]
cluster-private:
needs: [run-integration-tests, set-variables]
uses: ./.github/workflows/cluster_private.yaml
concurrency: # We support one build or nightly test to run at a time currently.
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
with:
cluster-name: '${{needs.set-variables.outputs.cluster-name}}'
tpu-type: '${{needs.set-variables.outputs.tpu-type || inputs.tpu-type}}'
@@ -92,9 +80,6 @@ jobs:
cluster-create:
needs: [run-integration-tests, set-variables]
uses: ./.github/workflows/cluster_create.yaml
concurrency: # We support one build or nightly test to run at a time currently.
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
with:
cluster-name: '${{needs.set-variables.outputs.cluster-name}}'
tpu-type: '${{needs.set-variables.outputs.tpu-type || inputs.tpu-type}}'
@@ -103,9 +88,6 @@ jobs:
secrets: inherit

workloads-tests:
concurrency: # We support one build or nightly test to run at a time currently.
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
needs: [cluster-create, set-variables]
uses: ./.github/workflows/workload_test.yaml
with:
@@ -114,9 +96,6 @@ jobs:
zone: ${{needs.set-variables.outputs.zone}}
secrets: inherit
batch-tests:
concurrency:
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
needs: [cluster-create, set-variables]
uses: ./.github/workflows/batch_test.yaml
with:
@@ -127,9 +106,6 @@ jobs:
if: always()
needs: [workloads-tests, batch-tests, set-variables]
uses: ./.github/workflows/cluster_delete.yaml
concurrency:
group: ${{needs.set-variables.outputs.group-name}}
cancel-in-progress: false
with:
cluster-name: ${{needs.set-variables.outputs.cluster-name}}
zone: ${{needs.set-variables.outputs.zone}}