Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge gcsfue into main #317

Open
wants to merge 52 commits into
base: develop
Choose a base branch
from
Open

Merge gcsfue into main #317

wants to merge 52 commits into from
+1,895 −68

Conversation

pawloch00
Copy link
Collaborator

@pawloch00 pawloch00 commented Jan 10, 2025 •
edited
Loading

Fixes / Features

  • Cherry pick commits from development branch regarding gcsfuse usage into main branch
  • Private cluster creation was commented out from build tests

Testing / Documentation

Testing details.

  • [ y/n ] Tests pass
  • [ y/n ] Appropriate changes to documentation are included in the PR

sharabiani and others added 17 commits January 10, 2025 14:36
@sharabiani @pawloch00
Enabling Workload Identity and GCSFuse driver flags added. (#177)
718d8de 
* Enabling Workload Identity and GCSFuse driver flags added.
@PBundyra @pawloch00
Introduce Storage API (#192)
a68961f 
* Introduce Storage API

* Fix Gh CI/CD

* Improve getting storages to mount to workload

* Improve handling exception while deleting storage

* Fix adding necessary args and cluster credentials

* Fix storage class, differentiate storage types in create and delete commands, add --type validation, reduce number of CRD installations
@pawloch00
fix
b8c419a 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
remove storage from pytype check
a7aaef6 
Signed-off-by: Piotr Pawłowski <[email protected]>
@PBundyra @pawloch00
Add GCS Fuse Storage to workload creation (#199)
7be30bc 
* Add Storage to workload creation

* Improve documentation
@PBundyra @pawloch00
Fix types (#219)
0b9ef41
@PBundyra @pawloch00
Fix get storage function (#221)
704dcad
@PBundyra @pawloch00
Add xpk/templates as a package to pyproject (#223)
5f7ff69
@pawloch00
Merge branch 'main' of https://github.com/AI-Hypercomputer/xpk into p…
8c73ccb 
…pawl-cherry-pick
@pawloch00
fix to make xpk run
ea2bbab 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Fix tests on development branch (#263)
10816ca 
* apply pathways fixes

Signed-off-by: Piotr Pawłowski <[email protected]>

---------

Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
remove duplicate env in tests
3f4b4d7 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
fix cluster creation in tests
6449df9 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
debug
523fa8d 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
add subnetwork
e0f8820 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
fix gcs removal
83d9b90 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
uncomment integration and unit tests
98424a2 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Copy link
Collaborator Author

This should be merged after #272

@pawloch00 pawloch00 marked this pull request as ready for review January 13, 2025 11:47
@pawloch00
Copy link
Collaborator Author

@44past4 @sharabiani please review

@pawloch00 pawloch00 added the release-features features label Jan 29, 2025
@pawloch00
Copy link
Collaborator Author

Does this PR address mounting gcs buckets onto the ramfs? That is something that gave pathways a significant perf boost at large scale. Relevant changes: 76547c4

I don't think this is in scope of this PR.

@pawloch00
fix unit tests
f7a3234 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Merge branch 'develop' of https://github.com/AI-Hypercomputer/xpk int…
3c543bb 
…o ppawl-cherry-pick
@pawloch00
fix
8bf25ba 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
fix
c6b7d33 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Merge branch 'develop' of https://github.com/AI-Hypercomputer/xpk int…
b6ccdd4 
…o ppawl-cherry-pick
@pawloch00
fix import
62d6f94 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Merge branch 'develop' into ppawl-cherry-pick
3e95437
@pawloch00
fix and cleanup
256a34d 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Merge branch 'ppawl-cherry-pick' of https://github.com/AI-Hypercomput…
cb036b6 
…er/xpk into ppawl-cherry-pick
@pawloch00
fix unit tests
d940e3a 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
cleanup
9a5eb61 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
cleanu[
a53a313 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
cleanup
50fa2e4 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
Copy link
Collaborator Author

This works for A3 mega and A3 ultra

@pawloch00 pawloch00 requested a review from sharabiani February 4, 2025 10:24
@pawloch00
disable flaky tests
84758bb 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
fix pathways
6edca3e 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
fix
3d8aa73 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
fix
242e0dd 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
debug
b60e16c 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
debug
87ebbaa 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00
restore all tests
fd9028e 
Signed-off-by: Piotr Pawłowski <[email protected]>
@pawloch00 pawloch00 requested a review from gcie as a code owner February 4, 2025 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Obliviour Obliviour Awaiting requested review from Obliviour Obliviour is a code owner

@44past4 44past4 Awaiting requested review from 44past4 44past4 is a code owner

@BluValor BluValor Awaiting requested review from BluValor BluValor is a code owner

@sharabiani sharabiani Awaiting requested review from sharabiani sharabiani is a code owner

@gcie gcie Awaiting requested review from gcie gcie is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
release-features features
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pawloch00 @shauryagup @sharabiani @PBundyra