Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

End-to-end tests for torch_xla llama and mixtral #100

Merged
merged 3 commits into from
Feb 11, 2025
Merged

Conversation

tengyifei
Copy link
Collaborator

@tengyifei tengyifei commented Feb 9, 2025

Fixes #52

E2E test design

The workflows in e2e_test.yml does a few things:

  • Set up gcloud credentials from a Service Account key managed in repo secrets.
  • Install torchprime.
  • Test tp use and point it to an XPK cluster hosted internally.
  • Test tp run on a few models.

After kicking off the training of some models, it starts a parallel job for each
model, and runs a few checks. This is implemented in
reusable_e2e_check.yml:

  • Stream the logs.
  • Check workload exit code.
  • Check for specific log strings that indicate training success.
  • Check that there is a profile .pb file.

@tengyifei tengyifei force-pushed the yifeit/e2e-test branch 24 times, most recently from 2bd79cf to 553cacc Compare February 10, 2025 02:53
@tengyifei tengyifei changed the title Test End-to-end tests for torch_xla llama and mixtral Feb 10, 2025
@tengyifei tengyifei force-pushed the yifeit/e2e-test branch 3 times, most recently from cb8a778 to cea98da Compare February 10, 2025 05:17
@tengyifei tengyifei marked this pull request as ready for review February 10, 2025 05:34
@bhavya01 bhavya01 self-requested a review February 10, 2025 18:25
@tengyifei tengyifei merged commit 6e91961 into main Feb 11, 2025
9 checks passed
@tengyifei tengyifei deleted the yifeit/e2e-test branch February 11, 2025 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

End-to-end tests
2 participants