[Epic] Release testing automation and frequent upstream releases

[ktam3]

Goal:

• We need integration testing on different hardware platforms as they come online.
• We need to have a better connection between our team and the platform team.
• We need to automate this testing
• To help alleviate last min testing, we should have frequent upstream releases, so testing can start early.

Currently:

• @JamesKunstle is working on a smoketest to indicate what needs to happen for a functional run.
• He won't have the capacity to take on long term, so we need to be able to build on what James is building
  • We need to have a better handoff so it's automated and running constantly so we always have feedback on how the logic we're building is interacting with drivers and lower levels.
• Nathan is working a lot of expanding the e2e test suite for upstream

Notes:

• As we get configs for different HW, the difference with upstream should only be models and lora layers. Bulk should happen in upstream runners. Could be a good smoke test.
• For 'full integration e2e' code compatibility, the root of that should be upstream first testing that will trickle into other configs.
  • Recently created a CI story that was full e2e on actual correct codepaths.
  • GPU enabled upstream runners that run either on most PRs or big ones that are every 12-24h… can expand that to use some abbreviate version of the system profiles we're working on - should cover a lot of the base functionlity for accelerated workflow.

Release testing automation will be split into two phases:
Phase 1:

• Assess and discuss the design of the script automation
  Understand: what is this, where is this running, where is the output?
• [Epic] Improvements to E2E testing #2196
• adds basic smoketests for main_ds and data_process CLI args training#280

Phase 2:

• Actual Automation

Frequent upstream releases:

• Release more ilab z streams on a regular cadence

[ktam3]

@JamesKunstle - i know there's work that you're currently doing related to this. Please link those here when you get a chance in the Phase 1 section!

[nathan-weinberg]

What is this actually meant for? The description isn't very descriptive

[ktam3]

I just updated with more info on the discussion

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.