adds upstream testing development document

[JamesKunstle]

Adds a dev doc describing our testing strategy for this repo.

[nathan-weinberg]

@JamesKunstle can you fix the markdown linting? can run make md-lint for debugging locally

[danmcp]

This is going to be a real challenge with runner availability to cover these on every PR.

[JamesKunstle]

Yeah, I figure that we'll scale out a little at a time, starting w/ CPU and nvidia, and adding runners to the matrix as possible.

[JamesKunstle]

The current CI failure seems to be because the workflow can't access vars.AWS_REGION even though the workflow itself hasn't changed in this PR and the branch is rebased onto main, for which the unit test works. I'm not sure what that's happening.
@danmcp @courtneypacheco do ya'll have any guidance? I was assuming that the protected variables would be exposed to the workflows once they were merged.

[danmcp]

The current CI failure seems to be because the workflow can't access vars.AWS_REGION even though the workflow itself hasn't changed in this PR and the branch is rebased onto main, for which the unit test works. I'm not sure what that's happening. @danmcp @courtneypacheco do ya'll have any guidance? I was assuming that the protected variables would be exposed to the workflows once they were merged.

I am looking at the difference between:

https://github.com/instructlab/training/blob/main/.github/workflows/unit-tests.yaml#L12

and

https://github.com/instructlab/training/blob/main/.github/workflows/e2e-nvidia-l4-x1.yml#L5

Were there reasons for the unit tests to be different?

See: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target

[JamesKunstle]

No there wasn't a reason for the two to be different apart from just writing the unit-test workflow from scratch. The two should have the same behavior, I'll amend that.

I'm confused about the vars context availability though because this CI workflow has merged to the main branch and doesn't seem to be able to access that context.

[danmcp]

No there wasn't a reason for the two to be different apart from just writing the unit-test workflow from scratch. The two should have the same behavior, I'll amend that.

I'm confused about the vars context availability though because this CI workflow has merged to the main branch and doesn't seem to be able to access that context.

pull_request runs from the context of your merge commit and pull_request_target runs from the context of instructlab/training. So when running from the context of your branch, it can't see the var from the instructlab/training repo.

[JamesKunstle]

That's super interesting, I totally missed that distinction in the docs. I'll look more closely at that.

I opened a PR #411 that mirrors the on and permissions usage between the e2e workflow invocation to the unit-tests invocation.

[JamesKunstle]

From the docs:
"For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission unless the permissions key is specified and the workflow can access secrets, even when it is triggered from a fork. Although the workflow runs in the context of the base of the pull request, you should make sure that you do not check out, build, or run untrusted code from the pull request with this event."

My new understanding is that we have to use pull_request_target because we are setting up a self-hosted ec2 runner and need secrets / vars context to do this. We then limit the permissions of the job that runs the untrusted code.

If we weren't using these secure resources we could use the pull_request invocation option instead.

[danmcp]

My new understanding is that we have to use pull_request_target because we are setting up a self-hosted ec2 runner and need secrets / vars context to do this. We then limit the permissions of the job that runs the untrusted code.

If we weren't using these secure resources we could use the pull_request invocation option instead.

That matches my understanding as well. I had originally made sure you had permissions set to {}, but I missed the pull_request vs pull_request_target at the top.

[JamesKunstle]

Here's the most informative doc IMO:
https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/approving-workflow-runs-from-public-forks