WIP: ci: add job to verify binary size #475
Conversation
|
How is this going to be different from #458? |
|
I wasn't aware of that thread. I'll look into getting a baseline from a pipeline and using it for comparison. |
|
I took a look at FluidFramework, which I used to work on and has a bundle size check as part of their PR workflow. From what I can tell, their way of doing this is to traverse |
|
Oooooh, prior art, nice. I think the commit we want to compare against is whatever the merge is based on. That would allow us to get as good a measurement of "this PR adds X bytes compared to not having it" as possible. If that commit is still running through CI maybe we just wait for it? If it fails though then walking backwards on main/release does seem like a reasonable fallback strategy. I think for ours we'd prefer to have a gate rather than just a comment, so long as there's some way for us to then override the block and say "yes this is acceptable". But a gate would prevent anyone from merging before the bot comments, for example. We could then have a dedicated size_override reviewers group that the gate requires sign off from to override or something. Also, I'd like to make sure we're actually storing the whole built file that we're using to compare against, not just a pre-computed summary of it. That frees us up to do more complex and involved analysis in the future. |
|
Tagging #76 |
justus-camp-microsoft
changed the title
|
Man this is exciting to see, the prior solution has been an annoyance for so long now. |
| if total_diff > 100 { | ||
| anyhow::bail!("{} size verification failed: The total difference ({} KiB) is greater than the allowed difference ({} KiB).", self.new.display(), total_diff, 100); | ||
| } |
There was a problem hiding this comment.
We'll need some way to override this check on a PR level, some way to say "Yes this size diff is acceptable". Not sure what github allows us to do here.
There was a problem hiding this comment.
GitHub doesn't really have a great way to do this. Ideally we could have this always be required and just hit an "override" button but afaik that's not possible. We're also unable to assign a review team through the action (as I painfully learned from trying to re-enable the unsafe reviewers assignment) because actions are scoped to the repo level and our review teams are scoped at the org level (no access).
My thought here is that we should have this action always succeed as long as it finishes all the way through and have it leave a comment on the PR with a summary of the size diff. The onus would be on the reviewer to look at the comment and make sure that the difference is acceptable.
There was a problem hiding this comment.
If that's the best we can do then it's the best we can do I guess. Maybe include some big warning text in the comment if the diff is over a threshold.
We really should figure out some way to get review groups working though. Then we could have the unsafe reviewers group back and create a new binary size reviewer group for large diffs or something.
There was a problem hiding this comment.
We'll need a PAT with org-level team read access and then the reviewer assignment would work. My understanding was that we don't want to deal with maintaining the PAT.
There was a problem hiding this comment.
Can we have review teams coped to the repo instead of the org? I'm really not familiar with github, so I'm just spitballing. But yeah, maintaining a PAT has bitten us in the past, and definitely isn't ideal.
There was a problem hiding this comment.
Hmm, good point. I can look into that and see if that's a possibility. I hadn't thought of bringing the teams down to the repo level.
| old_openhcl: ctx.use_artifact(&use_openhcl_igvm_extras), | ||
| new_openhcl: ctx.use_artifact(&use_openhcl_igvm_extras), |
There was a problem hiding this comment.
So at the moment it's diffing a binary against itself?
There was a problem hiding this comment.
Yeah this isn't ready for review yet. CI doesn't run on drafts so I added WIP to the title and opened it to see it run. Some of your comments are still valid though and will address as I wrap this up 😄
| .filter(|(_, size)| *size > 0) | ||
| .collect(); | ||
|
|
||
| for section in new.sections() { |
There was a problem hiding this comment.
What if a section was large in the original but is now 0 size or missing entirely in the new? Or vice versa?
WIP: This PR adds a job to verify the size of the output OpenHCL binary.