Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: detect regressions with build performance program #4973

Open
wants to merge 299 commits into
base: master
Choose a base branch
from

Conversation

zachary-bailey
Copy link
Collaborator

@zachary-bailey zachary-bailey commented Sep 18, 2024

What type of PR is this?

/kind test

What this PR does / why we need it:

This PR does the following:

  • Ingests local build performance data into kusto for later analysis
  • Queries aggregate performance data for the relevant SKU in kusto
    • Kusto will contain mean and stdev for every section in every script for every SKU
  • Compares queried data against performance data recorded for the local build
  • Detects regressions and prints them to logs, notifying user with a warning
    • Regression is currently defined as any time over 3 Stdev over the mean, but that can change if team would like a different
      metric
    • This is to prevent the logs from being too noisy and only showing regressions for sections that are significantly out of spec

Which issue(s) this PR fixes:

This PR will prevent VHD Build time regression by notifying software engineers of their code changes effect on VHD build times. Additionally, it can illuminate problem areas in the build when retry logic is causing the build to take longer than expected.

Requirements:

Reviewer Notes:

Open to any suggestions related to testing and error handling specifically, as I would like this program to be as robust as possible. Currently, I am wrapping all errors and sending them back to main.

Kusto functions are not currently tested because they are just wrappers around the go SDK functions to keep main clean and readable.

Once code is approved, I will create a new cluster and change a couple variables as I am currently only using a dev/test kusto instance. After that I will merge the code.

@coveralls
Copy link

coveralls commented Sep 18, 2024

Pull Request Test Coverage Report for Build 11297674229

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 70.165%

Totals Coverage Status
Change from base Build 11296795694: 0.0%
Covered Lines: 2808
Relevant Lines: 4002

💛 - Coveralls

go.sum Outdated
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the master branch, these mods are still listed but have yellow squiggly marks, citing that they are no longer needed. I ran go mod tidy in the overall directory and now can't get them back in the go sum because they automatically disappear. Are we keeping them around for some reason?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants