CI Test report does not match the result from scorecard

[godofredoc]

The scorecard action is reporting that most of the PRs are not running tests even though running scorecard command reports that 30/30 tests ran tests correctly:

Image from security tab report as generated by scorecard action:

Data from running the check with the same PAT as the action:

docker run -e GITHUB_AUTH_TOKEN=<pat> gcr.io/openssf/scorecard:v4.1.0@sha256:a1e9bb4a0976e800e977c986522b0e1c4e0466601642a84470ec1458b9fa6006 --show-details --repo=https://github.com/flutter/flutter --verbosity=debug --checks=CI-TESTS

Starting [CI-Tests]

RESULTS
-------
Aggregate score: 10.0 / 10

Check scores:
Finished [CI-Tests]
|---------|----------|--------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
|  SCORE  |   NAME   |             REASON             |                                            DETAILS                                             |                                        DOCUMENTATION/REMEDIATION                                        |
|---------|----------|--------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| 10 / 10 | CI-Tests | 30 out of 30 merged PRs        | Debug: CI test found: pr: 101645, context: flutter-dashboard:                                  | https://github.com/ossf/scorecard/blob/33f80c93dc79f860d874857c511c4d26d399609d/docs/checks.md#ci-tests |
|         |          | checked by a CI test -- score  | https://api.github.com/repos/flutter/flutter/check-runs/5956689942                             |                                                                                                         |
|         |          | normalized to 10               | Debug: CI test found: pr: 101641, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5955662569                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101638, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5955114082                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101634, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5954663793                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101625, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5953234632                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101200, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/f5a3c19f2ea7b765e5892d5c56535fa94ef8b62e |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101619, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5952277466                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101616, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5951947232                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101613, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5951119409                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101612, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5892676941                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101549, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5881408311                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 100794, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/d8435ca1e908b937c45ed54ba08bf97dc3312a1d |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 100787, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/cd4acce2deecb8c83a434d0742725d1d19900493 |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101607, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5892004554                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101592, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/531fd1f2feefe3e6c62d1e60b7a501f56cde8daa |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 98549, context: Google testing:                                      |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/71de516c5766c7aa24280062b81cc6e6ac5cf3c2 |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 100893, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5720651123                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101600, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/742570db066efab2bb914cfb7ddd2022f9d9c50a |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101537, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5877395802                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101567, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/f84cd8f66c6e90fc0bf6d62e1e7c7dda310cce8e |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101554, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5886492163                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101583, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5887260706                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101544, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5877504384                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101553, context: Google testing:                                     |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/statuses/3d3a80ac3e530b93f6a1346d15f76d0cce1d0fb8 |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101564, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5881851413                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101562, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5881102473                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101559, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5880429584                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101550, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5878652628                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101548, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5878154047                             |                                                                                                         |
|         |          |                                | Debug: CI test found: pr: 101545, context: flutter-dashboard:                                  |                                                                                                         |
|         |          |                                | https://api.github.com/repos/flutter/flutter/check-runs/5877594658                             |                                                                                                         |
|---------|----------|--------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|

[laurentsimon]

Interesting. Not sure what's going on here. I think we need a way to add debug input to the output. Created #176 for tracking

[godofredoc]

The alert information is included in the SARIF file, is caching used somewhere in between calling scorecards and the generation of the SARIF file?

[laurentsimon]

There's no caching. We always run scorecard and create the SARIF for each run.

[godofredoc]

Awesome, thanks!

[laurentsimon]

Are you still seeing this problem?

[godofredoc]

It replicates with gcr.io/openssf/scorecard@sha256:8165ad910019422f40c51cbb97ff6e7db0e2e2e11faebf59e0b6f1a2eb66ebd7 but not with the latest images. Seems like it will also get fixed with the next update.

[laurentsimon]

Great, so you'll get the fix in the next release.

[godofredoc]

This has been fixed, thanks @laurentsimon!