Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add coverage statistics #5907

Open
wants to merge 62 commits into
base: main
Choose a base branch
from

Conversation

mildm8nnered
Copy link
Collaborator

@mildm8nnered mildm8nnered commented Dec 22, 2024

WIP - Addresses #5906

Adds coverage reporting to SwiftLint, via a --report-coverage command line option, and a report_coverage configuration file setting.

% swiftlint --quiet --report-coverage
Enabled rules coverage: 0.995
    All rules coverage: 0.854

"Enabled rules coverage" is helpful for monitoring whether contributors are simply disabling SwiftLint for large sections of code or entire files (although blanket_disable_command, if not disabled, should help with that already).

"All rules coverage" is more useful - low values here indicate that a lot of rules are disabled (or suppressed), and that you could potentially be getting a lot more value out of SwiftLint.

Related changes - we now cache calls to file.regions(), as we'll call this at least twice if coverage is enabled.

From https://github.com/mildm8nnered/SwiftLint/blob/mildm8nnered-add-coverage-statistics/Source/SwiftLintFramework/Coverage.swift:

Coverage is defined as the sum of the number of rules applied to each line of input, divided by the product of the
number of input lines and the total number of rules.

If all rules are applied to every input line, then coverage would be 1 (or 100%). If half the rules are applied
to all input lines, or if all the rules are applied to half of the input lines, then coverage would be 0.5, and
if no rules are enabled, or there is no input, then coverage would be zero.

No distinction is made between actual lines of Swift source, versus blank lines or comments, as SwiftLint may
apply rules to those as well. Coverage is only calculated over input files, so if you exclude files in your
configuration, they will be ignored. Empty input files, or files containing a single empty line will be ignored, as
SwiftLint ignores these files automatically.

"All rules" can be defined either as all enabled rules, or all available rules, enabled or not, resulting
in two different coverage metrics:

  • "Enabled rules coverage" measures how frequently enabled rules are being disabled by swiftlint:disable
    commands.
  • "All rules coverage" measures how many of all possible rules are actually being applied.

Typically, enabled rules coverage will be close to 1, as swiftlint:disable is used sparingly. All rules
coverage will generally be much lower, as some rules are contradictory, and many rules are optin. With no opt-in
rules enabled, all rules coverage will be about 0.4, rising to 0.8 or more if many opt-in rules are enabled.

When calculating all rules coverage swiftlint:disable commands are still accounted for, but only for enabled
rules.

Coverage figures will be different for linting and analyzing as these use different sets of rules.

The number of enabled rules is determined on a per-file basis, so child and local configurations will be accounted
for.

Custom rules, if enabled and defined, will be counted as first class rules for both enabled and all rules coverage.
If not enabled, custom_rules will be counted as a single rule, even if a configuration exists for it.

When calculating enabled rules coverage, the custom rules in the configuration for each input file (e.g. including
child configurations) will be taken into account. When calculating all rules coverage, only the main configurations
custom rules settings will be used.

@SwiftLintBot
Copy link

SwiftLintBot commented Dec 22, 2024

1 Warning
⚠️ Big PR
18 Messages
📖 Building this branch resulted in a binary size of 24914.9 KiB vs 24893.21 KiB when built on main (0% larger).
📖 Linting Aerial with this PR took 0.93 s vs 0.95 s on main (2% faster).
📖 Linting Alamofire with this PR took 1.26 s vs 1.28 s on main (1% faster).
📖 Linting Brave with this PR took 8.48 s vs 8.56 s on main (0% faster).
📖 Linting DuckDuckGo with this PR took 5.51 s vs 5.55 s on main (0% faster).
📖 Linting Firefox with this PR took 11.06 s vs 11.12 s on main (0% faster).
📖 Linting Kickstarter with this PR took 10.37 s vs 10.36 s on main (0% slower).
📖 Linting Moya with this PR took 0.55 s vs 0.54 s on main (1% slower).
📖 Linting NetNewsWire with this PR took 2.96 s vs 2.96 s on main (0% slower).
📖 Linting Nimble with this PR took 0.79 s vs 0.79 s on main (0% slower).
📖 Linting PocketCasts with this PR took 8.77 s vs 8.81 s on main (0% faster).
📖 Linting Quick with this PR took 0.45 s vs 0.45 s on main (0% slower).
📖 Linting Realm with this PR took 4.51 s vs 4.53 s on main (0% faster).
📖 Linting Sourcery with this PR took 2.33 s vs 2.35 s on main (0% faster).
📖 Linting Swift with this PR took 4.54 s vs 4.56 s on main (0% faster).
📖 Linting VLC with this PR took 1.23 s vs 1.24 s on main (0% faster).
📖 Linting Wire with this PR took 18.33 s vs 18.43 s on main (0% faster).
📖 Linting WordPress with this PR took 11.56 s vs 11.61 s on main (0% faster).

Generated by 🚫 Danger

@SimplyDanny SimplyDanny linked an issue Dec 22, 2024 that may be closed by this pull request
2 tasks
@mildm8nnered mildm8nnered force-pushed the mildm8nnered-add-coverage-statistics branch 4 times, most recently from 761d4f4 to 3274fa8 Compare December 27, 2024 14:55
@mildm8nnered mildm8nnered marked this pull request as ready for review December 27, 2024 20:30
@mildm8nnered mildm8nnered force-pushed the mildm8nnered-add-coverage-statistics branch 2 times, most recently from 61d61b5 to e3be062 Compare December 30, 2024 13:53
@mildm8nnered mildm8nnered force-pushed the mildm8nnered-add-coverage-statistics branch from e3be062 to 60257b8 Compare January 9, 2025 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add coverage statistics
2 participants