br/cmd: added operator checksum-as|force-flush #58801

YuJuncen · 2025-01-08T10:06:56Z

What problem does this PR solve?

Issue Number: close #58798 #58799

Problem Summary:
See the issue description.

What changed and how does it work?

This PR added two new subcommands to br operator:

checksum-as: this generates rewrite rule from a storage and performs checksum.
force-flush: this allows user to force trigger a flush.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
I used these utilities when executing some internal tests. They works fine as far as I go.
The scenarios including:

Checksum in a cluster with log backup enabled. (Upstream)
Restore point in a new cluster. (Downstream)
Run checksum-as where -s is the --full-backup-storage in the restore point command.
Compare the checksum.

(With the compatibility solution between log backup and ingesting, which doesn't present in master branch yet)

Run checksum-as after restoring to a cluster, where -s is the restored backup.
After a point in time restore to another cluster, run checksum-as over the tables restored where -s is the restored backup in 1..

The force-flush part was used in integration tests.

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: hillium <[email protected]>

tiprow · 2025-01-08T10:07:16Z

Hi @YuJuncen. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-01-08T11:03:26Z

Codecov Report

Attention: Patch coverage is 25.49020% with 228 lines in your changes missing coverage. Please review.

Project coverage is 74.7206%. Comparing base (87867eb) to head (547de59).
Report is 4 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #58801        +/-   ##
================================================
+ Coverage   73.1124%   74.7206%   +1.6081%     
================================================
  Files          1678       1695        +17     
  Lines        464111     464746       +635     
================================================
+ Hits         339323     347261      +7938     
+ Misses       103907      95904      -8003     
- Partials      20881      21581       +700

Flag	Coverage Δ
integration	`46.2655% <25.4902%> (?)`
unit	`72.3398% <10.1307%> (+0.0348%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.4146% <25.4902%> (+15.6585%)`	⬆️

YuJuncen · 2025-01-09T01:23:11Z

/retest-required

tiprow · 2025-01-09T01:23:32Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: hillium <[email protected]>

YuJuncen · 2025-01-09T02:35:40Z

/retest-required

tiprow · 2025-01-09T02:36:02Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

YuJuncen · 2025-01-09T02:43:22Z

merge: 476f860c09e8b068b9b5ebffc2bf28a134bfa221 - not something we can merge ...?

/test build

tiprow · 2025-01-09T02:43:43Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

merge: 476f860c09e8b068b9b5ebffc2bf28a134bfa221 - not something we can merge ...?

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

YuJuncen · 2025-01-09T03:54:11Z

/test build

tiprow · 2025-01-09T03:54:32Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…e-new-opeartor-commands

br/pkg/task/operator/force_flush.go

BornChanger · 2025-01-13T08:25:52Z

br/pkg/task/operator/config.go

+type ForceFlushConfig struct {
+	task.Config
+
+	StoresPattern *regexp.Regexp


please give a simple sample

3pointer

rest LGTM

3pointer · 2025-01-14T07:23:27Z

br/pkg/task/operator/force_flush.go

+	log.Info("About to start force flushing.", zap.Stringer("stores-pattern", cfg.StoresPattern))
+	for _, s := range tikvs {
+		if !cfg.StoresPattern.MatchString(s.Address) || engine.IsTiFlash(s) {
+			log.Info("Skipping TiFlash or not matched TiKV.",


miss continue.

ti-chi-bot · 2025-01-14T09:11:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, BornChanger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [3pointer,BornChanger]
~~br/OWNERS~~ [3pointer,BornChanger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-01-14T09:11:24Z

[LGTM Timeline notifier]

Timeline:

2025-01-14 06:59:33.142729419 +0000 UTC m=+15444.597775568: ☑️ agreed by BornChanger.
2025-01-14 09:11:23.475521968 +0000 UTC m=+23354.930568117: ☑️ agreed by 3pointer.

close pingcap#58798

YuJuncen added 2 commits January 8, 2025 16:50

added operator checksum-as|force-flush

45d0c5b

added skip tiflash and utilized for integration test

94c29ae

Signed-off-by: hillium <[email protected]>

ti-chi-bot bot added do-not-merge/invalid-title release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 8, 2025

YuJuncen changed the title ~~added operator checksum-as|force-flush~~ br/cmd: added operator checksum-as|force-flush Jan 8, 2025

ti-chi-bot bot removed the do-not-merge/invalid-title label Jan 8, 2025

YuJuncen added 3 commits January 8, 2025 18:09

revert irrelative changes

292280b

make bazel_prepare

6f38e25

fix build

728145e

fix build

476f860

Signed-off-by: hillium <[email protected]>

YuJuncen closed this Jan 9, 2025

YuJuncen reopened this Jan 9, 2025

YuJuncen added 2 commits January 9, 2025 13:44

Merge branch 'master' of https://github.com/pingcap/tidb into add-som…

fd56324

…e-new-opeartor-commands

make linter happy

15664c5

BornChanger reviewed Jan 10, 2025

View reviewed changes

br/pkg/task/operator/force_flush.go Show resolved Hide resolved

BornChanger reviewed Jan 13, 2025

View reviewed changes

BornChanger approved these changes Jan 14, 2025

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 14, 2025

Merge branch 'master' into add-some-new-opeartor-commands

c0b96ea

3pointer reviewed Jan 14, 2025

View reviewed changes

YuJuncen added 2 commits January 14, 2025 15:34

fix build

40dfca5

address comments

547de59

3pointer approved these changes Jan 14, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 14, 2025

ti-chi-bot bot merged commit 0c6681f into pingcap:master Jan 14, 2025
29 checks passed

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Jan 16, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

a76c982

close pingcap#58798

YuJuncen mentioned this pull request Jan 16, 2025

br: Release 8.5 20250114 v8.5.0 #58981

Merged

13 tasks

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Jan 17, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

2ce950b

close pingcap#58798

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br/cmd: added operator checksum-as|force-flush #58801

br/cmd: added operator checksum-as|force-flush #58801

YuJuncen commented Jan 8, 2025

tiprow bot commented Jan 8, 2025

codecov bot commented Jan 8, 2025 •

edited

Loading

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

BornChanger Jan 13, 2025

3pointer left a comment

3pointer Jan 14, 2025

ti-chi-bot bot commented Jan 14, 2025

ti-chi-bot bot commented Jan 14, 2025

br/cmd: added operator checksum-as|force-flush #58801

br/cmd: added operator checksum-as|force-flush #58801

Conversation

YuJuncen commented Jan 8, 2025

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Jan 8, 2025

codecov bot commented Jan 8, 2025 • edited Loading

Codecov Report

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

YuJuncen commented Jan 9, 2025

tiprow bot commented Jan 9, 2025

BornChanger Jan 13, 2025

Choose a reason for hiding this comment

3pointer left a comment

Choose a reason for hiding this comment

3pointer Jan 14, 2025

Choose a reason for hiding this comment

ti-chi-bot bot commented Jan 14, 2025

ti-chi-bot bot commented Jan 14, 2025

[LGTM Timeline notifier]

codecov bot commented Jan 8, 2025 •

edited

Loading