Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dm: Add marker option to redact info log #12070

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

db-will
Copy link
Contributor

@db-will db-will commented Feb 23, 2025

What problem does this PR solve?

Issue Number: close #11489

What is changed and how it works?

Add marker option similar to other components(https://docs.pingcap.com/tidb/stable/log-redaction) for redacting info log

Check List

Tests

Manual test with following config to start dm worker:

...
# Log configuration.
log-level = "debug"
log-file = ".log/dm-worker.log"
redact-info-log = "marker"
...

By testing with a dm task, we can see the related logs:

[2025/02/23 02:04:21.316 -05:00] [DEBUG] [baseconn.go:184] ["execute statement"] [task=testdm] [unit="binlog replication"] [query="INSERT INTO `testdm`.`t1` (`id`,`uid`,`name`,`info`) VALUES (?,?,?,?)"] [argument="‹[20250223 3 Williama <nil>]›"]

We are expected to see sensitive data is wrapped by '‹' and '›' marker like other components.

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

Yes, here are a few documentation needed to be updated:

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

DM enhances log redaction and supports marking user data in DM worker logs with and . Based on the marked logs, you can decide whether to redact the marked information when displaying the logs, thus increasing the flexibility of log redaction.

Copy link
Contributor

ti-chi-bot bot commented Feb 23, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gmhdbjd, yudongusa for approval, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. area/dm Issues or PRs related to DM. contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Feb 23, 2025
Copy link
Contributor

ti-chi-bot bot commented Feb 23, 2025

Hi @db-will. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dm Issues or PRs related to DM. contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support query argument redaction in DM logging
1 participant