-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM is making multiple of same Alter again and again !! #60362
Comments
Another observation is when i check the DDL job queue, The DDL is being run by the DM for every 5 mins almost here is the jobs creation time
|
DM should be able to find a DDL is queuing. It will only retry executing DDL after it does not found the DDL is queuing. Can you provide DM worker log for us to locate the problem? also cc @OliverS929 |
|
I see that because
DM failed to check if DDL is queuing, so it retried again and again. PTAL @OliverS929 |
But the TiDB server is active, and its connecting |
Inside TIDB DDL queue lot of jobs struck in cancelling state for more than day and some even weeks |
we to recover we had to cancel the DDL and they got struck in cancelling state
|
Bug Report
We use Mysql in the upstream and replicate from there using DM cluster to the TiDB cluster.
When the DDL is executed in MySQL, when its replicating to TIDB via DM, we see the same DDL being executed in multiple threads and causing the replication to go for a lag indefinitely also DDL jobs goes to Queue state as below
1. Minimal reproduce step (Required)
we are seeing this issue for the 3rd time, with the same cluster
2. What did you expect to see? (Required)
We expect the alter to be executed once and proceed with the replication
3. What did you see instead (Required)
multiple execution of same alter and making it lag and stall the cluster
4. What is your version? (Required)
we use cluster version as 8.5.1 and DM cluster version as 8.5.1
The text was updated successfully, but these errors were encountered: