Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.8+

Compute Engine

Flink

Minimal reproduce step

1. Use cdc action and do not configure checkpoint intervals (Paimon will actively configure checkpoint via: [CDC] Refactor cdc to reduce redundant code and set default checkpoint interval #2461)
2. open commit.force-create-snapshot = true
3. Actively kill a taskmanager during a task run, triggering a flink failover.

What doesn't meet your expectations?

Exception

Chances are, we'll find that the task keeps restarting.

How this led to this Exception

• With commit.force-create-snapshot turned on, a commit will be triggered to generate a snapshot of paimon even if no data is written.

• flink's failover resumes from checkpoint, based on paimon's two-phase commit, and resuming from checkpoint may trigger a commit of the manifest file

• If there are no new data files in the commit (via the streamingCheckpointEnabled variable in the CommitterOperator), the manifest file commit may be filtered out because we didn't actively set the checkpoint. streamingCheckpointEnabled= false, then the commit will be ignored (because no new data file has been generated). Then, after the second job restart, the process will be repeated again, and we will be stuck in a dead loop.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!