feature : rocksdb phase4 consolidated - #8208
Conversation
feature : rocksdb phase1 config and spi
Feat/rocksdb phase2 engine
Feat/rocksdb phase3 query index
Add production configuration controls, diagnostics, maintenance foundations, snapshot support, and benchmark coverage.
Add configurable WAL synchronization, shutdown durability handling, recovery safeguards, and focused tests.
Add fair bounded status scans, storage tuning controls, verification modes, and workload comparison coverage.
Optimize high fan-out transaction cleanup and add resumable orphan cleanup, crash recovery, and interference coverage.
Add lock-index verification and repair planning, resumable progress, safety gates, and maintenance tests.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.x #8208 +/- ##
============================================
- Coverage 73.26% 73.13% -0.14%
- Complexity 1146 1148 +2
============================================
Files 1153 1153
Lines 42348 42511 +163
Branches 5061 5063 +2
============================================
+ Hits 31028 31090 +62
- Misses 8832 8942 +110
+ Partials 2488 2479 -9
🚀 New features to boost your workflow:
|
| /** | ||
| * RocksDB lock mode for file store engine. | ||
| */ | ||
| ROCKSDB("rocksdb"); |
There was a problem hiding this comment.
Why does lockMode have a separate RocksDB implementation? I think the better approach would be to abstract the underlying storage operations of the existing file implementation into an SPI, with the current file-based implementation serving as the default engine and RocksDB provided as an alternative implementation.
In other words, we could introduce a configuration such as:
store.file.engine=default/rocksdb
This way, RocksDB would be treated as a pluggable storage engine for the existing file storage mode, rather than introducing a separate storage mode specifically for RocksDB.
There was a problem hiding this comment.
I think the frontend changes should be handled in a separate PR.
| findBackgroundSessionsBySingleStatus(status, lazyLoadBranch, backgroundSessionQueryLimit)); | ||
| } | ||
|
|
||
| private List<GlobalSession> findBackgroundSessionsBySingleStatus( |
There was a problem hiding this comment.
The coordination layer should not contain so much implementation-specific logic. Instead, those details should be encapsulated within the respective storage implementations.
Ⅰ. Describe what this PR did
This PR continues to optimize RocksDB FileMode in #8149, focusing on handling the upper bound of background workload, Seata lifecycle performance, WAL/recovery semantics, benchmark evidence, and protected maintenance processes.
The main content is as follows:
noneandperiodicWAL sync strategies, and supplement with clean-shutdown marker, strict sync before shutdown, exception-safe resource release, scheduling shutdown race handling, and process-crash recovery harness.maxTotalWalSize, compaction, and WAL-related parameters, but did not upgrade profiles that failed the benchmark gate to their default values.GLOBAL_REMOVE, branch/lock cleanup, and lock release within the lifecycle boundary of xid. Use RangeDelete within a provably safe and ordered index range, and continue to use a singleWriteBatchto maintain atomicity of state and index updates.Main benchmark results:
Machine parameters:
-Xms2g -Xmx4g;a small number of early matrices use-Xms1g -Xmx3gglobal_remove_with_branches,1M×2×2,triplicated+53.4%global_remove,1M×2×2,triplicated+56.8%,p99-34.0%lock.release_branch,1M×2×2,triplicated+41.0%,p99-26.3%lock.release_global,1M×2×2,triplicated+45.7%,p99-28.0%GLOBAL_REMOVE,50 branch×10 lock+30.7%,p99-26.6%The performance improvement mentioned here is derived from repeated tests conducted on the same machine, with the same scale, workload, and independent JVM. Since bounded scan and full scan yield different return sizes, it is only used to demonstrate the upper bound of a single workload, without calculating the multiple improvement.
Ⅱ. Does this pull request fix one issue?
fixes#8114
Ⅲ. Why don't you add test cases (unit test/integration test)?
Test coverage:
WriteBatch.Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews
This PR is a follow-up optimization of #8149 and depends on #8149