Remote store recovery support for Optimized and Lucene Indices#20365

Open

ask-kamal-nayan wants to merge 40 commits intoopensearch-project:feature/datafusionfrom

ask-kamal-nayan:feature/rep_rec

Commits on Dec 24, 2025

Recovery flow building and updated with feature branch
Kamal Nayan
committed
working on replica promotion to primary
Kamal Nayan
committed
Changes to make replica promote as primary
Kamal Nayan
committed
IT test Replica is getting promoted to primary is passing
Kamal Nayan
committed
Recovery flow is working
Kamal Nayan
committed
Cherry-picked my changes for remote store recovery, not in working state
Kamal Nayan
committed
All recovery tests are working
Kamal Nayan
committed

Commits on Dec 29, 2025

Implementing docs, commits and segments stats in CompositeEngine

Shashank Gowri
authored and
raghuvanshraj
committed
Implementing docs, commits and segments stats in CompositeEngine

Shashank Gowri
authored and
raghuvanshraj
committed
Using isOptimizedIndex setting in InternalEngine as interim solution

Shashank Gowri
authored and
raghuvanshraj
committed
Using isOptimizedIndex setting in InternalEngine to select translog manager

Shashank Gowri
authored and
raghuvanshraj
committed
Refactored code and added some extra assertion checks, nit fixes
Kamal Nayan
committed
Merge remote-tracking branch 'upstream/feature/datafusion' into recovery-flow-prod
Kamal Nayan
committed

Commits on Dec 30, 2025

Updated the code to use Indexer and made some other changes
Kamal Nayan
committed
Changes to get remote upload and replication to work for lucene indices
raghuvanshraj
committed
merged changes related to lucene index
Kamal Nayan
committed

Commits on Dec 31, 2025

Working on lucene replication IT's
Kamal Nayan
committed
merged with datafusion branch
Kamal Nayan
committed
Primary is getting files
Kamal Nayan
committed

Commits on Jan 1, 2026

Commits on Jan 2, 2026

Commits on Jan 4, 2026

Lucene and Parquet recovery working
Kamal Nayan
committed

Commits on Jan 5, 2026

Minor fix for null Indexer during recovery
Kamal Nayan
committed
Merge remote-tracking branch 'upstream/feature/datafusion' into feature/rep_rec
Kamal Nayan
committed
Minor fix for updated DatafusionReader constructer
Kamal Nayan
committed
Removed local changes from run.gradle
Kamal Nayan
committed
Refresh after Replica promotion is working
Kamal Nayan
committed

Commits on Jan 6, 2026

Removed changes from DatafusionContext changes which was not needed
Kamal Nayan
committed
Fixing the deleteUnrefrencedFiles logic to look for files at correct directory
Kamal Nayan
committed
parquet-data-format:integTest test working
Kamal Nayan
committed
Minor change to close one catalogsnapshot refrence
Kamal Nayan
committed
Unrefrenced files are getting deleted from primary and replica
Kamal Nayan
committed

Commits on Jan 7, 2026

Added IT to test recovery with translogs
Kamal Nayan
committed
Updated DataFusionRemoteStoreRecoveryTests test to track file count from directory and doc counts as well
Kamal Nayan
committed

Commits on Jan 8, 2026

Fixing repeated uploads for existing files due to incorrect contains check
raghuvanshraj
committed

Commits on Jan 9, 2026

Fixed metadata path and enabled S3 GC for CompsositeEngine
raghuvanshraj
committed