Timestamp/globalIndex ordering consistency

[bigpandamx]

Investigation and Root Cause Analysis for Issue #259

Fixes #259

🎯 Problem Statement

Issue #259 reports inconsistent indexing where transactions from 2023 appear before transactions from 2024 when ordering by timestamp and globalIndex properties, despite the chronological ordering being incorrect. The issue occurs "at some point" (intermittently) at the indexer/database level.

🔍 Root Cause: Chain Reorganization Bug

After thorough investigation, I identified the root cause:

The bug occurs during blockchain reorganizations (reorgs). When a fork block becomes the main chain:

1. Initial State:

   • Main chain block A at height 100: transactions get globalIndex 1000-1010
   • Fork chain block B at height 100: transactions get globalIndex 1011-1020

2. Reorg Occurs: Block B becomes the new main chain

3. Current Behavior (BUGGY):

   • Code path: ChainIndexer.updateChainStatus() → repos.txs.updateChainStatusByHeaderId()
   • SQL executed: UPDATE node_transactions SET main_chain = true WHERE header_id = blockB
   • Result: Only main_chain flag is updated, globalIndex values are NOT recalculated

4. Consequence:

   • Block B transactions keep globalIndex 1011-1020 (from original insertion order)
   • But chronologically, they should have globalIndex 1000-1010 (as the new main chain)
   • Now: ORDER BY timestamp ≠ ORDER BY globalIndex
   • This causes transactions to appear out of chronological order when sorting by globalIndex

Evidence in codebase:

• ChainIndexer.scala:226-233 - updateChainStatus() only updates mainChain flag
• TransactionQuerySet.scala:178-182 - SQL only updates main_chain, not global_index
• No globalIndex recalculation logic exists anywhere in reorg handling code

✅ This PR: Foundation for the Fix

This PR provides the essential first step toward fixing issue #259:

Added: Comprehensive test suite (TimestampGlobalIndexConsistencySpec.scala) that validates:

1. ✅ Monotonic ordering - GlobalIndex strictly increases across transactions
2. ✅ Sort equivalence - sortBy(timestamp) must equal sortBy(globalIndex)
3. ✅ Cross-year boundaries - 2023/2024 transactions maintain consistent ordering
4. ✅ Same-block handling - Multiple transactions within same block maintain order
5. ✅ Gap detection - No missing values in globalIndex sequences
6. ✅ Duplicate prevention - No collisions in globalIndex values

The fix requires:

• Adding recalculateGlobalIndexAtHeight() method to recalculate globalIndex during reorgs
• Complex SQL queries to reorder transactions chronologically
• Updates to ChainIndexer.updateChainStatus() to call recalculation
• Extensive testing to ensure correctness

This test suite provides:

• Validation that any fix maintains timestamp/globalIndex consistency
• Regression prevention for future changes
• Specification of correct behavior that the fix must satisfy
• Foundation for implementing and verifying the reorg fix

📊 Technical Details

Current correct behavior (when no reorg occurs):

// extractors/package.scala:71-84
val lastTxGlobalIndex = parentOpt.map(_.maxTxGix).getOrElse(-1L)
val globalIndex = lastTxGlobalIndex + i + 1