Indexer chokes on too big transaction request blocks

[MHHukiewitz]

Problem

While debugging the Pyth indexer, we noticed that the standard behavior of fetching one-hour blocks of transaction data could cause:

• Slowdowns when querying over long stretches of empty timeframes
• Choking of other processes when having stretches of a huge amount of txn (in some cases >300k per hour)

This leads to sporadic updating of stats, as the indexer always needs to fetch the whole block of txns before processing and potentially discarding them again.

Solution

When the indexer is receiving a list of signatures by the fetcher, try to page them in smaller increments (like blocks of 1000 txns) and resume processing of stats, etc. in between these pages being fetched.