feat: add "pagination" to http-based event log poller #131
Conversation
Modify the HTTP log poller to fetch blocks in pages or chunks. Instead of trying to fetch all the logs from the RPC in one go (which fails if there are too many blocks between the specified "initial block" until the last one), we now fetch logs in sequential "pages" of N blocks. For instance, say we're using a page size of 1000: ``` # before GetLogs(FromBlock: 0, ToBlock: "<CurrentBLock>") # after GetLogs(FromBlock: 0, ToBlock: 1000) GetLogs(FromBlock: 1000, ToBlock: 2000) GetLogs(FromBlock: 2000, ToBlock: 3000) ... GetLogs(FromBlock: 99000, ToBlock: "<CurrentBlock>") ``` The page size is controlled by a new flag "event-listener-poll-size" (or the corresponding environment variable "EVENT_LISTENER_POLL_SIZE"). The recommended value based on tests with Monad is 1000.
| select { | ||
| case logCh <- log: | ||
| tw.logger.With("log", log).Debug("dispatching log") | ||
| case <-ctx.Done(): | ||
| tw.logger.Debug("stopped while dispatching logs: incomplete retrieval.") | ||
| break | ||
| return toBlock |
There was a problem hiding this comment.
just checking toBlock is inclusive or exclusive? So we dont fetch the last block again in the next fetch
There was a problem hiding this comment.
I think it's inclusive. We are fetching the boundary block twice indeed.
But I did not have enough time to test this thoroughly with Monad and started wondering if other chains might behave differently, so I ended up leaving the "double fetching" under the assumption that it won't cause any harm (other then minor ineficiency).
| if toBlock.Cmp(currentChainBlock) < 0 { | ||
| // we haven't reached the current block; re-run same procedure with | ||
| // the 'toBlock` as the start block | ||
| return tw.fetchAndDispatchLogs(ctx, logCh, toBlock, currentChainBlock) |
There was a problem hiding this comment.
we could, in theory, get rid of the duplication simply by using toBlock + 1 here. But, again, I'm not 100% confident we won't miss any blocks. I'll try to dig into the EVM rpc docs to see what I can find.
Modify the HTTP log poller to fetch blocks in pages or chunks.
Instead of trying to fetch all the logs from the RPC in one go (which fails if there are too many blocks between the specified "initial block" until the last one), we now fetch logs in sequential "pages" of N blocks. For instance, say we're using a page size of 1000:
The page size is controlled by a new flag "event-listener-poll-size" (or the corresponding environment variable "EVENT_LISTENER_POLL_SIZE"). The recommended value based on tests with Monad is 1000.
JIRA: https://smartcontract-it.atlassian.net/browse/DPA-1529