What happens when the indexer is unavailable or innaccurate?

[ryanchristo]

Opening this issue for further discussion and tracking as it may not be a concern but something we should take into consideration when leveraging the indexer within the application.

With the latest use of the credit class issuers to determine if the user can create a project and which credit classes are available to choose from, we are dependent on the indexer being available and the information being accurate (no missed events, all historical events included). While spinning up the local development environment with credit classes included in the genesis file (no events emitted, therefore no record of the credit class issuers in the indexer), I was unable to create and edit projects:

We don't need to worry about genesis state and no events being emitted (this is only an issue with a new chain and therefore only with a local testnet) but we should be tracking when and where we are dependent on the indexer and what we do in a scenario where the indexer is unavailable or what the implications are if the indexer is inaccurate.

The obvious solution is making sure the indexer is always available, it includes all the necessary events, and the event information is accurate, but we should consider the alternative scenarios. I thought the issue above was an interesting example because we are using the indexer to determine what the user can view within the application. This is not a security concern given permissions are still in place on the ledger and in the database but in a way we are relying on the indexer for permissions.

---

For Admin Use

• Not duplicate issue
• Appropriate labels and zenhub epics applied
• Appropriate contributors tagged

[ryanchristo]

We've discussed "pruning" or "resetting" the redwood data stored in the indexer to prevent unnecessary storage costs. A full reset would result in projects being unavailable to edit or credit classes being unavailable to select from.

[ryanchristo]

Cross-posting this review comment from #2049 as it seems relevant:

It doesn't seem related to this code but rather the indexer: why does it only return 2 classes for which regen1df675r9vnf7pdedn4sf26svdsem3ugavgxmy46 is an issuer while in reality there are other classes? cc/ @wgwz catching on this work so not sure if that's a bug or just some limitation we put for the staging indexer...

[wgwz]

Aside from just making the indexer infrastructure more reliable (regen-server is an important piece to), one thing I can think of is to have fallback's in place. I.e. for the credit class issuers permission, first we go to the indexer, if it fails in a critical way, fallback back to the slow but functional query we used previously. Same could apply for the activity page.

In terms of correctness, I think about things like improved testing and alerting. We already have some alerting when errors occur via sentry. But in terms of automated testing we don't have anything yet for this repo. Health checks come to mind as well, i.e. the indexer uses an RDS database in production (AWS supports alerts for memory/cpu usage etc. etc.)

Just some ideas, it's a pretty complicated system that might be good to visualize/diagram out.

[ryanchristo]

Sharing this issue as it seems relevant. The executor result for indexed group proposals may not be accurate in some scenarios (i.e. proposal pruned upon being executed): regen-network/indexer#42