Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What happens when the indexer is unavailable or innaccurate? #2046

Open
3 tasks
ryanchristo opened this issue Aug 24, 2023 · 4 comments
Open
3 tasks

What happens when the indexer is unavailable or innaccurate? #2046

ryanchristo opened this issue Aug 24, 2023 · 4 comments
Labels
dev question Further information is requested

Comments

@ryanchristo
Copy link
Member

ryanchristo commented Aug 24, 2023

Opening this issue for further discussion and tracking as it may not be a concern but something we should take into consideration when leveraging the indexer within the application.

With the latest use of the credit class issuers to determine if the user can create a project and which credit classes are available to choose from, we are dependent on the indexer being available and the information being accurate (no missed events, all historical events included). While spinning up the local development environment with credit classes included in the genesis file (no events emitted, therefore no record of the credit class issuers in the indexer), I was unable to create and edit projects:

Screenshot from 2023-08-24 15-51-11

We don't need to worry about genesis state and no events being emitted (this is only an issue with a new chain and therefore only with a local testnet) but we should be tracking when and where we are dependent on the indexer and what we do in a scenario where the indexer is unavailable or what the implications are if the indexer is inaccurate.

The obvious solution is making sure the indexer is always available, it includes all the necessary events, and the event information is accurate, but we should consider the alternative scenarios. I thought the issue above was an interesting example because we are using the indexer to determine what the user can view within the application. This is not a security concern given permissions are still in place on the ledger and in the database but in a way we are relying on the indexer for permissions.


For Admin Use

  • Not duplicate issue
  • Appropriate labels and zenhub epics applied
  • Appropriate contributors tagged
@ryanchristo
Copy link
Member Author

We've discussed "pruning" or "resetting" the redwood data stored in the indexer to prevent unnecessary storage costs. A full reset would result in projects being unavailable to edit or credit classes being unavailable to select from.

@ryanchristo
Copy link
Member Author

Cross-posting this review comment from #2049 as it seems relevant:

It doesn't seem related to this code but rather the indexer: why does it only return 2 classes for which regen1df675r9vnf7pdedn4sf26svdsem3ugavgxmy46 is an issuer while in reality there are other classes? cc/ @wgwz catching on this work so not sure if that's a bug or just some limitation we put for the staging indexer...

@wgwz
Copy link
Contributor

wgwz commented Aug 29, 2023

Aside from just making the indexer infrastructure more reliable (regen-server is an important piece to), one thing I can think of is to have fallback's in place. I.e. for the credit class issuers permission, first we go to the indexer, if it fails in a critical way, fallback back to the slow but functional query we used previously. Same could apply for the activity page.

In terms of correctness, I think about things like improved testing and alerting. We already have some alerting when errors occur via sentry. But in terms of automated testing we don't have anything yet for this repo. Health checks come to mind as well, i.e. the indexer uses an RDS database in production (AWS supports alerts for memory/cpu usage etc. etc.)

Just some ideas, it's a pretty complicated system that might be good to visualize/diagram out.

@ryanchristo ryanchristo added the question Further information is requested label Aug 31, 2023
@ryanchristo
Copy link
Member Author

Sharing this issue as it seems relevant. The executor result for indexed group proposals may not be accurate in some scenarios (i.e. proposal pruned upon being executed): regen-network/indexer#42

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants