soroban-rpc: Implement Captive core usage

[paulbellamy]

epic - stellar/stellar-rpc#43

Acceptance Criteria: implement the decision from #4681

[12/16/22] scope of the data backend design in #4681 has widened to cover all data model of rpc and include usage of captive core and ingestion, this came about after RPC design discussion had consensus to go with 0.5 option

• Horizon usage in rpc to be removed in favor of captive core and new custom ingestion to a data model and persistence layer identified by design.

[jcx120]

This will be based on the final design of data backend (post v1)

[tamirms]

In #4681 (comment) we determined that SQLite would be the best option to store events in soroban-rpc. https://github.com/tamirms/soroban-events has a prototype which ingests ethereum events into SQLite. This code from this prototype can be lifted and adapted for use in soroban-rpc

[tamirms]

We will need to implement the following tasks to close this issue:

• integrate captive core with soroban-rpc
• use captive core to ingest soroban events from tx-meta into SQLite DB
• update getEvents rpc endpoint to query events from the SQLite DB

Here is a schema we can use for the SQLite DB:

https://github.com/tamirms/soroban-events/blob/main/migrations/01_init.sql

We should consider the following schema optimizations noted in the design doc:

https://docs.google.com/document/d/1F60xINIbfKrdnGeBBxygeHU3ZlgthRJw8DLXaYDakl0/edit?disco=AAAAkizX2ek
https://docs.google.com/document/d/1F60xINIbfKrdnGeBBxygeHU3ZlgthRJw8DLXaYDakl0/edit?disco=AAAAkizX2eA

For the ingestion component we will need to implement the following sub-tasks:

• discard any events from the SQLite DB older than the 24 hour threshold so that we retain only 24 hours of data
• ensure soroban-rpc resumes ingestion from last recorded event in the SQLite DB when it starts up
• include the latest ingested ledger sequence in the soroban-rpc response (similar to the Latest-Ledger header in horizon)
• Update the health handler to include a field which indicates whether soroban-rpc has fully ingested its 24 hour window of events and is in sync with the latest ledger in the network

[tsachiherman]

per discussion on Friday:
We need to add captive core ( but not remote captive core ) to soroban-rpc. This means that soroban-rpc need to have all the command line arguments needed to support the captive core + adding the invocation of captive core. (#4682)

[sreuland]

@tamirms , is this still blocked or has #4681 allowed this to proceed? If so, it can go to In Progress or Current Sprint if it's just not actively being worked.

[sreuland]

@tamirms , @2opremio , is this effectively done now? since #322 PR on tools added captive core to rpc.

[2opremio]

This issue is for events (although it doesn’t specify it the title). The tools PR implements captive core support for ledger entries #4717 .

[tamirms]

I'm closing this issue because we already have #4718 which is tracking supporting events in soroban-rpc