services/horizon/ingest: express parsing routines for Payment as public, reusable functions

[sreuland]

What problem does your feature solve?

New application development that needs to run ingestion pipeline will tend to implement(repeat) many similar parsing routines to transform network data into derived payment data models that Horizon ingestion does internally.

Horizon has many routines that are package private, internal to ingestion which parse and derive payments from stellar tx meta, so, they can't be re-used externally as-is.

What would you like to see?

Follows design proposal for reusable transformer functions for a common model
A few of these private ingestion parsing routines in Horizon exposed as pure(no side effects) functions and package public so they can be re-used by external apps to derive payment model quickly in their ingestion pipelines.

We limit focus on derived models to those considered ubiquitous and therefore most widely accepted such as a payment.

The presence of public functions for parsing payment model can encourage adoption of proposed streaming ingestion pipeline of which operators within a stream can invoke these functions to include payments info in-stream.

Create new pure function and model for payment in the shared ingest package. Horizon internal ingestion should not be changed or refactored as part of this effort, later we may decide different.

• transactionOperationWrapper.IsPayment() -> ingest.HasPayment(op xdr.Operation) bool

• for all operations that qualified as payment from HasPayment(), pull up the 'details' parsing routines of these operations from transactionOperationWrapper.Details() and fold into -> ingest.DerivePayment(tx xdr.Transaction, op xdr.Operation) []PaymentEvent:

  • payment
  • strictsend/strictreceive
  • createaccount/mergeaccount
  • invokehostfn - SAC events for mint/xfer/burn/claw

• Payment info modeled as an event in struct:

       type PaymentEvent struct {
          ToAmount:        "10.0000000"
          ToAsset:           "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
          FromAmount:   "10.0000000"
          FromAsset:       "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
          From:                 "GANFZDRBCNTUXIODCJEYMACPMCSZEVE4WZGZ3CZDZ3P2SXK4KH75IK6Y"
          To:                      "GANFZDRBCNTUXIODCJEYMACPMCSZEVE4WZGZ3CZDZ3P2SXK4KH75IK6Y"
          Timestamp:       "1723153694"
        }
  

• Unit test coverage on all new functions added to ingest package

Visualization of where the payment transform function fits in the larger CDP design for data pipeline:

• Relates to:
  • Update 'Build Custom Ingestion Pipeline' sample code to use more CDP  stellar-docs#889
  • services/horizon/ingest: express parsing routines for Offers as public, reusable functions #5414
  • ingest/pipeline: Create functional producer for BufferedStorageBackend #5412

What alternatives are there?

we don't publicize the routines, external apps use horizon code as reference, and will clone from this to their codebase as they choose to re-implement.