services/horizon/ingest: express parsing routines for Offers as public, reusable functions

[sreuland]

What problem does your feature solve?

New application development that needs to run ingestion pipeline will tend to implement(repeat) many similar parsing routines to generate derived 'Offer' models that Horizon ingestion does internally.

Horizon has many routines that are package private, internal to ingestion which parse and derive 'Offer' from stellar tx meta, so, they can't be re-used externally as-is.

What would you like to see?

Follows same design proposal for reusable transformer function for a common model

A few of these private ingestion parsing routines in Horizon exposed as pure(no side effects) functions and package public so they can be re-used by external apps to derive the same 'Offer' data models quickly in their ingestion pipelines.

The presence of public functions for parsing 'Offer' model can further encourage external adoption of proposed streaming ingestion pipeline of which operators within a stream invoke these functions directly to include offers data in their stream processing.

Create new pure function and model for 'Offer' in the shared ingest package. Horizon internal ingestion should not be changed or refactored, later this may be revisited.

• ingest.HasOffer(op xdr.Operation) bool

• for all operations that qualified as offer in HasOffer(), pull up the 'details' parsing routines of these operations from transactionOperationWrapper.Details() and fold into -> ingest.DeriveOffer(tx xdr.Transaction, op xdr.Operation) []OfferEvent:

  • managebuy
  • managesell
  • createsell

• Offer data is modeled as a derived event representing the life cycle events of an offer, this is initial estimate of event schema, during effort, can change these:

  type OfferPartiallyFilledEvent struct {    
        Price:                         "8.0000000"
        Amount:                    "89.9342000"
        PriceAssetRatioD:    "1"
        PriceAssetRatioN:    "8"
        BuyingAsset:            "native"
        SellingAsset:             "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
        Timestamp:               "1723153694"
   }

type OfferCreatedEvent struct {    
        Price:                         "8.0000000"
        Amount:                    "89.9342000"
        PriceAssetRatioD:    "1"
        PriceAssetRatioN:    "8"
        BuyingAsset:            "native"
        SellingAsset:             "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
        Timestamp:               "1723153694"
   }

type OfferClosedEvent struct {    
        BuyingAsset:            "native"
        SellingAsset:             "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
        Timestamp:               "1723153694"
   }

• Unit test coverage on new functions added to ingest package

Visualization of where this transform function fits in the larger CDP design for data pipeline:

• Relates to:
  • Update 'Build Custom Ingestion Pipeline' sample code to use more CDP  stellar-docs#889
  • services/horizon/ingest: express parsing routines for Payment as public, reusable functions #5413
  • ingest/pipeline: Create functional producer for BufferedStorageBackend #5412

What alternatives are there?

we don't publicize the routines, external apps use horizon code as reference, and will clone from this to their codebase as they choose to re-implement.

[karthikiyer56]

Is there a notion of OfferCancelled at all?

[karthikiyer56]

This ticket has been closed since the work is now captured in #5543, which extends the sentiment of offers to include trades.

In that sense, the new way of approaching this to think about an offer and liqudity pool events/updates as a sequence of trades