Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add getTransactions endpoint #136

Merged
merged 65 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
c00ddae
Add cursor for getTransactions
aditya1702 Apr 12, 2024
2467dbe
Add validation of pagination and range
aditya1702 Apr 12, 2024
6aa348d
implement getTransactions handler - 1
aditya1702 Apr 15, 2024
7928c78
Add getLedgers
aditya1702 Apr 15, 2024
355b1e2
Read a single ledger instead of getting all ledgers
aditya1702 Apr 15, 2024
11baf9e
Add ParseCursor method for tx cursor
aditya1702 Apr 15, 2024
81eceff
Cursor changes - 1
aditya1702 Apr 15, 2024
83918f2
Cursor changes - 2
aditya1702 Apr 16, 2024
3d7f7d3
Cursor changes - 3
aditya1702 Apr 16, 2024
f509457
revert go-mod changes
aditya1702 Apr 16, 2024
f813d01
Use reader.Seek()
aditya1702 Apr 16, 2024
0e8b9da
Use reader.Seek() - 2
aditya1702 Apr 16, 2024
5156b9e
Add config-options for tx pagination limits
aditya1702 Apr 16, 2024
e4b34e5
Merge branch 'main' into get-transactions
aditya1702 Apr 16, 2024
35da2a5
Fix failing cursor test
aditya1702 Apr 16, 2024
35ed84d
Go mod changes
aditya1702 Apr 16, 2024
fce765b
Go mod changes - 2
aditya1702 Apr 16, 2024
60055da
Add unittests - 1
aditya1702 Apr 17, 2024
90f6b5b
Update go.mod and go.sum
aditya1702 Apr 17, 2024
36205ce
Merge branch 'main' into get-transactions
aditya1702 Apr 17, 2024
ba254f0
Add integration tests
aditya1702 Apr 18, 2024
18dbbd1
Merge branch 'main' into get-transactions
aditya1702 Apr 18, 2024
cb3eae9
Update go.mod and go.sum
aditya1702 Apr 18, 2024
a75205c
Add ledgerSeq to error string
aditya1702 Apr 18, 2024
65d2ab8
Add docstrings
aditya1702 Apr 18, 2024
dae0624
Change transactions limits
aditya1702 Apr 18, 2024
f1a0a87
add defensive check for error other than EOF
aditya1702 Apr 18, 2024
364ee92
add defensive check for error other than EOF - 2
aditya1702 Apr 18, 2024
aeb0a84
Change ledger sequence to uint32
aditya1702 Apr 18, 2024
2c5b2a4
Add comments/docstrings
aditya1702 Apr 18, 2024
95ce052
Include only cursor in response
aditya1702 Apr 18, 2024
c27d20c
Use toid instead of new cursor
aditya1702 Apr 22, 2024
6837f79
Revert cursor changes
aditya1702 Apr 22, 2024
9239dff
Merge branch 'main' into get-transactions
aditya1702 Apr 23, 2024
0f098fb
Return cursor as string in result
aditya1702 Apr 29, 2024
5f273a7
Merge branch 'main' into get-transactions
aditya1702 Apr 29, 2024
c9ddee4
Refactor reader.Seek error handling
aditya1702 Apr 29, 2024
d426361
Merge remote-tracking branch 'origin/get-transactions' into get-trans…
aditya1702 Apr 29, 2024
1c44f63
Small refactoring
aditya1702 Apr 30, 2024
850f208
Remove startLedger check
aditya1702 May 3, 2024
c45d333
Merge branch 'refs/heads/main' into get-transactions
aditya1702 May 3, 2024
be21a95
Remove endLedger
aditya1702 May 3, 2024
bd81a39
Merge branch 'refs/heads/database-backend' into get-transactions
aditya1702 May 3, 2024
088dff0
Import fix
aditya1702 May 3, 2024
74e2941
Fix failing tests
aditya1702 May 3, 2024
ca6c4ca
Refactor to use new transaction db
aditya1702 May 3, 2024
6d415dd
Refactor mocks
aditya1702 May 3, 2024
acaae61
Refactor unittests for using the new db changes
aditya1702 May 6, 2024
af41b32
Refactor integration test
aditya1702 May 6, 2024
90afd1e
Add config vars for max requests and request duration
aditya1702 May 6, 2024
31c80f7
Fix failing test
aditya1702 May 6, 2024
0a00590
Use transactionInfo struct instead of db.Transactions
aditya1702 May 8, 2024
0b19b17
Start indexing from 1 instead of 0 for toid
aditya1702 May 8, 2024
f85bdfb
Operation index from 1
aditya1702 May 9, 2024
4442348
Add lines to make sure structs implement interfaces
aditya1702 May 9, 2024
2d66084
Remove omitempty
aditya1702 May 9, 2024
f08dc00
rename test func
aditya1702 May 9, 2024
bc6cbba
make txInfo struct public and convert string cursor to int
aditya1702 May 10, 2024
68e45d5
Use sendSuccessfulTransaction helper func
aditya1702 May 10, 2024
21da905
Convert cursor to string in request
aditya1702 May 10, 2024
7e1a471
Change jrpc response codes
aditya1702 May 10, 2024
c63f8e1
Change ledger close meta code to invalid-params
aditya1702 May 13, 2024
a6a77b9
Revert back to InvalidParams error for reader.Read()
aditya1702 May 13, 2024
7509c6e
Refactor if-else statement
aditya1702 May 13, 2024
d4f9b54
Refactor if-else statement - 2
aditya1702 May 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cmd/soroban-rpc/internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ type Config struct {
CheckpointFrequency uint32
CoreRequestTimeout time.Duration
DefaultEventsLimit uint
DefaultTransactionsLimit uint
EventLedgerRetentionWindow uint32
FriendbotURL string
HistoryArchiveURLs []string
Expand All @@ -33,6 +34,7 @@ type Config struct {
LogFormat LogFormat
LogLevel logrus.Level
MaxEventsLimit uint
MaxTransactionsLimit uint
MaxHealthyLedgerLatency time.Duration
NetworkPassphrase string
PreflightWorkerCount uint
Expand Down
22 changes: 22 additions & 0 deletions cmd/soroban-rpc/internal/config/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,28 @@ func (cfg *Config) options() ConfigOptions {
return nil
},
},
{
Name: "max-transactions-limit",
Usage: "Maximum amount of transactions allowed in a single getTransactions response",
ConfigKey: &cfg.MaxTransactionsLimit,
DefaultValue: uint(10000),
},
{
Name: "default-transactions-limit",
Usage: "Default cap on the amount of transactions included in a single getTransactions response",
ConfigKey: &cfg.DefaultTransactionsLimit,
DefaultValue: uint(100),
aditya1702 marked this conversation as resolved.
Show resolved Hide resolved
Validate: func(co *ConfigOption) error {
if cfg.DefaultTransactionsLimit > cfg.MaxTransactionsLimit {
return fmt.Errorf(
"default-transactions-limit (%v) cannot exceed max-transactions-limit (%v)",
cfg.DefaultTransactionsLimit,
cfg.MaxTransactionsLimit,
)
}
return nil
},
},
{
Name: "max-healthy-ledger-latency",
Usage: "maximum ledger latency (i.e. time elapsed since the last known ledger closing time) considered to be healthy" +
Expand Down
8 changes: 8 additions & 0 deletions cmd/soroban-rpc/internal/daemon/interfaces/interfaces.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,11 @@ type CoreClient interface {
Info(ctx context.Context) (*proto.InfoResponse, error)
SubmitTransaction(context.Context, string) (*proto.TXResponse, error)
}

type Cursor interface {
String() string
MarshalJSON() ([]byte, error)
UnmarshalJSON(b []byte) error
Cmp(other Cursor) int
ParseCursor(input string) (Cursor, error)
}
41 changes: 25 additions & 16 deletions cmd/soroban-rpc/internal/events/cursor.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import (
"strings"

"github.com/stellar/go/toid"

"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/daemon/interfaces"
)

// Cursor represents the position of a Soroban event.
Expand Down Expand Up @@ -37,7 +39,7 @@ func (c Cursor) String() string {
}

// MarshalJSON marshals the cursor into JSON
func (c Cursor) MarshalJSON() ([]byte, error) {
func (c *Cursor) MarshalJSON() ([]byte, error) {
return json.Marshal(c.String())
}

Expand All @@ -48,35 +50,40 @@ func (c *Cursor) UnmarshalJSON(b []byte) error {
return err
}

if parsed, err := ParseCursor(s); err != nil {
parsed, err := c.ParseCursor(s)
if err != nil {
return err
}

if eventCursor, ok := parsed.(*Cursor); ok {
*c = *eventCursor
} else {
*c = parsed
return fmt.Errorf("parsed cursor could not be converted to Event cursor")
}
return nil
}

// ParseCursor parses the given string and returns the corresponding cursor
func ParseCursor(input string) (Cursor, error) {
func (c *Cursor) ParseCursor(input string) (interfaces.Cursor, error) {
parts := strings.SplitN(input, "-", 2)
if len(parts) != 2 {
return Cursor{}, fmt.Errorf("invalid event id %s", input)
return &Cursor{}, fmt.Errorf("invalid event id %s", input)
}

// Parse the first part (toid)
idInt, err := strconv.ParseInt(parts[0], 10, 64) //lint:ignore gomnd
if err != nil {
return Cursor{}, fmt.Errorf("invalid event id %s: %w", input, err)
return &Cursor{}, fmt.Errorf("invalid event id %s: %w", input, err)
}
parsed := toid.Parse(idInt)

// Parse the second part (event order)
eventOrder, err := strconv.ParseUint(parts[1], 10, 32) //lint:ignore gomnd
if err != nil {
return Cursor{}, fmt.Errorf("invalid event id %s: %w", input, err)
return &Cursor{}, fmt.Errorf("invalid event id %s: %w", input, err)
}

return Cursor{
return &Cursor{
Ledger: uint32(parsed.LedgerSequence),
Tx: uint32(parsed.TransactionOrder),
Op: uint32(parsed.OperationOrder),
Expand All @@ -98,17 +105,19 @@ func cmp(a, b uint32) int {
// 0 is returned if the c is equal to other.
// 1 is returned if c is greater than other.
// -1 is returned if c is less than other.
func (c Cursor) Cmp(other Cursor) int {
if c.Ledger == other.Ledger {
if c.Tx == other.Tx {
if c.Op == other.Op {
return cmp(c.Event, other.Event)
func (c Cursor) Cmp(other interfaces.Cursor) int {
otherCursor := other.(*Cursor)

if c.Ledger == otherCursor.Ledger {
if c.Tx == otherCursor.Tx {
if c.Op == otherCursor.Op {
return cmp(c.Event, otherCursor.Event)
}
return cmp(c.Op, other.Op)
return cmp(c.Op, otherCursor.Op)
}
return cmp(c.Tx, other.Tx)
return cmp(c.Tx, otherCursor.Tx)
}
return cmp(c.Ledger, other.Ledger)
return cmp(c.Ledger, otherCursor.Ledger)
}

var (
Expand Down
8 changes: 4 additions & 4 deletions cmd/soroban-rpc/internal/events/cursor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
)

func TestParseCursor(t *testing.T) {
for _, cursor := range []Cursor{
for _, cursor := range []*Cursor{
aditya1702 marked this conversation as resolved.
Show resolved Hide resolved
{
Ledger: math.MaxInt32,
Tx: 1048575,
Expand All @@ -29,7 +29,7 @@ func TestParseCursor(t *testing.T) {
Event: 1,
},
} {
parsed, err := ParseCursor(cursor.String())
parsed, err := cursor.ParseCursor(cursor.String())
assert.NoError(t, err)
assert.Equal(t, cursor, parsed)
}
Expand Down Expand Up @@ -97,12 +97,12 @@ func TestCursorCmp(t *testing.T) {
b := testCase.b
expected := testCase.expected

if got := a.Cmp(b); got != expected {
if got := a.Cmp(&b); got != expected {
t.Fatalf("expected (%v).Cmp(%v) to be %v but got %v", a, b, expected, got)
}
a, b = b, a
expected *= -1
if got := a.Cmp(b); got != expected {
if got := a.Cmp(&b); got != expected {
t.Fatalf("expected (%v).Cmp(%v) to be %v but got %v", a, b, expected, got)
}
}
Expand Down
13 changes: 7 additions & 6 deletions cmd/soroban-rpc/internal/events/events.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ func (m *MemoryStore) Scan(eventRange Range, f ScanFunction) (lastLedgerInWindow
timestamp := bucket.LedgerCloseTimestamp
for _, event := range events {
cur := event.cursor(bucket.LedgerSeq)
if eventRange.End.Cmp(cur) <= 0 {
if eventRange.End.Cmp(&cur) <= 0 {
return
}
var diagnosticEvent xdr.DiagnosticEvent
Expand All @@ -153,26 +153,26 @@ func (m *MemoryStore) validateRange(eventRange *Range) error {
}
firstBucket := m.eventsByLedger.Get(0)
min := Cursor{Ledger: firstBucket.LedgerSeq}
if eventRange.Start.Cmp(min) < 0 {
if eventRange.Start.Cmp(&min) < 0 {
if eventRange.ClampStart {
eventRange.Start = min
} else {
return errors.New("start is before oldest ledger")
}
}
max := Cursor{Ledger: min.Ledger + m.eventsByLedger.Len()}
if eventRange.Start.Cmp(max) >= 0 {
if eventRange.Start.Cmp(&max) >= 0 {
return errors.New("start is after newest ledger")
}
if eventRange.End.Cmp(max) > 0 {
if eventRange.End.Cmp(&max) > 0 {
if eventRange.ClampEnd {
eventRange.End = max
} else {
return errors.New("end is after latest ledger")
}
}

if eventRange.Start.Cmp(eventRange.End) >= 0 {
if eventRange.Start.Cmp(&eventRange.End) >= 0 {
return errors.New("start is not before end")
}

Expand All @@ -184,7 +184,8 @@ func (m *MemoryStore) validateRange(eventRange *Range) error {
// events must be sorted in ascending order.
func seek(events []event, cursor Cursor) []event {
j := sort.Search(len(events), func(i int) bool {
return cursor.Cmp(events[i].cursor(cursor.Ledger)) <= 0
cur := events[i].cursor(cursor.Ledger)
return cursor.Cmp(&cur) <= 0
})
return events[j:]
}
Expand Down
7 changes: 7 additions & 0 deletions cmd/soroban-rpc/internal/jsonrpc.go
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,13 @@ func NewJSONRPCHandler(cfg *config.Config, params HandlerParams) Handler {
queueLimit: cfg.RequestBacklogGetTransactionQueueLimit,
requestDurationLimit: cfg.MaxGetTransactionExecutionDuration,
},
{
methodName: "getTransactions",
underlyingHandler: methods.NewGetTransactionsHandler(params.Logger, params.LedgerReader, params.LedgerEntryReader, cfg.MaxTransactionsLimit, cfg.DefaultTransactionsLimit, cfg.NetworkPassphrase),
longName: "get_transactions",
queueLimit: cfg.RequestBacklogSendTransactionQueueLimit,
aditya1702 marked this conversation as resolved.
Show resolved Hide resolved
requestDurationLimit: cfg.MaxSendTransactionExecutionDuration,
},
{
methodName: "sendTransaction",
underlyingHandler: methods.NewSendTransactionHandler(params.Daemon, params.Logger, params.TransactionStore, cfg.NetworkPassphrase),
Expand Down
8 changes: 4 additions & 4 deletions cmd/soroban-rpc/internal/methods/get_events.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ type EventInfo struct {
}

type GetEventsRequest struct {
StartLedger uint32 `json:"startLedger,omitempty"`
Filters []EventFilter `json:"filters"`
Pagination *PaginationOptions `json:"pagination,omitempty"`
StartLedger uint32 `json:"startLedger,omitempty"`
Filters []EventFilter `json:"filters"`
Pagination *EventsPaginationOptions `json:"pagination,omitempty"`
}

func (g *GetEventsRequest) Valid(maxLimit uint) error {
Expand Down Expand Up @@ -288,7 +288,7 @@ func (s *SegmentFilter) UnmarshalJSON(p []byte) error {
return nil
}

type PaginationOptions struct {
type EventsPaginationOptions struct {
Cursor *events.Cursor `json:"cursor,omitempty"`
Limit uint `json:"limit,omitempty"`
}
Expand Down
10 changes: 5 additions & 5 deletions cmd/soroban-rpc/internal/methods/get_events_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,7 @@ func TestGetEventsRequestValid(t *testing.T) {
assert.EqualError(t, (&GetEventsRequest{
StartLedger: 1,
Filters: []EventFilter{},
Pagination: &PaginationOptions{Cursor: &events.Cursor{}},
Pagination: &EventsPaginationOptions{Cursor: &events.Cursor{}},
}).Valid(1000), "startLedger and cursor cannot both be set")

assert.NoError(t, (&GetEventsRequest{
Expand All @@ -422,7 +422,7 @@ func TestGetEventsRequestValid(t *testing.T) {
assert.EqualError(t, (&GetEventsRequest{
StartLedger: 1,
Filters: []EventFilter{},
Pagination: &PaginationOptions{Limit: 1001},
Pagination: &EventsPaginationOptions{Limit: 1001},
}).Valid(1000), "limit must not exceed 1000")

assert.EqualError(t, (&GetEventsRequest{
Expand Down Expand Up @@ -932,7 +932,7 @@ func TestGetEvents(t *testing.T) {
results, err := handler.getEvents(GetEventsRequest{
StartLedger: 1,
Filters: []EventFilter{},
Pagination: &PaginationOptions{Limit: 10},
Pagination: &EventsPaginationOptions{Limit: 10},
})
assert.NoError(t, err)

Expand Down Expand Up @@ -1016,7 +1016,7 @@ func TestGetEvents(t *testing.T) {
defaultLimit: 100,
}
results, err := handler.getEvents(GetEventsRequest{
Pagination: &PaginationOptions{
Pagination: &EventsPaginationOptions{
Cursor: id,
Limit: 2,
},
Expand Down Expand Up @@ -1048,7 +1048,7 @@ func TestGetEvents(t *testing.T) {
assert.Equal(t, GetEventsResponse{expected, 5}, results)

results, err = handler.getEvents(GetEventsRequest{
Pagination: &PaginationOptions{
Pagination: &EventsPaginationOptions{
Cursor: &events.Cursor{Ledger: 5, Tx: 2, Op: 0, Event: 1},
Limit: 2,
},
Expand Down
Loading
Loading