Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exp/services/ledgerexporter: Initial draft #5160

Merged
merged 78 commits into from
Feb 27, 2024
Merged

exp/services/ledgerexporter: Initial draft #5160

Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
bc44ebd
Ledgerexporter xdr changes
urvisavla Jan 9, 2024
28ca57a
Rewrite ledgerexporter using new xdr
urvisavla Jan 9, 2024
a6f9d99
Add basic test
urvisavla Jan 9, 2024
9aed8dc
Fixes
urvisavla Jan 10, 2024
a66a24f
Start upload in a separate go routine
urvisavla Jan 10, 2024
2822040
Naming changes and other refactoring
urvisavla Jan 11, 2024
19a5bad
Add condition to prevent file overwrite for GCS
urvisavla Jan 12, 2024
836661f
Adjust range based on ledgersperfile config parameter
urvisavla Jan 12, 2024
06015f7
Improve handling of ledger range and added a unit test.
urvisavla Jan 12, 2024
e8b4859
Update comments
urvisavla Jan 12, 2024
0eedb07
Added error handling and addressing review comments
urvisavla Jan 16, 2024
4ede9e2
Minor changes
urvisavla Jan 16, 2024
a113d5b
Update exp/services/ledgerexporter/exportmanager.go
urvisavla Jan 17, 2024
a6341fa
Update exp/services/ledgerexporter/exportmanager.go
urvisavla Jan 17, 2024
9ad4746
Update exp/services/ledgerexporter/uploader.go
urvisavla Jan 17, 2024
3f8a83b
Addressing review comments
urvisavla Jan 18, 2024
a8b403a
Improved code organization.
urvisavla Jan 18, 2024
509861e
Add README.md
urvisavla Jan 18, 2024
b87fce6
Improve validations and error handling
urvisavla Jan 18, 2024
5fb16d9
Add unit tests for ledger range adjustments based on config and fix a…
urvisavla Jan 18, 2024
d00cbd7
Removing redundant data from the LedgerCloseMetaObject. Adding checks…
urvisavla Jan 19, 2024
2b1106c
Fix static check error
urvisavla Jan 19, 2024
9d4a5b6
Disable ledgerexporter job from github workflow
urvisavla Jan 19, 2024
888ab26
Fix linter errors
urvisavla Jan 19, 2024
84e4245
Remove conditional puts for now to avoid potential disruption for exi…
urvisavla Jan 19, 2024
d7eddbe
fix ci error
urvisavla Jan 19, 2024
e24fa9b
Disable ledger-exporter job from ci
urvisavla Jan 24, 2024
a18b14a
update logging
urvisavla Jan 24, 2024
e88d565
Exportmanager code refactoring
urvisavla Jan 25, 2024
397d881
Add gzip compression
urvisavla Jan 25, 2024
4938e89
Add start-from-last-n-ledger option
urvisavla Jan 25, 2024
3fb9f07
refactoring export object
urvisavla Jan 30, 2024
90f7cef
Add conditional uploads
urvisavla Jan 30, 2024
166c4cc
Validate supplied ledge range and add unit tests
urvisavla Jan 30, 2024
946a673
Update README
urvisavla Jan 30, 2024
85675bc
Update exp/services/ledgerexporter/internal/app.go
urvisavla Jan 30, 2024
3f73670
Update exp/services/ledgerexporter/internal/datastore.go
urvisavla Jan 30, 2024
4df4989
Addressing review comments
urvisavla Feb 1, 2024
f352552
Add unit test coverage for the Uploader
urvisavla Feb 1, 2024
e923c13
Generate default captive-core config based on the specified 'network'
urvisavla Feb 1, 2024
f592361
Files missed in previous commit
urvisavla Feb 1, 2024
e380b5c
Add default testnet and pubnet configs for captive-core
urvisavla Feb 1, 2024
a66e11b
Fix ci errors
urvisavla Feb 1, 2024
a51e959
Update cloud.google.com/go/storage package to latest
urvisavla Feb 1, 2024
e648557
Use idiomatic go receiver pattern for config
urvisavla Feb 8, 2024
c566989
Remove captive_core_use_db config param.
urvisavla Feb 8, 2024
5d847fb
Set useragent
urvisavla Feb 8, 2024
e51467d
Set captive-core on-disk mode by default
urvisavla Feb 8, 2024
1ffb57e
Update exp/services/ledgerexporter/README.md
urvisavla Feb 8, 2024
49b7ce1
The captive-core config section is omitted as it's automatically deri…
urvisavla Feb 13, 2024
7ad8ada
Update exp/services/ledgerexporter/internal/app.go
urvisavla Feb 13, 2024
46655b8
Fix unittests
urvisavla Feb 13, 2024
c531e76
Addressing review comment
urvisavla Feb 13, 2024
97bc1a0
Addressing review comments
urvisavla Feb 13, 2024
4d39d2d
Remove doneCh
urvisavla Feb 14, 2024
69480ab
Pass app context to the datastore
urvisavla Feb 14, 2024
ead1bde
Resolve merge conflict
urvisavla Feb 20, 2024
42e2545
Regenerate xdr after Makefile update
urvisavla Feb 20, 2024
83f0429
Add XDRGzipEncoder and XDRGzipDecoder
urvisavla Feb 20, 2024
ca1c285
Addressing review comments
urvisavla Feb 20, 2024
977bb53
Addressing review comments
urvisavla Feb 21, 2024
24d2fad
Move captive-core default configs to ledgerbackend package.
urvisavla Feb 21, 2024
c02f09a
Addressing review comment
urvisavla Feb 21, 2024
a68d9b2
Update exp/services/ledgerexporter/internal/utils.go
urvisavla Feb 21, 2024
d2ba02d
Update exp/services/ledgerexporter/internal/utils.go
urvisavla Feb 21, 2024
03a6dbb
removed unused code and other minor changes
urvisavla Feb 21, 2024
2a97dc5
Addressing review comments
urvisavla Feb 21, 2024
9ed7ee2
Do not store ctx in App struct
urvisavla Feb 21, 2024
59093c8
Handle canceled context in AddLedgerCloseMeta
urvisavla Feb 22, 2024
8d91ef3
Improved uploader shutdown logic
urvisavla Feb 22, 2024
ec4283b
Address review comments
urvisavla Feb 22, 2024
44eb740
Export manager holds only a single current meta archive
urvisavla Feb 22, 2024
ba15276
Update exp/services/ledgerexporter/internal/uploader.go
urvisavla Feb 22, 2024
1bf8c45
Address review comments
urvisavla Feb 22, 2024
bea88e5
Update exp/services/ledgerexporter/internal/config.go
urvisavla Feb 24, 2024
92b7392
Misc. minor improvements
urvisavla Feb 26, 2024
717c3dc
Use replace instead of assert in unit tests
urvisavla Feb 27, 2024
cbb3a5b
Update README
urvisavla Feb 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Addressing review comment
  • Loading branch information
@urvisavla
urvisavla committed Feb 22, 2024
commit c02f09a7204891d1fd1ec7e35743d8629d40f947
1 change: 0 additions & 1 deletion exp/services/ledgerexporter/internal/exportmanager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ func (e *exportManager) AddLedgerCloseMeta(ledgerCloseMeta xdr.LedgerCloseMeta)

if ledgerSeq >= metaArchive.GetEndLedgerSequence() {
// Current archive is full, send it for upload
// This is a blocking call!
e.metaArchiveCh <- metaArchive
// Remove it from the map
delete(e.metaArchiveMap, objectKey)
Expand Down