Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export index and id in history_effects table #264

Merged
merged 5 commits into from
Jul 10, 2024

Conversation

amishas157
Copy link
Contributor

@amishas157 amishas157 commented Jul 9, 2024

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with the jira ticket associated with the PR.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated the README with the added features, breaking changes, new instructions on how to use the repository. I updated the description of the fuction with the changes that were made.

Release planning

  • I've decided if this PR requires a new major/minor/patch version accordingly to
    semver, and I've changed the name of the BRANCH to release/_ , feature/_ or patch/* .

What

This PR exports two additional params for operation effects:

  • Effect Index - These are generated in order for each operation
  • Effect id - This is essentially concatenation of operation_id-effect_index

Why

To keep data parity between Horizon and Hubble. Read https://stellarorg.atlassian.net/browse/HUBBLE-460

Known limitations

None

I additionally tested using docker container to ensure above props are getting outputted. Sample output below:

root@af62b592b099:/etl/data# stellar-etl export_effects --start-ledger 1000 \
> --end-ledger 10000 --output exported_effects.txt
INFO[2024-07-10T00:17:33.122Z] Number of bytes written: 6442                 pid=9
INFO[2024-07-10T00:17:33.125Z] {"attempted_transforms":8,"failed_transforms":0,"successful_transforms":8}  pid=9
INFO[2024-07-10T00:17:33.125Z] No cloud provider specified for upload. Skipping upload.  pid=9
root@af62b592b099:/etl/data# head exported_effects.txt
{"address":"GAP2KHWUMOHY7IO37UJY7SEBIITJIDZS5DRIIQRPEUT4VUKHZQGIRWS4","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"starting_balance":"20.0000000"},"id":"33676838572033-0","index":0,"ledger_sequence":7841,"operation_id":33676838572033,"type":0,"type_string":"account_created"}
{"address":"GALPCCZN4YXA3YMJHKL6CVIECKPLJJCTVMSNYWBTKJW4K5HQLYLDMZTB","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"amount":"20.0000000","asset_type":"native"},"id":"33676838572033-1","index":1,"ledger_sequence":7841,"operation_id":33676838572033,"type":3,"type_string":"account_debited"}
{"address":"GAP2KHWUMOHY7IO37UJY7SEBIITJIDZS5DRIIQRPEUT4VUKHZQGIRWS4","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"public_key":"GAP2KHWUMOHY7IO37UJY7SEBIITJIDZS5DRIIQRPEUT4VUKHZQGIRWS4","weight":1},"id":"33676838572033-2","index":2,"ledger_sequence":7841,"operation_id":33676838572033,"type":10,"type_string":"signer_created"}
{"address":"GCZTBYH66ISTZKUPVJWTMHWBH4S4JIJ7WNLQJXCTQJKWY3FIT34BWZCJ","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"starting_balance":"20.0000000"},"id":"33676838572034-0","index":0,"ledger_sequence":7841,"operation_id":33676838572034,"type":0,"type_string":"account_created"}
{"address":"GALPCCZN4YXA3YMJHKL6CVIECKPLJJCTVMSNYWBTKJW4K5HQLYLDMZTB","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"amount":"20.0000000","asset_type":"native"},"id":"33676838572034-1","index":1,"ledger_sequence":7841,"operation_id":33676838572034,"type":3,"type_string":"account_debited"}
{"address":"GCZTBYH66ISTZKUPVJWTMHWBH4S4JIJ7WNLQJXCTQJKWY3FIT34BWZCJ","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"public_key":"GCZTBYH66ISTZKUPVJWTMHWBH4S4JIJ7WNLQJXCTQJKWY3FIT34BWZCJ","weight":1},"id":"33676838572034-2","index":2,"ledger_sequence":7841,"operation_id":33676838572034,"type":10,"type_string":"signer_created"}
{"address":"GBEZOC5U4TVH7ZY5N3FLYHTCZSI6VFGTULG7PBITLF5ZEBPJXFT46YZM","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"starting_balance":"20.0000000"},"id":"33676838572035-0","index":0,"ledger_sequence":7841,"operation_id":33676838572035,"type":0,"type_string":"account_created"}
{"address":"GALPCCZN4YXA3YMJHKL6CVIECKPLJJCTVMSNYWBTKJW4K5HQLYLDMZTB","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"amount":"20.0000000","asset_type":"native"},"id":"33676838572035-1","index":1,"ledger_sequence":7841,"operation_id":33676838572035,"type":3,"type_string":"account_debited"}
{"address":"GBEZOC5U4TVH7ZY5N3FLYHTCZSI6VFGTULG7PBITLF5ZEBPJXFT46YZM","address_muxed":null,"closed_at":"2015-10-01T04:15:01Z","details":{"public_key":"GBEZOC5U4TVH7ZY5N3FLYHTCZSI6VFGTULG7PBITLF5ZEBPJXFT46YZM","weight":1},"id":"33676838572035-2","index":2,"ledger_sequence":7841,"operation_id":33676838572035,"type":10,"type_string":"signer_created"}
{"address":"GALPCCZN4YXA3YMJHKL6CVIECKPLJJCTVMSNYWBTKJW4K5HQLYLDMZTB","address_muxed":null,"closed_at":"2015-10-01T04:16:11Z","details":{"inflation_destination":"GAP2KHWUMOHY7IO37UJY7SEBIITJIDZS5DRIIQRPEUT4VUKHZQGIRWS4"},"id":"33736968114177-0","index":0,"ledger_sequence":7855,"operation_id":33736968114177,"type":7,"type_string":"account_inflation_destination_updated"}

@amishas157 amishas157 changed the title Export index and id in effects table Export index and id in history_effects table Jul 10, 2024
@amishas157 amishas157 marked this pull request as ready for review July 10, 2024 00:19
@amishas157 amishas157 requested a review from a team as a code owner July 10, 2024 00:19
Amisha Singla and others added 2 commits July 10, 2024 08:51
@amishas157 amishas157 merged commit e6a47f9 into master Jul 10, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants