fix: improve data fetcher response serialisation #216
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
API E2E Test Results207 tests 207 ✅ 29s ⏱️ Results for commit 5397280. ♻️ This comment has been updated with latest results. |
|
Visit the preview URL for this PR (updated for commit 5397280):
(expires Mon, 01 Apr 2024 17:59:27 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: e508f9012944951194447cb8885950b451a24403 |
Unit Test Results 4 files 263 suites 13m 52s ⏱️ Results for commit 5397280. ♻️ This comment has been updated with latest results. |
Romsters
approved these changes
Mar 26, 2024
|
🎉 This PR is included in version 2.45.6 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
SantiagoPittella
pushed a commit
to lambdaclass/block-explorer
that referenced
this pull request
May 28, 2024
chore: fix dev discussions link (matter-labs#143) - Updates dev discussions link <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> - We updated the link <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. Co-authored-by: Vasyl Ivanchuk <[email protected]> test: add allure public reports (matter-labs#68) updated workflow to make public allure reports current reports provide a link to a private url <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ + ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ n/a ] Tests for the changes have been added / updated. - [ n/a ] Documentation comments have been added / updated. --------- Co-authored-by: Vasyl Ivanchuk <[email protected]> feat: initial changes to show erc20 as native token in explorer feat: use nativechain instead of eth feat: finishing touches to show ERC20 as native token chore: rename .png chore: rename png asset feat: default to ethereum when rpc method is missing fix: for error of missing method fix: worker tests fix: worker tests fix: api tests and some e2e imports fix: move token.ts file to work for imports fix: common imports feat: adapt to base-token-addr-endpoint fix: worker tests fix: docs links (matter-labs#150) Fix links to documentation. The documentation has been updated so we should use new links. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. feat: add data fetcher service (matter-labs#145) Add data fetcher service that can be vertically scaled. Having data fetching vertically scaled we will be able to keep up with higher TPS. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. fix: remove portal from header (matter-labs#159) - Remove Portal link from the top menu - Do not show an error reason for the failed transaction if it's empty - Docker compose starts services in production and not in development mode - Add zkVM filter option for solc compilers on the verification page To fix bug reports / implement feature requests. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. --------- Co-authored-by: Oleh Bairak <[email protected]> fix: add data fetcher debug logs (matter-labs#165) Add more Data Fetcher debug logs. For easier troubleshooting. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. fix: fix data-fetcher tests chore: remove unused png chore: replace "Native Token" with "Base Token" chore: rename chainNative to baseToken feat: move base token fetching to separate module chore: remove old, unused code fix: use base token instead of base token fix: rename native token to base token chore: remove old, unused code fix: fix build, add comment fix: restore postgres default password on config feat: use base token from config in more places chore: update .env.example feat: move base-token fetching to a script fix: data-fetcher unit tests fix: api unit tests fix: worker tests fix: more tests chore: properly import config feat: rename to BaseToken feat: remove baseTokenData function feat: add base token address to configs feat: use base token from config instead of constant chore: remove unused scriped script fix: add transaction error reason transformer (matter-labs#186) Add transaction error reason transformer. To unsure error and reason are transformed correctly. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). fix: add graceful shutdown timeout (matter-labs#196) Add graceful shutdown timeout. To ensure that we can gracefully process all incoming HTTP requests before shutting down the service. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. feat: add transfer type filter for address transfers BFF API (matter-labs#195) Add filter by transfer type to the address transfers endpoint It is useful to be able to get a list of transfers of certain type for address. For example get list of withdrawals. feat: add address transfer type migration (matter-labs#199) - add script to migrate address transfer type - standard migration would not be able to complete, script uses parallel connections and updates data in small chunks. fix: use new DataSource instance for migration script (matter-labs#200) fix: use logger in migration scripts (matter-labs#201) fix: update address transfers type index order (matter-labs#197) fix: update goerli deprecation message (matter-labs#207) New goerli deprecation message with a specific deprecation date: <img width="1143" alt="image" src="https://github.com/matter-labs/block-explorer/assets/6553665/3bb8b99e-4998-40c4-8100-e4144fa8aa81"> To notify users when the deprecation is gonna happen. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). fix: improve fetcher usage (matter-labs#209) 1. Log timestamp in ISO8601 format. 2. Internal retries for `DataFetcherService`. 1. To have timestamps properly processed so logs are in the correct order. 2. Otherwise in case of an error the whole batch is retried which is unnecessary. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. fix: improve data fetcher response serialisation (matter-labs#216) Get rid of the separate interceptor that transforms `BigNumber` to string, accomplish the same result overriding `BigNumber` `toJSON` function. Separate interceptor is not optimal especially for blocks with lots of related data, which we btw expect to see more often with higher TPS. There are blocks where all related info weights 2Mb+ and it takes ~200ms to transform the whole response object by the interceptor. Since after the transformation JSON serialisation happens anyway, we can just override `toJSON` for `BigNumber` and it will be picked up during JSON serialisation. I checked that with this change generated response is the same. Performance improvement for large payloads (2Mb+): 1. Serialisation with interceptor (interceptor + json serialiser): avg 290ms 2. Serialisation with overridden `toJSON` (json serialiser): avg 34ms But most important event loop is not blocked for a long time. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. feat: increase request body size limit (matter-labs#217) feat: remove goerli network (matter-labs#219) Remove goerli network. Clean up the codebase since we stop supporting Goerli network. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. - [X] Documentation comments have been added / updated. fix: update baseToken to baseTokenData in E2E tests fix(e2e tests): fix end-to-end test to use base token, modify eth address fix(e2e tests): fix eth address in address e2e tests fix(e2e tests): change missing apparences of l1 address fix(e2e tests): non existing var name fix: increase default timeout settings (matter-labs#228) Increase default timeout settings. To make sure explorer can handle large transactions. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. fix(e2e tests): valid address format fix(e2e tests): add missing fields in select fix(e2e tests): remove unused modules fix(e2e tests): typecheck fix(e2e tests): increase amount of selects in test fix(e2e tests): add missing values in tests test(coverage): api config with base token test: uncomment tests fix: remove non existing script fix: address PR comments fix: lint errors fix: rpc method name fix: rollback address capitalization fix: use checksum address fix: rename root hash to block hash (matter-labs#249) Rename root hash on the block screen to block hash. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. fix: use both version of the rpc response for getBridgeContract method Revert "fix(e2e tests): add missing fields in select" This reverts commit d3916ed. Revert "fix(e2e tests): add missing fields in select" This reverts commit d3916ed. fix(e2e): account tests fix: use correct amount of selects test: add tests for old response chore: fix docker compose (matter-labs#256) Fix docker compose by applying the same updates that were done in [local-setup](https://github.com/matter-labs/local-setup.) Old docker compose doesn't work with the latest `local-node` docker image. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). fix: explorer doesn't require defined bridge address (matter-labs#257) Fix explorer so that a defined bridge address is not required. In-memory node doesn't have L1 network and bridge addresses defined. This fix make explorer work with the in-memory node. <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Get rid of the separate interceptor that transforms
BigNumberto string, accomplish the same result overridingBigNumbertoJSONfunction.Why ❔
Separate interceptor is not optimal especially for blocks with lots of related data, which we btw expect to see more often with higher TPS. There are blocks where all related info weights 2Mb+ and it takes ~200ms to transform the whole response object by the interceptor. Since after the transformation JSON serialisation happens anyway, we can just override
toJSONforBigNumberand it will be picked up during JSON serialisation. I checked that with this change generated response is the same.Performance improvement for large payloads (2Mb+):
toJSON(json serialiser): avg 34msBut most important event loop is not blocked for a long time.
Checklist