Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deduplicate data when encoding nested Cadence composites to reduce memory and storage #347

Closed
fxamacker opened this issue Oct 6, 2023 · 1 comment
Assignees
Labels
performance storage breaking change breaks existing stored data (requires storage migration)

Comments

@fxamacker
Copy link
Member

Updates #292 onflow/flow-go#1744 onflow/cadence#1854

When encoding nested Cadence composites, it is possible for redundant data to get encoded. For example, the same Cadence composite type and its field names can get encoded more than once when the same composite is included within the slab (aka payload, register, segment) more than once.

Suggested Solution

Deduplicate data (such as field names and types) within same segment when encoding nested composites.

Include data deduplication in PR #342 since the redesign of Atree to support inlining segments in PR #342 and #345 already requires changing the encoding format.

Combining this work with Atree Inlining avoids creating redundant work and saves time by implementing it within PR #342.

What is Outside Scope

This data deduplication is limited to the scope of a slab. It is intentionally not global deduplication. See onflow/cadence#1854 for other aspects.

@fxamacker fxamacker added performance storage breaking change breaks existing stored data (requires storage migration) labels Oct 6, 2023
@fxamacker fxamacker self-assigned this Oct 6, 2023
@fxamacker
Copy link
Member Author

Closed by #342 #345

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance storage breaking change breaks existing stored data (requires storage migration)
Projects
None yet
Development

No branches or pull requests

1 participant