Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Severe performance degradation after migrating to cramjam LZ4 #1028

Open
y4n9squared opened this issue Jul 9, 2024 · 0 comments
Open

Severe performance degradation after migrating to cramjam LZ4 #1028

y4n9squared opened this issue Jul 9, 2024 · 0 comments

Comments

@y4n9squared
Copy link
Contributor

y4n9squared commented Jul 9, 2024

Describe the bug

In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In python-lz4, the default level was 0. Now it is 9.

We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch codec.lz4_compress.

Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the link included in the function documentation:

image

the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.

Expected behaviour

Our "expected" behavior is that, since one cannot explicitly control compression settings and only compression.type, that settings are identical to they were when aiokafka directly used python-lz4. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant