You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In python-lz4, the default level was 0. Now it is 9.
We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch codec.lz4_compress.
Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the link included in the function documentation:
the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.
Expected behaviour
Our "expected" behavior is that, since one cannot explicitly control compression settings and only compression.type, that settings are identical to they were when aiokafka directly used python-lz4. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?
The text was updated successfully, but these errors were encountered:
Describe the bug
In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In
python-lz4
, the default level was 0. Now it is 9.We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch
codec.lz4_compress
.Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the link included in the function documentation:
the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.
Expected behaviour
Our "expected" behavior is that, since one cannot explicitly control compression settings and only
compression.type
, that settings are identical to they were when aiokafka directly usedpython-lz4
. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?The text was updated successfully, but these errors were encountered: