update #1

dawn2zhang · 2021-05-27T02:14:05Z

Motivation:

Explain here the context, and why you're making that change.
What is the problem you're trying to solve.

Modification:

Describe the modifications you've done.

Result:

Fixes #.

If there is no issue then describe the changes introduced by this PR.

Quarkus disables JNDI by default, which breaks the (default) DNS servers lookup in `DirContextUtils`, so DNS lookups default to the public Google DNS servers. This means, that looking up k8s service names, or any internal names, isn't possible without either enabling JNDI, which is not such a great option, or specifying DNS servers manually, which is awkward, since there is `/etc/resolv.conf`. This change prefers reading the nameservers from `/etc/resolv.conf` over looking those up on JNDI on Linux + macOS. This is effectively a noop-change when JNDI is enabled, because the nameservers available via JNDI are read from `/etc/resolv.conf`. Behavior for Windows (still requires JNDI) and Android (neither JNDI nor `/etc/resolv.conf`) is not changed. Fixes #13883

#13898) …ten. Motivation: ByteToMessageDecoder.channelReadComplete(...) should only call read() once if channelRead(...) did not produce any message. Modifications: - Correctly reset variable - Add unit test Result: Not break flow-control

Motivation: According to the specification for parsing of application/x-www-form-urlencoded content at https://url.spec.whatwg.org/#application/x-www-form-urlencoded, a key without an = should be able to be parsed and given an empty value. The current implementation of HttpPostStandardRequestDecoder fails to parse these no-value keys when they are the last value in the sequence. Modifications: HttpPostStandardRequestDecoder is modified to include a key with no value that is at the end of the undecoded chunk in the existing "special empty FIELD" code path that currently only handles such fields when they are followed by a '&' character. Additional tests are provided to throroughly exercise variations of content bodies with such empty fields. Result: Keys with no value that appear at the end of a x-www-form-urlencoded sequence will be parsed according to the spec.

Reverts #13904

…13908) Motivation: According to the specification for parsing of application/x-www-form-urlencoded content at https://url.spec.whatwg.org/#application/x-www-form-urlencoded, a key without an = should be able to be parsed and given an empty value. The current implementation of HttpPostStandardRequestDecoder fails to parse these no-value keys when they are the last value in the sequence. Modifications: HttpPostStandardRequestDecoder is modified to include a key with no value that is at the end of the undecoded chunk in the existing "special empty FIELD" code path that currently only handles such fields when they are followed by a '&' character. Additional tests are provided to throroughly exercise variations of content bodies with such empty fields. A test has also been added to verify that the change works with an empty last chunk, as suggested in the original PR #13904 Result: Keys with no value that appear at the end of a x-www-form-urlencoded sequence will be parsed according to the spec.

Motivation: The implementation of `PoolArena#numPinnedBytes()` uses a lock to compute the result of the method. This only purpose of this lock is to provide an accurate computation of metric of the list of chunks: the `chunkListMetrics` is final and an unmodifiable list. Note that the returned value is already an estimate since the computation also sums `activeBytesHuge` outside of the lock. In addition this method is used by the allocator and is already an estimate as it sums the list of arenas which can change during the iteration. Modifications: This removes the lock around the `chunkListMetrics` iteration in the `PoolArena#numPinnedBytes()` method. Result: Less locking

…13897) A lenient approach regarding end of lines can result in a parser differential. Example of such attacks: https://sec-consult.com/blog/detail/smtp-smuggling-spoofing-e-mails-worldwide/ As recommended in a private advisory we should document this and a possible mitigation in LineBasedFrameDecoder Motivation: Avoid new implementers of line based protocol to write vulnerable applications. Modification: Simple JavaDoc modification --------- Co-authored-by: Norman Maurer <[email protected]>

Motivation: Many browsers support cookie values separated by ';' instead of '; ', even if they violate the HTTP spec. HttpConversionUtil.toHttp2Headers would, however, silently drop a character of a second cookie in this case ('one=foo;two=bar' -> 'one=foo' and 'wo=bar'). Modification: Add a verification pass that checks that the cookie value can be split. If there is a semicolon without a space, or a character that `AsciiString` cannot handle, keep the cookie value as-is without splitting. Result: An invalid cookie is transmitted as-is instead.

#13876) …ine.getHandshakeStatus() returns the correct value all the time Motivation: We should call SSL_do_handshake(...) after we did run the delegating task so SSLEngine.getHandshakeStatus() will be able to see data that we produce as part of finish the handshake and return NEEDS_WRAP. Modifications: - Add SSL_do_handshake(...) call Result: Correctly reflect HandshakeStatus after running delegating tasks

Motivation: Zstandard(https://facebook.github.io/zstd/) is a high performance, high compression ratio compression algorithm,This pr is to add netty support for the zstandard algorithm,The implementation of zstandard algorithm relies on zstd-jni (https://github.com/luben/zstd-jni), which is an openSource third-party library,Apache Kafka is also using this library for message compression processing. This is the copy of #10422 Modification: Add ZstdDecoder and test case. Result: netty supports ZSTD with zstdDecoder --------- Signed-off-by: xingrufei <[email protected]> Co-authored-by: xingrufei <[email protected]> Co-authored-by: Norman Maurer <[email protected]> Co-authored-by: Chris Vest <[email protected]>

Motivation: Now that `ZstdDecoder` has been added, we should consider supporting ztsd content decompression Modification: Add support for zstd http and http2 content decompression Result: Netty supports zstd http and http2 content decompression Signed-off-by: xingrufei <[email protected]>

This reverts commit 57c579f.

Motivation: Java 22 has been released. Modification: Add a JDK 22 profile to the `pom.xml` file, and add JDK 22 to the PR build matrix. Result: We now build PRs on JDK 22, in addition to the existing Java versions.

* The InterfaceHttpPostRequestDecoder form implementations does not provide hard limits for the number of fields a form can have and the number of accumulated bytes. The former can be used by sending a large amount of fields that will fill the bodyListHttpData list, the later can be used by sending a very large field filling the undecodedChunk buffer since the decoder implementation buffers the field before handling it. This provides hard limits for both: maxFields defines the maximum number of fields the form can have, maxBufferedBytes defines the maximum number of bytes a field can cumulate. When a limit is reached, a decoder exception is thrown, letting the decoder controller take care of it. * Set default limits for maxFields/maxUnbufferedBytes (breaking change) * Update codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoder.java * Update codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoder.java --------- Co-authored-by: Julien Viet <[email protected]>

Motivation: Fix the javadoc rendering Modification: add `{@code }` to the code block. Result: <img width="698" alt="image" src="https://github.com/user-attachments/assets/46d6d1f7-fbf1-4ea9-a48c-515ef96cf8b8">

Motivation: 914ba08 made the methods public but did not change the naming of the methods. Usually we don't use the get* prefix in netty. Modifications: Remove get prefix from methods Result: Cleanup

…4472) Motivation: Corretto does not use "likes" and so we need to ensure we compute it without these. Modifications: Manually compose the classifier to use Result: Always use the correct classifier no matter on which linux system we compile

Motivation: PcapWriteHandler.Builder has a boolean for writePcapGlobalHeader so that the handler can support functionality like appending to an existing PCAP file. This boolean is currently not being checked when writing the global header, preventing appending from working. Modifications: PcapWriter checks PcapWriteHandler.writePcapGlobalHeader() returned true. Added unit test cases for when writePcapGlobalHeader is set to false. Result: PcapWriteHandler only writes the global PCAP header on initialization if writePcapGlobalHeader is true and if sharedOutputStream is false.

…ThreadLocalThread.willCleanupFastThreadLocals()` returns false (#14486) Motivation: The method `AdaptiveByteBufAllocator.usedHeapMemory()` does not return a correct value when `FastThreadLocalThread.willCleanupFastThreadLocals(Thread.currentThread()) == false` by using a customized `FastThreadLocalThread` thread. Modification: If `FastThreadLocalThread.willCleanupFastThreadLocals(Thread.currentThread()) == false`, then we will NOT use threadLocal magazine. Result: Fixes #14483. --------- Co-authored-by: laosijikaichele <laosijikaichele>

…oQueue(...) (#14495) Motivation: There is a possible race condition between methods `AdaptivePoolingAllocator.offerToQueue(...)` and `AdaptivePoolingAllocator.free()`: Assume 'thread-1' goes into method `offerToQueue(...)`, firstly check the `freed` flag is `false`, then 'thread-1' sleep 10 seconds. Then 'thread-2' call `AdaptivePoolingAllocator.free()` and finished the call. Then 'thread-1' wake up and continue to execute `offerToQueue(...)`, it successfully offered the chunk buffer to the `centralQueue`, and will not be freed by `finalized()`, which caused a leak. Modification: Check the `freed` flag again after offered the chunk buffer to the `centralQueue`, help to free the `centralQueue` if `freed == true`. Result: Avoid the possible race condition which may cause a memory leak. Co-authored-by: laosijikaichele <laosijikaichele>

…nd MAGAZINE_BUFFER_QUEUE_CAPACITY (#14493) Motivation: The customizable configurations `AdaptivePoolingAllocator.CENTRAL_QUEUE_CAPACITY` and `AdaptivePoolingAllocator.MAGAZINE_BUFFER_QUEUE_CAPACITY` MUST NOT less than 2. Modification: Add range check in the static block, to fail fast in class load stage with clearer error messages. Result: Fixes #14489. Co-authored-by: laosijikaichele <laosijikaichele>

Motivation: AsciiString objects can be windows onto byte array slices, and need not start at the beginning of the array. The header validation code assumed that AsciiString objects always started at index zero in the underlying byte array. Modification: Fix the end-index computation for the token validation loop, and add tests. Result: Header value token validation will no longer skip the end of values in carefully crafted AsciiString objects. Fixes #14482 --------- Co-authored-by: Norman Maurer <[email protected]>

) Motivation: The sentinel object `Magazine.MAGAZINE_FREED` should not be replaced. Modification: Check the `NEXT_IN_LINE` to make sure the `Magazine.MAGAZINE_FREED` not be replaced. Result: Fixes #14498. Co-authored-by: laosijikaichele <laosijikaichele>

#14494) Motivation: It's better to narrow the lock scope and avoid using nested lock when possible. The lock scope of `magazineExpandLock` in method `AdaptivePoolingAllocator.tryExpandMagazines(...)` can be narrowed. The `magazineExpandLock` guard the current(newest) version of `magazines`, but no need to guard the old version of `magazines`. We can get out of the `magazineExpandLock` immediately once the newest `magazines` is assigned, which means we can move the 'free-old-magazines' operation out of the `magazineExpandLock` scope. Another reason for doing this is that the 'free-old-magazines' operation required another lock(`Magazine.allocationLock`), which make it becomes a nested lock inside `magazineExpandLock`, it's better to avoid using the nested lock. Modification: Moved the 'free-old-magazines' operation out of the `magazineExpandLock` scope. Result: Narrowed the `magazineExpandLock` scope and avoided nest the `Magazine.allocationLock`. Co-authored-by: laosijikaichele <laosijikaichele>

…#14507 (#14509) Motivations: * it's not the responsibility of MqttPublishMessage, which is a ByteBufHolder to ensure the accessibility of the ByteBuf. The ByteBuf checks it himself. * the current design is broken and there's no way to check if the content is accessible as refCnt crashes with IllegalReferenceCountException instead of returning 0. Modification: Remove ByteBufUtil.ensureAccessible from MqttPublishMessage#content. ByteBuf's methods will check themselves. Result: Fixes #14507 It's now possible to check if the payload of a MqttPublishMessage is accessible.

…ed to central queue (#14508) Motivation: For `AdaptivePoolingAllocator`, the `magazine.usedMemory` should be decreased when the chunk be deallocated or be offered to the `centralQueue`. Modification: Decreased the `magazine.usedMemory` when the chunk be deallocated or be offered to the `centralQueue`. Result: The `magazine.usedMemory` becomes more accurate. --------- Co-authored-by: laosijikaichele <laosijikaichele> Co-authored-by: Norman Maurer <[email protected]>

…ompressor` (#14466) Motivation: When HttpContentCompressor is created using the default empty constructor or `HttpContentCompressor(int...)`, then some compression algorithms, like Brotli or Zstd, will never work regardless of their presence in the classpath because `factories` will never be initialized. Modification: Replaced default empty constructor parameters with `StandardCompressionOptions` and `HttpContentCompressor(int, int, int)` constructor. Result: Enabled the usage of Brotli and Zstd even when using the default empty constructor. --------- Co-authored-by: Norman Maurer <[email protected]>

Motivation: We should only use Zstd and Brotli by default if we can load the native lib. Modifications: Correctly detect if we can use the native libs or not Result: No issues when native libs cant be loaded.

#14524) …… (#14516) …l cases Motivation: The allocator uses different strategies when it comes to reusing previous allocated Chunks. In some cases we did not always correctly increment / degrement the zsed memory counters. Modifications; - Correctly update counters in all cases - Null out attached Magazine before adding Chunk to central queue Result: Fixes #14513

Motivation: A BoundedInputStream was recently introduced to protect against attacks where a large file was placed on the server. Unfortunately, this has an issue preventing it from reading the last bytes of a file where the buffer size would take the total bytes over the set bound, when reading just the remaining bytes in the file fits underneath the set bound. Modification: Adjusted the behaviour of BoundedInputStream to, where appropriate, read 1 more byte than the set bound, to see if the file terminates at this point, in which case there is no need to throw the exception Result: Fixes #14479.

Motivation: BlockHound version 1.0.10.RELEASE comes with newer byte-buddy dependency Modification: - Bump BlockHound version - Enable BlockHound tests on Java 18 and above as byte-buddy dependency is updated Result: Enable BlockHound tests on Java 18 and above

Motiviation: We should always add some details on why the exception was thrown to make things easier to debug. Modifications: Add details where we missed to do so Result: Fixes #14559

Motivation: To make our codec more flexible we should add support for unknown frames in SPDY. Modifications: - Add SpdyFrameDecoderExtendedDelegate that supports unknown frames - Add new constructor and protected method to SpdyFrameCodec that makes it easy to support it Result: More flexible SPDY implementation. --------- Co-authored-by: 虎鸣 <[email protected]>

Motivation: Support customized of the **unknwon** frame Modification: Change the returning type of `newSpdyUnknownFrame` from `SpdyUnknownFrame` to `SpdyFrame` Result: Users can override to fire a `CustomFrame` instead of the `SpdyUnknownFrame`

Motivation: Make the code style the same with `try finally release` when encoding SpdyDataFrame Modifications: Use same coding style Result: Same code style.

Motivation: `AdaptivePoolingAllocator` uses internally `CopyOnWriteArraySet`. This will cause the errors below when `BlockHound` is enabled. ``` reactor.blockhound.BlockingOperationError: Blocking call! sun.misc.Unsafe#park at sun.misc.Unsafe.park(Unsafe.java) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:624) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:615) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:261) at io.netty.buffer.AdaptivePoolingAllocator$1.initialValue(AdaptivePoolingAllocator.java:164) at io.netty.util.concurrent.FastThreadLocal.initialize(FastThreadLocal.java:177) at io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:142) at io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:223) at io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:215) at io.netty.buffer.AdaptiveByteBufAllocator.newDirectBuffer(AdaptiveByteBufAllocator.java:78) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) ``` ``` reactor.blockhound.BlockingOperationError: Blocking call! sun.misc.Unsafe#park at sun.misc.Unsafe.park(Unsafe.java) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at java.util.concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:537) at java.util.concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:528) at java.util.concurrent.CopyOnWriteArraySet.remove(CopyOnWriteArraySet.java:245) at io.netty.buffer.AdaptivePoolingAllocator$1.onRemoval(AdaptivePoolingAllocator.java:163) at io.netty.util.concurrent.FastThreadLocal.remove(FastThreadLocal.java:259) at io.netty.util.concurrent.FastThreadLocal.removeAll(FastThreadLocal.java:67) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1050) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) ``` Modification: Allow blocking calls in some methods in `AdaptivePoolingAllocator` Result: No `BlockHound` exceptions Fixes #14560

…14578) Motivation: We need to restore the allocatedBytes value to the previous value on failure to ensure we not mess up state. Modifications: Restore value on failure Result: Correctly recover from failure

Motivation: We can only add a Chunk to the central Queue if it is completely unused as otherwise it still is exclusively used by a Magazine Modifications: Ensure we only use the queue once the Chunk is not used anymore Result: Correctly reuse Chunks

Motivation: We require that the minimum queue size is 2 and use the number of cores as default value which can be less. Modifications: If we see less then 2 cores just use 2 as default size Result: Fixes #14579

Motivation: We can use readRetainedSlice to reduce the copy of bytes in `SpdyFrameDecoder`. Modification: Make use of `readRetainedSlice` instead of new a byteBuf and copy data to it. Result: Less copy

#14586) … are Magazine local cached Chunks (#14581) Motivation: Due a bug we did not re-use central queued Chunks if a Magazine already has a Chunk local cached that is too small to full-fill the allocation. This could lead to completely filling the queue. Modifications: Correctly try to reuse central queued chunks in all cases Result: Fixes #14553

Motivation: PcapWriteHandler should be able to support large files but is currently limited to 2GB due to the usage of int without wrapping. Modifications: segmentNumber and ackNumber are now long values to support uint32 values. segmentNumber and ackNumber wrap around when the uint32 max is reached. Ported unit test from apple/swift-nio-extras#85 Result: Fixes #11543

Motivation: 46b11cc did use Math.min(...) to guard while it has to be Math.max(...) Modifications: Use correct Math method Result: Fix problem when running on systems with 1 core. Fixes #14579

normanmaurer force-pushed the 4.1 branch from efa8f98 to 7903a44 Compare August 16, 2021 08:22

normanmaurer force-pushed the 4.1 branch from 28b869e to a0c9b2f Compare September 9, 2021 13:09

normanmaurer force-pushed the 4.1 branch 2 times, most recently from f9023b9 to 54a2c6d Compare February 8, 2022 09:13

chrisvest force-pushed the 4.1 branch from c2b2724 to 1ecc053 Compare April 12, 2022 18:03

normanmaurer force-pushed the 4.1 branch from f005043 to 9993e07 Compare February 13, 2023 13:39

normanmaurer force-pushed the 4.1 branch from 21df4a1 to 57d536c Compare May 24, 2023 13:05

slandelle force-pushed the 4.1 branch from 702cc4a to ea801cf Compare September 6, 2023 14:02

normanmaurer force-pushed the 4.1 branch 3 times, most recently from f3b47e5 to 58f75f6 Compare October 10, 2023 13:30

normanmaurer force-pushed the 4.1 branch from 16ea959 to 684dfd8 Compare January 16, 2024 09:01

snazy and others added 18 commits March 8, 2024 20:58

Revert "Fix x-www-form-urlencoded parsing for no-value key" (#13907)

3cd3641

Reverts #13904

Revert "Add support for zstd http content decompression (#13909)"

d67dcb2

This reverts commit 57c579f.

Add Java 22 build support (#13914)

d4a640d

Motivation: Java 22 has been released. Modification: Add a JDK 22 profile to the `pom.xml` file, and add JDK 22 to the PR build matrix. Result: We now build PRs on JDK 22, in addition to the existing Java versions.

Fix checkstyle error

bd26566

[maven-release-plugin] prepare release netty-4.1.108.Final

ff9f7d2

[maven-release-plugin] prepare for next development iteration

c5902c9

[maven-release-plugin] rollback the release of netty-4.1.108.Final

cd3c1b0

He-Pin and others added 30 commits November 18, 2024 09:06

chore: Fix javadoc in Http2FrameCodec. (#14468)

9f9581c

Motivation: Fix the javadoc rendering Modification: add `{@code }` to the code block. Result: <img width="698" alt="image" src="https://github.com/user-attachments/assets/46d6d1f7-fbf1-4ea9-a48c-515ef96cf8b8">

Rename methods to match naming conventations (#14470)

8c04c78

Motivation: 914ba08 made the methods public but did not change the naming of the methods. Usually we don't use the get* prefix in netty. Modifications: Remove get prefix from methods Result: Cleanup

Only try to use Zstd and Brotli if we can load the native libs. (#14519)

225adac

Motivation: We should only use Zstd and Brotli by default if we can load the native lib. Modifications: Correctly detect if we can use the native libs or not Result: No issues when native libs cant be loaded.

Add details to TooLongFrameException message (#14562)

9b13234

Motiviation: We should always add some details on why the exception was thrown to make things easier to debug. Modifications: Add details where we missed to do so Result: Fixes #14559

chore: use try finally to encode SpdyDataFrame (#14572)

67c50a1

Motivation: Make the code style the same with `try finally release` when encoding SpdyDataFrame Modifications: Use same coding style Result: Same code style.

Adaptive: Correctly restore allocatedBytes value on failure (#14577) (#…

6c3041f

…14578) Motivation: We need to restore the allocatedBytes value to the previous value on failure to ensure we not mess up state. Modifications: Restore value on failure Result: Correctly recover from failure

Adapt: Don't fail when we run on a host with 1 core (#14582) (#14584)

46b11cc

Motivation: We require that the minimum queue size is 2 and use the number of cores as default value which can be less. Modifications: If we see less then 2 cores just use 2 as default size Result: Fixes #14579

chore: use readRetainedSlice to avoid copy in SpdyFrameDecoder (#14573)

fdc10c4

Motivation: We can use readRetainedSlice to reduce the copy of bytes in `SpdyFrameDecoder`. Modification: Make use of `readRetainedSlice` instead of new a byteBuf and copy data to it. Result: Less copy

Correctly gard aginst failure when running on systems with 1 core

ad104c6

Motivation: 46b11cc did use Math.min(...) to guard while it has to be Math.max(...) Modifications: Use correct Math method Result: Fix problem when running on systems with 1 core. Fixes #14579

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update #1

update #1

dawn2zhang commented May 27, 2021

update #1

Are you sure you want to change the base?

update #1

Conversation

dawn2zhang commented May 27, 2021