Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][misc] Sync commits from apache into 3.1_ds #307

Merged
merged 11 commits into from
Aug 23, 2024
Merged

[improve][misc] Sync commits from apache into 3.1_ds #307

merged 11 commits into from
Aug 23, 2024

Conversation

nikhil-ctds
Copy link
Collaborator

Motivation

This PR sync all commits from apache/branch-3.0 into 3.1_ds which are not present till 30th June.

Modifications

  • Cherry-picked commits from branch-3.0 which are not present on 3.1_ds till 30th June.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

@nikhil-ctds nikhil-ctds self-assigned this Aug 22, 2024
BewareMyPower and others added 10 commits August 22, 2024 10:46
@BewareMyPower @nikhil-ctds
[improve][broker] Support customized shadow managed ledger implementa…
f8b8aac 
…tion (apache#23179)

(cherry picked from commit 7f04364)
(cherry picked from commit 4196195)
@Demogorgon314 @nikhil-ctds
[improve][broker] Should notify bundle ownership listener onLoad even…
cfa1e0c 
…t when ServiceUnitState start (ExtensibleLoadManagerImpl only) (apache#23152)

(cherry picked from commit 3053b64)
(cherry picked from commit 9a090f7)
@poorbarcode @nikhil-ctds
[improve] [broker] Optimize performance for checking max topics when …
29d155a 
…the topic is a system topic (apache#23185)

(cherry picked from commit 73433cd)
(cherry picked from commit 09bc877)
@dao-jun @nikhil-ctds
[fix][broker] Skip reading entries from closed cursor. (apache#22751)
3c8c8ab 
(cherry picked from commit aa8226f)
(cherry picked from commit 93e09ae)
@poorbarcode @nikhil-ctds
[fix] [broker] Topic can never be loaded up due to broker maintains a…
ecdcb52 
… failed topic creation future (apache#23184)

(cherry picked from commit 9edaa85)
(cherry picked from commit 34c6fa0)
@coderzc @nikhil-ctds
[improve][broker] Optimize high CPU usage when consuming from topics …
b227aca 
…with ongoing txn (apache#23189)

(cherry picked from commit 94e1341)
(cherry picked from commit b7ffa73)
@TakaHiR07 @nikhil-ctds
[improve][broker] Improve pulsar_topic_load_failed metric to record c…
bc3bfea 
…orrect failed time (apache#23199)

Co-authored-by: fanjianye <[email protected]>
(cherry picked from commit b661ec8)
(cherry picked from commit b7f5026)
@BewareMyPower @nikhil-ctds
[improve][client] Don't print info logs for each schema loaded by cli…
12c07c9 
…ent (apache#23206)

(cherry picked from commit 66e1a06)
(cherry picked from commit 06ebe2a)
@liangyepianzhou @nikhil-ctds
[fix][client] the nullValue in msgMetadata should be true by default (a…
f805fb3 
…pache#22372)

Co-authored-by: xiangying <[email protected]>
### Motivation
When a message is not set value, the `nullValue` message metadata should be true and change to false after the value is set. Otherwise, the message data will be set as a [] when the value is not set, that would cause the message data to be encoded and throw a `SchemaSerializationException` when calling `reconsumerLater`.
```

org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.SchemaSerializationException: Size of data received by IntSchema is not 4

	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1131)
	at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLater(ConsumerBase.java:467)
	at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLater(ConsumerBase.java:452)
	at org.apache.pulsar.client.api.ConsumerRedeliveryTest.testRedeliverMessagesWithoutValue(ConsumerRedeliveryTest.java:445)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
Caused by: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.SchemaSerializationException: Size of data received by IntSchema is not 4
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLater(ConsumerBase.java:462)
	... 29 more
Caused by: org.apache.pulsar.client.api.SchemaSerializationException: Size of data received by IntSchema is not 4
	at org.apache.pulsar.client.impl.schema.IntSchema.validate(IntSchema.java:49)
	at org.apache.pulsar.client.impl.schema.AutoProduceBytesSchema.encode(AutoProduceBytesSchema.java:80)
	at org.apache.pulsar.client.impl.schema.AutoProduceBytesSchema.encode(AutoProduceBytesSchema.java:32)
	at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.lambda$value$3(TypedMessageBuilderImpl.java:157)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.value(TypedMessageBuilderImpl.java:156)
	at org.apache.pulsar.client.impl.ConsumerImpl.doReconsumeLater(ConsumerImpl.java:689)
	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.doReconsumeLater(MultiTopicsConsumerImpl.java:550)
	at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLaterAsync(ConsumerBase.java:574)
```
### Modifications
When a message is not set value, the `nullValue` message metadata should be true and change to false after the value is set.

(cherry picked from commit f3c177e)
(cherry picked from commit 0541176)
@poorbarcode @nikhil-ctds
[fix] [log] Do not print warn log when concurrently publishing and sw…
334f190 
…itching ledgers (apache#23209)

(cherry picked from commit 0a5cb51)
(cherry picked from commit 57b0ca4)
@nikhil-ctds nikhil-ctds marked this pull request as ready for review August 23, 2024 03:57
@srinath-ctds srinath-ctds merged commit 7b7d335 into 3.1_ds Aug 23, 2024
46 checks passed
@srinath-ctds srinath-ctds deleted the cherry-picks branch August 23, 2024 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srinath-ctds srinath-ctds srinath-ctds approved these changes

Assignees

@nikhil-ctds nikhil-ctds

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@nikhil-ctds @srinath-ctds @lhotari @BewareMyPower @Demogorgon314 @poorbarcode @dao-jun @coderzc @TakaHiR07 @liangyepianzhou