[Bug] System Topics throws IncompatibleSchemaException

[danoshi]

Search before asking

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

Pulsar version 3.0.8.1 LTS
Pulsar client version 3.0.8
Java version openjdk 17.0.13

Minimal reproduce step

Create a Namespace with the property schemaValidationEnforced: true.

What did you expect to see?

SystemTopics are skipped when doing the schemaValidation.

What did you see instead?

IncompatibleSchemaException.

Anything else?

Hi,
we observe a strange behaviour in our pulsar cluster which I would like to clarify if this is intended.
We ran in a HA setup which means when I create a topology this will be replicated in the second cluster as well.
So one of our customers created a namespace with the property schemaValidationEnforced: true but this seems to cause some issues for the system topics.

[pulsar-io-6-4] WARN org.apache.pulsar.broker.service.AbstractReplicator - [persistent://tenant/namespace/__change_events | pulsar-prod-ha1-->pulsar-prod-ha2] Failed to create remote producer (org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: {"errorMsg":"org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Producers cannot connect or send message without a schema to topics with a schema caused by org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Producers cannot connect or send message without a schema to topics with a schema","reqId":3006917652050389956, "remote":"our.pulsar.cluster/:6651", "local":"/:34314"}), retrying in 57.934 s

2025-01-29T14:41:26,689+0000 [pulsar-io-6-4] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://tenant/namespace/__change_events] [pulsar.repl.pulsar-prod-ha1-->pulsar-prod-ha2] Failed to create producer: {"errorMsg":"org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Producers cannot connect or send message without a schema to topics with a schema caused by org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Producers cannot connect or send message without a schema to topics with a schema","reqId":3006917652050389956, "remote":"our.pulsar.cluster/:6651", "local":"/"}

025-01-29T14:41:26,689+0000 [pulsar-io-6-4] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x755f1150, L:/100.70.32.147:34314 - R:rour.pulsar.cluster/:6651] Received error from server: org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Producers cannot connect or send message without a schema to topics with a schema caused by org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Producers cannot connect or send message without a schema to topics with a schema

I turned off schemaValidationEnforced off on namespace level and i can confirm that the error is gone and I do not see it for any system topic for that specific namespace.

Can you confirm to me that this is intended? I guess the schemaValidationEnforced: true is inherited to the topic but I would be curious how this is solved when it comes to the system topics?

We ran currently the pulsar LTS version 3.0.8.1.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

We ran currently the pulsar LTS version 3.0.8.1.

@danoshi Is this a custom build of Pulsar? Just wondering since in Apache Pulsar we don't have a version tagged 3.0.8.1 .

[danoshi]

@lhotari sorry I mean 3.0.8, the 1 afterwards was added from our pipeline when tagging it and pushing to our artifactory.

We ran the 3.0.8 LTS version of pulsar.
https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.8/