Could not find request context for stream id -1 received from TARGET-CONNECTOR. It either timed out or a protocol error occurred

[icodeiexist]

Hi, we're DSE users and started demoing ZDM for a migration scenario: DSE 5.1 to DSE 6.8, app testers say that writes are failing and we're detecting too many messages of the type:
time="2023-11-13T22:47:04Z" level=warning msg="Could not find request context for stream id -1 received from TARGET-CONNECTOR. It either timed out or a protocol error occurred."
and occasional ones like these:
time="2023-11-13T22:47:29Z" level=warning msg="No cached entry for prepared-id = 'f3aa403a5bf3d64ef1efada57fb5a8fa' for OpCode EXECUTE [0x0A]."

No apparent errors in the target nodes, any ideas on how to troubleshoot ?
Any special configs zdm or driver side, to consider for prepared statements?

Some config info:
time="2023-11-09T10:04:30Z" level=info msg="Starting ZDM proxy version 2.1.0" time="2023-11-09T10:04:30Z" level=info msg="Parsed configuration: {\"PrimaryCluster\":\"ORIGIN\",\"ReadMode\":\"PRIMARY_ONLY\",\"ReplaceCqlFunctions\":false,\"AsyncHandshakeTimeoutMs\":4000,\"LogLevel\":\"INFO\",\"ProxyTopologyIndex\":0,\"ProxyTopologyAddresses\":\"REDACTED\",\"ProxyTopologyNumTokens\":8,\"OriginContactPoints\":\"REDACTED\",\"OriginPort\":9042,\"OriginSecureConnectBundlePath\":\"\",\"OriginLocalDatacenter\":\"\",\"OriginUsername\":\"REDACTED\",\"OriginConnectionTimeoutMs\":30000,\"OriginTlsServerCaPath\":\"REDACTED\",\"OriginTlsClientCertPath\":\"REDACTED\",\"OriginTlsClientKeyPath\":\"REDACTED\",\"TargetContactPoints\":\"REDACTED\",\"TargetPort\":9042,\"TargetSecureConnectBundlePath\":\"\",\"TargetLocalDatacenter\":\"\",\"TargetUsername\":\"REDACTED\",\"TargetConnectionTimeoutMs\":30000,\"TargetTlsServerCaPath\":\"REDACTED\",\"TargetTlsClientCertPath\":\"REDACTED\",\"TargetTlsClientKeyPath\":\"REDACTED\",\"ProxyListenAddress\":\"REDACTED\",\"ProxyListenPort\":9042,\"ProxyRequestTimeoutMs\":10000,\"ProxyMaxClientConnections\":1000,\"ProxyMaxStreamIds\":2048,\"ProxyTlsCaPath\":\"REDACTED\",\"ProxyTlsCertPath\":\"REDACTED\",\"ProxyTlsKeyPath\":\"REDACTED\",\"ProxyTlsRequireClientAuth\":false,\"MetricsEnabled\":true,\"MetricsAddress\":\"REDACTED\",\"MetricsPort\":14001,\"MetricsPrefix\":\"zdm\",\"MetricsOriginLatencyBucketsMs\":\"1, 4, 7, 10, 25, 40, 60, 80, 100, 150, 250, 500, 1000, 2500, 5000, 10000, 15000\",\"MetricsTargetLatencyBucketsMs\":\"1, 4, 7, 10, 25, 40, 60, 80, 100, 150, 250, 500, 1000, 2500, 5000, 10000, 15000\",\"MetricsAsyncReadLatencyBucketsMs\":\"1, 4, 7, 10, 25, 40, 60, 80, 100, 150, 250, 500, 1000, 2500, 5000, 10000, 15000\",\"HeartbeatIntervalMs\":30000,\"HeartbeatRetryIntervalMinMs\":250,\"HeartbeatRetryIntervalMaxMs\":30000,\"HeartbeatRetryBackoffFactor\":2,\"HeartbeatFailureThreshold\":1,\"SystemQueriesMode\":\"ORIGIN\",\"ForwardClientCredentialsToOrigin\":false,\"OriginEnableHostAssignment\":true,\"TargetEnableHostAssignment\":true,\"RequestWriteQueueSizeFrames\":128,\"RequestWriteBufferSizeBytes\":4096,\"RequestReadBufferSizeBytes\":32768,\"ResponseWriteQueueSizeFrames\":128,\"ResponseWriteBufferSizeBytes\":8192,\"ResponseReadBufferSizeBytes\":32768,\"RequestResponseMaxWorkers\":-1,\"WriteMaxWorkers\":-1,\"ReadMaxWorkers\":-1,\"ListenerMaxWorkers\":-1,\"EventQueueSizeFrames\":12,\"AsyncConnectorWriteQueueSizeFrames\":2048,\"AsyncConnectorWriteBufferSizeBytes\":4096}"

Thank you.

┆Issue is synchronized with this Jira Task by Unito
┆Components: Proxy
┆Issue Number: ZDM-599

[joao-r-reis]

Hi can you provide a larger sample of the log? The full log file would be ideal.

The "stream id -1" warning might be a red herring, it might be caused by #109 if no writes are sent via that particular connection during a time interval specified by the heartbeat interval configuration setting.

No cached entry for prepared-id this warning is a bit weird, does it occur very often? It shouldn't be a big issue though because the driver will re-prepare the statement automatically and shouldn't result in an error.

What errors are you actually getting in the client? Can you share driver logs? Maybe a stacktrace?

Do you have zdm monitoring set up? zdm proxy collects a lot of metrics that are very useful to troubleshoot issues.

[icodeiexist]

Hi can you provide a larger sample of the log? The full log file would be ideal.

Sure, logs are private to you right? My client is a Datastax customer, would it work if a opened a datastax support case and refer it to you?
It's support case 00088486, I've uploaded the logs there.

[icodeiexist]

Hey there, been receiving support from Datastax support. Just wanted to document for posterity that the issue of the app not being able to write via zdm had to do with compression being enabled. We can close this.