Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration tests fail using Scylla 5.2.x #126

Closed
vponomaryov opened this issue Jun 8, 2023 · 17 comments
Closed

Integration tests fail using Scylla 5.2.x #126

vponomaryov opened this issue Jun 8, 2023 · 17 comments
Assignees
@avelanarius

Comments

@vponomaryov
Copy link

vponomaryov commented Jun 8, 2023 •
edited
Loading

Please answer these questions before submitting your issue. Thanks!

What version of Scylla or Cassandra are you using?

5.2.1 and 5.2.0

What version of Gocql are you using?

v1.7.3

What version of Go are you using?

1.20.4

What did you do?

I explicitly defined the Scylla version:

SCYLLA_IMAGE=scylladb/scylla:5.2.1

Then ran tests:

./integration.sh integration scylla

And got following result:

==> Running integration scylla tests with args: -gocql.timeout=60s -proto=4 -rf=1 -clusterSize=1 -autowait=2000ms -compressor=snappy -gocql.cversion=3.11.4 -cluster=192.168.100.11
2023/06/08 12:42:51 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"38da3ff8-79e5-4515-96d6-9edbe943df37\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: EOF
2023/06/08 12:42:51 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"38da3ff8-79e5-4515-96d6-9edbe943df37\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: Keyspace 'invalidKeyspace' does not exist
2023/06/08 12:42:56 logger.go:35: gocql: unable to dial control conn 127.0.0.1:1: dial tcp :0->127.0.0.1:1: connect: connection refused
panic: test timed out after 5m0s
running tests:
        TestBatchObserve (4m38s)

goroutine 54910 [running]:
testing.(*M).startAlarm.func1()
        /usr/local/go/src/testing/testing.go:2241 +0x219
created by time.goFunc
        /usr/local/go/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc000103380, {0xbb06fc, 0x10}, 0xbea210)
        /usr/local/go/src/testing/testing.go:1630 +0x82e
testing.runTests.func1(0x0?)
        /usr/local/go/src/testing/testing.go:2036 +0x8e
testing.tRunner(0xc000103380, 0xc00016fb48)
        /usr/local/go/src/testing/testing.go:1576 +0x217
testing.runTests(0xc000140dc0?, {0xf95a40, 0x94, 0x94}, {0x1c?, 0x4a8b19?, 0xf99780?})
        /usr/local/go/src/testing/testing.go:2034 +0x87d
testing.(*M).Run(0xc000140dc0)
        /usr/local/go/src/testing/testing.go:1906 +0xb45
main.main()
        _testmain.go:345 +0x2ea

goroutine 125 [select, 1 minutes]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc000472000, 0xdf8475800?)
        /home/kiparis/projects/gocql/session.go:374 +0x177
created by github.com/gocql/gocql.(*Session).init
        /home/kiparis/projects/gocql/session.go:321 +0x108c

goroutine 1360 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x47943a?, {0xc9b758, 0xc0003614f0})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 1381 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000591c80, 0xbea210?)
        /home/kiparis/projects/gocql/conn.go:880 +0x245
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 1374 [select]:
github.com/gocql/gocql.(*Conn).prepareStatement(0xc0003a8900, {0xc9b790, 0xc000126018}, {0xc0000a9040, 0x38}, {0x0?, 0x0})
        /home/kiparis/projects/gocql/conn.go:1091 +0x44c
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1349 +0x7bf
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:1629 +0x806

goroutine 194 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc00033e300?, {0xc9b758, 0xc0001b4c30})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 75 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000386780, 0x30d40?)
        /home/kiparis/projects/gocql/conn.go:880 +0x245
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 181 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc0000e6b40)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 1345 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0xc000498940)
        /home/kiparis/projects/gocql/control.go:76 +0x1a9
created by github.com/gocql/gocql.(*controlConn).connect
        /home/kiparis/projects/gocql/control.go:265 +0x2cf

goroutine 161 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867b070, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0001e6a98, 0xc000346000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001e6a80, {0xc000346000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0001e6a80, {0xc000346000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000192060, {0xc000346000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0001b0e40, {0xc00033e348, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0001b0e40}, {0xc00033e348, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0001b0e40?}, {0xc00033e348?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc00033e300, {0xc9b758, 0xc0001b4c30})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0xc0000e6ba0?, {0xc9b758, 0xc0001b4c30})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 1375 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc0005918c0)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 182 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc0000e6c00)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 128 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0001b1320, 0x30d40?)
        /home/kiparis/projects/gocql/conn.go:880 +0x245
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 1351 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867b160, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00045c598, 0xc00030c000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00045c580, {0xc00030c000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc00045c580, {0xc00030c000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc0000140b8, {0xc00030c000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0005919e0, {0xc0000c0348, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0005919e0}, {0xc0000c0348, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0005919e0?}, {0xc0000c0348?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0000c0300, {0xc9b758, 0xc00040af50})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x47943a?, {0xc9b758, 0xc00040af50})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 77 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc0003a8180?, {0xc9b758, 0xc000388500})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 76 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867af80, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0003a0198, 0xc0003d2000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0003a0180, {0xc0003d2000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0003a0180, {0xc0003d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000380040, {0xc0003d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0003866c0, {0xc0003a81c8, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0003866c0}, {0xc0003a81c8, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0003866c0?}, {0xc0003a81c8?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0003a8180, {0xc9b758, 0xc000388500})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0xc00038e4e0?, {0xc9b758, 0xc000388500})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 70 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0xc0001e1100)
        /home/kiparis/projects/gocql/control.go:76 +0x1a9
created by github.com/gocql/gocql.(*controlConn).connect
        /home/kiparis/projects/gocql/control.go:265 +0x2cf

goroutine 1352 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x47943a?, {0xc9b758, 0xc00040af50})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 129 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867ae90, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0003a0518, 0xc0003e4000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0003a0500, {0xc0003e4000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0003a0500, {0xc0003e4000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000380048, {0xc0003e4000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0003867e0, {0xc0003a8348, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0003867e0}, {0xc0003a8348, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0003867e0?}, {0xc0003a8348?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0003a8300, {0xc9b758, 0xc000388b90})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x4aa657?, {0xc9b758, 0xc000388b90})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 210 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc0003a8300?, {0xc9b758, 0xc000388b90})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 1376 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc000591980)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 1383 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x47943a?, {0xc9b758, 0xc0004de7d0})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 1359 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867b250, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0002d0a18, 0xc0002d2000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002d0a00, {0xc0002d2000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0002d0a00, {0xc0002d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000192068, {0xc0002d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc000317b60, {0xc0003a8948, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc000317b60}, {0xc0003a8948, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc000317b60?}, {0xc0003a8948?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0003a8900, {0xc9b758, 0xc0003614f0})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x47943a?, {0xc9b758, 0xc0003614f0})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 1357 [select]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc000600800, 0x0?)
        /home/kiparis/projects/gocql/session.go:374 +0x177
created by github.com/gocql/gocql.(*Session).init
        /home/kiparis/projects/gocql/session.go:321 +0x108c

goroutine 1358 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0004f8b40, 0x0?)
        /home/kiparis/projects/gocql/conn.go:888 +0x310
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 1382 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867ada0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0001e7698, 0xc0000b8000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001e7680, {0xc0000b8000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0001e7680, {0xc0000b8000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc00037a008, {0xc0000b8000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0004f8840, {0xc0000ba048, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0004f8840}, {0xc0000ba048, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0004f8840?}, {0xc0000ba048?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0000ba000, {0xc9b758, 0xc0004de7d0})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x47943a?, {0xc9b758, 0xc0004de7d0})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 54909 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0001e0a50, 0x1a19c)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0001e0a40)
        /usr/local/go/src/sync/cond.go:70 +0xa5
github.com/gocql/gocql.(*writeCoalescer).Write(0xc0004f8b40, {0xc001c68900, 0x47, 0x80})
        /home/kiparis/projects/gocql/conn.go:857 +0x39f
github.com/gocql/gocql.(*Conn).Write(0xc0003a8900, {0xc001c68900, 0x47, 0x80})
        /home/kiparis/projects/gocql/conn.go:332 +0x6e
github.com/gocql/gocql.(*framer).finishWrite(0xc001c489c0)
        /home/kiparis/projects/gocql/frame.go:779 +0x63c
github.com/gocql/gocql.(*writePrepareFrame).writeFrame(0xc001d8ede0, 0xc001c489c0, 0x6680)
        /home/kiparis/projects/gocql/frame.go:861 +0xe16
github.com/gocql/gocql.(*Conn).exec(0xc0003a8900, {0xc9b758, 0xc0003614f0}, {0xc981e0, 0xc001d8ede0}, {0x0, 0x0?})
        /home/kiparis/projects/gocql/conn.go:934 +0x4d9
github.com/gocql/gocql.(*Conn).prepareStatement.func2()
        /home/kiparis/projects/gocql/conn.go:1048 +0x293
created by github.com/gocql/gocql.(*Conn).prepareStatement
        /home/kiparis/projects/gocql/conn.go:1035 +0x3a5
exit status 2
FAIL    github.com/gocql/gocql  300.076s

What did you expect to see?

I expect it to work ok like with all other more older Scylla versions 5.1.x, 5.0.x and 4.6.x which provide following result:

==> Running integration scylla tests with args: -gocql.timeout=60s -proto=4 -rf=1 -clusterSize=1 -autowait=2000ms -compressor=snappy -gocql.cversion=3.11.4 -cluster=192.168.100.11
2023/06/08 11:46:57 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"09183270-1045-4cb8-adde-5aed8578c76b\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: context canceled
2023/06/08 11:46:57 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"09183270-1045-4cb8-adde-5aed8578c76b\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: Keyspace 'invalidKeyspace' does not exist
2023/06/08 11:47:01 logger.go:35: gocql: unable to dial control conn 127.0.0.1:1: dial tcp :0->127.0.0.1:1: connect: connection refused
2023/06/08 11:47:18 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"09183270-1045-4cb8-adde-5aed8578c76b\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: context canceled
2023/06/08 11:47:37 logger.go:35: scylla: 192.168.100.11:9042 connection to shard-aware address 192.168.100.11:19042 resulted in wrong shard being assigned; please check that you are not behind a NAT or AddressTranslater which changes source ports; falling back to non-shard-aware port for 5m0s
PASS
ok      github.com/gocql/gocql  43.496s

What did you see instead?

I see constant connection 300s timeouts.

If you are having connectivity related issues please share the following additional information

Describe your Cassandra cluster

please provide the following information

  • output of 'nodetool status'
  • output of 'SELECT peer, rpc_address FROM system.peers'
  • rebuild your application with the 'gocql_debug' tag and post the output

nodetool status:

root@d0e0d92882dd:/# nodetool status
nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused)
See 'nodetool help' or 'nodetool help <command>'.

CQLSH:

root@d0e0d92882dd:/# cqlsh
Connected to  at 192.168.100.11:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT peer, rpc_address FROM system.peers;

 peer | rpc_address
------+-------------

(0 rows)

With gocql_debug:
gocql_debug_for_issue_126.log
@vponomaryov vponomaryov changed the title Tests do work using Scylla 5.2.x Integration tests fail using Scylla 5.2.x Jun 8, 2023
@mykaul
Copy link

mykaul commented Jun 8, 2023

127.0.0.1:1 looks wrong to me and can explain the connection refused.

@vponomaryov
Copy link
Author

127.0.0.1:1 looks wrong to me and can explain the connection refused.

Only Scylla docker image version was changed.
So, it can be said now that the gocql integration tests are not compatible with the latest Scylla releases.

@fruch
Copy link

fruch commented Jun 8, 2023

running the same with master, and in 300s it got killed by my kernel:

Jun 08 14:20:13 fruch-laptop kernel: Out of memory: Killed process 3160814 (gocql.test) total-vm:13869872kB, anon-rss:11142664kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22996

@roydahan
Copy link
Collaborator

@avelanarius / @zimnx can someone please take a look at this?

@avelanarius
Copy link

I will look into it.

@dkropachev
Copy link
Collaborator

dkropachev commented Jun 20, 2023 •
edited
Loading

This test failure due to the unabling execute prepared statement due to the getting 0x2500 error from the scylla server:
https://github.com/apache/cassandra/blob/7337fc0/doc/native_protocol_v5.spec#L1414-L1417

Same problem from java-driver scylladb/java-driver#124

@fruch
Copy link

fruch commented Jun 20, 2023

This test failure due to the unabling execute prepared statement due to the getting 0x2500 error from the scylla server: https://github.com/apache/cassandra/blob/7337fc0/doc/native_protocol_v5.spec#L1414-L1417

Same problem from java-driver scylladb/java-driver#124

that issue in java-driver wasn't fixed, and was fixed on scylla end in some way.
so I'm guessing it's not exactly the same.

@dkropachev
Copy link
Collaborator

I have run this test locally on 5.2.3 and 5.2.1 100 times and it did not fail even once.
Also I run whole suite for 10 times and it also did not fail even once.
Which go version are we using there ?

@dkropachev
Copy link
Collaborator

I ran it via integration.sh and it did fail, will look at the difference tomorrow

@dkropachev dkropachev mentioned this issue Jun 21, 2023
Open
1 task
@dkropachev
Copy link
Collaborator

dkropachev commented Jun 21, 2023 •
edited
Loading

Most likely it is driver issue, it's logic does not cover the case when prepared statement cache on scylla is under high pressure which could lead to being unable to run big batches as prepared statements, because scylla is expunging statements while driver preparing them.

If this is true, then all our and upstream drivers have exact same issue, which is not regression and only regression there is that scylla tightened prepared statement buffer.

I have opened issue on the scylla repo to confirm it.
Fast fix would be to increase scylla memory limit in docker-compose.yaml

@avelanarius
Copy link

This test failure due to the unabling execute prepared statement due to the getting 0x2500 error from the scylla server: https://github.com/apache/cassandra/blob/7337fc0/doc/native_protocol_v5.spec#L1414-L1417
Same problem from java-driver scylladb/java-driver#124

that issue in java-driver wasn't fixed, and was fixed on scylla end in some way. so I'm guessing it's not exactly the same.

That particular one was fixed by: scylladb/scylladb#10456 (seehttps://github.com/scylladb/scylladb/issues/10440)

@dkropachev
Copy link
Collaborator

dkropachev commented Jun 21, 2023 •
edited
Loading

From scylladb/scylladb#14338 :

Refs #10440 - a similar issue from the past. That issue was fixed by changing the eviction behavior of loading_cache (note that the fix should be improved: #10674).

One possible improvement to prevent infinite spining:

The way it works now (looking at Rust Driver, but others are similar): let's suppose that there is a batch statement of N prepared statements and the driver sends this statement to Scylla. Scylla might respond that some statement in that batch is not prepared (not present in prepared statements cache). The driver will then re-prepare all statements in that batch. However, if the cache is too small, it won't be possible for all statements to be in the cache at the same time.

Improvement:

The driver(s) should not infinite loop, instead impose some limit of how many times it will try to re-prepare (some draft PR from the past: scylladb/java-driver#143)
When Scylla receives a batch of N prepared statements, it should check if the prepared statements cache is large enough and return some error if it's not.

Which confirms my assumption, it is needed to be fixed on gocql side

dkropachev pushed a commit to dkropachev/gocql that referenced this issue Jun 21, 2023
fix(docker-compose.yaml): increase memory limit to avaoid hitting scy…
f325aa5 
…lladb#126
@dkropachev dkropachev mentioned this issue Jun 21, 2023
Merged
@dkropachev
Copy link
Collaborator

PR to increase memory limit - #128 to avoid hitting the issue

avelanarius added a commit that referenced this issue Jun 27, 2023
@avelanarius
Merge pull request #128 from dkropachev/dk/fix-intergration-sh
8bf4a23 
fix(docker-compose.yaml): increase memory limit to avoid hitting #126
sylwiaszunejko pushed a commit to sylwiaszunejko/gocql that referenced this issue Jul 18, 2023
@sylwiaszunejko
fix(docker-compose.yaml): increase memory limit to avaoid hitting scy…
0ef91ee 
…lladb#126
@roydahan
Copy link
Collaborator

roydahan commented Nov 9, 2023

@fruch / @k0machi (or anyone who work on the go-driver-matrix), is this issue still relevant?

@vponomaryov
Copy link
Author

@fruch / @k0machi (or anyone who work on the go-driver-matrix), is this issue still relevant?

@roy , I reviewed the fix and verified it works. So, this should have been closed with the merge of the fix.

@roydahan
Copy link
Collaborator

roydahan commented Nov 9, 2023

Thanks, closing.

@roydahan roydahan closed this as completed Nov 9, 2023
@fruch
Copy link

fruch commented Nov 9, 2023

@fruch / @k0machi (or anyone who work on the go-driver-matrix), is this issue still relevant?

We didn't have that issue, we are using ccm and not docker with limited memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@avelanarius avelanarius

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@fruch @vponomaryov @mykaul @avelanarius @roydahan @dkropachev