New Cluster Bootstrap Borked

[tpetracca]

Bug intro'ed in #97

If we have a brand new cluster that has never been turned on before, and then you turn multiple nodes on at once, they all remove themselves from the seed list, but none of them can communicate with any other ones so you fail bootstrap.

Example of brand new 2-node cluster where seeds list is both nodes.

INFO  [2019-11-19T21:52:04.042Z] org.apache.cassandra.net.OutboundTcpConnection: Handshaking version with other-node-hostname/other-node-ip (0: other-node-hostname/other-node-ip)
ERROR [2019-11-19T21:52:34.608Z] org.apache.cassandra.service.CassandraDaemon: Exception encountered during startup (throwable0_message: Unable to gossip with any seeds)
java.lang.RuntimeException: Unable to gossip with any seeds
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1360)
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:523)
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:759)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:678)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:564)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:322)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:560)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:670)

WARN  [2019-11-19T21:52:34.609Z] org.apache.cassandra.gms.Gossiper: No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [2019-11-19T21:52:34.609Z] org.apache.cassandra.net.MessagingService: Waiting for messaging service to quiesce
INFO  [2019-11-19T21:52:34.609Z] org.apache.cassandra.net.MessagingService: MessagingService has terminated the accept() thread

[Mattheo28]

What should be happening? 1 node picks only itself as the seed, and the others gossip with that one? If so, can't we just make it a requirement that when turning a cluster on, you turn on one node by itself, and then turn on the rest?