Super-cluster, geolocation and temporary broken gateway link

[kzozulya1]

Hello,
We have 2 clusters, east_az1 (gw), east_az2, east_az3 and west_az1(gw), west_az2, west_az3
gateways east_az1 and west_az1 are linked together.
Jetstream enabled.

We subscribe to messages in east cluster, and publish messages to west cluster, and when east_az1 <-> west_az1 link got broken (temporary) the messages are lost.

How should we configure servers in order to found some kind of messages queue, to prevent messages lost ?
Can we do this by means of NATs or should we implement it in pub client only?

Thanks!

[ripienaar]

If you're publishing to streams they would persist, however note a stream lives in one cluster only and can be mirrored to the other.

For JetStream you need 3 clusters to keep things up during an outage.

Best you give us some more information and check out these videos https://www.synadia.com/screencast

[kzozulya1]

Thanks for quick answer.
I've watched https://www.youtube.com/watch?v=6O_sNSJ2p70 (How to build a NATS Supercluster | Rethink Connectivity Episode 5)
So my configuration is 3 clusters (3 nodes in each, each node stands for gw).
I run my configuration in docker.
east cluster has 7222, 7223 and 7224 gateway ports,
west cluster has 7225, 7226 and 7227 gateway ports,
west cluster has 7228, 7229 and 7230 gateway ports

I subscribe into east cluster, publish into west cluster. While link is ok - all works.
Then I simulate broken gw link by means of iptables:

sudo iptables -A INPUT -p tcp --dport 7222 -j DROP
sudo iptables -A INPUT -p tcp --dport 7223 -j DROP
sudo iptables -A INPUT -p tcp --dport 7224 -j DROP

Now subscriber is not available.

Then I publish messages to west cluster again.
And got message nats: no response from stream (I use golang written pub/sub client).

After that I restore broken connection:

sudo iptables -D INPUT -p tcp --dport 7222 -j DROP
sudo iptables -D INPUT -p tcp --dport 7223 -j DROP
sudo iptables -D INPUT -p tcp --dport 7224 -j DROP

But subscriber has got no messages, but I expect it would.

Below attached all configaration files:

docker-compose.yml

services:
  
  east_az1:
    image: nats
    command: -c /etc/my-server.conf --name east_az1
    volumes:
      - ./nats_conf/east_az1.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"
  east_az2:
    image: nats
    command: -c /etc/my-server.conf --name east_az2
    volumes:
      - ./nats_conf/east_az2.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"
  east_az3:
    image: nats
    command: -c /etc/my-server.conf --name east_az3
    volumes:
      - ./nats_conf/east_az3.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"


  west_az1:
    image: nats
    command: -c /etc/my-server.conf --name west_az1
    volumes:
      - ./nats_conf/west_az1.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"
  west_az2:
    image: nats
    command: -c /etc/my-server.conf --name west_az2
    volumes:
      - ./nats_conf/west_az2.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"
  west_az3:
    image: nats
    command: -c /etc/my-server.conf --name west_az3
    volumes:
      - ./nats_conf/west_az3.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"

  north_az1:
    image: nats
    command: -c /etc/my-server.conf --name north_az1
    volumes:
      - ./nats_conf/north_az1.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"
  north_az2:
    image: nats
    command: -c /etc/my-server.conf --name north_az2
    volumes:
      - ./nats_conf/north_az2.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"

  north_az3:
    image: nats
    command: -c /etc/my-server.conf --name north_az3
    volumes:
      - ./nats_conf/north_az3.conf:/etc/my-server.conf
      - ./nats_conf/sys.conf:/etc/sys.conf
    network_mode: "host"    

golang client config.json

{
   "nats_sub": {
        "endpoints": [
            "127.0.0.1:5222",
            "127.0.0.1:5223",
            "127.0.0.1:5224"
        ]
    },

   "nats_pub": {
        "endpoints": [
            "127.0.0.1:5225",
            "127.0.0.1:5226",
            "127.0.0.1:5227"
        ]
    }

nats_conf/east_az1.conf

server_name: east-az1
server_tags: [az:1]
port: 5222
http_port: 8222

include sys.conf

cluster: {
  name: east
  port: 6222
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}

gateway: {
    name: east,
    port: 7222,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/east_az2.conf

server_name: east-az2
server_tags: [az:2]
port: 5223
http_port: 8223

include sys.conf

cluster: {
  name: east
  port: 6223
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}

gateway: {
    name: east,
    port: 7223,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/east_az3.conf

server_name: east-az3
server_tags: [az:3]
port: 5224
http_port: 8224

include sys.conf

cluster: {
  name: east
  port: 6224
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}

gateway: {
    name: east,
    port: 7224,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/west_az1.conf

server_name: west-az1
server_tags: [az:1]
port: 5225
http_port: 8225

include sys.conf

cluster: {
  name: west
  port: 6225
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}

gateway: {
    name: west,
    port: 7225,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/west_az2.conf

server_name: west-az2
server_tags: [az:2]
port: 5226
http_port: 8226

include sys.conf

cluster: {
  name: west
  port: 6226
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}

gateway: {
    name: west,
    port: 7226,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/west_az31.conf

server_name: west-az3
server_tags: [az:3]
port: 5227
http_port: 8227

include sys.conf

cluster: {
  name: west
  port: 6227
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}

gateway: {
    name: west,
    port: 7227,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/north_az1.conf

server_name: north-az1
server_tags: [az:1]
port: 5228
http_port: 8228

include sys.conf

cluster: {
  name: north
  port: 6228
  routes: [
    nats-route://127.0.0.1:6228,
    nats-route://127.0.0.1:6229,
    nats-route://127.0.0.1:6230,
  ]
}

gateway: {
    name: north,
    port: 7228,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/north_az2.conf

server_name: north-az2
server_tags: [az:2]
port: 5229
http_port: 8229

include sys.conf

cluster: {
  name: north
  port: 6229
  routes: [
    nats-route://127.0.0.1:6228,
    nats-route://127.0.0.1:6229,
    nats-route://127.0.0.1:6230,
  ]
}


gateway: {
    name: north,
    port: 7229,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/north_az3.conf

server_name: north-az3
server_tags: [az:3]
port: 5230
http_port: 8230

include sys.conf

cluster: {
  name: north
  port: 6230
  routes: [
    nats-route://127.0.0.1:6228,
    nats-route://127.0.0.1:6229,
    nats-route://127.0.0.1:6230,
  ]
}


gateway: {
    name: north,
    port: 7230,
     gateways: [
        {name: east, urls: [ nats://localhost:7222, nats://localhost:7223,  nats://localhost:7224 ]},
        {name: west, urls: [ nats://localhost:7225, nats://localhost:7226,  nats://localhost:7227 ]},
        {name: north, urls: [ nats://localhost:7228, nats://localhost:7229, nats://localhost:7230 ]}
     ]
}

nats_conf/sys.conf

accounts: {
    $SYS: {
      users: [
         { user:admin, password: password }
      ]
    }
}

jetstream {}

Thanks in advance

[jnmoyne]

The behavior you describe is what I would expect if the stream you are using is located in the east cluster. Streams are located in one and only one cluster (you can mirror/source them to other streams located in other clusters if you want though).