docker-compose setup: Velocity proxy refuses to connect to Paper hub world on same machine #1703

klaribot · 2022-08-31T07:01:06Z

klaribot
Aug 31, 2022

Alright, so I have a bit of an advanced problem I've been struggling to solve for the past 5 hours or so.

I'm building a larger-scale Minecraft network using Docker Compose (or preparing to anyway), and I'm trying to find out why the Velocity container is refusing to forward players to the Hub container.

Here's my docker-compose.yml:

version: '3.8'

services:

  #######################################################
  # hub Profile - The main Minecraft hub world. Includes a Velocity proxy configured for modern forwarding, and a void world running on Paper.

  main-proxy:
    hostname: main-proxy
    container_name: main-proxy
    profiles: ["hub"]
    image: itzg/bungeecord:java17
    ports:
      - "25565:25565"
    environment:
      UID: 1002 #Minecraft user
      GID: 1003 #Minecraft group
      TYPE: VELOCITY
      VELOCITY_VERSION: latest
      MEMORY: 512m
      JVM_XX_OPTS: "-XX:+UseG1GC -XX:G1HeapRegionSize=4M -XX:+UnlockExperimentalVMOptions -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:MaxInlineLevel=15"
      PLUGINS: https://github.com/ViaVersion/ViaVersion/releases/download/4.4.2/ViaVersion-4.4.2.jar,https://github.com/ViaVersion/ViaBackwards/releases/download/4.4.2/ViaBackwards-4.4.2.jar
    restart: always
    volumes:
      - ./servers/main-proxy:/server
    links:
      - main-hub

  main-hub:
    hostname: main-hub
    container_name: main-hub
    profiles: ["hub"]
    image: itzg/minecraft-server:java17
    ports:
      - "30065:25565"
    environment:
      UID: 1002 #Minecraft user
      GID: 1003 #Minecraft group
      TYPE: PAPER
      VERSION: latest
      MEMORY: 1024m
      USE_AIKAR_FLAGS: "TRUE"
      EULA: "TRUE"
      SPIGET_RESOURCES: 19254,27448,25391,102902 #ViaVersion, ViaBackwards, VoidGen, NoEncryption
      #EXEC_DIRECTLY: true
      # enable env variable replacement
      REPLACE_ENV_VARIABLES: "true"
      # define an optional prefix for env variables we want to replace
      ENV_VARIABLE_PREFIX: "CFG_"
      # actual variables
      CFG_FORWARDING_SECRET_FILE: "/forwarding.secret"
    tty: true
    stdin_open: true
    restart: always
    volumes:
      - ./servers/main-hub:/data
secrets:
  forwarding_secret:
    file: /forwarding.secret

velocity.toml already knows about the forwarding.secret file (at least it should?):

. . .

# If you are using modern or BungeeGuard IP forwarding, configure a file that contains a unique secret here.
# The file is expected to be UTF-8 encoded and not empty.
forwarding-secret-file = "forwarding.secret"

. . .

and paper-global.yml is configured to have an environment variable replaced at container start:

. . .

proxies:
  bungee-cord:
    online-mode: false
  proxy-protocol: false
  velocity:
    enabled: true
    online-mode: true
    secret: ${CFG_FORWARDING_SECRET_FILE}

. . .

This SHOULD just work, right?

In testing, it does not.

main-proxy  | [06:55:36 INFO]: [connected player] Klari (/{IP REDACTED}) has connected
main-proxy  | [06:55:36 ERROR]: [connected player] Klari (/{IP REDACTED}): unable to connect to server main-hub
main-proxy  | io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: main-hub/172.28.0.2:30065
main-proxy  | Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
main-proxy  |   at io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.unix.Socket.finishConnect(Socket.java:359) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at java.lang.Thread.run(Thread.java:833) [?:?]
main-proxy  | [06:55:36 INFO]: [connected player] Klari (/{IP REDACTED}) has disconnected: Unable to connect you to main-hub. Please try again later.

...although I'm 100% expecting to be pointed towards an obvious and/or incomplete misconfiguration in my config.
Any feedback would be appreciated to the highest degree. 🙏

Answered by itzg

Aug 31, 2022

This might be a question better asked on Velocity forums.

Without knowing Velocity, all I can question is

Why is velocity trying port 30065 of the main-hub host?
Docker mounts secrets under the path /run/secrets but I'm not seeing that path mentioned anywhere

View full answer

itzg · 2022-08-31T12:38:55Z

itzg
Aug 31, 2022
Maintainer

This might be a question better asked on Velocity forums.

Without knowing Velocity, all I can question is

Why is velocity trying port 30065 of the main-hub host?
Docker mounts secrets under the path /run/secrets but I'm not seeing that path mentioned anywhere

8 replies

itzg Sep 1, 2022
Maintainer

I updated the documentation about _FILE variables.

klaribot Sep 1, 2022
Author

That's half your problem. The external port is external. The container port within the docker network is still 25565 for main-hub. In fact you don't (and probably shouldn't) even need to declare a port mapping for that container. Refer to the example.

As for the secret, you cannot pick the path. It will put it under /run/secrets so you need to set that velocity secret with the correct path. See https://docs.docker.com/engine/swarm/secrets/#how-docker-manages-secrets

Really?
I thought that each container had its own internal network and that everything had to be routed externally.
How will I guarantee that every container will be able to connect on its own port, even if a given server container is on a completely different machine?

And thank you for the _FILE clarification! :D

klaribot Sep 1, 2022
Author

New configuration, new problems.

docker-compose.yml

version: '3.8'

services:
  # hub Profile - The main Minecraft hub world. Includes a Velocity proxy configured for modern forwarding, and a void world running on Paper.
  main-proxy:
    hostname: main-proxy
    container_name: main-proxy
    profiles: ["hub"]
    image: itzg/bungeecord:java17
    ports:
      - "25565:25577"
    environment:
      UID: 1002 #Minecraft user
      GID: 1003 #Minecraft group
      TYPE: VELOCITY
      VELOCITY_VERSION: latest
      MEMORY: 512m
      JVM_XX_OPTS: "-XX:+UseG1GC -XX:G1HeapRegionSize=4M -XX:+UnlockExperimentalVMOptions -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:MaxInlineLevel=15"
      PLUGINS: https://github.com/ViaVersion/ViaVersion/releases/download/4.4.2/ViaVersion-4.4.2.jar,https://github.com/ViaVersion/ViaBackwards/releases/download/4.4.2/ViaBackwards-4.4.2.jar
    secrets:
      - forwarding_secret
    restart: always
    volumes:
      - ./servers/main-proxy:/server
    links:
      - main-hub

  main-hub:
    hostname: main-hub
    container_name: main-hub
    profiles: ["hub"]
    image: itzg/minecraft-server:java17
    #ports:
    #  - "30065:25565"
    environment:
      UID: 1002 #Minecraft user
      GID: 1003 #Minecraft group
      TYPE: PAPER
      VERSION: latest
      MEMORY: 1024m
      USE_AIKAR_FLAGS: "TRUE"
      EULA: "TRUE"
      SPIGET_RESOURCES: 19254,27448,25391,102902 #ViaVersion, ViaBackwards, VoidGen, NoEncryption
      #EXEC_DIRECTLY: true
      # enable env variable replacement
      REPLACE_ENV_VARIABLES: "true"
      # define an optional prefix for env variables we want to replace
      ENV_VARIABLE_PREFIX: "CFG_"
      # actual variables
      CFG_FORWARDING_SECRET_FILE: "/run/secrets/forwarding.secret"
    secrets:
      - forwarding_secret
    tty: true
    stdin_open: true
    restart: always
    volumes:
      - ./servers/main-hub:/data
secrets:
  forwarding_secret:
    file: forwarding.secret

velocity.toml

forwarding-secret-file = "${CFG_FORWARDING_SECRET_FILE}"

paper-global.yml

proxies:
  bungee-cord:
    online-mode: false
  proxy-protocol: false
  velocity:
    enabled: true
    online-mode: true
    secret: ${CFG_FORWARDING_SECRET}

The new problem:

main-hub    | [mc-image-helper] 23:19:10.068 ERROR : Failed to process directory=/data: /run/secrets/forwarding.secret
main-hub exited with code 2
main-hub    | [init] Running as uid=1002 gid=1003 with /data as 'drwxrwxr-x 9 1002 1003 4096 Sep  1 23:13 /data'
main-proxy  | [23:19:11 INFO]: Booting up Velocity 3.1.2-SNAPSHOT (git-6547ccaa-b179)...
main-proxy  | [23:19:11 INFO]: Loading localizations...
main-proxy  | [23:19:11 INFO]: Connections will use epoll channels, libdeflate (Linux x86_64) compression, OpenSSL 1.1.x (Linux x86_64) ciphers
main-proxy  | [23:19:11 ERROR]: Unable to read/load/save your velocity.toml. The server will shut down.
main-proxy  | java.lang.RuntimeException: The forwarding-secret-file does not exists.
main-proxy  |   at com.velocitypowered.proxy.config.VelocityConfiguration.read(VelocityConfiguration.java:499) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at com.velocitypowered.proxy.VelocityServer.doStartupConfigLoad(VelocityServer.java:311) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at com.velocitypowered.proxy.VelocityServer.start(VelocityServer.java:217) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy  |   at com.velocitypowered.proxy.Velocity.main(Velocity.java:64) ~[velocity-3.1.2-SNAPSHOT-179.jar:3.1.2-SNAPSHOT (git-6547ccaa-b179)]
main-proxy exited with code 1

Do I now need to replace mounting the entire /data directory with the individual /config and /plugins mounts, etc.?

itzg Sep 1, 2022
Maintainer

I thought that each container had its own internal network and that everything had to be routed externally.

Actually, each container has its own networking namespace and gets its own container IP. They are on the same virtual network though (by default).

Your secret is called forwarding_secret so the filename needs to match exactly as in /run/secrets/forwarding_secret.

      CFG_FORWARDING_SECRET_FILE: "/run/secrets/forwarding.secret"
    secrets:
      - forwarding_secret

klaribot Sep 2, 2022
Author

OH RIGHT SORRY
Sometimes I'm literally just blind.
Thank you for pointing out the obvious typo LMAO

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker-compose setup: Velocity proxy refuses to connect to Paper hub world on same machine #1703

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

docker-compose setup: Velocity proxy refuses to connect to Paper hub world on same machine #1703

klaribot Aug 31, 2022

Replies: 1 comment · 8 replies

itzg Aug 31, 2022 Maintainer

itzg Sep 1, 2022 Maintainer

klaribot Sep 1, 2022 Author

klaribot Sep 1, 2022 Author

itzg Sep 1, 2022 Maintainer

klaribot Sep 2, 2022 Author

klaribot
Aug 31, 2022

Replies: 1 comment 8 replies

itzg
Aug 31, 2022
Maintainer

itzg Sep 1, 2022
Maintainer

klaribot Sep 1, 2022
Author

klaribot Sep 1, 2022
Author

itzg Sep 1, 2022
Maintainer

klaribot Sep 2, 2022
Author