RustDesk Server Configuration & Deployment

[tremor021]

General

This is a followup guide on how the RustServer works and how its configured. I'm gonna make it short, because there is not really that much to it. All the "good stuff" is behind a paywalll aka needs you to buy a licence from RustDesk, that includes the nice config UI that we're not getting with the self-hosted free version of the software.

Script link: https://community-scripts.github.io/ProxmoxVE/scripts?id=rustdesk-server

Installation

As you can tell, LXC installs everything automatically and there is no need for user interaction during the install process.
There are technically two executables (servers):

hbbs - RustDesk ID (rendezvous / signaling) server, listen on TCP (21114 - for http in Pro only, 21115, 21116, 21118 for web socket) and UDP (21116)
hbbr - RustDesk relay server, listen on TCP (21117, 21119 for web socket)
Our script installs both of these servers and two services are created:

rustdesk-hbbr.service
rustdesk-hbbs.service

Server & Client configurations

The first time server is started, if no key is defined, it will create private and public key pair in the working directory. To check which one is that, you can check the service file:

root@rustdeskserver:~# systemctl status rustdesk-hbbr.service
● rustdesk-hbbr.service - Rustdesk Relay Server
     Loaded: loaded (/lib/systemd/system/rustdesk-hbbr.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-02-13 00:53:34 CET; 1 day 8h ago
   Main PID: 4061 (hbbr)
      Tasks: 4 (limit: 77006)
     Memory: 536.0K
        CPU: 9.738s
     CGroup: /system.slice/rustdesk-hbbr.service
             └─4061 /usr/bin/hbbr

Feb 13 00:53:34 rustdeskserver systemd[1]: Started rustdesk-hbbr.service - Rustdesk Relay Server.
root@rustdeskserver:~#

So, we can see that service is loaded from /lib/systemd/system/rustdesk-hbbr.service. If we cat the service file, we get this:

root@rustdeskserver:~# cat /lib/systemd/system/rustdesk-hbbr.service 

[Unit]
Description=Rustdesk Relay Server

[Service]
Type=simple
LimitNOFILE=1000000
ExecStart=/usr/bin/hbbr
WorkingDirectory=/var/lib/rustdesk-server/
User=
Group=
Restart=always
StandardOutput=append:/var/log/rustdesk-server/hbbr.log
StandardError=append:/var/log/rustdesk-server/hbbr.error
# Restart service after 10 seconds if node service crashes
RestartSec=10

[Install]
WantedBy=multi-user.target

root@rustdeskserver:~# 

Service file states that the WorkingDirectory is /var/lib/rustdesk-server/. This is where the keypair is created. Let's look:

root@rustdeskserver:~# ls -la /var/lib/rustdesk-server/
total 136
drwxr-xr-x  2 root root  4096 Feb 14 09:10 .
drwxr-xr-x 19 root root  4096 Feb 13 00:53 ..
-rw-r--r--  1 root root  4096 Feb 13 00:53 db_v2.sqlite3
-rw-r--r--  1 root root 32768 Feb 14 08:28 db_v2.sqlite3-shm
-rw-r--r--  1 root root 82432 Feb 14 08:28 db_v2.sqlite3-wal
-rw-r--r--  1 root root    88 Feb 13 00:53 id_ed25519
-rw-r--r--  1 root root    44 Feb 13 00:53 id_ed25519.pub
root@rustdeskserver:~# 

Files id_ed25519 and id_ed25519.pub are the ones we're looking for. These are the keys used to encrypt communication between clients and server. You can create NEW keypair if you want by deleting these two files and restarting the services.

id_ed25519.pub contains the public key that clients need to know in order to connect. So, let's cat that:

root@rustdeskserver:~# cat /var/lib/rustdesk-server/id_ed25519.pub; echo
kMtF+J8VMGEk2iLmUQ6AbVDkVRKce9r+p8n4YuQfl+c=
root@rustdeskserver:~#

So, the servers public key is kMtF+J8VMGEk2iLmUQ6AbVDkVRKce9r+p8n4YuQfl+c=. We will use this to connect clients to our server. We will use Windows client as example. Get it from RustDesk GitHub:
https://github.com/rustdesk/rustdesk/releases/download/1.3.7/rustdesk-1.3.7-x86_64.exe

In order for RustDesk Client to know the server address and key used to encrypt communications, RustDesk uses client's exe file name. This means the .exe file you just downloaded needs to have a specific name, and it goes like this:

rustdesk-host=<IP of server or URL>,key=kMtF+J8VMGEk2iLmUQ6AbVDkVRKce9r+p8n4YuQfl+c=.exe

Let's assume our server IP address is 192.168.1.243, the file name should be like:

rustdesk-host=192.168.1.243,key=kMtF+J8VMGEk2iLmUQ6AbVDkVRKce9r+p8n4YuQfl+c=.exe

If we run client after we rename the .exe to look like the example above, the client window will show up. If you have other machines running, they will be autodiscovered as shown in this screenshot:

The end

Congratulations, you're done with the configuration and everything should be working. PLEASE read the docs if you have any questions!

https://rustdesk.com/docs/en/self-host/

[alhaddar]

Thank you for this effort. I have a question: Where can I find the compose.yml file? It was not there. Can I create it or not?

Regards.

[tremor021]

This is not a Docker app

[akhdanfadh]

Would you mind to elaborate on what I should do if I have a separate nginx-proxy-manager LXC so that I can use rustdesk.mydomain.com for the relay server? Relevant guide: https://gist.github.com/Lvdwardt/707d9c3fb4581d38102399f91a21c9c6

I keep getting 'Internal Error' when adding stream with rustdesk proxmox's IP and hosts are those 2111*.

[tremor021]

Thats a question for RustDesk github, we are not their support

[akhdanfadh]

Aight, thanks for the pointer. Probably a bug on NPM LXC themselves, submitted an issue.

[lukaspavelka]

Hi thx for manual, btw: how will be rustdesk in this LXC container updated in future? What do you see as right way to do it?

[tremor021]

just type update in the LXC shell

[lukaspavelka]

Can it be just added to cron as a job for every night ?

[tremor021]

i'm not sure you would really want to get automatic updates for any LXC. many apps introduce breaking changes that often crash your LXC