-
Notifications
You must be signed in to change notification settings - Fork 44
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
beb0a28
commit c06dd97
Showing
12 changed files
with
1,119 additions
and
13 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
--- | ||
title: "Prerequisites" | ||
draft: false | ||
weight: 10 | ||
--- | ||
|
||
## Prerequisites | ||
|
||
We deploy GoCast on our own hardware in VMs. Any cloud hosting provider works just as well. | ||
You will need the following hardware configuration: | ||
|
||
- 1 VM for the GoCast server and database. This can be a small VM if you are not expecting a lot of users. | ||
- At least 1 VM as an edge server. This server serves the videos to the users. Network throughput is important here. If you serve lots of users, you can spin up more of these. | ||
- At least 1 Worker VM. This server produces the stream, transcodes the vod and much more. CPU performance is important here. As you start streaming more, you can spin up more of these. | ||
- Optional: At least 1 NVIDIA CUDA equipped Server that transcribes streams using the Whisper LLM. | ||
- Optional: 1 VM for monitoring (grafana, prometheus, influx...). This can be a small VM as well. | ||
|
||
## Storage | ||
|
||
GoCast produces lots of large files. They'll need to be accessed by all workers and edge servers. | ||
Thus, you'll need a shared storage solution. We use [Ceph](https://www.ceph.com/en/). | ||
The reliability and performance of the storage solution is critical for the performance of GoCast, setting it up and running it is not trivial. | ||
Operating a network storage solution is out of scope for this documentation. | ||
|
||
For this documentation, we assume that you have some sort of high performance shared filesystem mounted to the same directory on all your servers. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
--- | ||
title: "Setup Docker" | ||
draft: false | ||
weight: 20 | ||
--- | ||
|
||
## Software | ||
|
||
Install Docker on all servers/vms: https://docs.docker.com/engine/install/ | ||
|
||
## Create Swarm | ||
|
||
On one of the servers, initialize the swarm: | ||
|
||
```bash | ||
$ docker swarm init | ||
|
||
> Swarm initialized: current node (bvz81updecsj6wjz393c09vti) is now a manager. | ||
> | ||
> To add a worker to this swarm, run the following command: | ||
> | ||
> docker swarm join \ | ||
> --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \ | ||
> 172.17.0.2:2377 | ||
> | ||
> To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. | ||
``` | ||
|
||
On the other servers, join the swarm: | ||
|
||
```bash | ||
$ docker swarm join \ | ||
--token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \ | ||
172.17.0.2:2377 | ||
``` | ||
|
||
Read the administration guide for docker swarm carefully and make the appropriate adjustments for your environment: | ||
https://docs.docker.com/engine/swarm/admin_guide/ | ||
|
||
Verify that all nodes are in the swarm: | ||
|
||
```bash | ||
$ docker node ls | ||
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION | ||
ko66mqj76xo9ftunxq78luc8p vm01 Ready Active Reachable 23.0.1 | ||
ogziph0qxfeivly5fnekepwx0 vm02 Ready Active 23.0.1 | ||
1prl8b1m7xw2ph5b8dnh98glk vm03 Ready Active 23.0.1 | ||
8utl07361ocn5xvzqh27z0c8s * vm04 Ready Active Reachable 23.0.1 | ||
hdsuhlwecidor7khbcfn4gni3 vm05 Ready Active Reachable 23.0.1 | ||
hj6fkl3j5hwho40uiehc7ikq5 vm06 Ready Active Leader 23.0.1 | ||
ctfdd9mtkse2yxid8zku2wx1f vm07 Ready Active 23.0.1 | ||
u391iukj6nljosaaygcfkzy2s vm08 Ready Active 23.0.1 | ||
wkxct5tvzclvc4uqm8w573dlf vm09 Ready Active 23.0.1 | ||
72weo6nozra1cdgjs5wghe7gh vm10 Ready Active 23.0.1 | ||
``` | ||
|
||
## Tag nodes | ||
|
||
We use labels to tag our nodes and to deploy services to appropriate nodes. | ||
|
||
This commands adds the label worker to the node vm02, instructing our deployment to deploy workers on this node: | ||
|
||
```bash | ||
docker node update --label-add worker=true vm02 | ||
``` | ||
This is a configuration you should aim for: | ||
|
||
```bash | ||
docker node ls -q | xargs docker node inspect -f '{{ .ID }} [{{ .Description.Hostname }}]: {{ .Spec.Labels }}' | ||
|
||
kwgmm6sxb9nqwojoclxuy4mpt [vmgpu01]: map[voiceservice:true] # optional, this is a server with a GPU for transcription | ||
ko66mqj76xo9ftunxq78luc8p [vm01]: map[db:true traefik:true tumlive:true] # this server is important, it runs the database and the reverse proxy. Don't under-provision it. | ||
hj6fkl3j5hwho40uiehc7ikq5 [vm02]: map[grafana:true influx:true meilisearch:true monitoring:true prometheus:true] # these services are not critical - and optional | ||
ctfdd9mtkse2yxid8zku2wx1f [vm03]: map[worker:true] # the number of workers depends on the number of concurrent streams you want to process. 1 worker can process around 5 stream in our environment. | ||
u391iukj6nljosaaygcfkzy2s [vm04]: map[worker:true] | ||
wkxct5tvzclvc4uqm8w573dlf [vm05]: map[worker:true] | ||
72weo6nozra1cdgjs5wghe7gh [vm06]: map[worker:true] | ||
f7ik66qq6tzhsbwphfpdp2vm1 [vm07]: map[worker:true] | ||
i4l8ouumms96qu96evkb6srol [vm08]: map[worker:true] | ||
vq5cw2bgwncenr5cp89xzsi32 [vm09]: map[worker:true] | ||
q4as4i27z2hnwypgzj8ql2dz1 [vm10]: map[worker:true] | ||
lfged5ra1a7z9wlstxa2bml5c [vm11]: map[worker:true] | ||
3wu812ybzynnunrpoqdsay0bf [vm12]: map[worker:true] | ||
itdbo77gempnl251lakioe5y1 [vm13]: map[worker:true] | ||
zcplsihexr88plf0t8q25tdn7 [vm14]: map[worker:true] | ||
fbi92hp7s0u3c2x13tgrb6fd6 [vm15]: map[worker:true] | ||
o6k2egpupik3qjgq2w0azv70o [vm16]: map[worker:true] | ||
urac70xjf1kx5op39kyulykad [vm17]: map[worker:true] | ||
wpue8f384h7z71mngov5j72c1 [vm18]: map[worker:true] | ||
th77fn3s91s06sy4ciprita3s [vm19]: map[edge:true] # the number of edge nodes depends on the number of concurrent viewers you want to support. | ||
5bqr01nyefxqmkd3luzhh3sne [vm20]: map[edge:true] | ||
vrroo1k8kgk8n557pos5wlz5k [vm21]: map[edge:true] | ||
b6m40kbtg1sctwq5p4vmtghxd [vm22]: map[edge:true] | ||
|
||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
--- | ||
title: "Networking" | ||
draft: false | ||
weight: 30 | ||
--- | ||
|
||
|
||
## Networking | ||
|
||
The following ports need to be exposed to the public: | ||
|
||
| Server (label) | Port | | ||
|----------------------------------|-----------------| | ||
| GoCast Server (tumlive, traefik) | 80 TCP, 443 TCP | | ||
| Worker (worker) | 1935 TCP | | ||
| Edge (edge) | 80 TCP, 443 TCP | | ||
|
||
Between the individual servers, communication should not be firewalled. Auditorium hardware should also be in the same vlan. |
Oops, something went wrong.