Skip to content

Commit

Permalink
some changes to deployment docs
Browse files Browse the repository at this point in the history
  • Loading branch information
joschahenningsen committed Jul 5, 2023
1 parent beb0a28 commit c06dd97
Show file tree
Hide file tree
Showing 12 changed files with 1,119 additions and 13 deletions.
10 changes: 0 additions & 10 deletions docs/content/docs/about.md

This file was deleted.

25 changes: 25 additions & 0 deletions docs/content/docs/deployment/00-prerequisites.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
title: "Prerequisites"
draft: false
weight: 10
---

## Prerequisites

We deploy GoCast on our own hardware in VMs. Any cloud hosting provider works just as well.
You will need the following hardware configuration:

- 1 VM for the GoCast server and database. This can be a small VM if you are not expecting a lot of users.
- At least 1 VM as an edge server. This server serves the videos to the users. Network throughput is important here. If you serve lots of users, you can spin up more of these.
- At least 1 Worker VM. This server produces the stream, transcodes the vod and much more. CPU performance is important here. As you start streaming more, you can spin up more of these.
- Optional: At least 1 NVIDIA CUDA equipped Server that transcribes streams using the Whisper LLM.
- Optional: 1 VM for monitoring (grafana, prometheus, influx...). This can be a small VM as well.

## Storage

GoCast produces lots of large files. They'll need to be accessed by all workers and edge servers.
Thus, you'll need a shared storage solution. We use [Ceph](https://www.ceph.com/en/).
The reliability and performance of the storage solution is critical for the performance of GoCast, setting it up and running it is not trivial.
Operating a network storage solution is out of scope for this documentation.

For this documentation, we assume that you have some sort of high performance shared filesystem mounted to the same directory on all your servers.
95 changes: 95 additions & 0 deletions docs/content/docs/deployment/01-setup-docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
title: "Setup Docker"
draft: false
weight: 20
---

## Software

Install Docker on all servers/vms: https://docs.docker.com/engine/install/

## Create Swarm

On one of the servers, initialize the swarm:

```bash
$ docker swarm init

> Swarm initialized: current node (bvz81updecsj6wjz393c09vti) is now a manager.
>
> To add a worker to this swarm, run the following command:
>
> docker swarm join \
> --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \
> 172.17.0.2:2377
>
> To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
```

On the other servers, join the swarm:

```bash
$ docker swarm join \
--token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \
172.17.0.2:2377
```

Read the administration guide for docker swarm carefully and make the appropriate adjustments for your environment:
https://docs.docker.com/engine/swarm/admin_guide/

Verify that all nodes are in the swarm:

```bash
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ko66mqj76xo9ftunxq78luc8p vm01 Ready Active Reachable 23.0.1
ogziph0qxfeivly5fnekepwx0 vm02 Ready Active 23.0.1
1prl8b1m7xw2ph5b8dnh98glk vm03 Ready Active 23.0.1
8utl07361ocn5xvzqh27z0c8s * vm04 Ready Active Reachable 23.0.1
hdsuhlwecidor7khbcfn4gni3 vm05 Ready Active Reachable 23.0.1
hj6fkl3j5hwho40uiehc7ikq5 vm06 Ready Active Leader 23.0.1
ctfdd9mtkse2yxid8zku2wx1f vm07 Ready Active 23.0.1
u391iukj6nljosaaygcfkzy2s vm08 Ready Active 23.0.1
wkxct5tvzclvc4uqm8w573dlf vm09 Ready Active 23.0.1
72weo6nozra1cdgjs5wghe7gh vm10 Ready Active 23.0.1
```

## Tag nodes

We use labels to tag our nodes and to deploy services to appropriate nodes.

This commands adds the label worker to the node vm02, instructing our deployment to deploy workers on this node:

```bash
docker node update --label-add worker=true vm02
```
This is a configuration you should aim for:

```bash
docker node ls -q | xargs docker node inspect -f '{{ .ID }} [{{ .Description.Hostname }}]: {{ .Spec.Labels }}'

kwgmm6sxb9nqwojoclxuy4mpt [vmgpu01]: map[voiceservice:true] # optional, this is a server with a GPU for transcription
ko66mqj76xo9ftunxq78luc8p [vm01]: map[db:true traefik:true tumlive:true] # this server is important, it runs the database and the reverse proxy. Don't under-provision it.
hj6fkl3j5hwho40uiehc7ikq5 [vm02]: map[grafana:true influx:true meilisearch:true monitoring:true prometheus:true] # these services are not critical - and optional
ctfdd9mtkse2yxid8zku2wx1f [vm03]: map[worker:true] # the number of workers depends on the number of concurrent streams you want to process. 1 worker can process around 5 stream in our environment.
u391iukj6nljosaaygcfkzy2s [vm04]: map[worker:true]
wkxct5tvzclvc4uqm8w573dlf [vm05]: map[worker:true]
72weo6nozra1cdgjs5wghe7gh [vm06]: map[worker:true]
f7ik66qq6tzhsbwphfpdp2vm1 [vm07]: map[worker:true]
i4l8ouumms96qu96evkb6srol [vm08]: map[worker:true]
vq5cw2bgwncenr5cp89xzsi32 [vm09]: map[worker:true]
q4as4i27z2hnwypgzj8ql2dz1 [vm10]: map[worker:true]
lfged5ra1a7z9wlstxa2bml5c [vm11]: map[worker:true]
3wu812ybzynnunrpoqdsay0bf [vm12]: map[worker:true]
itdbo77gempnl251lakioe5y1 [vm13]: map[worker:true]
zcplsihexr88plf0t8q25tdn7 [vm14]: map[worker:true]
fbi92hp7s0u3c2x13tgrb6fd6 [vm15]: map[worker:true]
o6k2egpupik3qjgq2w0azv70o [vm16]: map[worker:true]
urac70xjf1kx5op39kyulykad [vm17]: map[worker:true]
wpue8f384h7z71mngov5j72c1 [vm18]: map[worker:true]
th77fn3s91s06sy4ciprita3s [vm19]: map[edge:true] # the number of edge nodes depends on the number of concurrent viewers you want to support.
5bqr01nyefxqmkd3luzhh3sne [vm20]: map[edge:true]
vrroo1k8kgk8n557pos5wlz5k [vm21]: map[edge:true]
b6m40kbtg1sctwq5p4vmtghxd [vm22]: map[edge:true]

```
18 changes: 18 additions & 0 deletions docs/content/docs/deployment/02-networking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: "Networking"
draft: false
weight: 30
---


## Networking

The following ports need to be exposed to the public:

| Server (label) | Port |
|----------------------------------|-----------------|
| GoCast Server (tumlive, traefik) | 80 TCP, 443 TCP |
| Worker (worker) | 1935 TCP |
| Edge (edge) | 80 TCP, 443 TCP |

Between the individual servers, communication should not be firewalled. Auditorium hardware should also be in the same vlan.
Loading

0 comments on commit c06dd97

Please sign in to comment.