Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Worker with Runner #1233

Draft
wants to merge 37 commits into
base: dev
Choose a base branch
from
Draft

Replace Worker with Runner #1233

wants to merge 37 commits into from

Conversation

joschahenningsen
Copy link
Member

@joschahenningsen joschahenningsen commented Nov 12, 2023

Motivation and Context

The current worker implementation is hard to maintain, we therefore want to replace it with the new runner. The goal is to have a clean code base that replaces everything the worker does.

todos:

  • All API endpoints the current worker offers should be available in the runner
    • Request Stream
    • Request Stream End
    • Generate live preview
    • Generate Section images
    • Delete Section images (low prio)
    • Combine Thumbnails
  • All API endpoints TUM-Live currently offers for the runner
    • Register
    • Heartbeat (nice to have)
    • request selfstream
    • notify vodUpload finished
    • Notify Silence results
    • Notify Stream Started
    • Notify Thumbnails finished
    • Get stream info for upload
    • notify Transcoding Failure

Functionality wise, the runner should:

  • When request stream is called:
    • grab the url with ffmpeg and create a hls stream on disk and notify tum-live that the stream is available
    • After the stream is done, convert the files, append them together and upload them to the vod service.
    • create a thumbnail and the thumnails for the seekbar
    • detect silences in the video and send them to tum-live (not high prio for now)
  • when a vod is uploaded, everything that happens after a livestream should also happen to the uploaded file
  • when a selfstreamer starts streaming
    • ask tum-live for the details of the token and continue just like with a normal stream

Copy link

Your Testserver will be ready at https://1233.test.live.mm.rbg.tum.de in a few minutes.

Logins
Kurs1 Kurs2 Kurs3 Kurs4
public public loggedin enrolled
prof1 prof1 prof2 prof1
prof2
student1
student2
student3
student1
student2
student2
student3
student1
student2

- Actions refined and now with postman, working selfstream requests
- A site page development has begun with the needed functionality of heartbeats still in development for the response
- the model/runner.go and dao/runner.go have been edited to work better with existing functionalities

TODO:
- Thumbnail.go still needs tweaking
- rest implementations need to be done
- having the runners be displayed onto the webpage

please text me over element or github if there is anything I forgot to mention in this push
SebiWrn and others added 14 commits February 1, 2024 14:46
# Conflicts:
#	runner/actions/stream.go
#	runner/runner.go
… for the preventions of crashes and fail saves for runners( Jobs.go and Action.go. for now, the notify runner method only looks for streams like the notifyWorker func but I'm working on the overlooking of jobs on tumlive side with the models and dao files I added
* Realised silence detect action

* Apply silence detection in the job

* align method of setting a new value in the context with existing actions

* send silence results to server on success
* Add commands for audio normalization; pass 1 realised in a new action

* Pass 1 realised in a new action

* Make sure transcoding with audio normalization does the same to video.

* Integrate audio normalization into transcode. Apply it only when exactly 1 video exists for the stream.

* Pass 1: properly return error message

* Pass 1: 1. synchronise reading from stdin before extracting info; 2. make sure to close everything in case of error

* Audio normalization: log when applied

* Remove the unused action AudioNormalization

---------

Co-authored-by: Dawin <[email protected]>
…web page but not fully finished the pop up window. working on the models and their schemas.
DawinYurtseven and others added 10 commits October 22, 2024 17:23
# Conflicts:
#	.golangci.yml
#	cmd/tumlive/tumlive.go
#	go.work.sum
#	tools/stream-signing.go
#	web/admin.go
#	web/template/partial/footer.gohtml
#	web/template/user-settings.gohtml
jetzt sollte tumlive auf die richtige vm zugreifen müssen
…usted the cmd.yaml for less clutter during hls stream
Copy link
Collaborator

@SebiWrn SebiWrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole generating part of the URL should be done on TUM-Live rather than on the runner

runner/actions/stream.go Outdated Show resolved Hide resolved

localhls := fmt.Sprintf("%v/%d/%d/%s/%s/playlist.m3u8", hostname, courseID, streamID, version, end.Format("15-04-05"))

edgeURL, err := url.Parse(fmt.Sprintf("http://%v:8089/%v",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge Server isn't hosted on Port 8089 when not in development (port needs to be a config option)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the config option has been added. mind that the port is rn only one port and if needed to have multiple edge servers, we need to think of a selection option to not swarm one server full

@DawinYurtseven
Copy link
Contributor

Rn worked on the upload part and looked at the vod service of the worker, trying to see how it works and orientate myself after that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants