SRT Timecode via SEI Message

[bchah]

Hello! Avid Media Composer sends SRT directly which is a really attractive streaming workflow. According to Avid it also sends timecode information (so remote viewers can monitor the timecode from the application). The video streaming to OME works great but I notice this error in the log for what looks like each frame or packet:

E [Dech264:809] FFmpeg | third_parties.cpp:111 | [AVCodecContext: 0xffff8d45a000] SEI type 5 size 176 truncated at 171

Does this imply that the extra data (SEI message) they are sending with the stream is declaring a different length than it is sending? Or some other issue with OME decoding the data?

Finally my question leads to this: if we can receive data or messages via the stream, is there a way for OME to make those data or messages available either via an output file or API call?

[getroot]

OvenMediaEngine does not parse SEI. However, if you bypass transcoding, I think you can use it if the player supports it. Let’s think together about what it means to support timecode responses from the API.

The error in the decoder is output by ffmpeg, and I am not sure of its exact meaning.

[bchah]

Thank you @getroot! From my limited understanding, we can find the SEI messages within certain NAL units, ones that are not part of the Video Coding Layer. Perhaps it would be useful to OME developers to have API access to the non-VCL NAL units (and parse them ourselves)?

[bchah]

Hi @getroot checking in to see if you have given this some thought? I am discovering more encoders that support sending SEI timecode with the stream, like the Magewell Ultra Encode for instance:

Could this be a request for @SangwonOh ? Maybe there is something that can be done with the original data track in OvenPlayer if the original tracks are passed, like you suggested. Would love to find a way to do this :)

[getroot]

OvenPlayer relies on hls.js for HLS playback. This is especially true for deep features such as NAL unit parsing. If you search on hls.js, you will find some information.

[bchah]

Following up on this, I found out where SEI messages are supposed to be in Hls.js.

OvenPlayer.on('hlsPrepared', (hls)=> {
            hls.on(Hls.Events.FRAG_PARSING_USERDATA, (event, data) => {
                console.log(data.samples) // these are the SEI messages
        });
    });

The problem is, data.samples is empty when playing back with OvenPlayer. Testing with the Magewell encoder above and AVID Media Composer (both of which send SEI Timecode), I tried passing the original tracks as you suggested but could not access the data. Is there a chance that OME is not including the Data track when processing HLS? I see it in the Origin stream info:

>> Origin Stream Info
	id(101), output(helloWorld), SourceType(SRT), Created Time (Fri Mar 29 21:58:50 2024)

	Data  Track #0: Public Name(Data_2) Variant Name(Data) Codec(0,Unknown,Passthrough) BSF(Unknown) timebase(1/1000)
	Video Track #1: Public Name(Video_259) Variant Name(original_video) Bitrate(2.38Mb) Codec(1,H264,Passthrough:0) BSF(H264_ANNEXB) Resolution(1920x1080) Framerate(16.96) KeyInterval(30/frame) SkipFrames(-1) BFrames(0) timebase(1/90000)
	Audio Track #2: Public Name(Audio_260) Variant Name(original_audio) Bitrate(74.12Kb) Codec(6,AAC,Passthrough) BSF(AAC_ADTS) Samplerate(48.0K) Format(none, 0) Channel(stereo, 2) timebase(1/90000)

Thank you for any clues you can provide 🙏

[Morpheus235]

@bchah Using Bypass will definitely pass SEI Chunks to the Browser but only feasible be accessible by using hls.js. My Company employs Haivision Encoders which support SEI Timecode as well and I have been sucessfully parsing the Incoming TC based on ffmpegs pic_timing Code with a little addition I added into hls.js myself. Natively these SEI User Messages / NAL Units seem to be skipped entirely by hls.js.
After doing this I basically hook the Callbacks the same way you tried to:

player.on("hlsPrepared", (hls) => {
    hls.on("hlsFragChanged", function (event, data) {
      ...		
    });
    
   hls.on("hlsFragParsingUserdata", function (event, data) {
      ...
   });
})

For hls.js You will have to implement the parsing of these packages into parseSEIMessageFromNALu inside hls.js
FFmpeg Documentation

[bchah]

Thanks for the tips @Morpheus235 - I think this will remain a dream for me, since my hope is really to implement live timecode display with the WebRTC publisher (to avoid the latency of HLS).