-
Notifications
You must be signed in to change notification settings - Fork 1
Network Transport
Streaming SDK provides an implementation of a robust network transport, which can stream video and audio over a UDP or a TCP connection with minimal latency. However, in some cases you might want to implement a different transport protocol, such as, for example, WebRTC. Streaming SDK is designed in a way that allows to completely decouple the transport from the application and video/audio pipelines. This is achieved by means of two interfaces that abstract the transport layer:
- ssdk::transport_common::ServerTransport - an abstraction of the transport layer on the server
- ssdk::transport_common::ClientTransport - an abstraction of the transport layer on the client
The AMD Transport implements these interfaces in the ssdk::transport_amd::ServerTransportImpl and ssdk::transport_amd::ClientTransportImpl classes respectively. Both implementations are located in the sdk/transports/transport_amd/ directory.
The Network Transport layer delivers the following data between the client and the server:
- Multiple streams of compressed video frames and codec initialization blocks
- Multiple streams of compressed or uncompressed audio buffers and codec initialization blocks
- Messages carrying controller events
- Messages carrying streaming statistics and Quality-of-Service data
- Application-defined messages
All of the above messages can be transmitted in either direction. In this sense, the difference between the client and the server is that the client initiates a connection, while the server is listening and accepting connection requests from the client. The server can also send the same video and audio streams to multiple clients simultaneously. Aside from these two aspects, the ServerTransport and the ClientTransport interfaces are symmetrical.
The underlying medium transferring data between the server and the clients is completely opaque to the application. For example, the Network Transport implementation is responsible for fragmentation and reassembly of large messages that do not fit into a maximum transmission unit of the underlying network protocol when communicating over a packet-oriented protocol such as UDP. It is responsible for error handling, retransmission of lost packets and other low-level aspects. It is also responsible for encryption and decryption of messages sent across the network, where applicable.
Both the ServerTransport and the ClientTransport interfaces define a number of callback interfaces that must be implemented by the application. Please note that while the ServerTransport and the ClientTransport interfaces themselves are reference-counted and define the smart pointer Ptr type, the callback interfaces are not reference-counted to avoid memory leaks caused by circular references. The application must manage the life cycle of these callback interfaces in such a way that the objects implementing them are destroyed after the objects implementing the ServerTransport and the ClientTransport interfaces go out of scope and are destroyed.
The ServerTransport Interface is the main interface between the network transport implementation and the server application. Defined in sdk/transports/transport-common/ServerTransport.h and has the following methods:
Start the server and initialize with specified parameters. The AMD Transport defines its parameters in the ssdk::transport_amd::ServerTransportImpl.*. The initialization parameters include pointers to all callbacks as well as parameters such as the UDP or TCP port the server would listen for incoming connections on, carrier protocol-specific parameters and so on
Terminates the server. It is safe to release the pointer to the server object after calling Shutdown().
Queries the number of active sessions. Returns the number of active connections.
Checks if a session identified by a SessionHandle is connected. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection. Returns true if the session is active, false otherwise.
Forcefully disconnects a specific session. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection. Returns a Result indicating success or failure of the operation.
Forcefully disconnects all active sessions. Returns a Result indicating success or failure of the operation.
SendVideoInit(SessionHandle session, const char* codec, StreamID streamID, InitID initID, const AMFSize& streamResolution, const AMFRect& viewport, uint32_t bitDepth, bool stereoscopic, bool foveated, const void* initBlock, size_t initBlockSize);
Send an decoder initialization block for a video stream for the specified session. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection. codec is the video codec ID string (e.g., "H264"). Each unique initialization block which requires the decoder to be reinitialized, must be assigned a unique initID. initBlock contains a pointer to the initialization block data (aka ExtraData) and can be optional and set to nullptr depending on the codec. initBlockSize contains the size of the initialization block data.
Sends a video frame to the specified session. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection.
The frame parameter contains video frame to be transmitted. A frame is organized as a collection of sub-frames. A properly formed TransmittableVideoFrame must contain at least one sub-frame. Sub-frames can be used to carry any kind of additional information. For example, the left and the right eye views in a stereoscopic stream could be represented with two sub-frames. Alternatively, if you need to transmit an alpha channel or a transparency mask while using a codec that does not support them, or transmit any additional binary metadata.
When constructing a TransmittableVideoFrame, simply call the TransmittableVideoFrame::AddSubframe() method for every sub-frame to add it to the TransmittableVideoFrame object. Once ready, call TransmittableVideoFrame::CalculateRequiredBufferSize() to calculate the size of the buffer that can contain all sub-frames, allocate it and call the TransmittableVideoFrame::ConstructFrame() method passing a pointer to the buffer as a parameter.
Each frame also carries a sequence number which is incremented by 1 for every subsequent frame. This sequence number is used to detect frame loss on the receiving end. It also carries a boolean discontinuity flag, which is set to true when the sequence of frames is broken for any reason, such as, for example, a seek operation in a video file.
SendAudioInit(SessionHandle session, const char* codec, StreamID streamID, InitID initID, uint32_t channels, uint32_t layout, amf::AMF_AUDIO_FORMAT format, uint32_t samplingRate, const void* initBlock, size_t initBlockSize)
Initializes an audio stream for the specified session. This is similar to SendVideoInit for video. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection. codec is the video codec ID string. Note that audio codecs might use numerical FFMPEG IDs (still represented as strings) as well as codec names. Each unique initialization block which requires the decoder to be reinitialized, must be assigned a unique initID. initBlock contains a pointer to the initialization block data (aka ExtraData) and can be optional and set to nullptr depending on the codec. initBlockSize contains the size of the initialization block data.
Sends an audio buffer to the specified session. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection. This is the audio equivalent of SendVideoFrame(). Similarly to SendVideoFrame() the audio buffer is passed as an instance of the TransmittableAudioBuffer class, which includes an amf::AMFBuffer object containing the audio data (compressed or uncompressed), a sequence number and a discontinuity flag similar to the ones on a video frame.
Sends a controller event to the server. controlID is the ID of the control to which the event pertains. Control IDs are defined as strings of the following format:
[/host]/controller/input
The input portion of the ID can be further composed of multiple elements separated with a slash. For instance, some controls on game and VR controllers can generate separate events when they're touched and when they are pressed, resulting in several input substrings. All control ID strings are defined in the ssdk/controllers/UserInput.* files.
pEvent contains a pointer to the control event structure, which stores a control-specific value in a form of an amf::AMFVariant.
Sends the cursor bitmap to clients, invoked when the mouse pointer on the server changes shape. cursor is the cursor object containing bitmap data.
Sends absolute mouse coordinates in the [0.0:1.0] range to clients.
Sends a notification to clients to hide the cursor.
Sends an application-defined message to the specified session. session The session handle to which the message is sent. A session handle is passed to the ConnectionManagerCallback::OnConnectionRequest() callback when a client attempts to establish a connection. msg contains a pointer to a buffer with the message data. size contains the size of the message data.
The AMD Network Transport allows to encrypt all network traffic using the standard AES algorithm. The encryption key is calculated as a SHA-256 hash of the pre-shared passphrase. Encryption is disabled when cipherPassphrase is set to nullptr.
You can set a different encryption key for each session.
The ServerTransport interface defines a number of callback interfaces which are supposed to be implemented by the application. They notify the application of various events occurring on the server.
Called when the server receives a discovery request. The server can either accept or refuse the request based on the client ID. With the AMD Transport, refusing the discovery request would effectively make the server invisible.
Each client should generate a unique ID, which is passed to this method as clientID. It can have any format, as long as it uniquely identifies a specific client device. It can be used to filter out which clients can connect to a specific server and which cannot. The ID could be made either persistent on the client, or can be assigned dynamically by the server (through a separate, application-specific protocol, which is outside of the scope of Streaming SDK) as an access token. The same ID can be used with the OnConnectionRequest method (described below) to authorize a connection for a specific client.
Invoked when a client attempts to establish a connection. The server can choose to accept or refuse the connection request based on the client ID and desired role (either CONTROLLER or VIEWER). The role parameter determines whether this client can send input events or can just receive video and audio as a passive viewer.
A connection to the specific client is identified by the session handle. A handle is a unique opaque identifier assigned to every new session.
Called when a client has successfully subscribed and is ready to receive video and/or audio. This indicates that the connection setup is complete.
Triggered when a client is disconnected. The reason for disconnection (either CLIENT_DISCONNECTED or TIMEOUT) is provided for context.
In a typical streaming scenario the server sends video to the client(s). The VideoSenderCallback interface must be implemented by the server application to receive notifications about events that require the pipeline responsible for sending video to take action.
Called when a client subscribes to a specific video stream identified by streamID. The video sender pipeline should perform the necessary initializations for the specified stream to be able to stream it. The server should send the video decoder initialization block to the specified client identified by the session handle by calling ssdk::trabsport_common::ServerTranbsport::SendVideoInit().
Triggered when a client unsubscribes from a specific video stream, indicating that the client no longer wishes to receive video data.
Called when a client is ready to accept video frames after initialization or reinitialization of the video codec. The initID parameter indicates that the client's video decoder has been initialized with the specific initialization block.
It is up to the transport protocol implementation to ensure that the proper initialization of the client has been performed and it is ready to receive a video stream. The AMD Network Transport waits for an acknowledgement from the client before triggering this callback, other transport implementations may employ different methods to achieve this.
Invoked when a key or I/IDR frame must be submitted, either requested by the client or determined necessary by the server. The video encoder must generate a key frame as soon as possible in response. Typically, this method would be called when a new client subscribes to an existing stream or when a client detects a frame loss and needs a key frame to continue decoding.
Called when the video initialization block needs to be sent or resent to the client. The video transmitter pipeline should resend the last generated initialization block.
Triggered when a client requests a change in the video bitrate, allowing for adjustments based on network conditions.
Called when a client requests a change in the video framerate, facilitating optimization for their playback experience.
OnResolutionChangeReceiverRequest(SessionHandle session, StreamID streamID, const AMFSize& resolution)
Invoked when a client requests a change in video resolution, enabling adaptive streaming based on client capabilities or network conditions.
The VideoStatsCallback interface allows the application to be notified when video statistics data is received from a client.
OnVideoStats(SessionHandle session, StreamID streamID, amf_pts lastStatsTime, float framerate, int64_t forceIDRReqCount, float sendTime, int64_t decoderQueueDepth)
Called when video statistics are updated for a specific client and stream. This provides insights into performance metrics like framerate, force IDR request counts, and buffer depths.
Both the server and the client can send and receive video data to and from each other. For example, the server can stream its desktop to the client, while the client could stream video from a webcam. The VideoReceiverCallback notifies the server application about events pertaining to the server's video receiver function.
OnVideoInit(SessionHandle session, StreamID streamID, const char* codec, InitID initID, const AMFSize& streamResolution, const AMFRect& viewport, uint32_t bitDepth, bool stereoscopic, bool foveated, const void* initBlock, size_t initBlockSize)
Triggered when a video codec initialization block for the specific video stream is received. This includes codec details, stream resolution, and viewport information necessary for setting up the video stream.
Called when a video frame is received from a client, indicating that new video data is ready for processing. The ReceivableVideoFrame is complimentary to the TransmittableVideoFrame. It parses the buffer containing multiple sub-frames serialized by TransmittableVideoFrame.
The AudioSenderCallback interface is similar to VideoSenderCallback and is used to notify the audio transmitter pipeline about events that require action.
Called when a client subscribes to a specific audio stream.
Triggered when a client unsubscribes from a specific audio stream, indicating that they no longer wish to receive audio data.
Called when a client is ready to accept audio buffers after initialization or reinitialization of the video codec. The initID parameter indicates that the client's audio decoder has been initialized with the specific initialization block.
Called when the audio initialization block needs to be sent or resent, ensuring the client has the necessary setup data.
Similarly to the VideoReceiverCallback interface, the AudioReceiverCallback interface notifies the server application about events pertaining to the server's audio receiver function.
OnAudioInit(SessionHandle session, StreamID streamID, const char* codec, InitID initID, uint32_t channels, uint32_t layout, uint32_t samplingRate, const void* initBlock, size_t initBlockSize)
Called when an audio codec initialization block is received, providing essential parameters like codec type, channel configuration, and sampling rate.
Triggered when an audio buffer is received from a client, indicating that new audio data is ready for processing. Similarly to ReceivableVideoFrame, the ReceivableAudioBuffer class is complimentary to the TransmittableAudioBuffer class, performing the reverse transformationю
The InputControllerCallback callback informs the server application about events related to input devices connected to the client, such as keyboards, mice, game controllers, etc.
Called when a controller is enabled, allowing for interaction with the server. The deviceID parameter contains the device portion of the event ID string, see the Controller Messaging section for more details on the device ID format.
Triggered when a controller is disabled, indicating it can no longer send input data to the server.
Triggered when an input event has been received by the network transport, but before the message has been fully parsed. The type of the value expected depends on controlID and the network transport layer is unaware of it. This method is implemented by the application and returns a value of type amf::AMF_VARIANT_TYPE, which determines the type of data expected for this specific event. Whn the client has an incompatible controller connected, which sends events that the server is not aware of, this method should return amf::AMF_VARIANT_TYPE::AMF_VARIANT_EMPTY and the event will be ignored. Otherwise the subsequent call to OnControllerInputEvent() will receive the event parameter containing the value of the specified type, encapsulated in the amf::AMFVariant object.
OnControllerInputEvent(SessionHandle session, const char* controlID, const ssdk::ctls::CtlEvent& event)
Invoked when an input event is received from a controller connecting to the client, providing the session handle, control ID, and event details for processing. This callback is complimentary to the SendControllerEvent method.
Triggered when the pose of a trackable device, such as a head-mounted display (HMD) or a VR controller, changes allowing the server to update its state based on device movements.
Called when an application-specific message is received, allowing for custom communication between the application and the server.
The ClientTransport interface provides functionalities for client-side communication, managing connections to servers, video/audio streaming, and device interaction. It also offers several callback mechanisms for handling various client-server events.
The ClientTransport interface is the client's counterpart of the ServerTransport interface described above. They are almost symmetrical, with one noteable difference that the ClientTransport interface does not expose session ID since a client maintains a single connection to the server only, making session management unnecessary.
Starts the client transport using the provided initialization parameters.
Shuts down the client transport and releases resources.
Initiates the discovery process to find available servers.
Attempts to connect to the specified server using the provided URL.
Disconnects the client from the current server.
Subscribes the client to a video stream identified by the streamID.
Subscribes the client to an audio stream identified by the streamID.
Unsubscribes the client from a video stream identified by the StreamID.
Unsubscribes the client from an audio stream identified by the StreamID.
SendVideoInit(const char* codec, StreamID streamID, InitID initID, const AMFSize& streamResolution, const AMFRect& viewport, uint32_t bitDepth, const void* initBlock, size_t initBlockSize)
Sends the initial video stream configuration to the server.
Sends a compressed video frame to the server.
SendAudioInit(const char* codec, StreamID streamID, InitID initID, amf::AMF_AUDIO_FORMAT format, uint32_t channels, uint32_t layout, uint32_t samplingRate, const void* initBlock, size_t initBlockSize)
Sends the initial audio stream configuration to the server.
Sends an audio buffer to the server.
Sends a control event from an input device to the server.
Sends a pose update for a trackable device to the server.
SendDeviceConnected(const char* deviceID, ssdk::ctls::CTRL_DEVICE_TYPE eType, const char* description)
Notifies the server that a device has been connected.
Notifies the server that a device has been disconnected.
Sends streaming statistics to the server.
Sends a custom application-defined message to the server.
The ConnectionManagerCallback callback notifies the client application of events occuring during server discovery, connection establishment and termination.
Callback invoked when a server is discovered during the discovery process. This method can be called multiple times, once for every connection point discovered on the network. With AMD Network Transport it will be called once for every protocol supported by every server on the same IP subnet. For example, when a server supports streaming over TCP and UDP, the OnServerDiscovered callback will be invoked twice for such server, each time with a different URL in the server descriptor.
This callback can instruct the client to either continue or terminate further discovery. When OnServerDiscovered returns DiscoveryAction::STOP, further responses from other servers that might potentially respond to the discovery broadcast will be ignored.
Once a server is discovered, call Connect() passing it a URL from the server descriptor.
Callback invoked when the server discovery process is complete. There will be no more calls to OnServerDiscovered after that.
Callback invoked when a connection to a server is successfully established.
Callback invoked when the connection is terminated, providing the reason for termination.
As mentioned previously, client devices can also send video and audio to the server, for instance, when emulating a webcam and a microphone. The VideoSenderCallback interface notifies the client application of events that require actions in the vide transmitter pipeline.
Callback invoked when a receiver subscribes to a video stream.
Callback invoked when a receiver unsubscribes from a video stream.
Called when a receiver is ready to accept video frames after initialization or reinitialization of the video codec. The initID parameter indicates that the receiver's video decoder has been initialized with the specific initialization block.
It is up to the transport protocol implementation to ensure that the proper initialization of the receiver has been performed and it is ready to receive a video stream. The AMD Network Transport waits for an acknowledgement from the server before triggering this callback, other transport implementations may employ different methods to achieve this.
Callback invoked when a receiver requests a bitrate change for a video stream.
Callback invoked when a receiver requests a framerate change for a video stream.
Callback invoked when a receiver requests a resolution change for a video stream.
Callback invoked when a receiver requests a keyframe for the video stream.
Callback invoked when a Quality of Service (QoS) policy triggers a bitrate change for a video stream.
Callback invoked when a QoS policy triggers a framerate change for a video stream.
Callback invoked when a QoS policy triggers a resolution change for a video stream.
The server application must implement this callback to receive the video receiver pipeline statistics from the client and video sender pipeline statistics running locally on the server. It also receives notifications when the server receives origin timestamps from the client used to calculate round-trip latency.
Callback invoked when the server receives a message with client's statistics from a specific client identified by the session parameter for a specific video stream identified by the streamID parameter. The statistical parameters are contained inside the stats structure.
Callback invoked when the server receives a frame origin timestamp used to calculate the roundtrip latency.
Since the client and the server run in independent time domains, their local timestamps cannot be compared to each other. Therefore, the roundtrip latency is calculated using the following algorithm:
- The the client sends a timestamp in its local time domain, called the Origin timestamp to the server periodically. The frequency of these timestamps must exceed the framerate for accurate measurements.
- The server renders a frame and attaches the most current origin timestamp available at the moment before the frame is rendered to the frame.
- The origin timestamp is transmitted to the client along with the frame.
- When the frame is decoded, post-processed and is ready to be presented, the client obtains another timestamp in its local time domain. Since both the origin timestamp and the current timestamp are in the same time domain (client), they can be directly compared and the difference between them represents the full roundtrip latency.
Both the server and the client can send and receive video data to and from each other. For example, the server can stream its desktop to the client, while the client could stream video from a webcam. VideoReceiverCallback notifies the server application about events pertaining to the server's video receiver function. The ClientTrabsport::VideoReceiverCallback interface is very similar to the ServerTransport::VideoReceiverCallback with the main distinction being that no session ID is passed to it as a parameter since only a single connection is maintained from the client side.
bool OnVideoInit(const char* codec, StreamID streamID, int64_t initID, const AMFSize& streamResolution, const AMFRect& viewport, uint32_t bitDepth, bool stereoscopic, bool foveated, const void* initBlock, size_t initBlockSize)
Callback invoked when the video codec initialization block is received. Perform video decoder initialization.
Callback invoked when a video frame is received from the server.
Callback invoked when the server's video statistics are updated.
Callback invoked when the server subscribes to an audio stream.
Callback invoked when the server unsubscribes from an audio stream.
Callback invoked when the server is ready to receive an audio stream.
The ClientTransport::AudioReceiverCallback interface is the client's counterpart of the ServerTransport::AudioReceiverCallback interface and behaves similarly to the ClientTrabsport::VideoReceiverCallback interface.
bool OnAudioInit(const char* codec, StreamID streamID, int64_t initID, uint32_t channels, uint32_t layout, uint32_t samplingRate, amf::AMF_AUDIO_FORMAT format, const void* initBlock, size_t initBlockSize)
Callback invoked when the audio codec initialization block is received.
Callback invoked when an audio buffer is received.
The InputControllerCallback notifies the client of controller-related events originating on the server that need to be reflected on the client. This includes mouse movements on the server or haptic events sent to the client.
Callback invoked when an input controller is enabled on the server.
Callback invoked when an input controller is disabled on the server.
Callback invoked when an input event is received from a controller. Usually this
The CursorCallback notifies the client of the changes to the mouse cursor occuring on the client, such as changes to the cursor bitmap and cursor being hidden or shown.
Callback invoked when the cursor is updated. The cursor parameter contains the new cursor bitmap
Callback invoked when the cursor is hidden. Client should remove the mouse cursor from the screen
Callback invoked when a custom application message is received.