Skip to content

Implementing Video Transmitter Pipelines

Jump to bottom
Kolesnik, Gennadiy edited this page Nov 25, 2024 · 1 revision

For general information about transmitter pipelines refer to the Transmitter Pipelines section section.

Video Transmitter Pipeline Architecture

The video transmitter pipeline enables efficient streaming of video data to multiple subscribed clients by processing frames through a sequence of video conversion and encoding steps, ultimately packaging the output for network transmission. This pipeline is encapsulated in the ssdk::video::MonoscopicVideoOutput class, which handles color space conversion, scaling, and encoding, and finally sends encoded frames to clients via the VideoTransmitterAdapter.

Core Components

MonoscopicVideoOutput

The ssdk::video::MonoscopicVideoOutput class serves as the main interface for preprocessing and encoding video frames. It provides:

  • Input Frame Submission: Frames are submitted to MonoscopicVideoOutput through the SubmitInput() method.
  • Pipeline Processing: Frames undergo color space conversion, optional scaling, and encoding within the pipeline, preparing them for transmission to clients.

Pipeline Components

The video transmitter pipeline consists of the following main components, each responsible for a critical step in video processing:

1. Video Converter (AMFVideoConverter)

The Video Converter, an AMF component, prepares video frames for encoding by performing necessary color space conversions and scaling as follows:

  • Color Space Conversion: Converts frames from RGB (or another format) to YUV, the color space expected by the video encoder.
  • Scaling: Adjusts the resolution of frames to match encoding or bandwidth requirements when needed. This helps ensure compatibility with different client display settings and optimizes resource usage.

Note that the Video Converter component is instantiated only when:

  • the resolution of the input frames is different from the resolution of the encoded stream, and
  • when the format and color characteristics of the input surfaces are not compatible with the video encoder input.

Whenever possible, match the input resolution of the surfaces submitted to MonoscopicVideoOutput to the resolution of the encoded stream. The AMF VideoConverter component performs scaling and color space conversions using shaders and may interfere with a high intensity graphics content, such as a game, that is being rendered on the same GPU. When avoiding scaling is not possible, prefer the Direct3D12 API on Windows as it allows for a more efficient GPU job scheduling, to minimize interference.

2. Video Encoder

After conversion, frames are passed to the encoder, which compresses video data using one of the following codecs:

  • Supported Codecs:

    • H.264: A widely compatible codec for most client devices and network conditions.
    • HEVC (H.265): A more efficient codec for lower bitrates with higher quality, suited for clients supporting HEVC decoding.
    • AV1: The latest codec option, offering even better compression but requiring client support for AV1 decoding.

  • Encoding Efficiency: Each codec is selected based on network conditions, client capabilities, and quality requirements, balancing video quality with data size to optimize streaming performance.

VideoTransmitterAdapter

The VideoTransmitterAdapter receives encoded frames from MonoscopicVideoOutput and manages the client-side transmission. This component:

  • Client Subscription Management: Maintains a list of clients subscribed to the video stream, ensuring they receive video data based on their network and device capabilities.
  • Network Transmission:
    • SendVideoInit: Called by VideoTransmitterAdapter to initialize each client’s video stream, sending information about the codec, resolution, and frame rate.
    • SendVideoFrame: Sends encoded frames to each client in the subscription list, ensuring synchronized delivery to all participants.

Data Flow and Processing in the Transmitter Pipeline

  1. Input Frame Submission: Frames are submitted to MonoscopicVideoOutput via SubmitInput().
  2. Sequential Processing: The frames are processed through the Video Converter and then passed to the encoder.
  3. Client Transmission:
    • Initialization: VideoTransmitterAdapter uses SendVideoInit to establish the video stream with each subscribed client.
    • Frame Delivery: Encoded frames are sent to clients using SendVideoFrame, ensuring efficient data flow and synchronized delivery.
Clone this wiki locally