View Source Membrane.LiveCompositor (Membrane LiveCompositor Plugin v0.10.0)

Membrane SDK for LiveCompositor.

Input streams

Each input pad has a format Pad.ref(:video_input, input_id) or Pad.ref(:audio_input, input_id), where input_id is a string. input_id needs to be unique for all input pads, in particular you can't have audio and video input pads with the same id.

See Membrane.LiveCompositor.Lifecycle for input stream lifecycle notifications.

Output streams

Each output pad has a format Pad.ref(:video_output, output_id) or Pad.ref(:audio_output, output_id), where output_id is a string. output_id needs to be unique for all output pads, in particular you can't have audio and video output pads with the same id.

After registering and linking an output stream the LiveCompositor will notify the parent with output_registered/0.

Composition specification - video

To specify what LiveCompositor should render you can:

  • Define initial option when connecting :video_output pad.
  • Send Request.UpdateVideoOutput notification to update a scene on an already connected pad.

For example, code snippet bellow will update content of a stream from Pad.ref(:video_output, "output_0") to include streams from Pad.ref(:video_input, "input_0") and Pad.ref(:video_input, "input_1") side by side using Tiles component.

scene_update_request =  %Request.UpdateVideoOutput{
  output_id: "output_0"
  root: %{
    type: :tiles,
    children: [
      { type: "input_stream", input_id: "input_0" },
      { type: "input_stream", input_id: "input_1" }
    ]
  }
}

{[notify_child: {:live_compositor, scene_update_request}]}

:root option specifies root component of a scene that will be rendered on the output stream. Concept of a component is explained here. For actual component definitions see LiveCompositor documentation e.g. View, Tiles, ...

Composition specification - audio

  • Define initial option when connecting :audio_output pad.
  • Send Request.UpdateAudioOutput notification to audio composition on an already connected pad.

For example, code snippet bellow will update content of a stream from Pad.ref(:video_output, "output_0") to include streams from Pad.ref(:video_input, "input_0") and Pad.ref(:video_input, "input_1") mixed together, where "input_0" is mixed at half volume.

audio_update_request =  %Request.UpdateAudioOutput{
  output_id: "output_0"
  inputs: [
    { input_id: "input_0", volume: 0.5 },
    { input_id: "input_1" }
  ]
}

{[notify_child: {:live_compositor, audio_update_request}]}

Notifications

LiveCompositor bin can send following notifications to the parent process.

LiveCompositor documentation

This documentation covers mostly Elixr/Membrane specific API. For platform/language independent topics that are not covered here check LiveCompositor documentation.

Bin options

Passed via struct Membrane.LiveCompositor.t/0

  • framerate

    Membrane.RawVideo.framerate_t()

    Required
    Framerate of LiveCompositor outputs.

  • output_sample_rate

    output_sample_rate()

    Default value: 48000
    Sample rate of audio on LiveCompositor outputs.

  • api_port

    :inet.port_number() | port_range()

    Default value: 8081
    Port number or port range where API of a LiveCompositor will be hosted.

  • stream_fallback_timeout

    Membrane.Time.t()

    Default value: 500000000
    Timeout that defines when the LiveCompositor should switch to fallback on the input stream that stopped sending frames.

  • composing_strategy

    :real_time_auto_init | :real_time | :offline_processing

    Default value: :real_time_auto_init
    Specifies LiveCompositor mode for composing frames:

    • :real_time - Frames are produced at a rate dictated by real time clock. The parent process has to send :start_composing message to start.

    • :real_time_auto_init - The same as :real_time, but the pipeline starts automatically and sending :start_composing message is not necessary.

    • :offline_processing

      • Output streams will be produced faster than in real time if input streams are ready.

      • Never drop output frames, even if the encoder or rendering process is not able to process data in real time.

        When using this option, make sure to register the output stream before starting; otherwise, the compositor will run in a busy loop processing data far into the future.

  • server_setup

    :already_started | :start_locally | {:start_locally, path :: String.t()}

    Default value: :start_locally
    Defines how the LiveCompositor bin should start-up a LiveCompositor server.

    Available options:

    • :start_locally - LC server is automatically started.
    • {:start_locally, path} - LC server is automatically started, but different binary is used to spawn the process.
    • :already_started - LiveCompositor bin assumes, that LC server is already started and is available on a specified port. When this option is selected, the api_port option need to specify an exact port number (not a range).
  • init_requests

    list(Request.t())

    Default value: []
    Requests that will be sent on startup to the LC server. It's main use case is to register renderers that will be needed in the scene from the very beginning.

    Example:

    [%Request.RegisterShader{
      shader_id: "example_shader_1",
      source: "<shader sources>"
    }]

Pads

:audio_input

Accepted formats:

%Opus{self_delimiting?: false}
%RemoteStream{type: :packetized, content_format: Opus}
%RemoteStream{type: :packetized, content_format: nil}
Direction::input
Availability::on_request

Pad options:

  • required

    boolean()

    Default value: false
    If stream is marked required the LiveCompositor will delay processing new frames until frames are available. In particular, if there is at least one required input stream and the encoder is not able to produce frames on time, the output stream will also be delayed. This delay will happen regardless of whether required input stream was on time or not.

  • offset

    Membrane.Time.t() | nil

    Default value: nil
    An optional offset used for stream synchronization. This value represents how PTS values of the stream are shifted relative to the start request. If not defined streams are synchronized based on the delivery times of initial frames.

  • port

    :inet.port_number() | port_range()

    Default value: {10000, 60000}
    Port number or port range.

    Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

:video_input

Accepted formats:

%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction::input
Availability::on_request

Pad options:

  • required

    boolean()

    Default value: false
    If stream is marked required the LiveCompositor will delay processing new frames until frames are available. In particular, if there is at least one required input stream and the encoder is not able to produce frames on time, the output stream will also be delayed. This delay will happen regardless of whether required input stream was on time or not.

  • offset

    Membrane.Time.t() | nil

    Default value: nil
    An optional offset used for stream synchronization. This value represents how PTS values of the stream are shifted relative to the start request. If not defined streams are synchronized based on the delivery times of initial frames.

  • port

    :inet.port_number() | port_range()

    Default value: {10000, 60000}
    Port number or port range.

    Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

:audio_output

Accepted formats:

%RemoteStream{type: :packetized, content_format: Opus}
Direction::output
Availability::on_request

Pad options:

  • port

    :inet.port_number() | port_range()

    Default value: {10000, 60000}
    Port number or port range.

    Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

  • encoder

    Encoder.Opus.t()

    Required

  • send_eos_when

    send_eos_condition()

    Default value: nil
    Condition for automatically finishing output stream in response to end of input streams.

    • {:any_of, input_ids} - End the output stream if any of the inputs from the list finished or if they don't exist.
    • {:all_of, input_ids} - End the output stream if all of the inputs from the list finished or if they don't exist.
    • :any_input - End the output stream when any input stream finishes.
    • :all_inputs - End the output stream when all of the input streams have finished. This also includes a case where no inputs are were ever connected.
  • initial

    any()

    Required
    Initial audio mixer configuration that will be produced on this output.

    Example:

    %{
      inputs: [
        %{ input_id: "input_0" },
        %{ input_id: "input_0", volume: 0.5 }
      ]
    }

    To change the scene after the registration you can send %Request.UpdateAudioOutput{} notification.

:video_output

Accepted formats:

%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction::output
Availability::on_request

Pad options:

  • port

    :inet.port_number() | port_range()

    Default value: {10000, 60000}
    Port number or port range.

    Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

  • width

    non_neg_integer()

    Required

  • height

    non_neg_integer()

    Required

  • encoder

    Encoder.FFmpegH264.t()

    Required

  • send_eos_when

    send_eos_condition()

    Default value: nil
    Condition for automatically finishing output stream in response to end of input streams.

    • {:any_of, input_ids} - End the output stream if any of the inputs from the list finished or if they don't exist.
    • {:all_of, input_ids} - End the output stream if all of the inputs from the list finished or if they don't exist.
    • :any_input - End the output stream when any input stream finishes.
    • :all_inputs - End the output stream when all of the input streams have finished. This also includes a case where no inputs are were ever connected.
  • initial

    any()

    Required
    Initial scene that will be rendered on this output.

    Example:

    %{
      root: %{
        type: :view,
        children: [
          %{ type: :input_stream, input_id: "input_0" }
        ]
      }
    }

    To change the scene after the registration you can send %Request.UpdateVideoOutput{} notification.

    Format of the :root field is documented here. For specific options see documentation pages for each component e.g. View, Tiles, ...

Summary

Types

Options for pad :audio_input

Options for pad :audio_output

Input stream id, uniquely identifies an input pad.

Output stream id, uniquely identifies an output pad.

Supported output sample rates.

Range of ports.

Condition that defines when output stream should end depending on the EOS received on inputs.

t()

Struct containing options for Membrane.LiveCompositor

Options for pad :video_input

Options for pad :video_output

Functions

Returns description of options available for this module

Types

Link to this type

audio_input_pad_opts()

View Source
@type audio_input_pad_opts() :: [
  required: boolean(),
  offset: Membrane.Time.t() | nil,
  port: :inet.port_number() | port_range()
]

Options for pad :audio_input

Link to this type

audio_output_pad_opts()

View Source
@type audio_output_pad_opts() :: [
  port: :inet.port_number() | port_range(),
  encoder: Membrane.LiveCompositor.Encoder.Opus.t(),
  send_eos_when: send_eos_condition(),
  initial: any()
]

Options for pad :audio_output

@type input_id() :: String.t()

Input stream id, uniquely identifies an input pad.

@type output_id() :: String.t()

Output stream id, uniquely identifies an output pad.

@type output_sample_rate() :: 8000 | 12000 | 16000 | 24000 | 48000

Supported output sample rates.

@type port_range() ::
  {lower_bound :: :inet.port_number(), upper_bound :: :inet.port_number()}

Range of ports.

@type send_eos_condition() ::
  nil
  | :any_input
  | :all_inputs
  | {:any_of, [input_id()]}
  | {:all_of, [input_id()]}

Condition that defines when output stream should end depending on the EOS received on inputs.

@type t() :: %Membrane.LiveCompositor{
  api_port: :inet.port_number() | port_range(),
  composing_strategy: :real_time_auto_init | :real_time | :offline_processing,
  framerate: Membrane.RawVideo.framerate_t(),
  init_requests: [Membrane.LiveCompositor.Request.t()],
  output_sample_rate: output_sample_rate(),
  server_setup:
    :already_started | :start_locally | {:start_locally, path :: String.t()},
  stream_fallback_timeout: Membrane.Time.t()
}

Struct containing options for Membrane.LiveCompositor

Link to this type

video_input_pad_opts()

View Source
@type video_input_pad_opts() :: [
  required: boolean(),
  offset: Membrane.Time.t() | nil,
  port: :inet.port_number() | port_range()
]

Options for pad :video_input

Link to this type

video_output_pad_opts()

View Source
@type video_output_pad_opts() :: [
  port: :inet.port_number() | port_range(),
  width: non_neg_integer(),
  height: non_neg_integer(),
  encoder: Membrane.LiveCompositor.Encoder.FFmpegH264.t(),
  send_eos_when: send_eos_condition(),
  initial: any()
]

Options for pad :video_output

Functions

@spec options() :: keyword()

Returns description of options available for this module