Membrane SDK for LiveCompositor.

Input streams

Each input pad has a format Pad.ref(:video_input, input_id) or Pad.ref(:audio_input, input_id), where input_id is a string. input_id needs to be unique for all input pads, in particular you can't have audio and video input pads with the same id.

See Membrane.LiveCompositor.Lifecycle for input stream lifecycle notifications.

Output streams

Each output pad has a format Pad.ref(:video_output, output_id) or Pad.ref(:audio_output, output_id), where output_id is a string. output_id needs to be unique for all output pads, in particular you can't have audio and video output pads with the same id.

After registering and linking an output stream the LiveCompositor will notify the parent with output_registered/0.

Composition specification - video

To specify what LiveCompositor should render you can:

• Define initial option when connecting :video_output pad.
• Send Request.UpdateVideoOutput notification to update a scene on an already connected pad.

For example, code snippet bellow will update content of a stream from Pad.ref(:video_output, "output_0") to include streams from Pad.ref(:video_input, "input_0") and Pad.ref(:video_input, "input_1") side by side using Tiles component.

scene_update_request =  %Request.UpdateVideoOutput{
  output_id: "output_0"
  root: %{
    type: :tiles,
    children: [
      { type: "input_stream", input_id: "input_0" },
      { type: "input_stream", input_id: "input_1" }
    ]
  }
}

{[notify_child: {:live_compositor, scene_update_request}]}

:root option specifies root component of a scene that will be rendered on the output stream. Concept of a component is explained here. For actual component definitions see LiveCompositor documentation e.g. View, Tiles, ...

Composition specification - audio

• Define initial option when connecting :audio_output pad.
• Send Request.UpdateAudioOutput notification to audio composition on an already connected pad.

For example, code snippet bellow will update content of a stream from Pad.ref(:video_output, "output_0") to include streams from Pad.ref(:video_input, "input_0") and Pad.ref(:video_input, "input_1") mixed together, where "input_0" is mixed at half volume.

audio_update_request =  %Request.UpdateAudioOutput{
  output_id: "output_0"
  inputs: [
    { input_id: "input_0", volume: 0.5 },
    { input_id: "input_1" }
  ]
}

{[notify_child: {:live_compositor, audio_update_request}]}

Notifications

LiveCompositor bin can send following notifications to the parent process.

• Lifecycle.notification/0 - Notification about lifecycle of input/output streams.
• Request.result/0 - Result of a Membrane.LiveCompositor.Request sent from the parent process.

LiveCompositor documentation

This documentation covers mostly Elixr/Membrane specific API. For platform/language independent topics that are not covered here check LiveCompositor documentation.

Bin options

Passed via struct Membrane.LiveCompositor.t/0

• framerate

  Membrane.RawVideo.framerate_t()

  Required
  Framerate of LiveCompositor outputs.

• output_sample_rate

  output_sample_rate()

  Default value: 48000
  Sample rate of audio on LiveCompositor outputs.

• api_port

  :inet.port_number() | port_range()

  Default value: 8081
  Port number or port range where API of a LiveCompositor will be hosted.

• stream_fallback_timeout

  Membrane.Time.t()

  Default value: 500000000
  Timeout that defines when the LiveCompositor should switch to fallback on the input stream that stopped sending frames.

• composing_strategy

  :real_time_auto_init | :real_time | :offline_processing

  Default value: :real_time_auto_init
  Specifies LiveCompositor mode for composing frames:

  • :real_time - Frames are produced at a rate dictated by real time clock. The parent process has to send :start_composing message to start.

  • :real_time_auto_init - The same as :real_time, but the pipeline starts automatically and sending :start_composing message is not necessary.

  • :offline_processing

    • Output streams will be produced faster than in real time if input streams are ready.

    • Never drop output frames, even if the encoder or rendering process is not able to process data in real time.

      When using this option, make sure to register the output stream before starting; otherwise, the compositor will run in a busy loop processing data far into the future.

• server_setup

  :already_started | :start_locally | {:start_locally, path :: String.t()}

  Default value: :start_locally
  Defines how the LiveCompositor bin should start-up a LiveCompositor server.

  Available options:

  • :start_locally - LC server is automatically started.
  • {:start_locally, path} - LC server is automatically started, but different binary is used to spawn the process.
  • :already_started - LiveCompositor bin assumes, that LC server is already started and is available on a specified port. When this option is selected, the api_port option need to specify an exact port number (not a range).

• init_requests

  list(Request.t())

  Default value: []
  Requests that will be sent on startup to the LC server. It's main use case is to register renderers that will be needed in the scene from the very beginning.

  Example:

  [%Request.RegisterShader{
    shader_id: "example_shader_1",
    source: "<shader sources>"
  }]

Pads

:audio_input

Accepted formats:

%Opus{self_delimiting?: false}

%RemoteStream{type: :packetized, content_format: Opus}

%RemoteStream{type: :packetized, content_format: nil}

Pad options:

• required

  boolean()

  Default value: false
  If stream is marked required the LiveCompositor will delay processing new frames until frames are available. In particular, if there is at least one required input stream and the encoder is not able to produce frames on time, the output stream will also be delayed. This delay will happen regardless of whether required input stream was on time or not.

• offset

  Membrane.Time.t() | nil

  Default value: nil
  An optional offset used for stream synchronization. This value represents how PTS values of the stream are shifted relative to the start request. If not defined streams are synchronized based on the delivery times of initial frames.

• port

  :inet.port_number() | port_range()

  Default value: {10000, 60000}
  Port number or port range.

  Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

:video_input

Accepted formats:

%Membrane.H264{alignment: :nalu, stream_structure: :annexb}

Pad options:

• required

  boolean()

  Default value: false
  If stream is marked required the LiveCompositor will delay processing new frames until frames are available. In particular, if there is at least one required input stream and the encoder is not able to produce frames on time, the output stream will also be delayed. This delay will happen regardless of whether required input stream was on time or not.

• offset

  Membrane.Time.t() | nil

  Default value: nil
  An optional offset used for stream synchronization. This value represents how PTS values of the stream are shifted relative to the start request. If not defined streams are synchronized based on the delivery times of initial frames.

• port

  :inet.port_number() | port_range()

  Default value: {10000, 60000}
  Port number or port range.

  Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

:audio_output

Accepted formats:

%RemoteStream{type: :packetized, content_format: Opus}

Pad options:

• port

  :inet.port_number() | port_range()

  Default value: {10000, 60000}
  Port number or port range.

  Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

• encoder

  Encoder.Opus.t()

  Required

• send_eos_when

  send_eos_condition()

  Default value: nil
  Condition for automatically finishing output stream in response to end of input streams.

  • {:any_of, input_ids} - End the output stream if any of the inputs from the list finished or if they don't exist.
  • {:all_of, input_ids} - End the output stream if all of the inputs from the list finished or if they don't exist.
  • :any_input - End the output stream when any input stream finishes.
  • :all_inputs - End the output stream when all of the input streams have finished. This also includes a case where no inputs are were ever connected.

• initial

  any()

  Required
  Initial audio mixer configuration that will be produced on this output.

  Example:

  %{
    inputs: [
      %{ input_id: "input_0" },
      %{ input_id: "input_0", volume: 0.5 }
    ]
  }

  To change the scene after the registration you can send %Request.UpdateAudioOutput{} notification.

:video_output

Accepted formats:

%Membrane.H264{alignment: :nalu, stream_structure: :annexb}

Pad options:

• port

  :inet.port_number() | port_range()

  Default value: {10000, 60000}
  Port number or port range.

  Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.

• width

  non_neg_integer()

  Required

• height

  non_neg_integer()

  Required

• encoder

  Encoder.FFmpegH264.t()

  Required

• send_eos_when

  send_eos_condition()

  Default value: nil
  Condition for automatically finishing output stream in response to end of input streams.

  • {:any_of, input_ids} - End the output stream if any of the inputs from the list finished or if they don't exist.
  • {:all_of, input_ids} - End the output stream if all of the inputs from the list finished or if they don't exist.
  • :any_input - End the output stream when any input stream finishes.
  • :all_inputs - End the output stream when all of the input streams have finished. This also includes a case where no inputs are were ever connected.

• initial

  any()

  Required
  Initial scene that will be rendered on this output.

  Example:

  %{
    root: %{
      type: :view,
      children: [
        %{ type: :input_stream, input_id: "input_0" }
      ]
    }
  }

  To change the scene after the registration you can send %Request.UpdateVideoOutput{} notification.

  Format of the :root field is documented here. For specific options see documentation pages for each component e.g. View, Tiles, ...