View Source Membrane.LiveCompositor (Membrane LiveCompositor Plugin v0.10.0)
Membrane SDK for LiveCompositor.
Input streams
Each input pad has a format Pad.ref(:video_input, input_id)
or Pad.ref(:audio_input, input_id)
,
where input_id
is a string. input_id
needs to be unique for all input pads, in particular
you can't have audio and video input pads with the same id.
See Membrane.LiveCompositor.Lifecycle
for input stream lifecycle notifications.
Output streams
Each output pad has a format Pad.ref(:video_output, output_id)
or Pad.ref(:audio_output, output_id)
,
where output_id
is a string. output_id
needs to be unique for all output pads, in particular
you can't have audio and video output pads with the same id.
After registering and linking an output stream the LiveCompositor will notify the parent with
output_registered/0
.
Composition specification - video
To specify what LiveCompositor should render you can:
- Define
initial
option when connecting:video_output
pad. - Send
Request.UpdateVideoOutput
notification to update a scene on an already connected pad.
For example, code snippet bellow will update content of a stream from Pad.ref(:video_output, "output_0")
to include streams from Pad.ref(:video_input, "input_0")
and Pad.ref(:video_input, "input_1")
side by side using Tiles
component.
scene_update_request = %Request.UpdateVideoOutput{
output_id: "output_0"
root: %{
type: :tiles,
children: [
{ type: "input_stream", input_id: "input_0" },
{ type: "input_stream", input_id: "input_1" }
]
}
}
{[notify_child: {:live_compositor, scene_update_request}]}
:root
option specifies root component of a scene that will be rendered on the output stream.
Concept of a component is explained here. For
actual component definitions see LiveCompositor documentation e.g.
View
,
Tiles
, ...
Composition specification - audio
- Define
initial
option when connecting:audio_output
pad. - Send
Request.UpdateAudioOutput
notification to audio composition on an already connected pad.
For example, code snippet bellow will update content of a stream from Pad.ref(:video_output, "output_0")
to include streams from Pad.ref(:video_input, "input_0")
and Pad.ref(:video_input, "input_1")
mixed together, where "input_0"
is mixed at half volume.
audio_update_request = %Request.UpdateAudioOutput{
output_id: "output_0"
inputs: [
{ input_id: "input_0", volume: 0.5 },
{ input_id: "input_1" }
]
}
{[notify_child: {:live_compositor, audio_update_request}]}
Notifications
LiveCompositor bin can send following notifications to the parent process.
Lifecycle.notification/0
- Notification about lifecycle of input/output streams.Request.result/0
- Result of aMembrane.LiveCompositor.Request
sent from the parent process.
LiveCompositor documentation
This documentation covers mostly Elixr/Membrane specific API. For platform/language independent topics that are not covered here check LiveCompositor documentation.
Bin options
Passed via struct Membrane.LiveCompositor.t/0
framerate
Membrane.RawVideo.framerate_t()
Required
Framerate of LiveCompositor outputs.output_sample_rate
output_sample_rate()
Default value:
48000
Sample rate of audio on LiveCompositor outputs.api_port
:inet.port_number() | port_range()
Default value:
8081
Port number or port range where API of a LiveCompositor will be hosted.stream_fallback_timeout
Membrane.Time.t()
Default value:
500000000
Timeout that defines when the LiveCompositor should switch to fallback on the input stream that stopped sending frames.composing_strategy
:real_time_auto_init | :real_time | :offline_processing
Default value:
:real_time_auto_init
Specifies LiveCompositor mode for composing frames::real_time
- Frames are produced at a rate dictated by real time clock. The parent process has to send:start_composing
message to start.:real_time_auto_init
- The same as:real_time
, but the pipeline starts automatically and sending:start_composing
message is not necessary.:offline_processing
Output streams will be produced faster than in real time if input streams are ready.
Never drop output frames, even if the encoder or rendering process is not able to process data in real time.
When using this option, make sure to register the output stream before starting; otherwise, the compositor will run in a busy loop processing data far into the future.
server_setup
:already_started | :start_locally | {:start_locally, path :: String.t()}
Default value:
:start_locally
Defines how the LiveCompositor bin should start-up a LiveCompositor server.Available options:
:start_locally
- LC server is automatically started.{:start_locally, path}
- LC server is automatically started, but different binary is used to spawn the process.:already_started
- LiveCompositor bin assumes, that LC server is already started and is available on a specified port. When this option is selected, theapi_port
option need to specify an exact port number (not a range).
init_requests
list(Request.t())
Default value:
[]
Requests that will be sent on startup to the LC server. It's main use case is to register renderers that will be needed in the scene from the very beginning.Example:
[%Request.RegisterShader{ shader_id: "example_shader_1", source: "<shader sources>" }]
Pads
:audio_input
Accepted formats:
%Opus{self_delimiting?: false}
%RemoteStream{type: :packetized, content_format: Opus}
%RemoteStream{type: :packetized, content_format: nil}
Direction: | :input |
Availability: | :on_request |
Pad options:
required
boolean()
Default value:
false
If stream is marked required the LiveCompositor will delay processing new frames until frames are available. In particular, if there is at least one required input stream and the encoder is not able to produce frames on time, the output stream will also be delayed. This delay will happen regardless of whether required input stream was on time or not.offset
Membrane.Time.t() | nil
Default value:
nil
An optional offset used for stream synchronization. This value represents how PTS values of the stream are shifted relative to the start request. If not defined streams are synchronized based on the delivery times of initial frames.port
:inet.port_number() | port_range()
Default value:
{10000, 60000}
Port number or port range.Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.
:video_input
Accepted formats:
%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction: | :input |
Availability: | :on_request |
Pad options:
required
boolean()
Default value:
false
If stream is marked required the LiveCompositor will delay processing new frames until frames are available. In particular, if there is at least one required input stream and the encoder is not able to produce frames on time, the output stream will also be delayed. This delay will happen regardless of whether required input stream was on time or not.offset
Membrane.Time.t() | nil
Default value:
nil
An optional offset used for stream synchronization. This value represents how PTS values of the stream are shifted relative to the start request. If not defined streams are synchronized based on the delivery times of initial frames.port
:inet.port_number() | port_range()
Default value:
{10000, 60000}
Port number or port range.Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.
:audio_output
Accepted formats:
%RemoteStream{type: :packetized, content_format: Opus}
Direction: | :output |
Availability: | :on_request |
Pad options:
port
:inet.port_number() | port_range()
Default value:
{10000, 60000}
Port number or port range.Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.
encoder
Encoder.Opus.t()
Required
send_eos_when
send_eos_condition()
Default value:
nil
Condition for automatically finishing output stream in response to end of input streams.{:any_of, input_ids}
- End the output stream if any of the inputs from the list finished or if they don't exist.{:all_of, input_ids}
- End the output stream if all of the inputs from the list finished or if they don't exist.:any_input
- End the output stream when any input stream finishes.:all_inputs
- End the output stream when all of the input streams have finished. This also includes a case where no inputs are were ever connected.
initial
any()
Required
Initial audio mixer configuration that will be produced on this output.Example:
%{ inputs: [ %{ input_id: "input_0" }, %{ input_id: "input_0", volume: 0.5 } ] }
To change the scene after the registration you can send
%Request.UpdateAudioOutput{}
notification.
:video_output
Accepted formats:
%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction: | :output |
Availability: | :on_request |
Pad options:
port
:inet.port_number() | port_range()
Default value:
{10000, 60000}
Port number or port range.Internally LiveCompositor server communicates with this pipeline locally over RTP. This value defines which TCP ports will be used.
width
non_neg_integer()
Required
height
non_neg_integer()
Required
encoder
Encoder.FFmpegH264.t()
Required
send_eos_when
send_eos_condition()
Default value:
nil
Condition for automatically finishing output stream in response to end of input streams.{:any_of, input_ids}
- End the output stream if any of the inputs from the list finished or if they don't exist.{:all_of, input_ids}
- End the output stream if all of the inputs from the list finished or if they don't exist.:any_input
- End the output stream when any input stream finishes.:all_inputs
- End the output stream when all of the input streams have finished. This also includes a case where no inputs are were ever connected.
initial
any()
Required
Initial scene that will be rendered on this output.Example:
%{ root: %{ type: :view, children: [ %{ type: :input_stream, input_id: "input_0" } ] } }
To change the scene after the registration you can send
%Request.UpdateVideoOutput{}
notification.Format of the
:root
field is documented here. For specific options see documentation pages for each component e.g.View
,Tiles
, ...
Summary
Types
Options for pad :audio_input
Options for pad :audio_output
Input stream id, uniquely identifies an input pad.
Output stream id, uniquely identifies an output pad.
Supported output sample rates.
Range of ports.
Condition that defines when output stream should end depending on the EOS received on inputs.
Struct containing options for Membrane.LiveCompositor
Options for pad :video_input
Options for pad :video_output
Types
@type audio_input_pad_opts() :: [ required: boolean(), offset: Membrane.Time.t() | nil, port: :inet.port_number() | port_range() ]
Options for pad :audio_input
@type audio_output_pad_opts() :: [ port: :inet.port_number() | port_range(), encoder: Membrane.LiveCompositor.Encoder.Opus.t(), send_eos_when: send_eos_condition(), initial: any() ]
Options for pad :audio_output
@type input_id() :: String.t()
Input stream id, uniquely identifies an input pad.
@type output_id() :: String.t()
Output stream id, uniquely identifies an output pad.
@type output_sample_rate() :: 8000 | 12000 | 16000 | 24000 | 48000
Supported output sample rates.
@type port_range() :: {lower_bound :: :inet.port_number(), upper_bound :: :inet.port_number()}
Range of ports.
@type send_eos_condition() :: nil | :any_input | :all_inputs | {:any_of, [input_id()]} | {:all_of, [input_id()]}
Condition that defines when output stream should end depending on the EOS received on inputs.
@type t() :: %Membrane.LiveCompositor{ api_port: :inet.port_number() | port_range(), composing_strategy: :real_time_auto_init | :real_time | :offline_processing, framerate: Membrane.RawVideo.framerate_t(), init_requests: [Membrane.LiveCompositor.Request.t()], output_sample_rate: output_sample_rate(), server_setup: :already_started | :start_locally | {:start_locally, path :: String.t()}, stream_fallback_timeout: Membrane.Time.t() }
Struct containing options for Membrane.LiveCompositor
@type video_input_pad_opts() :: [ required: boolean(), offset: Membrane.Time.t() | nil, port: :inet.port_number() | port_range() ]
Options for pad :video_input
@type video_output_pad_opts() :: [ port: :inet.port_number() | port_range(), width: non_neg_integer(), height: non_neg_integer(), encoder: Membrane.LiveCompositor.Encoder.FFmpegH264.t(), send_eos_when: send_eos_condition(), initial: any() ]
Options for pad :video_output
Functions
@spec options() :: keyword()
Returns description of options available for this module