View Source Membrane.RTP.SessionBin (Membrane RTP plugin v0.17.0)

Bin handling one RTP session, that may consist of multiple incoming and outgoing RTP streams.

incoming-streams

Incoming streams

Incoming RTP streams can be connected via :rtp_input pads. As each pad can provide multiple RTP streams, they are distinguished basing on SSRC. Once a new stream is received, bin sends new_stream_notification_t/0 notification, meaning the parent should link Pad.ref(:output, ssrc) pad to consuming components. The stream is then depayloaded and forwarded via said pad.

outgoing-streams

Outgoing streams

To create an RTP stream, the source stream needs to be connected via Pad.ref(:input, ssrc) pad and the sink - via Pad.ref(:rtp_output, ssrc). At least one of :encoding or :payload_type options of :rtp_output pad must be provided too.

payloaders-and-depayloaders

Payloaders and depayloaders

Payloaders are Membrane elements that transform stream so that it can be put into RTP packets, while depayloaders work the other way round. Different codecs require different payloaders and depayloaders.

By default SessionBin will neither payload nor depayload incoming/outgoing streams, to do so a payloader/depayloader needs to be passed via Pad.ref(:input, ssrc) and Pad.ref(:output, ssrc) pads options.

Payloading/Depayloading is necessary if we need to somehow transform the streams. If SessionBins main role is to route packets then depayloading and payloading processes are redundant.

Payloaders and depayloaders can be found in membrane_rtp_X_plugin packages, where X stands for codec name. It's enough when such a plugin is added to dependencies. To determine which payloader/depayloader to use, one can use Membrane.RTP.PayloadFormatResolver which given an encoding name should resolve to proper payloader/depayloader modules (if those previously have been registered via mentioned plugins).

For payloading and depayloading, SessionBin uses respective bins Membrane.RTP.PayloaderBin and Membrane.RTP.DepayloaderBin which will be spawned once payloader/depayloader are passed explicitly via pads' options.

Important note

Payloaders and depayloaders are mostly needed when working with external media sources (in different formats than RTP). For applications such as an SFU it is not needed to either payload or depayload the RTP stream as we are always dealing with RTP format. In such a case, SessionBin will receive payloaded packets and work as a simple proxy just forwarding the packets (and decrypting them if necessary). Therefore it is possible to specify in newly added pads if payloaders/depayloaders should be used for the certain stream.

padding

Padding

Addition and removal of a padding from RTP packets is handled by the RTP plugin.

To send a packet with a padding, one should include padding_size field in Membrane.Buffer's metadata.

E.g.

metadata = %{rtp: %{padding_size: 20}}

will result in adding 20 bytes of padding at the end of packet's payload.

When parsing an RTP stream, a padding is stripped out and the padding_size field is set appropriately to the number of bytes that were removed.

For more information, please refer to the RFC 3550, sec. 5

rtcp

RTCP

RTCP packets for inbound stream can be provided either in-band or via a separate rtp_input pad instance. Corresponding receiver report packets will be sent back through rtcp_receiver_output with the same id as rtp_input for the RTP stream.

RTCP for outbound stream is not yet supported. # But will be :)

bin-options

Bin options

Passed via struct Membrane.RTP.SessionBin.t/0

  • fmt_mapping

    %{RTP.payload_type_t() => {RTP.encoding_name_t(), RTP.clock_rate_t()}}

    Default value: %{}
    Mapping of the custom payload types ( > 95)

  • rtcp_receiver_report_interval

    Membrane.Time.t() | nil

    Default value: nil
    Interval between sending subseqent RTCP receiver reports.

  • rtcp_sender_report_interval

    Membrane.Time.t() | nil

    Default value: nil
    Interval between sending subseqent RTCP sender reports.

  • receiver_ssrc_generator

    (local_ssrcs :: [pos_integer], remote_ssrcs :: [pos_integer] -> ssrc :: pos_integer)

    Default value: &Membrane.RTP.SessionBin.generate_receiver_ssrc/2
    Function generating receiver SSRCs. Default one generates random SSRC that is not in local_ssrcs nor remote_ssrcs.

  • secure?

    boolean

    Default value: false
    Specifies whether to use SRTP. Requires adding srtp dependency to work.

  • srtp_policies

    [ExLibSRTP.Policy.t()]

    Default value: []
    List of SRTP policies to use for decrypting packets. Used only when secure? is set to true. See ExLibSRTP.Policy.t/0 for details.

  • receiver_srtp_policies

    [ExLibSRTP.Policy.t()] | nil

    Default value: nil
    List of SRTP policies to use for encrypting receiver reports and other receiver RTCP packets. Used only when secure? is set to true. Defaults to the value of srtp_policies. See ExLibSRTP.Policy.t/0 for details.

pads

Pads

rtp_input

:rtp_input

Availability :on_request
Caps Membrane.RemoteStream, restrictions:
  type: :packetized,
  content_format: one_of([nil, Membrane.RTP])
Demand unit :buffers
Direction :input
Mode :pull
Name :rtp_input

input

:input

Availability :on_request
Caps :any
Demand unit :buffers
Direction :input
Mode :pull
Name :input

Options:

  • payloader

    module() | nil

    Default value: nil
    Payloader's module that should be used for a media stream flowing through the pad.

    If set to nil then the payloading process gets skipped.

  • rtp_extensions

    [rtp_extension_options_t()]

    Default value: []
    List of RTP extension options. RTP plugin ships with the following extensions for input pad:

    • :twcc (sender) - it will tag outgoing packets with transport-wide sequence numbers and estimate available bandwidth. For input pad, TWCC sender can only be spawned. For more information refer to Membrane.RTP.TWCCSender module documentation.

    There is no possibility to pass user-defined RTP extenions for input pad.

    Examples:

    • {:twcc, Mebrane.RTP.TWCCSender}

rtcp_sender_output

:rtcp_sender_output

Availability :on_request
Caps Membrane.RemoteStream, restrictions:
  type: :packetized,
  content_format: Membrane.RTCP
Demand unit :buffers
Direction :output
Mode :pull
Name :rtcp_sender_output

rtcp_receiver_output

:rtcp_receiver_output

Availability :on_request
Caps Membrane.RemoteStream, restrictions:
  type: :packetized,
  content_format: Membrane.RTCP
Demand unit :buffers
Direction :output
Mode :pull
Name :rtcp_receiver_output

rtp_output

:rtp_output

Availability :on_request
Caps Membrane.RemoteStream, restrictions:
  type: :packetized,
  content_format: Membrane.RTP
Demand unit :buffers
Direction :output
Mode :pull
Name :rtp_output

Options:

  • payload_type

    RTP.payload_type_t() | nil

    Default value: nil
    Payload type of output stream. If not provided, determined from :encoding.

  • encoding

    RTP.encoding_name_t() | nil

    Default value: nil
    Encoding name of output stream. If not provided, determined from :payload_type.

  • clock_rate

    integer() | nil

    Default value: nil
    Clock rate to use. If not provided, determined from :payload_type.

  • rtp_extension_mapping

    rtp_extension_mapping_t()

    Default value: nil
    Mapping from locally used rtp_extension_name_t() to integer identifiers expected by the receiver of a RTP stream.

output

:output

Availability :on_request
Caps :any
Demand unit :buffers
Direction :output
Mode :pull
Name :output

Options:

  • depayloader

    module() | nil

    Default value: nil
    Depayloader's module that should be used for an outgoing media stream flowing through the pad.

    If set to nil then the depayloading process gets skipped.

  • telemetry_label

    Membrane.TelemetryMetrics.label()

    Default value: []
    Label passed to Membrane.TelemetryMetrics functions

  • encoding

    RTP.encoding_name_t() | nil

    Default value: nil

  • clock_rate

    integer() | nil

    Default value: nil
    Clock rate to use. If not provided, determined from fmt_mapping or defaults registered by proper plugins i.e. Membrane.RTP.X.Plugin where X is the name of codec corresponding to encoding.

  • rtp_extensions

    [rtp_extension_options_t()]

    Default value: []
    List of RTP extension options. RTP plugin ships with the following RTP extensions for output pad:

    • :vad will turn on Voice Activity Detection mechanism firing appropriate notifications when needed. Should be set only for audio tracks. For more information refer to Membrane.RTP.VAD module documentation.
    • :twcc (receiver) will gather transport-wide information about received packets and generate feedbacks for sender. For output pad, TWCC receiver can only be spawned. For more information refer to Membrane.RTP.TWCCReceiver module documentation.

    User can also pass its own RTP extensions for output pad.

    RTP extensions (except :twcc) are applied in the same order as passed to the pad options.

    Examples:

    • {:vad, %Mebrane.RTP.VAD{vad_id: 1, time_window: 1_000_000}}
    • {:twcc, %Mebrane.RTP.TWCCReceiver{twcc_id: 1, report_interval: Membrane.Time.milliseconds(250)}}
  • extensions

    [extension_t()]

    Default value: []
    A list of general extensions that will be attached to the packets flow (added inside Membrane.RTP.StreamReceiveBin). In case of SRTP extensions are placed before the Decryptor. The order of provided elements is important as the extensions are applied in FIFO order.

    An extension can be responsible e.g. for dropping silent audio packets when encountered VAD extension data in the packet header.

Link to this section Summary

Types

A definition of a general extension inside Membrane.RTP.StreamReceiveBin. Each extension should have just a single input and output pad named accordingly.

Options for pad :input

Options for pad :output

A mapping between internally used rtp_extension_name_t() and extension identifiers expected by RTP stream receiver.

An atom that identifies an RTP extension in the bin. It will be used by the module implementing it to mark its header extension under Membrane.RTP.Header.Extension's identifier key.

A module representing an RTP extension that will be spawned and linked just after a newly created :input pad or before a newly created :output pad representing a single RTP stream.

Options for pad :rtp_output

t()

Struct containing options for Membrane.RTP.SessionBin

Functions

Returns pads descriptions for Membrane.RTP.SessionBin

Returns description of options available for this module

Link to this section Types

A definition of a general extension inside Membrane.RTP.StreamReceiveBin. Each extension should have just a single input and output pad named accordingly.

Extensions can implement different functionalities, for example a filter can be responsible for dropping silent audio packets when encountered VAD extension data in header extensions of a packet.

@type input_pad_opts_t() :: [
  payloader: module() | nil,
  rtp_extensions: [rtp_extension_options_t()]
]

Options for pad :input

Link to this type

new_stream_notification_t()

View Source
@type new_stream_notification_t() ::
  Membrane.RTP.SSRCRouter.new_stream_notification_t()
@type output_pad_opts_t() :: [
  depayloader: module() | nil,
  telemetry_label: Membrane.TelemetryMetrics.label(),
  encoding: Membrane.RTP.encoding_name_t() | nil,
  clock_rate: integer() | nil,
  rtp_extensions: [rtp_extension_options_t()],
  extensions: [extension_t()]
]

Options for pad :output

Link to this type

rtp_extension_mapping_t()

View Source
@type rtp_extension_mapping_t() :: %{required(rtp_extension_name_t()) => 1..14}

A mapping between internally used rtp_extension_name_t() and extension identifiers expected by RTP stream receiver.

Link to this type

rtp_extension_name_t()

View Source
@type rtp_extension_name_t() :: atom()

An atom that identifies an RTP extension in the bin. It will be used by the module implementing it to mark its header extension under Membrane.RTP.Header.Extension's identifier key.

Link to this type

rtp_extension_options_t()

View Source
@type rtp_extension_options_t() ::
  {extension_name :: rtp_extension_name_t(),
   extension_config :: Membrane.ParentSpec.child_spec_t()}

A module representing an RTP extension that will be spawned and linked just after a newly created :input pad or before a newly created :output pad representing a single RTP stream.

Given extension config must be a valid Membrane.Filter.

An extension will be spawned inside the bin under {extension_name, ssrc} name.

rtp-plugin-ships-with-the-following-extensions

RTP plugin ships with the following extensions:

twcc

TWCC

TWCC as a transport-wide extension is handled differently, and is linked from RTP.SSRCRouter to possibly many RTP.StreamReceiveBins. Only the first TWCC extension is initialized, and it will handle all RTP streams that have declared support for it. For outgoing streams, an RTP.TWCCSender element will be spawned and linked to all RTP.StreamSendBins.

Link to this type

rtp_output_pad_opts_t()

View Source
@type rtp_output_pad_opts_t() :: [
  payload_type: Membrane.RTP.payload_type_t() | nil,
  encoding: Membrane.RTP.encoding_name_t() | nil,
  clock_rate: integer() | nil,
  rtp_extension_mapping: rtp_extension_mapping_t()
]

Options for pad :rtp_output

@type t() :: %Membrane.RTP.SessionBin{
  fmt_mapping: %{
    required(Membrane.RTP.payload_type_t()) =>
      {Membrane.RTP.encoding_name_t(), Membrane.RTP.clock_rate_t()}
  },
  receiver_srtp_policies: [ExLibSRTP.Policy.t()] | nil,
  receiver_ssrc_generator:
    (local_ssrcs :: [pos_integer()], remote_ssrcs :: [pos_integer()] ->
       ssrc :: pos_integer()),
  rtcp_receiver_report_interval: Membrane.Time.t() | nil,
  rtcp_sender_report_interval: Membrane.Time.t() | nil,
  secure?: boolean(),
  srtp_policies: [ExLibSRTP.Policy.t()]
}

Struct containing options for Membrane.RTP.SessionBin

Link to this section Functions

@spec membrane_pads() :: [{Membrane.Pad.name_t(), Membrane.Pad.description_t()}]

Returns pads descriptions for Membrane.RTP.SessionBin

@spec options() :: keyword()

Returns description of options available for this module