Bin handling one RTP session, that may consist of multiple incoming and outgoing RTP streams.

Incoming streams

Incoming RTP streams can be connected via :rtp_input pads. As each pad can provide multiple RTP streams, they are distinguished basing on SSRC. Once a new stream is received, bin sends new_stream_notification_t/0 notification, meaning the parent should link Pad.ref(:output, ssrc) pad to consuming components. The stream is then depayloaded and forwarded via said pad.

Outgoing streams

To create an RTP stream, the source stream needs to be connected via Pad.ref(:input, ssrc) pad and the sink - via Pad.ref(:rtp_output, ssrc). At least one of :encoding or :payload_type options of :rtp_output pad must be provided too.

Payloaders and depayloaders

Payloaders are Membrane elements that transform stream so that it can be put into RTP packets, while depayloaders work the other way round. Different codecs require different payloaders and depayloaders.

By default SessionBin will neither payload nor depayload incoming/outgoing streams, to do so a payloader/depayloader needs to be passed via Pad.ref(:input, ssrc) and Pad.ref(:output, ssrc) pads options.

Payloading/Depayloading is necessary if we need to somehow transform the streams. If SessionBins main role is to route packets then depayloading and payloading processes are redundant.

Payloaders and depayloaders can be found in membrane_rtp_X_plugin packages, where X stands for codec name. It's enough when such a plugin is added to dependencies. To determine which payloader/depayloader to use, one can use Membrane.RTP.PayloadFormatResolver which given an encoding name should resolve to proper payloader/depayloader modules (if those previously have been registered via mentioned plugins).

For payloading and depayloading, SessionBin uses respective bins Membrane.RTP.PayloaderBin and Membrane.RTP.DepayloaderBin which will be spawned once payloader/depayloader are passed explicitly via pads' options.

Important note

Payloaders and depayloaders are mostly needed when working with external media sources (in different formats than RTP). For applications such as an SFU it is not needed to either payload or depayload the RTP stream as we are always dealing with RTP format. In such a case, SessionBin will receive payloaded packets and work as a simple proxy just forwarding the packets (and decrypting them if necessary). Therefore it is possible to specify in newly added pads if payloaders/depayloaders should be used for the certain stream.

Padding

Addition and removal of a padding from RTP packets is handled by the RTP plugin.

To send a packet with a padding, one should include padding_size field in Membrane.Buffer's metadata.

E.g.

metadata = %{rtp: %{padding_size: 20}}

will result in adding 20 bytes of padding at the end of packet's payload.

When parsing an RTP stream, a padding is stripped out and the padding_size field is set appropriately to the number of bytes that were removed.

For more information, please refer to the RFC 3550, sec. 5

RTCP

RTCP packets for inbound stream can be provided either in-band or via a separate rtp_input pad instance. Corresponding receiver report packets will be sent back through rtcp_receiver_output with the same id as rtp_input for the RTP stream.

RTCP sender reports from inbound streams are parsed and then dispatched as Membrane.RTCP.SenderReportPacket events on session bin :output pads. On each Pad.ref(:output, ssrc), only RTCP sender reports with the corresponding SSRC will be sent. These reports can be used, for instance, to synchronize different rtp streams.

RTCP packets for the outbound stream are available on rtcp_sender_output pad. RTCP packets will only be produced if rtcp_sender_report_interval option is set to a value other than nil.

Bin options

Passed via struct Membrane.RTP.SessionBin.t/0

• fmt_mapping

  %{RTP.payload_type_t() => {RTP.encoding_name_t(), RTP.clock_rate_t()}}

  Default value: %{}
  Mapping of the custom payload types ( > 95)

• rtcp_receiver_report_interval

  Membrane.Time.t() | nil

  Default value: nil
  Interval between sending subseqent RTCP receiver reports.

• rtcp_sender_report_interval

  Membrane.Time.t() | nil

  Default value: nil
  Interval between sending subseqent RTCP sender reports.

• receiver_ssrc_generator

  (local_ssrcs :: [pos_integer], remote_ssrcs :: [pos_integer] -> ssrc :: pos_integer)

  Default value: &Membrane.RTP.SessionBin.generate_receiver_ssrc/2
  Function generating receiver SSRCs. Default one generates random SSRC that is not in local_ssrcs nor remote_ssrcs.

• secure?

  any

  Default value: false
  Specifies whether to use SRTP. Requires adding srtp dependency to work.

• srtp_policies

  [ExLibSRTP.Policy.t()]

  Default value: []
  List of SRTP policies to use for decrypting packets. Used only when secure? is set to true. See ExLibSRTP.Policy.t/0 for details.

• receiver_srtp_policies

  [ExLibSRTP.Policy.t()] | nil

  Default value: nil
  List of SRTP policies to use for encrypting receiver reports and other receiver RTCP packets. Used only when secure? is set to true. Defaults to the value of srtp_policies. See ExLibSRTP.Policy.t/0 for details.

Pads

:rtp_input

Accepted formats:

%RemoteStream{type: :packetized, content_format: cf} when cf in [RTP, RTCP, nil]

:input

Accepted formats:

_any

Pad options:

• payloader

  module() | nil

  Default value: nil
  Payloader's module that should be used for a media stream flowing through the pad.

  If set to nil then the payloading process gets skipped.

• rtp_extensions

  [rtp_extension_options_t()]

  Default value: []
  List of RTP extension options. RTP plugin ships with the following extensions for input pad:

  • :twcc (sender) - it will tag outgoing packets with transport-wide sequence numbers and estimate available bandwidth. For input pad, TWCC sender can only be spawned. For more information refer to Membrane.RTP.TWCCSender module documentation.

  There is no possibility to pass user-defined RTP extenions for input pad.

  Examples:

  • {:twcc, Mebrane.RTP.TWCCSender}

• telemetry_label

  Membrane.TelemetryMetrics.label()

  Default value: []

:rtcp_sender_output

Accepted formats:

%RemoteStream{type: :packetized, content_format: RTCP}

:rtcp_receiver_output

Accepted formats:

%RemoteStream{type: :packetized, content_format: RTCP}

:rtp_output

Accepted formats:

%RemoteStream{type: :packetized, content_format: RTP}

Pad options:

• payload_type

  RTP.payload_type_t() | nil

  Default value: nil
  Payload type of output stream. If not provided, determined from :encoding.

• encoding

  RTP.encoding_name_t() | nil

  Default value: nil
  Encoding name of output stream. If not provided, determined from :payload_type.

• clock_rate

  integer() | nil

  Default value: nil
  Clock rate to use. If not provided, determined from :payload_type.

• rtp_extension_mapping

  rtp_extension_mapping_t()

  Default value: nil
  Mapping from locally used rtp_extension_name_t() to integer identifiers expected by the receiver of a RTP stream.

:output

Accepted formats:

_any

Pad options:

• depayloader

  module() | nil

  Default value: nil
  Depayloader's module that should be used for an outgoing media stream flowing through the pad.

  If set to nil then the depayloading process gets skipped.

• telemetry_label

  Membrane.TelemetryMetrics.label()

  Default value: []
  Label passed to Membrane.TelemetryMetrics functions

• encoding

  RTP.encoding_name_t() | nil

  Default value: nil

• clock_rate

  integer() | nil

  Default value: nil
  Clock rate to use. If not provided, determined from fmt_mapping or defaults registered by proper plugins i.e. Membrane.RTP.X.Plugin where X is the name of codec corresponding to encoding.

• rtp_extensions

  [rtp_extension_options_t()]

  Default value: []
  List of RTP extension options. RTP plugin ships with the following RTP extensions for output pad:

  • :vad will turn on Voice Activity Detection mechanism firing appropriate notifications when needed. Should be set only for audio tracks. For more information refer to Membrane.RTP.VAD module documentation.
  • :twcc (receiver) will gather transport-wide information about received packets and generate feedbacks for sender. For output pad, TWCC receiver can only be spawned. For more information refer to Membrane.RTP.TWCCReceiver module documentation.

  User can also pass its own RTP extensions for output pad.

  RTP extensions (except :twcc) are applied in the same order as passed to the pad options.

  Examples:

  • {:vad, %Mebrane.RTP.VAD{vad_id: 1, time_window: 1_000_000}}
  • {:twcc, %Mebrane.RTP.TWCCReceiver{twcc_id: 1, report_interval: Membrane.Time.milliseconds(250)}}

• extensions

  [extension_t()]

  Default value: []
  A list of general extensions that will be attached to the packets flow (added inside Membrane.RTP.StreamReceiveBin). In case of SRTP extensions are placed before the Decryptor. The order of provided elements is important as the extensions are applied in FIFO order.

  An extension can be responsible e.g. for dropping silent audio packets when encountered VAD extension data in the packet header.