Data Streams Ex
v1.2.2

View Source Datadog.DataStreams (Data Streams Ex v1.2.2)

This is a port of the data-streams-go library to Elixir.

Configuration

Configuration can be done via the Config module and your config/ files. We use OpenTelemetry configured values first, so if you are already using OpenTelemetry to instrument your application, your service name and environment should already be set.

import Config

config :opentelemetry, :resource,
  name: "my-elixir-service",
  env: "production"

If you are not using OpenTelemetry, you can set the service, environment, and primary tag via the :data_streams application like so:

import Config

config :data_streams, :metadata,
  service: "my-elixir-service",
  env: "production",
  primary_tag: "datacenter:d1"

Once that is configured, you will also want to set the configuration for accessing the Datadog agent. This can be done via:

import Config

config :data_streams, :agent,
  enabled?: true,
  host: "my-datadog-agent.local",
  port: 8125

By default, the agent is disabled and will not send data to Datadog. Without the host or port configured, we default to "localhost:8125".

For more information, view the Datadog.DataStreams.Config module.

Running

Once installed, if :dd_agent_streams :agent is enabled?, the Datadog.DataStreams.Aggregator will start automatically and start sending metrics. Just instrument your data pipelines!

Integrations

Telemetry

Similar to the golang implementation, we export a couple of :telemetry metrics that can (and should) be sent to Datadog. These can be done via how ever you are sending your application telemetry metrics to Datadog.

Available metrics include:

  • datadog.datastreams.aggregator.payloads_in.count - The number of Datadog.DataStreams.Aggregator.Points that were sent to the Datadog.DataStreams.Aggregator.

  • datadog.datastreams.aggregator.flushed_payloads.count - The number of successful payloads sent to Datadog.

  • datadog.datastreams.aggregator.flushed_buckets.count - The number of successful 10 second buckets sent to Datadog.

  • datadog.datastreams.aggregator.flush_errors.count - The number of failed requests to the Datadog agent (and dropped payloads).