Settings View Source OffBroadway.Splunk.Producer (off_broadway_splunk v1.2.0)

GenStage Producer for a Splunk Event Stream. Broadway producer acts as a consumer for the specified Splunk SID.

producer-options

Producer Options

• :sid - Required. The SID (Search ID) for the Splunk job we want to consume events from.

• :receive_interval (non_neg_integer/0) - The duration (in milliseconds) for which the producer waits before making a request for more messages. The default value is 5000.

• :shutdown_timeout (timeout/0) - The duration (in milliseconds) Broadway should wait before timing out when trying to stop the pipeline. The default value is :infinity.

• :on_success (atom/0) - Configures the acking behaviour for successful messages. See the "Acknowledgements" section below for all the possible values. The default value is :ack.

• :on_failure (atom/0) - Configures the acking behaviour for failed messages. See the "Acknowledgements" section below for all the possible values. The default value is :noop.

• :splunk_client - A module that implements the OffBroadway.Splunk.Client behaviour. This module is responsible for fetching and acknowledging the messages from Splunk. All options passed to the producer will also be forwarded to the client. The default value is OffBroadway.Splunk.SplunkClient.

• :config (keyword/0) - A set of config options that overrides the default config for the splunk_client module. Any option set here can also be configured in config.exs. The default value is [].

  • :base_url (String.t/0) - Base URL to Splunk instance.

  • :api_token (String.t/0) - API token used to authenticate on the Splunk instance.

  • :api_version - Some API endpoints are available in multiple versions. Sets the API version to use (where applicable). The default value is "v2".

  • :endpoint - Configures if output should be consumed from the events or results endpoint of the Splunk API. In Splunk terminology, results are the output after transformation commands has been executed. Consider the following search command:

        * | head 1000 | stats count by sourcetype
    
    The output from the command above is no longer the raw `events` because they have been
    transformed by the search command. Splunk call these transformed events `results`.
    To receive the computed values, you should choose to use the `results` endpoint.
    If you want to receive all events that produced this output, choose the `events` endpoint.

    The default value is :events.

  • :offset (integer/0) - Initial offset can be used to skip N before starting to consume messages. If using events endpoint, a negative value can be used to start consuming "from the end", where -1 is the last available offset. The default value is 0.

  • :max_events - If set to a positive integer, automatically shut down the pipeline after consuming max_events messages from the Splunk API.

acknowledgements

Acknowledgements

You can use the on_success and on_failure options to control how messages are acknowledged. You can set these options when starting the Splunk producer or change them for each message through Broadway.Message.configure_ack/2. By default, successful messages are acked (:ack) and failed messages are not (:noop).

The possible values for :on_success and :on_failure are:

• :ack - acknowledge the message. Splunk does not have any concept of acking messages, because we are just consuming messages from a web api endpoint. For now we are just executing a :telemetry event for acked messages.

• :noop - do not acknowledge the message. No action are taken.

telemetry

Telemetry

This library exposes the following telemetry events:

• [:off_broadway_splunk, :job_status, :start] - Dispatched before polling SID status from Splunk.

  • measurement: %{time: System.monotonic_time}
  • metadata: %{sid: string, progress: integer}

• [:off_broadway_splunk, :job_status, :stop] - Dispatched when polling SID status from Splunk is complete.

  • measurement: %{time: native_time}
  • metadata: %{sid: string, progress: integer}

• [:off_broadway_splunk, :job_status, :exception] - Dispatched after a failure while polling SID status from Splunk.

  • measurement: %{duration: native_time}

  • metadata:

    %{
      sid: string,
      kind: kind,
      reason: reason,
      stacktrace: stacktrace
    }

• [:off_broadway_splunk, :receive_messages, :start] - Dispatched before receiving messages from Splunk (c:receive_messages/2)

  • measurement: %{time: System.monotonic_time}
  • metadata: %{sid: string, demand: integer}

• [:off_broadway_splunk, :receive_messages, :stop] - Dispatched after messages have been received from Splunk and "wrapped".

  • measurement: %{time: native_time}

  • metadata:

    %{
      sid: string,
      received: integer,
      demand: integer
    }

• [:off_broadway_splunk, :receive_messages, :exception] - Dispatched after a failure while receiving messages from Splunk.

  • measurement: %{duration: native_time}

  • metadata:

    %{
      sid: string,
      demand: integer,
      kind: kind,
      reason: reason,
      stacktrace: stacktrace
    }

• [:off_broadway_splunk, :receive_messages, :ack] - Dispatched when acking a message.

  • measurement: %{time: System.system_time, count: 1}

  • meatadata:

    %{
      sid: string,
      receipt: receipt
    }