View Source Toolbox.Incident (toolbox v5.4.10)

Module extends Toolbox.Workflow and abstracts how regular incident behaves.

Wraps around Toolbox.Workflow and adds some additional callbacks to manage incident processing. This module works very much like regular workflow, but some additional properties can be specified. Contrary to a regular workflow, this automatically generates output actions to manage the incident data in the database and therefore syncs the general state of this workflow with the persistent representation of the incident.

Start by creating a definition (see new/0, add_transition/2 and build/1) which describes the workflow of the incident. Then, you can create a new instance based on that definition with new_instance/7.

The documentation mentions several data structures containing similar data but serving different purposes. These are:

  • params - keyword list passed to the functions new_instance/7 and add_transition/2. These parameters are used by workflow or by callback functions.
  • attributes - params which are not handled by the workflow. The workflow uses specific keys from params but you can add other keys, which become attributes. The workflow handles the following keys of the params: from, to, when, then, severity, subject, description, description_before, description_after, actors, side_effects, update_history_entry, update_possible_transition, user_actions, upsert_attributes. If params contains keys not listed above, they are put into the attributes. The attributes are part of the transition data. They can be used inside callbacks via transition parameter for any purpose. See example in add_transition/2 documentation how to use attributes.
  • state - is a map of incident working data.
    • It contains system keys used by the workflow engine and scenario specific keys added from state parameter of the new_instance/7 function.
    • Data from state are available inside callback functions and can be inserted with interpolation into text attributes (description, subject, description_before, description_after).
    • state is a part of incident instance so any code using this instance can read and write state data. It means the state can be used as a working data storage of the incident instance.
  • status - is a string representing the state of the underlying incident workflow state machine. The documentation uses the term status to refer to the state of the state machine. The term state is used to refer to the working data of an incident instance. Using two different terms avoids ambiguity.
  • additional_attributes - is part of the incident related output actions.
    • additional_attributes are maintained with upsert_attributes callback. See new_instance/7 or add_transition/2 for more details about upsert_attributes callback.
    • additional_attributes is part of the incident data stored in reality network so user can see it in user interface. This is the main purpose of the additional_attributes. See example in add_transition/2 documentation how to use additional_attributes and upsert_attributes callback.

Example

The following example shows simple usage of Toolbox.Incident module.

defmodule SimpleIncident do
  alias Toolbox.Incident

  def ok?(_, _, message) do
    message.type == :ok
  end

  def timeout?(_, _, message) do
    message.type == :timeout
  end

  def definition do
    Incident.new()
    |> Incident.add_transition(
      from: "open",
      to: "timeout",
      when: [{__MODULE__, :timeout?}],
      description_before: "Timeout will elapse",
      description_after: "Timeout elapsed"
    )
    |> Incident.add_transition(
      from: "timeout",
      to: "closed",
      when: [{__MODULE__, :ok?}],
      description_before: "Incident will be closed after timeout",
      description_after: "Incident was closed after timeout"
    )
    |> Incident.add_transition(
      from: "open",
      to: "closed",
      when: [{__MODULE__, :ok?}],
      description_before: "Incident will be closed",
      description_after: "Incident was closed"
    )
    |> Incident.build()
  end

  def new(definition, message) do
    Incident.new_instance(
      definition,
      "open",
      "incident_type",
      "id",
      %{},
      message,
      severity: 2,
      subject: "Incident on {{message.body.asset_id}}",
      description: "Incident was detected"
    )
  end
end

Usage:

Definition of the incident:

{:ok, definition} = SimpleIncident.definition

Creation of the new incident:

{:ok, oas, instance} = SimpleIncident.new(definition, %Runbox.Message{type: :ko, timestamp: 0, body: %{asset_id: "/assets/asset/a"}})

The function above returns the following result:

{:ok,
  [
    %Runbox.Scenario.OutputAction.Incident{
      type: "incident_type",
      id: "id",
      subject: "Incident on /assets/asset/a",
      status: "open",
      resolved: false,
      severity: 2,
      future: [
        %Runbox.Scenario.OutputAction.IncidentFuture{
          status: "timeout",
          severity: 2,
          timestamp: -1,
          description: "Timeout will elapse"
        },
        %Runbox.Scenario.OutputAction.IncidentFuture{
          status: "closed",
          severity: 2,
          timestamp: -1,
          description: "Incident will be closed"
        }
      ],
      history: [
        %Runbox.Scenario.OutputAction.IncidentHistory{
          status: "open",
          severity: 2,
          timestamp: 0,
          description: "Incident was detected",
          attributes: %{},
          event_type: nil,
          event_actors: nil,
          event_params: nil,
          event_origin_messages: nil
        }
      ],
      actors: [],
      user_actions: %{},
      additional_attributes: %{}
    }
  ],
  %Toolbox.Workflow.Instance{
    id: "id",
    type: "incident_type",
    state: %{
      :key => :value,
      "_prev_user_actions" => nil,
      "_user_actions" => %{},
      "severity" => 2
    },
    last_update: 0,
    status: "open",
    history: [
      %{
        "attributes" => %{},
        "description" => "Incident was detected",
        "severity" => 2,
        "status" => "open",
        "timestamp" => 0
      }
    ],
    possible_transitions: [
      %{
        "description" => "Timeout will elapse",
        "severity" => 2,
        "status" => "timeout",
        "timestamp" => -1
      },
      %{
        "description" => "incident will be closed",
        "severity" => 2,
        "status" => "closed",
        "timestamp" => -1
      }
    ],
    next_possible_transition_timestamp: nil,
    terminated?: false
 }}

The first member of the returned tuple is a result code.

The second member is the list of output actions created by function new_instance/7 which is called from SimpleIncident.new/2. The output actions consists of:

  • Output action which creates incident in reality network. Incident data is based on parameters of the new_instance/7 and input message.
    • The future is a list containing statuses which can be reached from status state.
    • The history is a list containing single item built from data passed to function new_instance/7.
  • possible other output actions created by callbacks.

The third member is the instance of the workflow. Similarly to output actions it also contains history created from data passed to function new_instance/7. It also contains initial state in status and list possible_transitions containing statuses reachable from initial status. possible_transitions can be manipulated using update_possible_transition callback. See new_instance/7 or add_transition/2 for more details about this callback.

Walking through incident workflow using messages:

{:ok, oas, instance} = Toolbox.Incident.handle_message(definition, instance, %Runbox.Message{type: :timeout, timestamp: 0, body: %{}})

{:terminated, oas, instance} = Toolbox.Incident.handle_message(definition, instance, %Runbox.Message{type: :ok, timestamp: 0, body: %{}})

Summary

Functions

Adds a new transition to incident workflow definition.

Finishes incident definition, validates all configured dependencies and incident structure. See Toolbox.Workflow.build/1 for more details.

Uses given incident workflow definition and message to update state of given instance.

Creates new blank incident workflow definition

Creates new incident instance for given workflow.

Functions

Link to this function

add_transition(wf, params)

View Source
@spec add_transition(Toolbox.Workflow.t(), Keyword.t()) :: Toolbox.Workflow.t()

Adds a new transition to incident workflow definition.

Incident workflow is a finite state machine.

The documentation uses the term status to refer to the state of the state machine. The term state is used to refer to the working data of an incident instance. Using two different terms avoids ambiguity.

Transition defines how workflow changes its status from one to another and what must be done during the status change. Incident transition is defined by params parameter. params is a keyword list where each transition parameter has its key.

If add_transition/2 succeeds it returns a Toolbox.Workflow.Transition.t/0 structure.

Parameters passed via the params parameter are either required or optional. Some parameters have reserved keys with predefined function. These predefined parameters are used to control workflow. All predefined keys are explained bellow:

  • from - the status of the workflow from which the transition starts.

    • required parameter
  • to - the status of the workflow after transition is completed.

    • required parameter
  • description_before - transition from from status to to status generates an output action. The output action contains simplified list of all possible statuses that can be reached from from status. The list is stored in body.attributes["future"]. Each entry of the list is a map. The description before is assigned to description key of that map. The description is intended to provide the human readable description of the conditions leading to the particular status.

    • required parameter
  • description_after - the aforementioned output action contains a list of all status changes that the workflow has already passed through. That list is stored in body.attributes["history']. Each entry of the list is a map containing details of the status. The description_after is copied to the "description" key of the map. The description is intended to provide human readable description about the past status change.

    • required parameter
  • severity - an integer from 1 to 4. The severity of the incident after transition to to status.

    • optional parameter
  • when - The when parameter contains a list of conditions that must be fulfilled to use this transition. The incident workflow responds to the incoming messages or elapsed timeouts. If message arrives or timeout elapses, the incident workflow goes through the transitions in the order in which they were defined. If it finds a transition where from matches the current status of the incident and all the conditions defined by the when parameter are met, it changes the incident state to the status specified by the to parameter and performs all actions and changes defined by the other parameters. The conditions are defined according to the following rules.

    • optional parameter
    • possible conditions definitions are:
      • {Module, function}, where function accepts transition, instance and message as args, and returns a boolean.
      • {:timeout, timeout}, where timeout is defined in milliseconds. Transition is automatically executed when timeout elapses.
      • {:=, [path, to, state, key], value} transition is executed if the specified field of state equals specified value. Similarly for the following condition definitions.
      • {:>, [path, to, state, key], value}
      • {:<, [path, to, state, key], value}
      • {:<=, [path, to, state, key], value}
      • {:>=, [path, to, state, key], value}
      • {:contains, [path, to, state, key], value}
      • {:is_in, [path, to, state, key], list_value}
    • If incident workflow reaches the transition with undefined when while searching for next transition from the from state, then workflow generates all the output actions defined by the transition and terminates.
  • then - List of callback definitions used to update workflow instance state during transition execution.

    • optional parameter
    • there can be multiple callback definitions in the list. Callbacks are executed in order they are listed.
    • callback definition:
      • {Module, function}, where function accepts transition, instance and message as args, and returns:
        • {:ok, state()} to update the instance state.
        • {:error, reason} - if any callback function returns error then state will not be updated.
  • side_effects - list of callback definitions used to generate side effects during transition execution.

    • optional parameter
    • there can be multiple callback definitions in list. Callbacks are executed in order they are listed.
    • callback definition:
      • {Module, function}, where function accepts transition, instance and message as args, and returns {:ok, [OA | OtherSideEffect]}
  • update_history_entry - list of callback definitions used to modify the entry of the transition in history of the incident.

    • optional parameter.
    • there can be multiple callback definitions in the list. Callbacks are executed in order they are listed.
    • this is usually used to interpolate description texts, or to add additional attributes to history
    • You can use this callback to set additional event parameters - event_* available in Runbox.Scenario.OutputAction.IncidentHistory (the keys are strings in this case).
    • callback definition:
      • {Module, function}, where function accepts history entry, transition, instance and message as args and returns {:ok, history_entry}
  • update_possible_transition - list of callback definitions used to modify possible future transitions of the incident.

    • optional parameter
    • there can be multiple callback definitions in the list. Callbacks are executed in the order they are listed.
    • the callback is executed for each possible future transition. Possible transitions are all transitions reachable from the status identified by from parameter.
    • note this only modifies the items of future attribute of the incident instance and output action. It has no effect on definition of the transitions.
    • callback definition:
      • {Module, function}, where function accepts possible transition, current transition, instance and message as args, and returns {:ok, future_transition}. Possible transition is a structure similar to Runbox.Scenario.OutputAction.IncidentFuture with keys "description", "severity", "status" and "timestamp". The callback function can modify value of any of the keys.
  • user_actions - specifies all possible user actions from the target state.

    • a map of user actions that should be enabled once the transition is executed and incident is in the target state.
    • optional parameter
    • user actions are automatically deleted if their key is not present in the next transition
      • keys are strings and are used to distinguish user actions on an incident from each other.
    • values are {module, function}, this specifies the function to be called to generate the user action token (since tokens are not known in advance, they are generated by the specified function)
      • the function takes transition, instance, message as arguments and is expected to return {:ok, binary_token} to register the user action token
      • the token should be generated using Runbox.Scenario.UserAction.pack/3.
  • upsert_attributes - list of callback definitions to add and maintain special attributes of the incident stored under the key additional_attributes. These attributes are stored in reality network and are visible to the user.

    • optional parameter
    • each callback produces a map of additional attributes and this is merged into a single map where the latter has priority over the former
    • callbacks can create and maintain only attributes in the additional_attributes attribute.
    • callback definition:
      • {Module, function}, where function accepts transition, instance and message as args, and should return {:ok, attribute_map}

    Any other keys of the params parameter not listed in the list above are collected into the map under the attribute key of the Toolbox.Workflow.Transition.t/0 data structure. They can be used inside callback functions for any purpose.

    Example of usage of not reserved key for debugging:

    Code snippets

    def definition do
      Incident.new()
      |> Incident.add_transition(
        from: "detected",
        to: "rare",
        when: [{__MODULE__, :rare_message_arrived?}],
        description_before: "Rare messages will arrive",
        description_after: "Rare message arrived",
        update_history_entry: [{__MODULE__, :update_history_entry}],
        my_debug_key: "This should be a very rare transition"
        )
        ...
      |> Incident.build()
    end
    
    def update_history_entry(history, transition, _workflow_instance, message) do
      Logger.info(transition.attributes.my_debug_key)
      ...
      {:ok, new_history}
    end

When a message is evaluated the callbacks above are run in the following order.

  1. when callbacks are evaluated to see if the current transition is ready to be executed. If not the next transition is tried.
  2. then callbacks are evaluated to update the instance state.
  3. user_actions is evaluated, all callbacks specified inside are executed and all user action tokens are calculated.
  4. update_history_entry callbacks are evaluated to update the new history entry
  5. update_possible_transition callbacks are evaluated to update the new possible future transitions
  6. upsert_attributes callbacks are evaluated to gather additional attributes
  7. side_effects callbacks are evaluated to acquire the list of all additional output actions

All text bearing attributes (such as subject, description_before, description_after) have access to incident metadata dictionary. This dictionary contains these, keys:

  • type - incident type
  • id - incident id
  • transition - transition attributes dictionary containing from, to, severity keys
  • state - dictionary containing user defined state
  • message - Runbox.Message which triggered given transition

Metadata can be accessed via interpolation defined as {{}}, e.g. {{state.foo.bar}}, {{message.body.foo}}. There is also an option to reference assets in those attributes, see Runbox.Scenario.OutputAction.interpolable/0.

@spec build(Toolbox.Workflow.t()) ::
  {:ok, Toolbox.Workflow.t()}
  | {:error, :transition_from_required}
  | {:error, :transition_to_required}
  | {:error, :description_after_required}
  | {:error, :description_before_required}
  | {:error, {:bad_callback, {atom(), atom()}}}
  | {:error, :multiple_init_statuses}
  | {:error,
     {:user_actions_invalid | :upsert_attributes_invalid, reason :: String.t()}}

Finishes incident definition, validates all configured dependencies and incident structure. See Toolbox.Workflow.build/1 for more details.

Link to this function

handle_message(wf, inc_inst, msg)

View Source
@spec handle_message(
  Toolbox.Workflow.t(),
  Toolbox.Workflow.Instance.t(),
  Runbox.Message.t()
) ::
  {:ok, [Runbox.Scenario.OutputAction.oa_params()],
   Toolbox.Workflow.Instance.t()}
  | {:terminated, [Runbox.Scenario.OutputAction.oa_params()],
     Toolbox.Workflow.Instance.t()}
  | {:error, :not_built_yet}
  | {:error, :status_mismatch}

Uses given incident workflow definition and message to update state of given instance.

If no configured workflow transition matches, nothing will happen = instance state will remain the same.

Order of callback execution:

  1. when definitions of transitions in definition order
  2. then definitions of matching transition
  3. user_actions definitions of matching transition
  4. update_history_entry definitions of matching transition
  5. update_possible_transition definitions of matching transition
  6. upsert_attributes definitions of matching transition
  7. side_effects definitions of matching transition
@spec new() :: Toolbox.Workflow.t()

Creates new blank incident workflow definition

Link to this function

new_instance(definition, status, type, id, state, msg, params)

View Source
@spec new_instance(
  Toolbox.Workflow.t(),
  Toolbox.Workflow.status(),
  String.t(),
  String.t(),
  map(),
  Runbox.Message.t(),
  Keyword.t()
) ::
  {:ok, [Runbox.Scenario.OutputAction.oa_params()],
   Toolbox.Workflow.Instance.t()}
  | {:terminated, [Runbox.Scenario.OutputAction.oa_params()],
     Toolbox.Workflow.Instance.t()}
  | {:error, :unknown_status}
  | {:error,
     {:user_actions_invalid | :upsert_attributes_invalid, reason :: String.t()}}

Creates new incident instance for given workflow.

The function receives following arguments:

  • definition - the definition of the workflow.

  • status - the initial status of the incident. It is a string representing the status of the underlying workflow state machine. It is copied to the output action uder keys:

    • status.
    • history.status of the first entry of the history. The documentation uses the term status to refer to the state of the state machine. The term state is used to refer to the working data of an incident instance. Using two different terms avoids ambiguity.
  • type - the incident type. It is copied to the output action uder key type.

  • id - the unique identifier of an incident. It is copied to the output action uder key id. id and type together reference an incident.

  • state - the initial state of the incident.

  • message - the input message that triggered the incident.

  • params - the following parameters are handled by the workflow.

    • severity - is an integer number representing the initial severity of the incident.
      • optional parameter.
      • if not defined, defaults to 1.
      • it is copied to the output action under two different keys:
        • severity.
        • history.severity of the first entry of the history.
    • subject - the short text description which is copied to the output action under the key subject.
      • optional parameter.
      • defaults to "Incident [incident_type] / [incident_id]" string
    • description - the initial description of the incident. It is copied into the history.description key of the output action.
      • optional parameter.
      • if not defined defaults to text "Incident was created".
    • actors - list of actors used when an incident is created.
      • optional key.
      • defaults to empty list.
      • It is a list of maps. Each map has two keys:
        • :type - type of the asset ID representing actor. Required key.
        • :id - last part of the asset ID representing actor. Required key.
    • side_effects - list of callback definitions used to generate side effects during transition execution.
      • optional parameter
      • there can be multiple callback definitions in the list. Callbacks are executed in the order they are listed in the list.
      • callback definition:
        • {Module, function}, where function accepts transition, instance and message as args, and returns {:ok, [OA | OtherSideEffect]}
    • update_history_entry - list of callback definitions used to modify the entry of the transition in history of the incident.
      • optional parameter
      • there can be multiple callback definitions in the list. All callbacks are executed in the order they are listed in the list.
      • this is usually used to interpolate description texts, or to add additional attributes to history
      • the callback can be used to set additional event parameters - event_* available in Runbox.Scenario.OutputAction.IncidentHistory (the keys are strings in this case).
      • callback definition:
        • {Module, function}, where function accepts history entry, transition, instance and message as args and returns {:ok, history_entry}
    • update_possible_transition - callback definitions used to modify possible future transitions.
      • optional parameter
      • there can be multiple callback definitions in the list. All callbacks are executed in the order they are listed in the list.
      • the callback is executed for each possible future transition. Possible transitions are all transitions reachable from the status identified by status parameter.
      • note this only modifies the items of future attribute of the incident instance and output action. It has no effect on definition of the transitions.
      • callback definition:
        • {Module, function}, where function accepts possible transition, current transition, instance and message as args, and returns {:ok, future_transition}. Possible transition is a structure similar to Runbox.Scenario.OutputAction.IncidentFuture with keys "description", "severity", "status" and "timestamp". The callback function can modify value of any of the keys.
    • user_actions - a map of user actions that should be enabled once the transition is executed and incident is in the target status defined by parameter to.
      • optional parameter.
      • user actions are automatically deleted if not present in the next transition.
      • keys are strings and are used to distinguish user actions on an incident from each other.
      • values are callback definitions {module, function}. Callback definition specifies the function to be called to generate the user action token. Since tokens are not known in advance, they are generated by the specified function.
        • the function takes transition, instance, message as arguments and is expected to return {:ok, binary_token} to register the user action token.
        • the token should be generated using Runbox.Scenario.UserAction.pack/3.
    • upsert_attributes - List of callback definitions to add and maintain special attributes of the incident stored under the key additional_attributes. These attributes are stored in reality network and are visible to the user.
      • optional parameter.
      • each callback produces a map of additional attributes and this is merged into a single map where the latter has priority over the former.
      • callbacks can create and maintain only attributes in the additional_attributes attribute.
      • possible callback definitions:
        • {Module, function}, where function accepts transition, instance and message as args, and should return {:ok, attribute_map}.

    Code snippets:

    def new(definition, message) do
      Incident.new_instance(
        definition,
        "open",
        "incident_type",
        "id",
        %{key: :value},
        message,
        severity: 2,
        subject: "Incident on {{message.body.asset_id}}",
        description: "Incident was detected",
        upsert_attributes: [{__MODULE__, :upsert_attribute}]
      )
    end
    
    def upsert_attribute(_transition, _workflow_instance, message) do
      {:ok, %{"user_attribute" => message.body.some_attribute}}
    end

All text bearing attributes (such as subject, description) have access to incident metadata dictionary. This dictionary contains these, keys:

  • type - incident type
  • id - incident id
  • transition - transition attributes dictionary containing from, to, severity keys
  • state - map containing user defined state
  • message - Runbox.Message which triggered given transition

Metadata can be accessed via interpolation defined as {{}}, e.g. {{state.foo.bar}}, {{message.body.foo}}. There is also an option to reference assets in those attributes, see Runbox.Scenario.OutputAction.interpolable/0.