smart_city
v6.0.0

View Source SmartCity.Data (smart_city v6.0.0)

Message struct shared amongst all SmartCity microservices.

const DataMessage = {
    "dataset_ids": "",        // list(UUID)
    "ingestion_id":"",       // UUID
    "extraction_start_time": "", // iso8601
    "payload": {},
    "_metadata": {           // cannot be used safely
        "orgName": "",       // ~r/^[a-zA-Z_]+$/
        "dataName": "",      // ~r/^[a-zA-Z_]+$/
        "stream": true
    },
    "operational": {
        "timing": [{
            "startTime": "", // iso8601
            "endTime": "",   // iso8601
            "app": "",       // microservice generating timing data
            "label": ""      // label for this particular timing data
        }]
    }
}

Link to this section Summary

Types

payload()
t()

Functions

add_timing(message, new_timing)

Adds a SmartCity.Data.Timing to the list of timings in this SmartCity.Data. The timing will be validated to ensure both start and end times have been set.

encode(message)

Encodes SmartCity.Data into JSON. Typically used right before sending as a Kafka message.

encode!(message)

Encodes SmartCity.Data into JSON. Typically used right before sending as a Kafka message.

end_of_data()

Defines the string that will be the payload of the last message in a dataset.

get_all_timings(data)

Get all timings on this Data

new(msg)

Returns a new SmartCity.Data struct. SmartCity.Data.Timing structs will be created along the way.

timed_new(msg, app)

Creates a new SmartCity.Data struct using new/1 and adds timing information to the message.

timed_transform(msg, app, function)

Transforms the SmartCity.Data payload field with the given unary function and replaces it in the message.

Link to this section Types

Link to this type

payload()

View Source 
@type payload() :: String.t()
Link to this type

t()

View Source 
@type t() :: %SmartCity.Data{
  _metadata: %{org: String.t(), name: String.t(), stream: boolean()},
  dataset_ids: [String.t()],
  extraction_start_time: DateTime.t(),
  ingestion_id: String.t(),
  operational: %{timing: [SmartCity.Data.Timing.t()]},
  payload: String.t(),
  version: String.t()
}

Link to this section Functions

Link to this function

add_timing(message, new_timing)

View Source 
@spec add_timing(
  t(),
  SmartCity.Data.Timing.t()
) :: t()

Adds a SmartCity.Data.Timing to the list of timings in this SmartCity.Data. The timing will be validated to ensure both start and end times have been set.

Returns a SmartCity.Data struct with new_timing prepended to existing timings list.

parameters

 Parameters

  • message: A SmartCity.Data
  • new_timing: A timing you want to add. Must have start_time and end_time set
Link to this function

encode(message)

View Source 
@spec encode(t()) ::
  {:ok, String.t()} | {:error, Jason.EncodeError.t() | Exception.t()}

Encodes SmartCity.Data into JSON. Typically used right before sending as a Kafka message.

Link to this function

encode!(message)

View Source 
@spec encode!(t()) :: String.t()

Encodes SmartCity.Data into JSON. Typically used right before sending as a Kafka message.

Raises an error if it fails to convert to a JSON string.

Link to this macro

end_of_data()

View Source (macro)

Defines the string that will be the payload of the last message in a dataset.

Link to this function

get_all_timings(data)

View Source 
@spec get_all_timings(t()) :: [SmartCity.Data.Timing.t()]

Get all timings on this Data

Returns a list of SmartCity.Data.Timing structs or []

parameters

 Parameters

  • data_message: The message to extract timings from
Link to this function

new(msg)

View Source 
@spec new(map() | String.t()) :: {:ok, t()} | {:error, String.t()}

Returns a new SmartCity.Data struct. SmartCity.Data.Timing structs will be created along the way.

Can be created from:

  • map with string keys
  • map with atom keys
  • JSON

examples

 Examples 

iex> SmartCity.Data.new(%{dataset_ids: ["a_guid"], ingestion_id: "b_guid", extraction_start_time: "2019-05-06T19:51:41+00:00", payload: "the_data", _metadata: %{org: "scos", name: "example"}, operational: %{timing: [%{app: "app name", label: "function name", start_time: "2019-05-06T19:51:41+00:00", end_time: "2019-05-06T19:51:51+00:00"}]}})
{:ok, %SmartCity.Data{
    dataset_ids: ["a_guid"],
    ingestion_id: "b_guid",
    extraction_start_time: "2019-05-06T19:51:41+00:00",
    payload: "the_data",
    _metadata: %{org: "scos", name: "example"},
    operational: %{
    timing: [%SmartCity.Data.Timing{ app: "app name", end_time: "2019-05-06T19:51:51+00:00", label: "function name", start_time: "2019-05-06T19:51:41+00:00"}]
  }
}}
Link to this function

timed_new(msg, app)

View Source 
@spec timed_new(map(), String.t()) :: {:ok, t()} | {:error, String.t()}

Creates a new SmartCity.Data struct using new/1 and adds timing information to the message.

Returns a SmartCity.Data struct with new_timing prepended to existing timings list.

parameters

 Parameters

Link to this function

timed_transform(msg, app, function)

View Source 
@spec timed_transform(
  t(),
  String.t(),
  (payload() -> {:ok, term()} | {:error, term()})
) :: {:ok, t()} | {:error, String.t()}

Transforms the SmartCity.Data payload field with the given unary function and replaces it in the message.

Additionally, returns a SmartCity.Data struct with new_timing prepended to existing timings list.

parameters

 Parameters

  • message: A SmartCity.Data
  • app: The application that is asking to create the new SmartCity.Data. Ex. reaper or voltron
  • function: an arity 1 (/1) function that will transform the payload in the provided message