scrape

v3.1.0

scrape v3.1.0 Scrape.Flow

Logic Module for implementing linear data processing workflows.

Uses a "token" approach to store/retrieve values and persists a pipeline state that can be halted at any time. In case that something goes wrong, the pipeline will be halted and an error object will be returned with the occured error. Therefore, the pipeline should never raise an actual exception.

Link to this section Summary

Types

flow()

Intermediate state object that holds everything relevant for the data processing work flow. state holds general processing information, assigns are the user-level data fields and options contains a keyword list for, well, configuration options.

Functions

assign(flow, arg2)

Declare a new value in the data flow.

finish(flow, keys \\ [])

Select keys from the flow assigns and return a map with the chosen fields.

start(opts \\ [])

Initiate a new data processing flow with optional configuration.

Link to this section Types

Link to this type

flow() 
flow() :: %Scrape.Flow{
  assigns: map(),
  options: [{atom(), any()}],
  state: %{halted: boolean(), error: nil | any()}
}

Intermediate state object that holds everything relevant for the data processing work flow. state holds general processing information, assigns are the user-level data fields and options contains a keyword list for, well, configuration options.

Link to this section Functions

Link to this function

assign(flow, arg2) 
assign(flow(), [{atom(), any()}]) :: flow()

Declare a new value in the data flow.

Will do nothing if the flow got halted previously. If a function is given, and it raises an exception, the pipeline will catch the error and transform into a halted state.

Link to this function

finish(flow, keys \\ []) 
finish(flow(), [atom()]) :: {:ok, map()} | {:error, any()}

Select keys from the flow assigns and return a map with the chosen fields.

Will result in an error object if the flow got halted previously.

Link to this function

start(opts \\ []) 
start([{atom(), any()}]) :: flow()

Initiate a new data processing flow with optional configuration.

NOTE: the options are currently not used but will be in upcoming versions.

Example 

    iex> Flow.start()
    %Flow{state: %{halted: false, error: nil}, assigns: %{}, options: []}