# Getting Started with Fred

```elixir
Mix.install([
  {:fred, "~> 0.4.0"},
  {:vega_lite, "~> 0.1"},
  {:kino_vega_lite, "~> 0.1"}
])
```

## Introduction

FRED® (Federal Reserve Economic Data) provides access to over 800,000 economic time series from 100+ sources including the Bureau of Labor Statistics, the Bureau of Economic Analysis, and the Federal Reserve Board. This library was written to allow readers of [Elixir For Finance](https://www.financialelixir.dev/) to collect, analyze and visualize economic data from Fred, but it is a complete Fred API client and can be used outside the context of the book.

To learn how you can analyze and visualize the financial markets using Livebook, Explorer, Scholar and Nx, be sure to pick up a copy of our book:

<a target="_blank" href="https://www.financialelixir.dev">
  <img
    src="https://financial-analytics-elixir-landing.vercel.app/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fcover.8c040087.png&w=1080&q=75"
    alt="Elixir For Finance Book Cover"
    width="350"
  />
</a>

## Setup

To being, start by installing the notebook dependencies. This notebook uses the `fred` library for
API access and `VegaLite` for charting.

You'll need a FRED API key before making any API calls. You can get your free API key from the
[FRED API website](https://fred.stlouisfed.org/docs/api/api_key.html). After you have a FRED API
key, add it to your Livebook secrets undet the key `FRED_API_KEY` so the the downstream code can
access it.

With your API key in place, you can set the application configuration for the Fred library and
attach the default logger.

```elixir
alias VegaLite, as: Vl

# API key pulled from Livebook secrets
Application.put_env(:fred, :api_key, System.fetch_env!("LB_FRED_API_KEY"))

# Attach the default logger to keep an eye on requests
Fred.Telemetry.Logger.attach(level: :info)
```

With that you are ready to rock and roll! Let's now take a look at some simple endpoints to see
what data we can extract from FRED.

## Exploring a Series

We'll begin by looking up metadata for the _Unemployment Rate_ series (which has the key
`UNRATE`). The meta data will include things like the frequency of the series, the human readable
name, and the units of the series.

```elixir
# We use the `Fred.Series.get/1` function to get metadata about the `UNRATE` series and pluck
# the head of the response list.
{:ok, %{"seriess" => [data | _]}} =
  Fred.Series.get("UNRATE")

# Print out some of the metadata from the series
IO.puts("""
Title:       #{data["title"]}
Frequency:   #{data["frequency"]}
Units:       #{data["units"]}
Seasonal:    #{data["seasonal_adjustment"]}
Last Update: #{data["last_updated"]}
""")

:ok
```

## Fetching Observations

Now let's pull the actual unemployment rate data from 2000-01-01 onward. We'll work with the
raw response data from the FRED API and massage it into a format that works with VegaLite.

```elixir
# Get the observation data for the `UNRATE` series after (and including) 2000-01-01
observation_start = ~D[2000-01-01]

{:ok, %{"observations" => observations}} =
  Fred.Series.observations("UNRATE",
    observation_start: observation_start,
    frequency: :m
  )

# Parse into a list of maps, extract the necessary data and cast the float values. FRED returns
# "." for dates without any data so we'll ignore those.
formatted_observations =
  observations
  |> Enum.reduce([], fn
    %{"value" => "."}, acc ->
      acc

    %{"date" => date, "value" => value}, acc ->
      [%{date: Date.from_iso8601!(date), value: String.to_float(value)} | acc]
  end)
  |> Enum.sort_by(
    fn %{date: date} ->
      date
    end,
    Date
  )

# Get the min and max Dates for the series
{%{date: min_date}, %{date: max_date}} = Enum.min_max_by(formatted_observations, &(&1.date), Date)

# Print some information about the series
IO.puts("Fetched #{length(formatted_observations)} observations")
IO.puts("Date range: #{min_date} to #{max_date}")

# Preview the first few rows
Enum.take(formatted_observations, 5)
```

With the `UNRATE` data formatted nicely inside of the `observations` variable, let's now see how
we can plot this data using the `VegaLite` library inside of Livebook.

## Plotting with VegaLite

With our unemployment data now formatted as a list of maps with the date and value, we
can now plot the simple time series data using a `VegaLite` line chart:

```elixir
Vl.new(width: 750, height: 400, title: "U.S. Unemployment Rate (#{min_date} - #{max_date})")
|> Vl.data_from_values(formatted_observations)
|> Vl.mark(:line, tooltip: true, color: "#2563eb")
|> Vl.encode_field(:x, "date",
  type: :temporal,
  title: "Date",
  axis: [format: "%Y"]
)
|> Vl.encode_field(:y, "value",
  type: :quantitative,
  title: "Unemployment Rate (%)",
  scale: [zero: false]
)
```

## Adding Recession Shading

FRED provides the National Bureau of Economic Research (NBER for short) recession indicator as
series `"USREC"`. The data value is a `1` during a recession and `0` otherwise. We'll convert this
time series data set into a list of tuples to mark the beginning and end of recession periods
which we'll then use to overlay on top of the unemployment data in `VegaLite`.

```elixir
# Fetch the recession indicator for the same date range
{:ok, %{"observations" => reccession_data}} =
  Fred.Series.observations("USREC",
    observation_start: observation_start,
    frequency: :m
  )

# Convert monthly 0/1 flags into recession periods: [{start, end}, ...]
recession_periods =
  reccession_data
  |> Enum.reject(fn %{"value" => value} ->
    value == "."
  end)
  |> Enum.chunk_while(
    nil,
    fn
      %{"value" => "1", "date" => date}, nil ->
        {:cont, date}

      %{"value" => "1"}, start_date ->
        {:cont, start_date}

      %{"value" => "0"}, nil ->
        {:cont, nil}

      %{"value" => "0", "date" => date}, start_date ->
        {:cont, %{start: Date.from_iso8601!(start_date), end: Date.from_iso8601!(date)}, nil}
    end,
    fn
      nil ->
        {:cont, nil}

      start_date ->
        {:cont, %{start: start_date, end: Date.utc_today()}, nil}
    end
  )
```

With the recession date ranges, aggregated, we can now create a new `VegaLite` chart using the
`VegaLite.layers/2` function in order to overlay the recession ranges along with the unemployment
data.

```elixir
# Layer recession bands behind the unemployment area chart
Vl.new(width: 750, height: 400, title: "U.S. Unemployment Rate with Recession Shading")
|> Vl.layers([
  # Gray recession bands
  Vl.new()
  |> Vl.data_from_values(recession_periods)
  |> Vl.mark(:rect, color: "#3f3f46", opacity: 0.25)
  |> Vl.encode_field(:x, "start", type: :temporal)
  |> Vl.encode_field(:x2, "end", type: :temporal),

  # Unemployment area
  Vl.new()
  |> Vl.data_from_values(formatted_observations)
  |> Vl.mark(:area,
    tooltip: true,
    color: "#2563eb",
    opacity: 0.5,
    line: [color: "#2563eb"]
  )
  |> Vl.encode_field(:x, "date",
    type: :temporal,
    title: "Date",
    axis: [format: "%Y"]
  )
  |> Vl.encode_field(:y, "value",
    type: :quantitative,
    title: "Unemployment Rate (%)",
    scale: [zero: true]
  )
])
```

## Using FRED's Built-in Transformations

The FRED API can also compute percent changes and format the response payload accordingly. You
can do this via the `:units` option. Let's look at _month-over-month percent change_ in the unemployment rate.

```elixir
{:ok, %{"observations" => pch_observations}} =
  Fred.Series.observations("UNRATE",
    observation_start: observation_start,
    frequency: :m,
    units: :pch
  )

formatted_pch_observations =
  pch_observations
  |> Enum.reduce([], fn
    %{"value" => "."}, acc ->
      acc

    %{"date" => date, "value" => value}, acc ->
      value = String.to_float(value)

      data = %{
        date: Date.from_iso8601!(date),
        value: value,
        direction: if(value >= 0, do: "Increase", else: "Decrease")
      }

      [data | acc]
  end)
  |> Enum.sort_by(
    fn %{date: date} ->
      date
    end,
    Date
  )
```

With our formatted percent change data, we can once again lean on the `VegaLite.layers/2` function
in order to overlay the recession periods along with the unemployment rate percent change.

```elixir
Vl.new(width: 750, height: 400, title: "Unemployment Rate - Month-over-Month % Change")
|> Vl.layers([
  # Gray recession bands
  Vl.new()
  |> Vl.data_from_values(recession_periods)
  |> Vl.mark(:rect, color: "#3f3f46", opacity: 0.25)
  |> Vl.encode_field(:x, "start", type: :temporal)
  |> Vl.encode_field(:x2, "end", type: :temporal),

  # Unemployment area
  Vl.new()
|> Vl.data_from_values(formatted_pch_observations)
|> Vl.mark(:bar, tooltip: true)
|> Vl.encode_field(:x, "date",
  type: :temporal,
  title: "Date",
  axis: [format: "%Y"]
)
|> Vl.encode_field(:y, "value",
  type: :quantitative,
  title: "% Change"
)
|> Vl.encode_field(:color, "direction",
  type: :nominal,
  scale: [domain: ["Increase", "Decrease"], range: ["#ef4444", "#22c55e"]],
  legend: nil
)
])
```

## Searching for Observation Series

In addition to fetching data for known observation series, you can search FRED's full catalog via
full text search. Let's find the most popular series related to "inflation".

```elixir
{:ok, %{"seriess" => search_results}} =
  Fred.Series.search("inflation",
    order_by: :popularity,
    sort_order: :desc,
    limit: 5
  )

search_results
|> Enum.map(fn s ->
  %{
    id: s["id"],
    title: s["title"],
    frequency: s["frequency_short"],
    popularity: s["popularity"]
  }
end)
```