Blink

Blink is a fast bulk data insertion library for Ecto and PostgreSQL.

It provides a callback-based pattern for seeding databases with dependent tables and shared context. Designed for scenarios where you need to insert test data, seed development databases, or populate staging environments quickly.

Installation

Add blink to your list of dependencies in mix.exs:

def deps do
  [
    {:blink, "~> 0.2.0"}
  ]
end

Then run mix deps.get to install it.

Example

defmodule MyApp.Seeder do
  use Blink

  def call do
    new()
    |> add_table("users")
    |> add_table("posts")
    |> insert(MyApp.Repo)
  end

  def table(_store, "users") do
    for i <- 1..1000 do
      %{
        id: i,
        name: "User #{i}",
        email: "user#{i}@example.com",
        inserted_at: DateTime.utc_now(),
        updated_at: DateTime.utc_now()
      }
    end
  end

  def table(store, "posts") do
    users = store.tables["users"]

    Enum.flat_map(users, fn user ->
      for i <- 1..5 do
        %{
          id: (user.id - 1) * 5 + i,
          title: "Post #{i}",
          user_id: user.id,
          inserted_at: DateTime.utc_now(),
          updated_at: DateTime.utc_now()
        }
      end
    end)
  end
end

# Inserts 1,000 users and 5,000 posts
MyApp.Seeder.call()

Features

Fast bulk inserts - Uses PostgreSQL's COPY FROM STDIN command for optimal performance
Dependent tables - Insert tables in order with access to previously inserted data
Shared context - Compute expensive operations once and share across tables
File loading - Built-in helpers for CSV and JSON file imports
Configurable batching - Adjust batch sizes for memory-efficient large dataset insertion
Transaction support - Automatic rollback on errors

Usage

Blink uses a callback-based pattern where you define:

Which tables to insert (via add_table/2)
What data goes in each table (via table/2 callback)
Optional shared context (via add_context/2 and context/2 callback)

Accessing Previously Inserted Tables

Tables are inserted in the order they're added. Access previous table data via store.tables:

def table(store, "posts") do
  users = store.tables["users"]  # Access users inserted earlier

  Enum.flat_map(users, fn user ->
    for i <- 1..3 do
      %{
        id: (user.id - 1) * 3 + i,
        title: "Post #{i}",
        user_id: user.id,
        inserted_at: DateTime.utc_now(),
        updated_at: DateTime.utc_now()
      }
    end
  end)
end

Using Data from Context

Use context to compute expensive operations once and share across all tables:

def call do
  new()
  |> add_context("timestamps")
  |> add_table("users")
  |> add_table("posts")
  |> insert(MyApp.Repo)
end

def context(_store, "timestamps") do
  base = ~U[2024-01-01 00:00:00Z]
  for day <- 0..29, do: DateTime.add(base, day, :day)
end

def table(store, "users") do
  timestamps = store.context["timestamps"]
  # Use shared timestamps...
end

def table(store, "posts") do
  timestamps = store.context["timestamps"]
  # Reuse same timestamps...
end

Loading from Files

Load data from CSV or JSON files:

def table(_store, "users") do
  Blink.from_csv("priv/seed_data/users.csv",
    transform: fn row ->
      row
      |> Map.update!("id", &String.to_integer/1)
      |> Map.put("inserted_at", DateTime.utc_now())
      |> Map.put("updated_at", DateTime.utc_now())
    end
  )
end

def table(_store, "products") do
  Blink.from_json("priv/seed_data/products.json",
    transform: fn product ->
      Map.put(product, "inserted_at", DateTime.utc_now())
    end
  )
end

CSV files use the first row as headers by default. Both helpers accept a :transform option for type conversion or data manipulation.

Configuring Batch Size

Adjust batch size for large datasets:

new()
|> add_table("users")
|> insert(MyApp.Repo, batch_size: 5_000)  # Default: 900

Using with ExMachina

Combine ExMachina's factory pattern with Blink's fast insertion:

defmodule MyApp.Seeder do
  use Blink
  import MyApp.Factory

  def call do
    new()
    |> add_table("users")
    |> add_table("posts")
    |> insert(MyApp.Repo)
  end

  def table(_store, "users") do
    for _i <- 1..100 do
      user = build(:user)
      Map.put(user, :id, Ecto.UUID.generate())
    end
  end

  def table(store, "posts") do
    user_ids = Enum.map(store.tables["users"], & &1.id)

    Enum.flat_map(user_ids, fn user_id ->
      for _i <- 1..5 do
        post = build(:post, user_id: user_id)
        Map.put(post, :id, Ecto.UUID.generate())
      end
    end)
  end
end

Learning Blink

Getting Started guide - Step-by-step tutorial building a complete seeding system
API documentation - Full reference for all functions and callbacks
Changelog - Version history and migration guides

Requirements

Requirement	Version
Elixir	1.14+
Ecto	3.0+
PostgreSQL	Any supported version

Known Limitations

Memory usage with large datasets - Blink loads all table data into memory before insertion. For very large datasets, consider splitting your seeder into multiple modules:

# Instead of one large seeder, use multiple smaller ones
organization_ids = OrganizationSeeder.call()
user_ids = UserSeeder.call(organization_ids)
PostSeeder.call(user_ids)

This limitation may be addressed in a future version.

License

Licensed under the MIT License. See LICENSE for details.

Next Page → Changelog