AshReports.Charts.DataSourceHelpers (ash_reports v0.1.0)
Helper functions for optimizing chart data source queries.
Avoiding N+1 Query Problems
When working with large datasets, eagerly loading relationships with Ash.Query.load/2
can cause severe performance issues known as "N+1 query problems". This occurs when
loading a relationship for each record in a collection, resulting in N individual queries.
Example Problem
# ❌ BAD - This creates an N+1 query problem
data_source(fn ->
InvoiceLineItem
|> Ash.Query.load(product: :category) # Loads product for EVERY line item!
|> Ash.read!(domain: MyApp.Domain)
end)With 325,000 line items, this performs 325,000+ individual lookups. This can take 8+ minutes on large datasets!
Solution Pattern
# ✅ GOOD - Load relationships separately and join in memory
data_source(fn ->
# 1. Load main records without relationships
items = InvoiceLineItem |> Ash.read!(domain: MyApp.Domain)
# 2. Get unique related IDs
product_ids = items |> Enum.map(& &1.product_id) |> Enum.uniq()
# 3. Load related records once
products =
Product
|> Ash.read!(domain: MyApp.Domain)
|> build_lookup_map(:id)
# 4. Join in memory using helper
chart_data = join_and_aggregate(items, products, :product_id)
end)This reduces execution time from 8 minutes to <1 second on large datasets!
Using the Helpers
This module provides helper functions to make the optimized pattern easier to implement:
build_lookup_map/2- Create fast lookup maps from collectionsextract_unique_ids/2- Extract unique IDs from a collectionload_related_batch/3- Load related records in a single queryjoin_with_lookup/3- Join collections using a lookup mapwarn_if_loaded/2- Runtime warning for potential N+1 issues
See individual function documentation for detailed usage examples.
Summary
Functions
Builds a fast lookup map from a collection of records.
Extracts unique ID values from a collection of records.
Joins source records with related records using a lookup map.
Loads related records in a single batch query.
Optimized pattern for loading records with a single relationship.
Warns if records were loaded with relationships (potential N+1 issue).
Functions
Builds a fast lookup map from a collection of records.
This creates a map where keys are the specified field values and values are the records themselves, enabling O(1) lookups instead of O(N) scans.
Parameters
records- Collection of records to indexkey_field- Field name to use as the map key (atom)
Returns
A map with key_field values as keys and records as values.
Examples
# Create lookup map by ID
products = [%Product{id: 1, name: "A"}, %Product{id: 2, name: "B"}]
lookup = build_lookup_map(products, :id)
# => %{1 => %Product{id: 1, name: "A"}, 2 => %Product{id: 2, name: "B"}}
# Create lookup map by name
lookup = build_lookup_map(products, :name)
# => %{"A" => %Product{id: 1, name: "A"}, "B" => %Product{id: 2, name: "B"}}Extracts unique ID values from a collection of records.
Filters out nil values and returns only unique IDs.
Parameters
records- Collection of recordsid_field- Field name containing the ID (atom)
Returns
List of unique, non-nil ID values.
Examples
line_items = [
%{product_id: 1},
%{product_id: 1},
%{product_id: 2},
%{product_id: nil}
]
extract_unique_ids(line_items, :product_id)
# => [1, 2]Joins source records with related records using a lookup map.
This performs an in-memory join by looking up related records for each source record.
Parameters
source_records- Collection of source recordslookup_map- Map created bybuild_lookup_map/2foreign_key- Field in source records containing the lookup key
Returns
List of tuples {source_record, related_record} for matching records.
Examples
products_map = build_lookup_map(products, :id)
joined = join_with_lookup(line_items, products_map, :product_id)
# => [
# {%LineItem{product_id: 1}, %Product{id: 1}},
# {%LineItem{product_id: 2}, %Product{id: 2}}
# ]@spec load_with_relationship(module(), module(), atom(), keyword()) :: {:ok, {list(), map()}} | {:error, term()}
Optimized pattern for loading records with a single relationship.
This is a convenience function that combines the common pattern of loading records, extracting IDs, loading related records, and creating a lookup map.
Parameters
source_module- Ash resource module for source recordsrelated_module- Ash resource module for related recordsforeign_key- Field in source records pointing to related recordsopts- Keyword list of options:domain- Ash domain (required):preload- Fields to preload on related records (optional)
Returns
{:ok, {source_records, lookup_map}} or {:error, reason}
Examples
{:ok, {line_items, products_map}} = load_with_relationship(
InvoiceLineItem,
Product,
:product_id,
domain: MyApp.Domain,
preload: :category
)
# Now use the lookup map
enriched_items = Enum.map(line_items, fn item ->
product = products_map[item.product_id]
%{item | product: product}
end)Warns if records were loaded with relationships (potential N+1 issue).
This can be used as a runtime check to detect potentially inefficient queries. Checks the first record in the collection to see if relationship fields are loaded.
Parameters
records- Collection of records to checkrelationship_fields- List of relationship field names (atoms)
Returns
The original records (passthrough for piping).
Examples
InvoiceLineItem
|> Ash.read!(domain: MyApp.Domain)
|> warn_if_loaded([:product, :invoice])
# Logs warning if product or invoice are loaded