Lazy stream-based pagination for all Aurinko list endpoints.
Instead of manually tracking next_page_token and next_delta_token,
wrap any list function in a stream and consume it lazily.
Usage
# Lazy stream — only fetches pages as consumed
stream = Aurinko.Paginator.stream(token, &Aurinko.Email.list_messages/2,
limit: 50,
q: "is:unread"
)
# Take the first 100 messages across all pages
messages = stream |> Stream.take(100) |> Enum.to_list()
# Or collect all into memory
all_messages = Enum.to_list(stream)
# Stream calendar events
events =
Aurinko.Paginator.stream(token, fn t, opts ->
Aurinko.Calendar.list_events(t, "primary", opts)
end, time_min: ~U[2024-01-01 00:00:00Z], time_max: ~U[2024-12-31 23:59:59Z])
|> Enum.to_list()Sync streaming
For delta-sync streams, use sync_stream/4:
# Streams all updated records (pages handled automatically)
{:ok, sync} = Aurinko.Email.start_sync(token, days_within: 30)
Aurinko.Paginator.sync_stream(token, sync.sync_updated_token, fn t, delta_or_page_token, is_page ->
if is_page do
Aurinko.Email.sync_updated(token, delta_or_page_token)
else
Aurinko.Email.sync_updated(token, delta_or_page_token)
end
end)
|> Stream.each(&process_message/1)
|> Stream.run()
Summary
Types
@type fetch_fn() :: (String.t(), keyword() -> {:ok, Aurinko.Types.Pagination.t()} | {:error, term()})
Functions
Collect all pages synchronously and return a flat list of records.
Convenience wrapper around stream/3 for when you want all results.
@spec stream(String.t(), fetch_fn(), keyword()) :: Enumerable.t()
Create a lazy Stream that automatically paginates through all pages.
The stream yields individual records (not page structs). Pages are fetched on demand as the stream is consumed.
Parameters
token— Aurinko access tokenfetch_fn— A function(token, opts) -> {:ok, %Pagination{}}. Any list API function works.opts— Options forwarded tofetch_fnon every call (e.g.:limit,:q)
Options
:on_error—:halt(default) or:skip— what to do on an API error mid-stream
@spec sync_stream( String.t(), String.t(), (String.t(), String.t() -> {:ok, Aurinko.Types.Pagination.t()} | {:error, term()}), keyword() ) :: Enumerable.t()
Create a lazy Stream over a sync-updated or sync-deleted endpoint.
Handles next_page_token (more pages in this sync batch) and
next_delta_token (batch complete — use this token next time).
Returns individual records. The final delta token is NOT yielded as a record —
use the :on_delta callback to capture it.
Options
:on_delta—(delta_token -> any)— called when a new delta token is received