Services.BackgroundIndexer (fnord v0.8.83)

View Source

Overview

The BackgroundIndexer is a silent, cancellable GenServer that indexes project files one at a time. It generates per-file derivatives (summary, outline, embeddings) and saves them to the project store.

This module is intentionally designed to be both:

  • sequential (one file at a time) for predictability and resource control
  • promptly cancellable so the parent command (e.g., ask) can stop it immediately

Strategy (literate walkthrough)

  • We do not pre-queue a large list of files. Instead, we:
    1. Optionally accept a small explicit files_queue from the caller (mainly used by tests)
    2. Otherwise fetch exactly one stale entry dynamically between tasks
  • Each file is processed in a linked Task so the GenServer stays responsive
  • We monitor the Task and, when it finishes, we schedule processing of the next file
  • If there is no next file, we stop the server with :normal
  • On shutdown, we kill any in-flight Task to guarantee prompt cancellation

Lifecycle checkpoints

  • init/1: set per-process HTTP pool, determine project & initial files_queue, prime state
  • handle_continue(:process_next): run the state machine to start the next Task or stop
  • handle_info({:DOWN, ...}): clear task state and schedule the next step
  • terminate/2: kill in-flight Task and clear HttpPool override

Why one-at-a-time?

  • Prevents overwhelming APIs (summaries, outlines, embeddings)
  • Simplifies cancellation and error isolation
  • Ensures the background indexer does not keep running long after ask completes

Summary

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

handle_continue(atom, state)

handle_continue(:process_next) operates in these modes

handle_info(msg, state)

When a monitored Task completes, we receive a :DOWN message. We clear the task state and trigger the next file via {:continue, :process_next}. This guarantees one-at-a-time processing and ensures prompt stop semantics.

init(opts)

init/1 sets up the GenServer state, performing the following steps

start_link(opts \\ [])
stop(pid)

Stop the BackgroundIndexer GenServer safely. This function is idempotent, swallows exits, and accepts non-pid values. Always returns :ok.

terminate(reason, state)

terminate/2 ensures prompt cancellation and cleanup

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

handle_continue(atom, state)

handle_continue(:process_next) operates in these modes:

  1. If a Task is already running, do nothing and wait for :DOWN
  2. If files_queue has entries, pop one and start a Task to process it
  3. If files_queue is empty but we have a project, fetch one stale entry and start a Task for it; if none remain, stop normally
  4. If there is no project and no files, stop normally

handle_info(msg, state)

When a monitored Task completes, we receive a :DOWN message. We clear the task state and trigger the next file via {:continue, :process_next}. This guarantees one-at-a-time processing and ensures prompt stop semantics.

init(opts)

init/1 sets up the GenServer state, performing the following steps:

  1. Configure HttpPool for AI indexer requests (efficient, reusable connections)
  2. Determine project context from opts or Store.get_project/0
  3. Build files_queue from opts (if provided) or default to []
  4. Initialize state and immediately continue to :process_next

start_link(opts \\ [])

@spec start_link(
  opts :: [project: Store.Project.t(), files: [Store.Project.Entry.t()]]
) ::
  GenServer.on_start()

stop(pid)

@spec stop(pid() | any()) :: :ok

Stop the BackgroundIndexer GenServer safely. This function is idempotent, swallows exits, and accepts non-pid values. Always returns :ok.

terminate(reason, state)

terminate/2 ensures prompt cancellation and cleanup:

  • Kills any in-flight Task to stop work immediately
  • Clears the HttpPool override for this process