Electric.ShapeCache.ShapeStatus.ShapeDb.WriteBuffer (electric v1.4.13)
View SourceBuffers SQLite metadata writes using ETS for immediate return to callers, then batches and flushes to SQLite in the background.
This prevents timeout cascades when many concurrent clients create shapes after a redeploy.
Architecture
Two ETS tables are used:
- Operations table (ordered_set) - queue of operations to flush to SQLite
- Shapes table (set) - buffered shapes, comparable index, and tombstones with namespaced keys
How it works
- When a shape is added, insert into shapes table and queue :add operation
- When a shape is removed, insert tombstone and queue :remove operation
- GenServer polls every 50ms and flushes pending operations to SQLite
- After successful flush, clean up entries from shapes table
Crash recovery
If the system crashes, all in-flight writes in the operations table will be
lost. On reboot clients of the in-flight shapes will receive must-refetch
responses and the shapes will be re-inserted into the buffer.
This will leave orphaned shape data in the storage implementation, as we are losing all references to the handle. We will need some background reconciliation process that culls orphaned storage data.
Operations table format
{{monotonic_time, unique_int}, operation, flushing}
Shapes table key formats
{:pending_count, integer}- count of shapes pending addition{{:shape, handle}, shape, comparable}- shape data{{:comparable, comparable}, handle}- reverse index for O(1) lookup{{:tombstone, handle}, timestamp}- handles marked for deletion
Summary
Functions
Add a shape to the buffer
Returns a specification to start this module under a supervisor.
Clear all data from all tables
Synchronously flush all pending writes. Useful for testing and graceful shutdown.
Returns handles from the buffer that match any of the given relations (by OID)
Check if a handle exists in the buffer. Returns
Check if a handle is in the tombstones (marked for deletion)
Returns all buffered shapes as a list of {handle, shape} tuples, excluding tombstoned handles, sorted by handle
Look up a handle by comparable shape binary. Returns {:ok, handle} or :not_found.
Look up a shape by its handle. Returns {:ok, shape} or :not_found.
Returns a unique, ordered key for operations table entries
Disable flushing of buffered actions to db. Useful for testing
Gives the change to the total count of shapes in the database once all buffered writes are applied
Returns the number of pending operations in the buffer
Queue a snapshot_complete operation
Mark a shape for removal
Re-enable flushing of buffered actions to db. Useful for testing
Returns a monotonic timestamp for ordering writes
Returns all tombstoned handles as a MapSet
Functions
Add a shape to the buffer
Returns a specification to start this module under a supervisor.
See Supervisor.
Clear all data from all tables
Synchronously flush all pending writes. Useful for testing and graceful shutdown.
Returns handles from the buffer that match any of the given relations (by OID)
Check if a handle exists in the buffer. Returns:
falseif tombstoned (being deleted)trueif in shapes table (buffered, not yet in SQLite):unknownif not in buffer (may or may not exist in SQLite)
Note: Checks tombstone last since it's the authoritative "delete in progress" signal, avoiding a race where tombstone is added between checks.
Check if a handle is in the tombstones (marked for deletion)
Returns all buffered shapes as a list of {handle, shape} tuples, excluding tombstoned handles, sorted by handle
Look up a handle by comparable shape binary. Returns {:ok, handle} or :not_found.
Look up a shape by its handle. Returns {:ok, shape} or :not_found.
Returns a unique, ordered key for operations table entries
Disable flushing of buffered actions to db. Useful for testing
Gives the change to the total count of shapes in the database once all buffered writes are applied
Returns the number of pending operations in the buffer
Queue a snapshot_complete operation
Mark a shape for removal
Re-enable flushing of buffered actions to db. Useful for testing
Returns a monotonic timestamp for ordering writes
Returns all tombstoned handles as a MapSet