datastream

datastream is a compositional, resource-safe stream library for Gleam. The cross-target core (datastream/source, datastream/stream, datastream/fold, datastream/sink, datastream/chunk, datastream/text, datastream/binary) runs on both the Erlang and JavaScript targets. Target-specific extensions live under datastream/erlang/* and datastream/javascript/*.

This module defines the foundational types every later module builds on: the opaque pipeline value Stream(a) and the pull-result enum Step(a, state).

Invalid-argument policy

Across the library, any size / count / prefix_size argument that cannot have a meaningful interpretation is rejected at construction time with a panic rather than silently normalised — silent normalisation produces results that look plausible but lose information the caller cannot recover. The per-function docstring states the exact accepted range; the general rule is:

A “size” argument that controls how many elements form one chunk / frame (stream.chunks_of, binary.fixed_size, binary.length_prefixed’s prefix_size) MUST be >= 1 (and prefix_size is further restricted to {1, 2, 4, 8}).
A “count” argument that says how many elements to take or drop (stream.take, stream.drop) MUST be >= 0. The 0 case has a single mathematically natural answer (the empty stream / the identity respectively) and is accepted; negative counts are programmer error.

Every violation is a programmer error: the only sensible response is to crash so a bogus value cannot quietly produce surprising data downstream.

Types

Step

</>

The result of pulling one element from a stream-like producer.

Next(element, state) carries the produced element together with the continuation needed to pull the next one. Done signals end-of-stream. Once Done is observed the producer MUST NOT be pulled again.

There is intentionally no Error variant: the core does not impose a failure model. Callers that need errors carry them as element-shaped values (Stream(Result(a, e)), Stream(Validated(a, e)), …).

Unlike Stream(a), Step is a transparent public type: callers return Next / Done from the next callback passed to source.resource and source.try_resource. The enum shape is stable from v0.1.0 onward.

pub type Step(a, state) {
  Next(element: a, state: state)
  Done
}

Constructors

```
Next(element: a, state: state)
```
```
Done
```

Stream

opaque </>

A lazy, pull-based pipeline that yields elements of type a.

Values of this type are pipeline definitions, not materialised collections. Each terminal operation (in fold / sink) re-runs the pipeline from its source — there is no implicit caching. Callers that want replay should materialise once with fold.to_list.

The internal representation is intentionally hidden so the library is free to change it (list-backed, function-batched, chunk-batched, …) without a breaking change. == / != carry no specified meaning; observational equality is via terminal reduction.

The library threads a cleanup callback through every stream value so resource-safe sources from source.resource can be released on every termination path (normal end, downstream early-exit, early-exit folds, try_each failure). Pure (unfold-built) streams use a no-op cleanup.

pub opaque type Stream(a)