Changelog

Copy Markdown View Source

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

0.3.9 - 2026-02-16

Improved

  • Decimal encoding performance — Fixed a bottleneck that caused scheduler contention and connection timeouts when encoding Decimal values under high throughput.
  • Error handling — The parser now returns descriptive errors for malformed unicode escapes and invalid exponents instead of risking NIF crashes.
  • Reduced allocations — Less memory pressure when decoding arrays of same-shape objects.

Changed

  • Removed unused serde and lazy_static dependencies.
  • Internal code quality improvements (named constants, idiomatic error propagation).

0.3.8 - 2026-02-12

Fixed

  • Duplicate key corruption — Fixed a critical bug where JSON objects containing duplicate keys had their entire parse result corrupted. Non-duplicate keys were silently dropped and values were assigned to wrong keys (e.g., {"a": 1, "b": 2, "b": 3, "c": 4} decoded to %{"a" => 4} instead of %{"a" => 1, "b" => 3, "c" => 4}). Root cause: build_map_with_duplicates decoded key Terms as Vec<u8> (Erlang list type) instead of Binary, causing all keys to silently decode to empty bytes via unwrap_or_default() and collapse into a single entry. The fix uses zero-copy Binary::as_slice() references into the BEAM heap, which is also faster than the original intended HashMap<Vec<u8>> approach.

Testing

  • Strengthened duplicate key test coverage — Existing tests only used all-duplicate inputs (e.g., {"a": 1, "a": 2}) which accidentally produced correct results despite the bug. Added tests with mixed duplicate and unique keys to catch key-collapse regressions.
  • Tightened assertions across test suite — Replaced = (Elixir subset match on maps) with == (exact equality) in map result assertions, replaced field-by-field checks with full map equality, and strengthened {:error, _} wildcards to verify %RustyJson.DecodeError{} struct type. These changes ensure tests fail on missing keys, extra keys, or wrong error types.

0.3.7 - 2026-02-12

Fixed

  • OrderedObject pretty-printing — OrderedObject now produces properly indented multi-line output with pretty: true. Previously, encoding always returned compact single-line JSON because the Encoder produced a Fragment that bypassed the NIF's inline pretty-printing. OrderedObject is now handled as a native struct in Rust, formatting directly from the values tuple list with zero intermediate serialization.

  • Fragment pretty-printing — Pre-encoded Fragment content is now reformatted with depth-aware indentation when pretty: true is active. This fixes pretty-printing for any struct values nested inside Maps or Lists (e.g., %{data: %OrderedObject{...}}). The reformatter streams through iodata byte-by-byte with zero buffer allocation, matching Jason's behavior where the Formatter reformats all output uniformly.

0.3.6 - 2026-02-11

Fixed

  • Alpine Linux / musl compatibility — Fixed a NIF loading crash on Alpine Linux caused by mimalloc's default initial-exec TLS model. The build configuration now automatically enables mimalloc's local_dynamic_tls feature when compiling for musl targets, ensuring correct behavior on Alpine while preserving maximum performance on other platforms.

0.3.5 - 2026-02-01

Major backend refactor: the Rust core was rewritten for safety, stability, and performance. All architecture-specific SIMD intrinsics replaced with portable std::simd, eliminating all unsafe code. No API breaking changes — the Elixir interface is fully backwards-compatible with 0.3.4.

Added

  • sort_keys encode option — sort_keys: true sorts map keys lexicographically in the JSON output. Useful for snapshot tests, caching, and diffing. Sorting is recursive (nested maps are also sorted). Default: false — JSON objects are unordered per RFC 8259, and RustyJson preserves this semantics by default. No overhead when disabled.
  • AVX2 precompiled binary variants — Each x86_64 target now ships two precompiled binaries: a baseline (SSE2, 16-byte SIMD) and an AVX2 variant optimized for Haswell+ CPUs (32-byte SIMD, built with -C target-cpu=x86-64-v3). At compile time, RustlerPrecompiled detects AVX2 support on the host CPU and downloads the optimal variant automatically. Total precompiled artifacts: 45 (30 baseline + 15 AVX2 variants across 5 x86_64 targets × 3 NIF versions).

Changed

  • Portable SIMD (std::simd) — Replaced all architecture-specific SIMD intrinsics (NEON, AVX2, SSE2) with Rust's portable SIMD. One codepath per pattern, zero unsafe, no #[cfg(target_arch)] branching. The compiler generates optimal instructions for each target automatically. Only uses std::simd APIs with stable semantics (comparisons, masks, splat/from_slice) — the critical stabilization blockers (swizzle, scatter/gather, mask element types) are unrelated to our usage and explicitly avoided.
  • Zero unsafe code — Eliminated all unsafe blocks. SIMD is now safe via std::simd, and make_subbinary_unchecked was replaced with the safe make_subbinary. The entire codebase is 100% safe Rust.
  • Nightly Rust toolchain — Required for #![feature(portable_simd)]. Pinned via rust-toolchain.toml.

Performance

  • SIMD-accelerated everything — Hardware acceleration across the entire pipeline: string scanning (decode), escape scanning (encode), structural character indexing, and whitespace skipping. 32-byte wide paths on AVX2, 16-byte on all other targets, with scalar tails.
  • Same-shape object detection — Reuses key sets for arrays of objects with identical schemas, providing 2-6x speedup on typical API responses.
  • Direct NIF binary writes — Encoded JSON is written once directly into BEAM-managed memory, eliminating intermediate buffer copies.
  • Fast-path integer parsing — Homogeneous integer arrays use a tighter, branchless loop that bypasses general-purpose parsing for small numbers.
  • Allocation reduction — Switched to SmallVec for formatting context, interned struct field atoms, and pre-allocated URI buffers to minimize heap pressure.
  • SIMD digit scanning — Number parsing skips contiguous digit runs in 16/32-byte chunks with partial-chunk handling via to_bitmask().trailing_zeros(). Applied across all digit-scanning sites in parse_number, parse_number_fast, and scan_number.
  • Bulk escaped string copy — decode_escaped_string uses SIMD find_escape_json to locate the next escape-worthy byte, then copies the entire safe region in one memcpy via extend_from_slice.
  • Precise SIMD exit positions — skip_whitespace and skip_ascii_digits use to_bitmask().trailing_zeros() to advance to the exact byte position within a partial chunk, eliminating redundant scalar fallback. (String scanning and structural indexing retain the simpler .any() + break pattern, which benchmarks faster for their typical workloads of frequent hits and dense JSON.)
  • Static error messages — Replaced dynamic string allocation with static Cow types for common parsing errors.

Fixed

  • Test suite cleanup — Removed redundant/flawed assertions and added explicit regression tests for SIMD boundary safety and structural index validation.

0.3.4 - 2026-01-30

Performance

  • Decode fast path — decode!/1 bypasses option parsing for the common no-options call (Phoenix, Plug, etc.). No API changes.
  • SIMD string scanning — String parsing uses portable SIMD (32 bytes/iter on AVX2, 16 bytes/iter elsewhere) combined with zero-copy sub-binary references for non-escaped strings. In local benchmarks, 2-4x faster on string-heavy payloads vs v0.3.3, 15-30% faster on small payloads. Results may vary by hardware and payload shape.

0.3.3 - 2026-01-30

Performance

  • Single-pass struct encoding — Struct-heavy data (e.g., lists of derived or custom structs) is now encoded in ~1 walk instead of 3. Previously the encoding pipeline performed three separate walks: protocol dispatch, fragment resolution, and NIF serialization. Now the Encoder protocol produces iodata directly, and the NIF writes it in O(1). RustyJson is now faster across all encoding workloads — plain data and struct-heavy data alike.
  • Compile-time derived encoder codegen — @derive RustyJson.Encoder now generates iodata templates at compile time with pre-escaped keys. Eliminates runtime Map.from_struct, Map.to_list, and key-escaping overhead.
  • NIF bypass for iodata Fragments — When the top-level encoding result is already iodata (no pretty-print or compression), IO.iodata_to_binary/1 is used directly instead of passing through the Rust NIF, avoiding unnecessary Erlang↔Rust term conversion.
  • Top-level-only fragment resolution — When protocol: true (the default), fragment function resolution is now O(1) instead of O(n). Nested fragments are resolved during the protocol walk itself, so only the top-level result needs checking.
  • Deep-nested decode fast path — ~27% faster decode for deeply nested JSON (e.g., 100 levels of {"nested": {...}}). Single-entry objects and arrays avoid heap allocation entirely.
  • No regressions on plain data workloads (maps, lists, primitives).

Changed

  • RustyJson.Encoder custom implementations should now return iodata via RustyJson.Encode functions (e.g., Encode.map/2) for best performance. Returning plain maps is still supported for backwards compatibility and will be re-encoded automatically.

Testing

  • 421 tests, all passing with 0 failures.

0.3.2 - 2026-01-29

Fixed

  • validate_strings now defaults to true — Jason's decoder always validates UTF-8 (its parser matches ::utf8 codepoints and rejects control bytes). The v0.3.1 release shipped validate_strings: false, which silently accepted invalid UTF-8 byte sequences — breaking the Jason-parity guarantee documented since v0.3.0. The default is now true to match Jason's behavior. Pass validate_strings: false to opt out for maximum throughput on trusted input.

Testing

  • 421 tests, all passing with 0 failures.
  • Moved 10 invalid-UTF-8 JSONTestSuite i_* fixtures from @implementation_accepts to @implementation_rejects to reflect the new default.

0.3.1 - 2026-01-29

Added - Robustness & Security

Decode Options

  • max_bytes — Maximum input size in bytes (default: 0, unlimited). The check uses IO.iodata_length/1 before IO.iodata_to_binary/1 to avoid allocating a contiguous binary for oversized input. Also enforced defensively in the NIF.

  • duplicate_keys: :last | :error — Opt-in strict duplicate key rejection (default: :last, preserving last-wins semantics). When :error, tracks seen keys via HashSet in the Rust parser and returns a DecodeError on the first duplicate. Performance note: adds per-key overhead when enabled — use only when strict validation is needed.

  • validate_strings: true | false — Opt-in UTF-8 validation for decoded strings (default: false). When true, calls std::str::from_utf8() on both escaped and non-escaped string paths, rejecting invalid byte sequences with a DecodeError.

  • dirty_threshold — Byte size threshold for auto-dispatching decode to a dirty CPU scheduler (default: 102_400 / 100 KB, configurable at compile time via Application.compile_env(:rustyjson, :dirty_threshold_bytes)). Set to 0 to disable. Prevents large inputs from blocking normal BEAM schedulers.

Encode Options

  • scheduler: :auto | :normal | :dirty — Controls which scheduler the encode NIF runs on (default: :auto). :auto promotes to dirty only when compress: :gzip is used (compression is always CPU-heavy). :dirty always uses the dirty CPU scheduler. :normal always uses the normal scheduler.

Changed

Scheduler Strategy

  • Added dirty CPU scheduler NIF variants (nif_encode_direct_dirty, nif_decode_dirty) alongside the existing normal-scheduler NIFs. Both variants share the same implementation — only the scheduler annotation differs.
  • Updated docs/ARCHITECTURE.md: renamed "Why We Don't Use Dirty Schedulers" to "Scheduler Strategy" explaining the hybrid approach (normal by default, dirty for large payloads or compression).

Security Hardening

  • Randomized FNV hasher seed — The FnvBuildHasher used for keys: :intern key caching now generates a per-parse random seed by mixing std::time::SystemTime with a stack address. This blocks precomputed hash collision tables without adding any dependencies or measurable overhead. Previously used a fixed FNV offset basis (0xcbf29ce484222325). Note: the seed is not cryptographic — an adaptive attacker measuring aggregate latency could still craft collisions. The intern cache cap (below) bounds the damage in that scenario.
  • Intern cache cap (4096 keys) — The keys: :intern cache stops accepting new entries after 4096 unique keys. Beyond that, new keys are allocated normally (no worse than default mode). This serves dual purposes: (1) bounds worst-case CPU time from hash collisions to O(4096²) operations regardless of hash quality; (2) stops paying cache overhead when the input clearly has too many unique keys for interning to help. The cap is internal and not user-configurable.

Testing

  • 412 tests, all passing with 0 failures.
  • New test coverage: key interning correctness, max_bytes limits (including iodata input), duplicate key detection (including nested objects), UTF-8 string validation (single-byte and multi-byte), dirty scheduler dispatch for both encode and decode.

Release Notes

Adding new NIF exports (nif_encode_direct_dirty, nif_decode_dirty) changes the compiled artifact. This requires rebuilding precompiled binaries and updating checksums.

0.3.0 - 2026-01-28

Breaking Changes

This release aims to achieve full Jason API parity. Three changes require action when upgrading:

1. decode/2 returns %DecodeError{} instead of a string

# Before (0.2.x)
{:error, message} = RustyJson.decode(bad_json)
Logger.error("Failed: #{message}")

# After (0.3.0)
{:error, %RustyJson.DecodeError{} = error} = RustyJson.decode(bad_json)
Logger.error("Failed: #{error.message}")
# Also available: error.position, error.data, error.token

2. encode/2 returns %EncodeError{} instead of a string

# Before (0.2.x)
{:error, message} = RustyJson.encode(bad_data)

# After (0.3.0)
{:error, %RustyJson.EncodeError{} = error} = RustyJson.encode(bad_data)

3. RustyJson.Encoder protocol changed from encode/1 to encode/2

# Before (0.2.x)
defimpl RustyJson.Encoder, for: MyStruct do
  def encode(value), do: Map.take(value, [:name])
end

# After (0.3.0)
defimpl RustyJson.Encoder, for: MyStruct do
  def encode(value, _opts), do: Map.take(value, [:name])
end

Added - Full Jason Feature Parity

RustyJson now matches Jason's public API 1:1 in signatures, return types, and behavior.

Decode Options

  • keys: :copy - Accepted for Jason compatibility. Equivalent to :strings since RustyJson NIFs always produce copied binaries.
  • keys: :atoms - Convert keys to atoms using String.to_atom/1 (matches Jason — unsafe with untrusted input).
  • keys: :atoms! - Convert keys to existing atoms using String.to_existing_atom/1 (matches Jason — safe, raises if atom doesn't exist).
  • keys: custom_function - Pass a function of arity 1 to transform keys recursively: keys: &String.upcase/1

  • strings: :copy | :reference - Accepted for Jason compatibility (both behave identically).

  • objects: :ordered_objects - Decode JSON objects as %RustyJson.OrderedObject{} structs that preserve key insertion order. Built in Rust during parsing for zero overhead. Key transforms (:atoms, :atoms!, custom functions) apply to OrderedObject keys as well.
  • floats: :decimals - Decode JSON floats as %Decimal{} structs for exact decimal representation. Decimal components are parsed in Rust.
  • decoding_integer_digit_limit - Configurable maximum digits for integer parsing (default: 1024, 0 to disable). Also configurable at compile time via Application.compile_env(:rustyjson, :decoding_integer_digit_limit, 1024). Enforced in the Rust parser.

Encode Options

  • protocol: true is now the default - Encoding always goes through the RustyJson.Encoder protocol first, matching Jason's behavior. Use protocol: false to bypass the protocol for maximum performance.
  • maps: :strict - Detect duplicate serialized keys (e.g. atom :a and string "a" in the same map). Tracked via HashSet in Rust.
  • Pretty print keyword opts - pretty: [indent: 4, line_separator: "\r\n", after_colon: ""] for full control over formatting. Separators are passed to and applied in Rust.
  • iodata indent - The :indent option now accepts strings/iodata (e.g. pretty: "\t" for tab indentation), matching Jason's Formatter behavior. Indent strings are passed to Rust and applied directly.

New Modules

  • RustyJson.Decoder - Thin wrapper matching Jason's Decoder API. Provides parse/2 that delegates to RustyJson.decode/2.
  • RustyJson.Encode - Low-level encoding functions (value/2, atom/2, integer/1, float/1, list/2, keyword/2, map/2, string/2, struct/2, key/2), compatible with Jason's Encode module. keyword/2 preserves insertion order (does not convert to map).
  • RustyJson.Helpers - Compile-time macros json_map/1 and json_map_take/2 that pre-encode JSON object keys at compile time for faster runtime encoding. Preserves key insertion order, propagates encoding options (escape, maps) at runtime via function-based Fragments.
  • RustyJson.Sigil - ~j sigil (runtime, supports interpolation) and ~J sigil (compile-time) for JSON literals. Modifiers: a (atoms), A (atoms!), r (reference), c (copy). Unknown modifiers raise ArgumentError.
  • RustyJson.OrderedObject - Order-preserving JSON object struct with Access behaviour and Enumerable protocol.

Error Factories

  • RustyJson.EncodeError.new/1 - Factory functions for creating structured encode errors: new({:duplicate_key, key}) and new({:invalid_byte, byte, original}).

Changed

Error Return Types (Breaking)

  • decode/2 now returns {:error, %RustyJson.DecodeError{}} instead of {:error, String.t()}. The DecodeError struct includes :message, :data, :position, and :token fields for detailed error diagnostics.

  • encode/2 now returns {:error, %RustyJson.EncodeError{} | Exception.t()} instead of {:error, String.t()}.

  • encode!/2 now wraps NIF errors in %RustyJson.EncodeError{} instead of leaking ErlangError.

Encoder Protocol (Breaking)

  • RustyJson.Encoder protocol changed from encode/1 to encode/2 with an opts parameter, matching Jason's Encoder protocol. The opts parameter carries encoder options (:escape, :maps) as a keyword list, enabling custom implementations like Fragment and OrderedObject to respect encoding context. All protocol implementations must update from def encode(value) to def encode(value, _opts).
  • protocol: true is now the default in encode/2 and encode!/2. Previously required explicit opt-in. This matches Jason, which always dispatches through its Encoder protocol.
  • Any fallback now raises Protocol.UndefinedError for structs without an explicit RustyJson.Encoder implementation, matching Jason. Previously, structs were silently encoded via Map.from_struct/1. There is no fallback to Jason's Encoder — RustyJson is a complete replacement, not a bridge.
  • MapSet and Range now raise Protocol.UndefinedError by default (matching Jason). Previously had pass-through encoder implementations. Use protocol: false to encode them via the Rust NIF directly.

Formatter API (Breaking)

  • RustyJson.Formatter.pretty_print/2 now returns binary() directly instead of {:ok, binary()} | {:error, String.t()}. Raises RustyJson.DecodeError on invalid input.

  • RustyJson.Formatter.minimize/2 now returns binary() directly. Same error behavior.
  • pretty_print_to_iodata/2 returns iodata() directly.
  • minimize_to_iodata/2 returns iodata() directly. No default for opts parameter (matching Jason).
  • Removed pretty_print!/2, pretty_print_to_iodata!/2, minimize!/2, minimize_to_iodata!/2 — Jason does not have bang variants for Formatter.
  • Stream-based rewrite — Formatter internals ported from Jason's stream-based approach. Preserves key order and number formatting during pretty-print and minimize operations.

OrderedObject

  • pop/2 changed to pop/3 with an optional default parameter (default: nil), matching Jason's OrderedObject pop/3.
  • Key type changed from String.t() to String.Chars.t() for Jason compatibility.

Fixed

  • Large integer precision — Integers exceeding u64::MAX are now decoded using arbitrary-precision BigInt (via num-bigint) instead of falling back to f64, which lost precision. Matches Jason's behavior of preserving exact integer values regardless of magnitude.
  • html_safe forward slash escaping — escape: :html_safe now correctly escapes / as \/, matching Jason. Previously / was only escaped in unicode/javascript safe modes.
  • Encoding options propagation — escape and maps options now flow correctly through the Encoder protocol, Fragment functions, Helpers macros, and OrderedObject encoding. Previously these options were consumed before reaching protocol implementations.
  • Nested Fragment encoding — Fragments nested inside maps or lists now encode correctly in both protocol: true and protocol: false modes. The Encoder protocol now resolves Fragment functions to iodata immediately instead of wrapping in another closure, and resolve_fragment_functions recursively traverses maps and lists to resolve any remaining function-based Fragments before the Rust NIF.
  • Helpers key validation regex — Fixed character class regex that incorrectly rejected alphabetic keys. Now uses hex escapes for correct ASCII range matching.

Performance

No regressions. Relative speedup vs Jason is unchanged from v0.2.0.

Testing

  • 394 tests, all passing with 0 failures.
  • New test files: encode_test.exs, helpers_test.exs, sigil_test.exs, ordered_object_test.exs, decoder_test_module_test.exs.
  • Updated all error pattern matches across test suite for new structured error returns.
  • Tightened formatter tests to use exact-match assertions instead of substring matches.
  • Added coverage for: large integer precision, html_safe / escaping, Fragment function opts propagation, Helpers opts flow, OrderedObject key transforms and encoding opts, atoms/atoms! key decoding, sigil unknown modifiers, MapSet/Range protocol errors.

0.2.0 - 2025-01-25

Added

  • Faster decoding for API responses and bulk data (keys: :intern) - ~30% speedup for the most common JSON patterns

    Modern APIs return arrays of objects with the same shape: paginated endpoints (GET /users), GraphQL queries, database results, webhook events, ElasticSearch hits. This is the vast majority of JSON most applications decode.

    With keys: :intern, RustyJson caches object keys during parsing so identical keys like "id", "name", "created_at" are allocated once and reused across all objects in the array.

    # Before: allocates "id", "name", "email" for every object
RustyJson.decode!(json)

# After: allocates each key once, reuses for all 10,000 objects
RustyJson.decode!(json, keys: :intern)  # ~30% faster

    Caution: Don't use for single objects or varied schemas—the cache overhead makes it 2-3x slower when keys aren't reused. Only use for homogeneous arrays of 10+ objects.

    See BENCHMARKS.md for detailed performance data.

Documentation

  • Error handling documentation - Added comprehensive documentation highlighting RustyJson's clear, actionable error messages and consistent {:error, reason} returns:

    # Clear error messages describe the problem
RustyJson.decode(~s({"key": "value\\'s"}))
# => {:error, "Invalid escape sequence: \\'"}

# Consistent error tuples for invalid input
RustyJson.encode(%{{:tuple, :key} => 1})
# => {:error, "Map key must be atom, string, or integer"}

  • Added error handling sections to README, moduledoc, and ARCHITECTURE.md

Testing

  • Automated JSONTestSuite regression tests - Added repeatable test suite that validates against JSONTestSuite on every test run. Ensures the documented 283/283 compliance doesn't regress. Fixtures are downloaded on first run to test/fixtures/ (gitignored).

  • Added keys: :intern validation against the full JSONTestSuite.

0.1.1 - 2025-01-24

Changed

  • Updated hex.pm description for better discoverability

0.1.0 - 2025-01-24

Added

  • High-performance JSON encoding - 3-6x faster than Jason for medium/large payloads
  • Memory efficient - 10-20x lower memory usage during encoding
  • Full JSON spec compliance - 89 tests covering RFC 8259
  • Drop-in Jason replacement - Compatible API with encode/2, decode/2, encode_to_iodata/2
  • Phoenix integration - Works with config :phoenix, :json_library, RustyJson

Encoding Features

  • Native Rust handling for DateTime, NaiveDateTime, Date, Time, Decimal, URI, MapSet, Range
  • Multiple escape modes: :json, :html_safe, :javascript_safe, :unicode_safe
  • Pretty printing with configurable indentation
  • Gzip compression with compress: :gzip option
  • lean: true option for maximum performance (skips struct type detection)
  • protocol: true/false option for custom encoding via RustyJson.Encoder protocol (default changed to true in v0.3.0)

Jason Compatibility

Safety

  • Zero unsafe code — all SIMD uses portable std::simd (safe), no raw intrinsics
  • 128-level nesting depth limit per RFC 7159
  • Safe Rust guarantees memory safety at compile time

Technical Details

  • Built with Rustler 0.37+ for modern OTP compatibility (24-27)
  • Uses mimalloc as default allocator
  • Uses itoa and ryu for fast number formatting
  • Uses lexical-core for number parsing
  • Zero-copy string handling in decoder for unescaped strings
  • SIMD-accelerated escape scanning via portable std::simd