RustyJson Benchmarks

Comprehensive benchmarks comparing RustyJson vs Jason across synthetic and real-world datasets.

Key Findings

1. Fast across all workloads — plain data, struct-heavy data, and decoding (including deeply nested and small payloads)
2. Encoding plain data shows the largest gains — 3-6x faster, 2-3x less memory
3. Struct encoding optimized in v0.3.3 via single-pass iodata pipeline with compile-time codegen (~2x improvement over v0.3.2)
4. Deep-nested decode optimized in v0.3.3 via single-entry fast path (~27% faster than v0.3.2 for 100-level nested JSON)
5. Larger payloads = bigger advantage — real-world 10 MB files show better results than synthetic benchmarks
6. BEAM scheduler load dramatically reduced — 100-28,000x fewer reductions

Test Environment

Real-World Benchmarks: Amazon Settlement Reports

These are production JSON files from Amazon SP-API settlement reports, representing real-world API response patterns with nested objects, arrays of transactions, and mixed data types.

Encoding Performance (Elixir → JSON)

Decoding Performance (JSON → Elixir)

BEAM Reductions (Scheduler Load)

This is the most dramatic difference - RustyJson offloads virtually all work to native code.

Synthetic Benchmarks: nativejson-benchmark

Using standard datasets from nativejson-benchmark:

Decode Performance (JSON → Elixir)

Roundtrip Performance (Decode + Encode)

BEAM Reductions by Dataset

Struct Encoding Benchmarks (v0.3.3+)

Encoding data that contains Elixir structs (e.g., @derive RustyJson.Encoder or custom defimpl) follows a different path than plain maps and lists. Structs require the RustyJson.Encoder protocol to convert them to JSON-serializable forms.

In v0.3.3, the struct encoding pipeline was rewritten from a three-pass approach (protocol dispatch → fragment resolution → NIF serialization) to a single-pass iodata pipeline with compile-time codegen for derived structs. This closed the last remaining performance gap, making RustyJson faster across all encoding workloads.

Struct Encoding Performance

Measured with protocol consolidation enabled (MIX_ENV=prod), which is the default for production builds.

How It Works

RustyJson's struct encoding produces iodata in a single pass:

1. Derived encoders (@derive RustyJson.Encoder) generate compile-time iodata templates with pre-escaped keys — no runtime Map.from_struct, Map.to_list, or key escaping.
2. Map/List impls detect struct-containing data and route through Encode.map/2 / Encode.list/2 to build iodata directly, wrapped in a Fragment.
3. NIF bypass — When the top-level result is an iodata Fragment (no pretty-print or compression), IO.iodata_to_binary/1 is used directly, avoiding Erlang↔Rust term conversion entirely.

For plain data (no structs), encoding still uses the fast Rust NIF path unchanged.

Why Encoding Shows Bigger Gains

iolist Encoding Pattern (Pure Elixir)

encode(data)
  → allocate "{" binary
  → allocate "\"key\"" binary
  → allocate ":" binary
  → allocate "\"value\"" binary
  → allocate list cells to link them
  → return iolist (many BEAM allocations)

RustyJson's Encoding Pattern (NIF)

encode(data)
  → [Rust: walk terms, write to single buffer]
  → copy buffer to BEAM binary
  → return binary (one BEAM allocation)

Pure-Elixir encoders create many small BEAM allocations. RustyJson creates one.

Why Decoding Memory is Similar

Both libraries produce identical Elixir data structures when decoding. The resulting maps, lists, and strings take the same space regardless of which library created them.

Why Benchee Memory Measurements Don't Work for NIFs

Important: Benchee's memory_time option gives misleading results for NIF-based libraries.

What Benchee Reports (Incorrect)

| Library   | Memory    |
|-----------|-----------|
| RustyJson | 0.00169 MB |
| Jason     | 20.27 MB   |

This suggests 12,000x less memory - which is wrong.

Why This Happens

Benchee measures memory using :erlang.memory/0, which only tracks BEAM allocations:

• BEAM process heap
• BEAM binary space
• ETS tables

RustyJson allocates memory in Rust via mimalloc, completely invisible to BEAM tracking. The 0.00169 MB is just NIF call overhead.

How We Measure Instead

We use :erlang.memory(:total) delta in isolated spawned processes:

spawn(fn ->
  :erlang.garbage_collect()
  before = :erlang.memory(:total)
  results = for _ <- 1..10, do: RustyJson.encode!(data)
  after_mem = :erlang.memory(:total)
  # Report (after_mem - before) / 10
end)

This captures BEAM allocations during the operation. For total system memory (including NIF), we verified with RSS measurements that Rust adds only ~1-2 MB temporary overhead.

Actual Memory Comparison

For a 10 MB settlement report encode:

Running Benchmarks

# 1. Download synthetic test data
mkdir -p bench/data && cd bench/data
curl -LO https://raw.githubusercontent.com/miloyip/nativejson-benchmark/master/data/canada.json
curl -LO https://raw.githubusercontent.com/miloyip/nativejson-benchmark/master/data/citm_catalog.json
curl -LO https://raw.githubusercontent.com/miloyip/nativejson-benchmark/master/data/twitter.json
cd ../..

# 2. Run memory benchmarks (no extra deps needed)
mix run bench/memory_bench.exs

# 3. (Optional) Run speed benchmarks with Benchee
# Add to mix.exs: {:benchee, "~> 1.0", only: :dev}
mix deps.get
mix run bench/stress_bench.exs

Key Interning Benchmarks

The keys: :intern option provides significant speedups when decoding arrays of objects with repeated keys (common in API responses, database results, etc.).

When Key Interning Helps: Homogeneous Arrays

Arrays where every object has the same keys:

[{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}, ...]

When Key Interning Hurts: Unique Keys

Single objects or heterogeneous arrays where keys aren't repeated:

Scaling: Benefit Increases with Object Count

With 5 keys per object, the benefit grows as more objects reuse the cached keys:

Usage Recommendation

# API responses, database results, bulk data
RustyJson.decode!(json, keys: :intern)

# Config files, single objects, unknown schemas
RustyJson.decode!(json)  # default, no interning

Rule of thumb: Use keys: :intern when you know you're decoding arrays of 10+ objects with the same schema.

Note: Keys containing escape sequences (e.g., "field\nname") are not interned because the raw JSON bytes differ from the decoded string. This is rare in practice and has negligible performance impact.

Summary

*For arrays of objects with repeated keys (API responses, DB results, etc.)

Bottom line: As of v0.3.3, RustyJson is fast across all encoding and decoding workloads, including deeply nested and small payloads. Plain data encoding shows the largest gains (5-6x, 2-3x less memory, dramatically fewer BEAM reductions). Struct encoding was rewritten in v0.3.3 with a single-pass iodata pipeline. Deep-nested decode was optimized in v0.3.3 with a single-entry fast path that avoids heap allocation for single-element objects and arrays. For decoding bulk data, enable keys: :intern for an additional 30% speedup.