# `RustyJson` [🔗](https://github.com/jeffhuen/rustyjson/blob/v0.3.9/lib/rustyjson.ex#L1) A high-performance JSON library for Elixir powered by Rust NIFs. RustyJson is designed as a **drop-in replacement for Jason** with significantly better performance characteristics: - **3-6x faster encoding** for medium/large payloads - **10-20x lower memory usage** during encoding - **Full JSON spec compliance** (RFC 8259) - **Native type support** for DateTime, Decimal, URI, and more ## Quick Start # Encoding iex> RustyJson.encode!(%{name: "Alice", age: 30}) ~s({"age":30,"name":"Alice"}) # Decoding iex> RustyJson.decode!(~s({"name":"Alice","age":30})) %{"age" => 30, "name" => "Alice"} # Pretty printing iex> RustyJson.encode!(%{items: [1, 2, 3]}, pretty: true) """ { "items": [ 1, 2, 3 ] } """ ## Phoenix Integration Configure Phoenix to use RustyJson as the JSON library: # config/config.exs config :phoenix, :json_library, RustyJson ## Why RustyJson? Traditional JSON libraries in Elixir create many intermediate binary allocations during encoding, which pressures the garbage collector. RustyJson eliminates this by walking the Erlang term tree directly in Rust and writing to a single buffer. For a detailed comparison, see the [README](readme.html). ## Module Overview | Module | Description | |--------|-------------| | `RustyJson` | Main encoding/decoding API | | `RustyJson.Encoder` | Protocol for custom type encoding | | `RustyJson.Encode` | Low-level encoding functions | | `RustyJson.Fragment` | Pre-encoded JSON injection | | `RustyJson.Formatter` | JSON pretty-printing utilities | | `RustyJson.Helpers` | Compile-time JSON macros (`json_map`, `json_map_take`) | | `RustyJson.Sigil` | `~j`/`~J` sigils for JSON literals | | `RustyJson.OrderedObject` | Order-preserving JSON object (for `objects: :ordered_objects`) | | `RustyJson.Decoder` | JSON decoding module (Jason.Decoder compatible) | | `RustyJson.DecodeError` | Decoding error exception | | `RustyJson.EncodeError` | Encoding error exception | ## Built-in Type Support RustyJson natively handles these Elixir types without protocol overhead: | Elixir Type | JSON Output | Example | |-------------|-------------|---------| | `map` | object | `%{a: 1}` → `{"a":1}` | | `list` | array | `[1, 2]` → `[1,2]` | | `tuple` | array | `{1, 2}` → `[1,2]` | | `binary` | string | `"hello"` → `"hello"` | | `integer` | number | `42` → `42` | | `float` | number | `3.14` → `3.14` | | `true/false` | boolean | `true` → `true` | | `nil` | null | `nil` → `null` | | `atom` | string | `:hello` → `"hello"` | | `DateTime` | ISO8601 string | `~U[2024-01-15 14:30:00Z]` → `"2024-01-15T14:30:00Z"` | | `NaiveDateTime` | ISO8601 string | `~N[2024-01-15 14:30:00]` → `"2024-01-15T14:30:00"` | | `Date` | ISO8601 string | `~D[2024-01-15]` → `"2024-01-15"` | | `Time` | ISO8601 string | `~T[14:30:00]` → `"14:30:00"` | | `Decimal` | string | `Decimal.new("123.45")` → `"123.45"` | | `URI` | string | `URI.parse("https://example.com")` → `"https://example.com"` | | structs | object | `%User{name: "Alice"}` → `{"name":"Alice"}` (requires `@derive RustyJson.Encoder` or explicit `defimpl`) | > **Note:** `MapSet` and `Range` are **not** natively encoded. They require an explicit > `RustyJson.Encoder` implementation or `protocol: false` to encode via the Rust NIF directly. > This matches Jason's behavior. ## Escape Modes RustyJson supports multiple escape modes for different security contexts: | Mode | Description | Use Case | |------|-------------|----------| | `:json` | Standard JSON escaping (default) | General use | | `:html_safe` | Escapes `<`, `>`, `&` as `\uXXXX` and `/` as `\/` | HTML embedding | | `:javascript_safe` | Escapes line/paragraph separators | JavaScript strings | | `:unicode_safe` | Escapes all non-ASCII as `\uXXXX` | ASCII-only output | ## Performance Tips 1. **Bypass the protocol**: The protocol is enabled by default for Jason compatibility. If you have no custom `RustyJson.Encoder` implementations, use `protocol: false` to bypass protocol dispatch for maximum speed. 2. **Use lean mode**: If you don't have DateTime/Decimal types, use `lean: true` to skip struct type detection in Rust. 3. **Use compression**: For large payloads over the network, `compress: :gzip` reduces output size 5-10x. 4. **Avoid `keys: :atoms` with untrusted input**: `keys: :atoms` uses `String.to_atom/1`, which can exhaust the atom table. Use `keys: :atoms!` (which uses `String.to_existing_atom/1`) or `keys: :strings` (default) instead. 5. **Use key interning for bulk data**: When decoding arrays of objects with the same schema (API responses, database results, webhooks), use `keys: :intern` for ~30% faster parsing: RustyJson.decode!(json, keys: :intern) The intern cache is capped at 4096 unique keys. Beyond that, new keys are allocated normally — this bounds worst-case CPU time and stops paying cache overhead when keys aren't being reused. **Caution**: Don't use for single objects or varied schemas—cache overhead makes it 2-3x *slower* when keys aren't reused. ## Error Handling RustyJson provides clear, actionable error messages. `encode/1` and `decode/1` consistently return `{:error, reason}` tuples for invalid input. # Error messages describe the problem RustyJson.decode(~s({"key": "value\\'s"})) # => {:error, "Invalid escape sequence: \\'"} # Unencodable values return error tuples RustyJson.encode(%{{:tuple, :key} => 1}) # => {:error, "Map key must be atom, string, or integer"} # Strict UTF-16 surrogate validation per RFC 7493 RustyJson.decode(~s("\\uD800")) # => {:error, "Lone surrogate in string"} This makes error handling predictable—pattern match on results without needing `try/rescue` blocks. # `compression_algorithm` ```elixir @type compression_algorithm() :: :gzip | :none ``` Supported compression algorithms for encoding. - `:gzip` - Standard gzip compression - `:none` - No compression (default) # `compression_level` ```elixir @type compression_level() :: 0..9 ``` Compression level from 0 (fastest, least compression) to 9 (slowest, best compression). # `compression_option` ```elixir @type compression_option() :: :gzip | {:gzip, compression_level()} | :none ``` Compression options tuple. Can be specified as: - `:gzip` - Use default compression level - `{:gzip, level}` - Use specific compression level (0-9) - `:none` - No compression # `compression_options` ```elixir @type compression_options() :: {compression_algorithm(), compression_level() | nil} ``` Internal compression options format passed to NIF. # `decode_opt` ```elixir @type decode_opt() :: {:keys, keys()} | {:strings, :copy | :reference} | {:objects, :maps | :ordered_objects} | {:floats, :native | :decimals} | {:decoding_integer_digit_limit, non_neg_integer()} | {:max_bytes, non_neg_integer()} | {:duplicate_keys, :last | :error} | {:validate_strings, boolean()} | {:dirty_threshold, non_neg_integer()} ``` Options for `decode/2` and `decode!/2`. - `:keys` - How to handle object keys (see `t:keys/0`). Default: `:strings` - `:strings` - How to handle decoded strings. `:copy` or `:reference`. Both produce copies (RustyJson always copies). Default: `:reference` - `:objects` - How to decode JSON objects. `:maps` (default) or `:ordered_objects` - `:floats` - How to decode JSON floats. `:native` (default) or `:decimals` - `:decoding_integer_digit_limit` - Maximum digits in integer part. 0 disables. Default: 1024, or the value of `Application.compile_env(:rustyjson, :decoding_integer_digit_limit)` - `:max_bytes` - Maximum input size in bytes. 0 means unlimited (default). The check is performed using `IO.iodata_length/1` *before* converting to binary, avoiding the memory spike from allocating the full binary. - `:duplicate_keys` - How to handle duplicate object keys. `:last` (default) uses last-wins semantics. `:error` rejects objects with duplicate keys. **Performance note**: `:error` adds per-key overhead from HashSet tracking. Use only when strict validation is needed. - `:validate_strings` - Whether to validate that decoded strings contain valid UTF-8. Default: `true`. When `true`, rejects strings with invalid UTF-8 byte sequences. Set to `false` to skip validation for maximum throughput on trusted input. - `:dirty_threshold` - Byte size threshold for auto-dispatching to dirty CPU scheduler. When input size >= this threshold, decode runs on a dirty scheduler to avoid blocking normal BEAM schedulers. Default: 102400 (100KB). Set to 0 to disable. # `encode_opt` ```elixir @type encode_opt() :: {:pretty, boolean() | pos_integer() | keyword()} | {:escape, escape_mode()} | {:compress, compression_option()} | {:protocol, boolean()} | {:lean, boolean()} | {:maps, :naive | :strict} | {:sort_keys, boolean()} | {:scheduler, :auto | :normal | :dirty} ``` Options for `encode/2` and `encode!/2`. - `:pretty` - Pretty print with indentation. `true` for 2 spaces, an integer for custom spacing, a string/iodata for custom indent (e.g. `"\t"` for tabs), or a keyword list with `:indent`, `:line_separator`, and `:after_colon` keys. - `:escape` - Escape mode (see `t:escape_mode/0`). Default: `:json` - `:compress` - Compression (see `t:compression_option/0`). Default: `:none` - `:protocol` - Use `RustyJson.Encoder` protocol. Default: `true` - `:lean` - Skip special struct handling. Default: `false` - `:maps` - Key uniqueness mode. `:naive` (default) allows duplicate serialized keys, `:strict` raises on duplicate keys (e.g. atom `:a` and string `"a"` in the same map). # `escape_mode` ```elixir @type escape_mode() :: :json | :html_safe | :javascript_safe | :unicode_safe ``` Escape mode for JSON string encoding. - `:json` - Standard JSON escaping (default) - `:html_safe` - Also escape `<`, `>`, `&`, `/` for safe HTML embedding - `:javascript_safe` - Also escape line/paragraph separators (U+2028, U+2029) - `:unicode_safe` - Escape all non-ASCII characters as `\uXXXX` # `keys` ```elixir @type keys() :: :strings | :atoms | :atoms! | :copy | :intern | (String.t() -> term()) ``` Options for decoding JSON object keys. - `:strings` - Keep keys as strings (default, safe) - `:atoms` - Convert to atoms using `String.to_atom/1` (unsafe with untrusted input) - `:atoms!` - Convert to existing atoms using `String.to_existing_atom/1` (safe, raises if missing) - `:copy` - Copy key binaries (same as `:strings` in RustyJson since NIFs always copy) - `:intern` - Cache repeated keys during parsing (~30% faster for arrays of objects). The cache is capped at 4096 unique keys — beyond this, new keys are allocated normally. This bounds worst-case CPU time and avoids cache overhead for pathological inputs with many distinct keys. The cap is internal and not user-configurable. - A function of arity 1 - Applied to each key string recursively # `decode` ```elixir @spec decode(iodata(), [decode_opt()]) :: {:ok, term()} | {:error, RustyJson.DecodeError.t()} ``` Decodes a JSON string to an Elixir term. Returns `{:ok, term}` on success or `{:error, reason}` on failure. ## Options * `:keys` - How to decode object keys. One of: * `:strings` - Keep as strings (default, safe) * `:atoms` - Convert to atoms (unsafe with untrusted input) * `:atoms!` - Convert to existing atoms only (safe, raises if atom missing) * `:copy` - Copy key binaries (equivalent to `:strings` in RustyJson) * `:intern` - Cache repeated keys during parsing. **~30% faster** for arrays of objects with the same schema (REST APIs, GraphQL, database results, webhooks). The cache is capped at 4096 unique keys; beyond that, new keys are allocated normally (no worse than default). This bounds worst-case CPU time and stops paying cache overhead when keys aren't being reused. **Caution**: 2-3x slower for single objects or varied schemas—only use for homogeneous arrays of 10+ objects. * A function of arity 1 - Applied recursively to each key string. Example: `keys: &String.upcase/1` * `:strings` - How to handle decoded strings. `:reference` (default) or `:copy`. Both produce copies in RustyJson (Rust NIFs always copy into BEAM binaries), so this option exists for Jason API compatibility. * `:objects` - How to decode JSON objects. `:maps` (default) or `:ordered_objects`. When `:ordered_objects`, returns `%RustyJson.OrderedObject{}` structs that preserve key insertion order. * `:floats` - How to decode JSON floats. `:native` (default) returns Elixir floats, `:decimals` returns `%Decimal{}` structs for exact decimal representation. * `:decoding_integer_digit_limit` - Maximum number of digits allowed in the integer part of a JSON number. Integers exceeding this limit cause a decode error. Default: `1024`, or the value of `Application.compile_env(:rustyjson, :decoding_integer_digit_limit)`. Set to `0` to disable the limit. ## Examples iex> RustyJson.decode(~s({"name":"Alice","age":30})) {:ok, %{"age" => 30, "name" => "Alice"}} iex> RustyJson.decode(~s([1, 2, 3])) {:ok, [1, 2, 3]} iex> RustyJson.decode("invalid") {:error, "Unexpected character at position 0"} ## Security Considerations **Avoid `keys: :atoms` with untrusted input.** Atoms are not garbage collected, so an attacker could exhaust your atom table by sending JSON with many unique keys. Use `keys: :atoms!` if you expect specific keys to exist, or `keys: :strings` (default). See `decode!/2` for a version that raises on error. # `decode!` ```elixir @spec decode!(iodata(), [decode_opt()]) :: term() ``` Decodes a JSON string to an Elixir term, raising on error. Same as `decode/2` but raises `RustyJson.DecodeError` on failure. The raised exception includes `:data`, `:position`, and `:token` fields for detailed error diagnostics. ## Options See `decode/2` for available options. ## Examples iex> RustyJson.decode!(~s({"x": [1, 2, 3]})) %{"x" => [1, 2, 3]} iex> RustyJson.decode!(~s({"x": 1}), keys: :atoms) %{x: 1} iex> RustyJson.decode!(~s({"x": 1}), keys: &String.upcase/1) %{"X" => 1} iex> RustyJson.decode!("null") nil iex> RustyJson.decode!("true") true iex> RustyJson.decode!(~s({"price":19.99}), floats: :decimals) %{"price" => Decimal.new("19.99")} ## JSON Types to Elixir | JSON | Elixir | |------|--------| | object | map (or `RustyJson.OrderedObject` with `objects: :ordered_objects`) | | array | list | | string | binary | | number (int) | integer | | number (float) | float (or `Decimal` with `floats: :decimals`) | | true | `true` | | false | `false` | | null | `nil` | ## Error Cases iex> RustyJson.decode!("invalid") ** (RustyJson.DecodeError) Unexpected character at position 0 iex> RustyJson.decode!("{trailing: comma,}") ** (RustyJson.DecodeError) Expected string key at position 1 # `encode` ```elixir @spec encode(term(), [encode_opt()]) :: {:ok, String.t()} | {:error, RustyJson.EncodeError.t() | Exception.t()} ``` Encodes an Elixir term to a JSON string. Returns `{:ok, json}` on success or `{:error, reason}` on failure. ## Options * `:pretty` - Pretty print with indentation. `true` uses 2 spaces, or pass an integer for custom spacing. Default: `false` * `:escape` - Escape mode for special characters. One of `:json` (default), `:html_safe`, `:javascript_safe`, or `:unicode_safe`. See `t:escape_mode/0`. * `:compress` - Compression algorithm. `:gzip` or `{:gzip, 0..9}` for specific level. Default: `:none` * `:protocol` - Enable `RustyJson.Encoder` protocol for custom types. Default: `true` * `:lean` - Skip struct type detection (DateTime, Decimal, etc. encoded as raw maps). Default: `false` * `:maps` - Key uniqueness mode. `:naive` (default) allows duplicate serialized keys, `:strict` errors on duplicate keys (e.g. atom `:a` and string `"a"`). * `:sort_keys` - Sort map keys lexicographically in output. Default: `false`. JSON objects are unordered per RFC 8259. Enable for deterministic output (useful for snapshot tests, caching, or diffing). Note: Jason always sorts keys; RustyJson does not by default for performance. ## Examples iex> RustyJson.encode(%{name: "Alice", scores: [95, 87, 92]}) {:ok, ~s({"name":"Alice","scores":[95,87,92]})} iex> RustyJson.encode(%{valid: true}, pretty: true) {:ok, "{\n \"valid\": true\n}"} iex> RustyJson.encode("invalid UTF-8: " <> <<0xFF>>) {:error, "Failed to decode binary as UTF-8"} ## Error Handling Common error cases: - Invalid UTF-8 binary - Non-finite float (NaN, Infinity) - Circular references (will cause stack overflow) See `encode!/2` for a version that raises on error. # `encode!` ```elixir @spec encode!(term(), [encode_opt()]) :: String.t() ``` Encodes an Elixir term to a JSON string, raising on error. Same as `encode/2` but raises `RustyJson.EncodeError` on failure. ## Options See `encode/2` for available options. ## Examples iex> RustyJson.encode!(%{hello: "world"}) ~s({"hello":"world"}) iex> RustyJson.encode!([1, 2, 3], pretty: 4) """ [ 1, 2, 3 ] """ iex> RustyJson.encode!(~D[2024-01-15]) ~s("2024-01-15") iex> RustyJson.encode!(%{html: "