Once is an Ecto type for locally unique (unique within your domain or application) 64-bit IDs generated by multiple Elixir nodes. Locally unique IDs make it easier to keep things separated, simplify caching and simplify inserting related things (because you don't have to wait for the database to return the ID).

Because a Once fits into an SQL bigint, they use little space and keep indexes small and fast. Because of their structure they have counter-like data locality, which helps your indexes perform well, unlike UUIDv4s.

Once IDs are based on counter, time-sortable or encrypted nonces. These underlying values can then be encoded in several formats, can be prefixed and can be masked. And you can combine all of these options, providing great flexibility.

The library has only Ecto and its sibling NoNoncense as dependencies. NoNoncense generates the actual values and performs incredibly well, hitting rates of tens of millions of nonces per second, and it also helps you to safeguard the uniqueness guarantees.

Quick Start

To get going, you need to set up a NoNoncense instance to generate the base unique values. Follow its documentation to do so. Once expects an instance with its own module name by default, like so:

# application.ex (read the NoNoncense docs!)
machine_id = NoNoncense.MachineId.id!(opts)
NoNoncense.init(name: Once, machine_id: machine_id)

# if you want to use encrypted/masked IDs, also pass a 256-bits key
NoNoncense.init(name: Once, machine_id: machine_id, base_key: System.get_env("ONCE_SECRET_KEY"))

In your Ecto schemas, you can then use the type:

schema "things" do
  field :id, Once, autogenerate: true
end

And that's it!

Core Concepts

ID Generation Types

Once supports three types of ID generation, controlled by the :nonce_type option:

• Counter (default): Really fast to generate, predictable, works well with b-tree indexes. IDs are generated using a machine init timestamp, machine ID and counter.
• Sortable: Time-sortable like a Snowflake ID. Use this when chronological ordering is important.
• Encrypted: Unique and unpredictable, like a UUIDv4 but shorter. Use when you need unpredictable IDs. Note that encrypted IDs cost you data locality, decrease index performance and are slightly slower to generate. Alternatively, use masked counter/sortable IDs.

Data Formats

IDs can be represented in multiple formats, both in your Elixir application (:ex_format) and in your database (:db_format). By default, IDs are url64-encoded 11-character strings in Elixir and signed bigints in the database.

These are the formats in which an ID can be rendered. All are equivalent and can be transformed to one another using to_format/3. The examples represent the same underlying value.

• :url64 a url64-encoded string of 11 characters, for example "zAjhfZyAAAE"
• :hex a hex-encoded string of 16 characters, for example "cc08e17d9c800001"
• :raw a bitstring of 64 bits, for example <<204, 8, 225, 125, 156, 128, 0, 1>>
• :signed a signed 64-bits integer between -(2^63) and 2^63-1, for example -3744495160545771519
• :unsigned an unsigned 64-bits integer between 0 and 2^64-1, for example 14702248913163780097
• :hex32 an extended-hex string of 13 characters, for example "pg4e2vcsg0002"

With the default formats, you might see "AAAAAACYloA" in Elixir and 10_000_000 in your database. The reasoning behind these defaults is that the encoded format is readable, short, and JSON-safe by default, while the signed format means you can use a standard bigint column type (resulting in a small, fast index).

The negative integers will not cause problems with Postgres and MySQL - they both happily accept them. Negative integers will only start to appear after ~70 years of usage. However, be careful if you wish to sort by ID (see Sorting Considerations).

Prefixed IDs

You can add prefixes to IDs to make them recognizable and self-documenting. This is useful for debugging, API clarity and type safety. A prefixed ID looks like "usr_AV7m9gAAAAU" or "prod_123" - a human-readable prefix followed by the actual ID value. Use the :prefix option to specify the desired prefix.

Note that prefixed IDs are always binaries - using :unsigned or :signed as :ex_format results in numeric strings like "prod_123", not raw integers. Autogenerated IDs will include the prefix.

The prefixes can optionally be persisted to the database. This is controlled by option :persist_prefix, the default is false. If the prefix is not persisted, it only exists in Elixir and is stripped before storing an ID in the database, and re-added on load. This allows us to still use 64-bit integer columns. So in Elixir you could have "usr_AV7m9gAAAAU" while in the database you would have 98770186085072901.

With persist_prefix: true, the full prefixed string is stored. This sacrifices storage efficiency for database-level readability. Your ID would look like "usr_AV7m9gAAAAU" in both Elixir and your database. You can only use db_format: :url64, :hex, hex32 or :raw with persist_prefix: true.

Use to_format/3 with the :prefix option to convert between formats while preserving the prefix.

Encrypted IDs

If you need unpredictable IDs, you can use encrypted nonce generation. To use encrypted IDs:

• Set option nonce_type: :encrypted
• Initialize NoNoncense with option base_key: <some 32-byte secret binary>
• (optional but recommended) Change the encryption algorithm using option :cipher64 from the default :blowfish to :speck (requires optional dependency SpeckEx).

To learn more about nonce encryption and the available ciphers, see the NoNoncense docs.

Masked IDs

Masked IDs provide a middle ground between plaintext and encrypted IDs. IDs are stored as plaintext in the database (preserving sequential writes and index performance) but encrypted when retrieved by the Ecto type. The application only sees encrypted values while the database maintains optimal performance.

To use masked IDs:

• Set option mask: true
• Initialize NoNoncense with option base_key: <some 32-byte secret binary>
• Works with nonce types :counter and :sortable (there's no point in masking an encrypted nonce)

Benefits: Masked IDs have the database performance of plaintext IDs, but the application sees encrypted values. ORDER BY id and keyset pagination work transparently (cast decrypts input). No database migration is needed for existing IDs.

Trade-offs: Database and application IDs look completely different (operational friction with SQL/BI tools), slight performance cost for encryption/decryption on every read/write, ordering info can leak into the application when using ORDER BY id queries.

Configuration

Options

The Ecto type takes the following optional parameters:

• :no_noncense name of the NoNoncense instance used to generate new IDs (default Once)
• :ex_format what an ID looks like in Elixir, one of format/0 (default :url64). Be sure to read the caveats.
• :db_format what an ID looks like in your database, one of format/0 (default :signed)
• :nonce_type how the nonce is generated, one of nonce_type/0 (default :counter)
• :mask use encrypted IDs in Elixir; encrypt on load, decrypt on dump (default false)
• :prefix a string prefix to prepend to IDs, for example "usr_" or "prod_" (default nil)
• :persist_prefix whether to store the prefix in the database; requires a binary :db_format (default false)

Format Caveats

Some caveats apply to the :ex_format option.

While JSON does not impose a precision limit on numbers, JavaScript can only represent integers up to 2^53. Once IDs encoded as integers are larger than that within 24 days of the epoch passed to NoNoncense.

That's because we can't disambiguate some binaries that are valid hex, hex32, url64 and raw binaries and also valid numeric strings. An example is "12345678901":

# interpret as numeric string, format as signed int
iex> {:ok, 12345678901}          = Once.to_format("12345678901", :signed, parse_int: true)

# interpret as url64-encoded binary, format as signed int
iex> {:ok, -2923406909136636083} = Once.to_format("12345678901", :signed)

The :ex_format setting and :parse_int option resolve this ambiguity, at the cost of some flexibility.

Sorting Considerations

The various formats have different sorting behaviors for the same underlying value, which becomes particularly problematic with values equivalent to negative integers:

# unsigned ints (0, max-signed, max-unsigned a.k.a. -1)
iex> unsigned = [0, 9223372036854775807, 18446744073709551615]
iex> ^unsigned = Enum.sort_by(unsigned, &Once.to_format!(&1, :hex))
iex> ^unsigned = Enum.sort_by(unsigned, &Once.to_format!(&1, :hex32))
iex> ^unsigned = Enum.sort_by(unsigned, &Once.to_format!(&1, :raw))
iex> Enum.sort_by(unsigned, &Once.to_format!(&1, :signed))
[18446744073709551615, 0, 9223372036854775807]                  # [max-unsigned, 0, max-signed]
iex> Enum.sort_by(unsigned, &Once.to_format!(&1, :url64))
[0, 18446744073709551615, 9223372036854775807]                  # [0, max-unsigned, max-signed]

IDs with nonce_type: :encrypted or :counter can't be meaningfully sorted. Encrypted IDs are effectively random, and counter IDs generated on different machines can interleave unpredictably when sorted.

Masked IDs can be meaningfully sorted in a database because they are stored in plaintext, so ORDER BY id works as usual. In Elixir they are encrypted, which randomizes their sort order.

Sorting in your database / ORDER BY id queries

• PostgreSQL: Only supports signed integers. Using "ORDER BY id" will work fine until you reach negative numbers (same as :signed in Elixir). The easiest way to deal with this is to ignore the problem and assume that a) your app will not reach that age or b) Postgres will support unsigned ints at some point in the next 70 years. Alternatively, use db_format: :hex or :hex32. These are fixed-length, human-readable formats that sort correctly throughout the range, at the cost of storage space.
• MySQL: Supports unsigned bigint, so :unsigned format works perfectly for sorting.