# Building and Publishing OTP Release Tarballs

`MobDev.OtpDownloader` pulls pre-built OTP runtimes from a GitHub release on
`GenericJam/mob`. This file documents how to build and publish those tarballs
when upgrading OTP.

> **TL;DR — runnable form:** every step below is also implemented as a script
> under [`scripts/release/`](scripts/release/). See [`scripts/release/README.md`](scripts/release/README.md)
> for the typical full-release flow. The markdown carries the narrative; the
> scripts carry the imperative.

---

## What's in each tarball

Both tarballs must extract (via `--strip-components=1`) into a flat directory
that looks like an OTP release root:

```
erts-<vsn>/
  bin/          # ERTS executables (erl_child_setup, inet_gethost, epmd, ...)
  include/      # Headers (iOS only — see below)
  lib/          # Static libs + ERTS internal libs
    libbeam.a
    libzstd.a       (iOS only)
    libepcre.a      (iOS only)
    libryu.a        (iOS only)
    asn1rt_nif.a    (iOS only)
    crypto.a              # OTP's crypto NIF, built with -DSTATIC_ERLANG_NIF
    libcrypto.a           # OpenSSL 3.x, statically linked into crypto.a
    internal/
      liberts_internal_r.a
      libethread.a
lib/            # OTP stdlib (kernel, stdlib, elixir, logger, ...)
              # Includes lib/crypto-VSN, lib/public_key-VSN, lib/ssl-VSN
              # — real crypto, not a shim.
releases/
  29/
    start_clean.boot
    start_sasl.boot
```

### Android arm64 (`otp-android-<hash>.tar.gz`)

Built from a full cross-compiled OTP release for `aarch64-unknown-linux-android`.
Needs the same extra static libs and headers as iOS (see Step 2).

The ERTS helper binaries (`erl_child_setup`, `inet_gethost`, `epmd`) must be
in `erts-<vsn>/bin/`; `mob_dev` copies them into the APK as `lib*.so` (required
for SELinux `execve` permission on Android).

### Android arm32 (`otp-android-arm32-<hash>.tar.gz`)

Built from a full cross-compiled OTP release for `arm-unknown-linux-androideabi`
(armeabi-v7a). Same structure as arm64. Required for 32-bit-only devices (e.g.
Motorola E 2020).

`asn1rt_nif.a` must be compiled separately — it is not emitted by the OTP build
system for arm32. Build it with:

```bash
NDK=~/Library/Android/sdk/ndk/27.2.12479018/toolchains/llvm/prebuilt/darwin-x86_64/bin
OTP_SRC=~/code/otp

$NDK/armv7a-linux-androideabi21-clang \
  -march=armv7-a -mfloat-abi=softfp -mthumb \
  -fvisibility=hidden -fno-common -fno-strict-aliasing \
  -fstack-protector-strong -O2 \
  -I "$OTP_SRC/erts/arm-unknown-linux-androideabi" \
  -I "$OTP_SRC/erts/include/arm-unknown-linux-androideabi" \
  -I "$OTP_SRC/erts/emulator/beam" \
  -I "$OTP_SRC/erts/include" \
  -DHAVE_CONFIG_H \
  -DSTATIC_ERLANG_NIF_LIBNAME=asn1rt_nif \
  -c "$OTP_SRC/lib/asn1/c_src/asn1_erl_nif.c" \
  -o /tmp/asn1rt_nif_arm32.o

$NDK/llvm-ar rc /tmp/asn1rt_nif_arm32.a /tmp/asn1rt_nif_arm32.o
$NDK/llvm-ranlib /tmp/asn1rt_nif_arm32.a
```

Then include it in the tarball at `erts-<vsn>/lib/asn1rt_nif.a` (see Step 2b).

### iOS device (`otp-ios-device-<hash>.tar.gz`)

Built from a cross-compiled OTP for `aarch64-apple-ios`. Same install-tree
contents as the simulator tarball (libs, headers, ERTS bin), **plus** EPMD
source files and iOS configure output that `mob_dev`'s `build_device.sh`
needs at native-build time to static-link EPMD into the app.

The extra files added on top of the install tree:

```
erts/
  epmd/src/{epmd,epmd_srv,epmd_cli}.c    # EPMD C sources
  aarch64-apple-ios/config.h             # generated by ./configure for iOS arm64
  include/                               # cross-platform ERTS headers
  include/internal/                      # ERTS-internal headers (ethread, etc.)
```

The path layout — `erts/...` next to `erts-<vsn>/...` — looks unusual but
intentional: `erts-<vsn>/` is the *install* tree (compiled artifacts), `erts/`
is a stripped-down *source* tree fragment carrying just what `build_device.sh`
needs to compile EPMD against the iOS arm64 SDK.

`MobDev.OtpDownloader` validates these files are present after extraction;
older tarballs without them are treated as invalid and re-downloaded.

### iOS simulator (`otp-ios-sim-<hash>.tar.gz`)

Built from a cross-compiled OTP for `aarch64-apple-iossimulator`. Needs:

1. **Headers** at `erts-<vsn>/include/` (same set for both platforms, arch-specific file differs):
   - `erl_nif.h` — from `erts/emulator/beam/erl_nif.h`
   - `erl_nif_api_funcs.h` — from `erts/emulator/beam/erl_nif_api_funcs.h`
   - `erl_drv_nif.h` — from `erts/emulator/beam/erl_drv_nif.h`
   - `erl_int_sizes_config.h` — iOS: `erts/include/aarch64-apple-iossimulator/erl_int_sizes_config.h` / Android: `erts/include/aarch64-unknown-linux-android/erl_int_sizes_config.h`
   - `erl_fixed_size_int_types.h` — from `erts/include/erl_fixed_size_int_types.h`

2. **Extra static libs** at `erts-<vsn>/lib/`:
   - `libzstd.a` — `erts/emulator/zstd/obj/aarch64-apple-iossimulator/opt/libzstd.a`
   - `libepcre.a` — `erts/emulator/pcre/obj/aarch64-apple-iossimulator/opt/libepcre.a`
   - `libryu.a` — `erts/emulator/ryu/obj/aarch64-apple-iossimulator/opt/libryu.a`
   - `asn1rt_nif.a` — `lib/asn1/priv/lib/aarch64-apple-iossimulator/asn1rt_nif.a`

---

## Prerequisites

- A cross-compiled OTP build. The OTP source tree at `~/code/otp` (commit `7721ab74`)
  has iOS simulator and Android targets already compiled.
- `gh` CLI authenticated to the `GenericJam` GitHub account.

---

## Step 1 — Locate the OTP commit hash

```bash
cd ~/code/otp
git rev-parse --short HEAD   # e.g. 7721ab74
```

Use this hash everywhere below as `<hash>`.

---

## Step 2 — Build the Android tarballs (arm64 + arm32)

The Android OTP release lives at the cross-compiled install dir (typically
`/tmp/otp-android` for arm64, `/tmp/otp-android-arm32` for arm32 — check
where the previous build left it).

**Before tarballing**, make sure you have a project with exqlite compiled
(`_build/dev/lib/exqlite/ebin/` must exist — run `mix deps.get && mix compile`
from any app that uses ecto_sqlite3). The exqlite BEAMs are platform-independent
bytecode and can be bundled directly; the native `.so` is already in the APK and
is symlinked at runtime by `mob_beam.c`.

### arm64

```bash
OTP_SRC=~/code/otp
OTP_RELEASE=/tmp/otp-android   # adjust if different
EXQLITE_BUILD=~/code/mob_test_liveview/_build/dev/lib/exqlite  # any project with exqlite
HASH=<hash>
STAGE=$(mktemp -d)

cp -r "$OTP_RELEASE/." "$STAGE"

# Add extra static libs
ERTS_LIB="$STAGE/erts-16.3/lib"   # update version as needed
cp "$OTP_SRC/erts/emulator/zstd/obj/aarch64-unknown-linux-android/opt/libzstd.a" "$ERTS_LIB/"
cp "$OTP_SRC/erts/emulator/pcre/obj/aarch64-unknown-linux-android/opt/libepcre.a" "$ERTS_LIB/"
cp "$OTP_SRC/erts/emulator/ryu/obj/aarch64-unknown-linux-android/opt/libryu.a"   "$ERTS_LIB/"
cp "$OTP_SRC/lib/asn1/priv/lib/aarch64-unknown-linux-android/asn1rt_nif.a"       "$ERTS_LIB/"

# Add required headers
ERTS_INC="$STAGE/erts-16.3/include"
mkdir -p "$ERTS_INC"
cp "$OTP_SRC/erts/emulator/beam/erl_nif.h"                                        "$ERTS_INC/"
cp "$OTP_SRC/erts/emulator/beam/erl_nif_api_funcs.h"                              "$ERTS_INC/"
cp "$OTP_SRC/erts/emulator/beam/erl_drv_nif.h"                                    "$ERTS_INC/"
cp "$OTP_SRC/erts/include/aarch64-unknown-linux-android/erl_int_sizes_config.h"   "$ERTS_INC/"
cp "$OTP_SRC/erts/include/erl_fixed_size_int_types.h"                             "$ERTS_INC/"

# Add host Elixir stdlib (elixir, logger, eex) so the device starts with the
# correct version. Without this, Elixir version drift causes Regex.safe_run
# crashes when the host is upgraded between native deploys.
ELIXIR_LIB=$(elixir -e "IO.puts(:code.lib_dir(:elixir))" | xargs dirname)
for app in elixir logger eex; do
    mkdir -p "$STAGE/lib/$app/ebin"
    cp "$ELIXIR_LIB/$app/ebin/"* "$STAGE/lib/$app/ebin/"
done
echo "Bundled Elixir $(elixir --version | grep Elixir | awk '{print $2}')"

# Add exqlite BEAMs. The .so NIF is in the APK (symlinked at runtime by mob_beam.c);
# only the ebin/ bytecode goes in the tarball.
EXQLITE_VSN=$(grep '"exqlite"' "$EXQLITE_BUILD/../../../mix.lock" | grep -o '"[0-9][^"]*"' | head -1 | tr -d '"')
EXQLITE_LIB="$STAGE/lib/exqlite-$EXQLITE_VSN"
mkdir -p "$EXQLITE_LIB/ebin" "$EXQLITE_LIB/priv"
cp "$EXQLITE_BUILD/ebin/"* "$EXQLITE_LIB/ebin/"
echo "Bundled exqlite $EXQLITE_VSN"

BASE=$(basename $STAGE)
tar czf "/tmp/otp-android-$HASH.tar.gz" -C "$(dirname $STAGE)" "$BASE"

# Verify
tar tzf "/tmp/otp-android-$HASH.tar.gz" | grep "\.a$\|\.h$"
tar tzf "/tmp/otp-android-$HASH.tar.gz" | grep "lib/elixir/ebin/elixir.app"
tar tzf "/tmp/otp-android-$HASH.tar.gz" | grep "lib/exqlite"
```

### arm32

Repeat the same steps with the arm32 OTP release and the arm32 asn1rt_nif.a (see
prerequisites above for how to build it). The Elixir and exqlite BEAMs are the same
— bytecode is architecture-independent.

```bash
OTP_RELEASE_ARM32=/tmp/otp-android-arm32   # adjust if different
STAGE32=$(mktemp -d)

cp -r "$OTP_RELEASE_ARM32/." "$STAGE32"

ERTS_LIB32="$STAGE32/erts-16.3/lib"
cp "$OTP_SRC/erts/emulator/zstd/obj/arm-unknown-linux-androideabi/opt/libzstd.a"   "$ERTS_LIB32/"
cp "$OTP_SRC/erts/emulator/pcre/obj/arm-unknown-linux-androideabi/opt/libepcre.a"  "$ERTS_LIB32/"
cp "$OTP_SRC/erts/emulator/ryu/obj/arm-unknown-linux-androideabi/opt/libryu.a"    "$ERTS_LIB32/"
cp /tmp/asn1rt_nif_arm32.a "$ERTS_LIB32/asn1rt_nif.a"

ERTS_INC32="$STAGE32/erts-16.3/include"
mkdir -p "$ERTS_INC32"
cp "$OTP_SRC/erts/emulator/beam/erl_nif.h"                                          "$ERTS_INC32/"
cp "$OTP_SRC/erts/emulator/beam/erl_nif_api_funcs.h"                                "$ERTS_INC32/"
cp "$OTP_SRC/erts/emulator/beam/erl_drv_nif.h"                                      "$ERTS_INC32/"
cp "$OTP_SRC/erts/include/arm-unknown-linux-androideabi/erl_int_sizes_config.h"     "$ERTS_INC32/"
cp "$OTP_SRC/erts/include/erl_fixed_size_int_types.h"                               "$ERTS_INC32/"

# Same Elixir and exqlite BEAMs (bytecode is arch-independent)
for app in elixir logger eex; do
    mkdir -p "$STAGE32/lib/$app/ebin"
    cp "$ELIXIR_LIB/$app/ebin/"* "$STAGE32/lib/$app/ebin/"
done
mkdir -p "$STAGE32/lib/exqlite-$EXQLITE_VSN/ebin" "$STAGE32/lib/exqlite-$EXQLITE_VSN/priv"
cp "$EXQLITE_BUILD/ebin/"* "$STAGE32/lib/exqlite-$EXQLITE_VSN/ebin/"

BASE32=$(basename $STAGE32)
tar czf "/tmp/otp-android-arm32-$HASH.tar.gz" -C "$(dirname $STAGE32)" "$BASE32"
tar tzf "/tmp/otp-android-arm32-$HASH.tar.gz" | grep "lib/elixir/ebin/elixir.app"
tar tzf "/tmp/otp-android-arm32-$HASH.tar.gz" | grep "lib/exqlite"
```

---

## Step 3 — Build the iOS simulator tarball

The iOS OTP runtime typically lives at `/tmp/otp-ios-sim`. App-specific BEAM
directories (created by `ios/build.sh` runs) must be excluded.

```bash
OTP_SRC=~/code/otp
OTP_ROOT=/tmp/otp-ios-sim
HASH=<hash>
STAGE=$(mktemp -d)

# Copy the OTP runtime
cp -r "$OTP_ROOT/." "$STAGE"

# Add extra static libs
ERTS_LIB="$STAGE/erts-16.3/lib"   # update version as needed
cp "$OTP_SRC/erts/emulator/zstd/obj/aarch64-apple-iossimulator/opt/libzstd.a" "$ERTS_LIB/"
cp "$OTP_SRC/erts/emulator/pcre/obj/aarch64-apple-iossimulator/opt/libepcre.a" "$ERTS_LIB/"
cp "$OTP_SRC/erts/emulator/ryu/obj/aarch64-apple-iossimulator/opt/libryu.a"   "$ERTS_LIB/"
cp "$OTP_SRC/lib/asn1/priv/lib/aarch64-apple-iossimulator/asn1rt_nif.a"       "$ERTS_LIB/"

# Add required headers
ERTS_INC="$STAGE/erts-16.3/include"
cp "$OTP_SRC/erts/emulator/beam/erl_nif.h"                                     "$ERTS_INC/"
cp "$OTP_SRC/erts/emulator/beam/erl_nif_api_funcs.h"                           "$ERTS_INC/"
cp "$OTP_SRC/erts/emulator/beam/erl_drv_nif.h"                                 "$ERTS_INC/"
cp "$OTP_SRC/erts/include/aarch64-apple-iossimulator/erl_int_sizes_config.h"   "$ERTS_INC/"
cp "$OTP_SRC/erts/include/erl_fixed_size_int_types.h"                          "$ERTS_INC/"

# Add host Elixir stdlib (same reasoning as Android — bake in so fresh installs
# start with the correct version and don't hit Regex.safe_run crashes).
ELIXIR_LIB=$(elixir -e "IO.puts(:code.lib_dir(:elixir))" | xargs dirname)
for app in elixir logger eex; do
    mkdir -p "$STAGE/lib/$app/ebin"
    cp "$ELIXIR_LIB/$app/ebin/"* "$STAGE/lib/$app/ebin/"
done
echo "Bundled Elixir $(elixir --version | grep Elixir | awk '{print $2}')"

# List any app-specific BEAM dirs to exclude (ls $OTP_ROOT | grep -vE "^(erts|lib|releases|misc|usr)$")
BASE=$(basename $STAGE)
tar czf "/tmp/otp-ios-sim-$HASH.tar.gz" \
    --exclude="$BASE/beamhello" \
    --exclude="$BASE/test_app" \
    --exclude="$BASE/test_app0" \
    -C "$(dirname $STAGE)" "$BASE"

# Verify
tar tzf "/tmp/otp-ios-sim-$HASH.tar.gz" | grep "\.a$"
tar tzf "/tmp/otp-ios-sim-$HASH.tar.gz" | grep "\.h$"
tar tzf "/tmp/otp-ios-sim-$HASH.tar.gz" | grep "lib/elixir/ebin/elixir.app"
```

---

## Step 3b — Build the iOS device tarball

The iOS device runtime is the cross-compiled install dir for `aarch64-apple-ios`.
On top of the runtime, this tarball must ship EPMD source files and the iOS-arm64
configure output so `mix mob.deploy --native` can static-link EPMD into the iOS
app at build time.

### 3b.0 — Cross-compile OTP for iOS arm64 (only needed once per OTP version)

> **One required source patch:** the iOS device sandbox blocks `fork()`,
> which the BEAM unconditionally calls at startup via `forker_start`. The
> simulator runs as a Mac process and allows fork, which is why iOS-sim
> builds work without modification. The patch lives at
> [`scripts/release/patches/0001-ios-device-skip-forker-fork.patch`](scripts/release/patches/0001-ios-device-skip-forker-fork.patch);
> `xcompile_ios_device.sh` applies it automatically (idempotent — detects
> if it's already in the source). Without it, the device app dies at
> launch with `Sandbox: <App>(<pid>) deny(1) process-fork` in the device
> log and tears down before any UI shows.

This mirrors the iOS-sim cross-compile (which produced `erts/aarch64-apple-iossimulator/`)
but uses the device xcomp config instead. OTP ships both configs out of the box —
see `xcomp/erl-xcomp-arm64-ios.conf` (device) vs `erl-xcomp-arm64-iossimulator.conf`
(simulator). The device conf differs only in the host triple (`arm64-apple-ios`)
and SDK (`iphoneos`); flow is identical to the sim one. OTP's own walkthrough is
at `HOWTO/INSTALL-IOS.md`.

```bash
cd ~/code/otp

# Sanity check: the iPhoneOS SDK must be installed.
xcrun --sdk iphoneos --show-sdk-path

# iOS doesn't allow shared libraries; this env var tells the build to emit
# libbeam.a (static) instead of libbeam.so. Same flag as for the sim build.
export RELEASE_LIBBEAM=yes

# Configure for the iOS arm64 device target.
#
# `--with-ssl=$OPENSSL_PREFIX` points at a previously cross-compiled
# OpenSSL 3.x install (see scripts/release/openssl/ios_device.sh).
# `--disable-dynamic-ssl-lib` keeps OpenSSL static-linked into the
# crypto NIF; no separate libcrypto.so/.dylib ends up in the app.
# `--enable-static-nifs` registers crypto (and asn1rt_nif) in the
# BEAM's static_nif_tab[] so `erlang:load_nif("crypto", ...)` resolves
# `crypto_nif_init` via dlsym(RTLD_DEFAULT) — no dlopen of crypto.so.
# The latter is required: Android's RTLD_LOCAL default makes a dlopen'd
# crypto.so unable to see libpigeon.so's enif_* symbols.
./otp_build configure \
    --xcomp-conf=./xcomp/erl-xcomp-arm64-ios.conf \
    --with-ssl="$OPENSSL_PREFIX" \
    --disable-dynamic-ssl-lib \
    --enable-static-nifs

# Build everything (ERTS, stdlib, and all OTP apps for the target).
./otp_build boot

# Assemble the install tree at /tmp/otp-ios-device — this is the dir that
# the staging step below copies from.
make release RELEASE_ROOT=/tmp/otp-ios-device
```

After this completes you should have:
- `erts/aarch64-apple-ios/config.h` (the configure output the staging step bundles)
- `erts/emulator/{zstd,pcre,ryu}/obj/aarch64-apple-ios/opt/lib*.a`
- `lib/asn1/priv/lib/aarch64-apple-ios/asn1rt_nif.a`
- `/tmp/otp-ios-device/{bin,erts-<vsn>,lib,releases,...}` (the install tree)

```bash
OTP_SRC=~/code/otp
OTP_RELEASE=/tmp/otp-ios-device   # wherever the iOS-arm64 install lives
HASH=<hash>
STAGE=$(mktemp -d)

# Copy the OTP runtime (install tree)
cp -r "$OTP_RELEASE/." "$STAGE"

# Add extra static libs (same set as the sim tarball, but iOS-device arch)
ERTS_LIB="$STAGE/erts-16.3/lib"   # update version as needed
cp "$OTP_SRC/erts/emulator/zstd/obj/aarch64-apple-ios/opt/libzstd.a"   "$ERTS_LIB/"
cp "$OTP_SRC/erts/emulator/pcre/obj/aarch64-apple-ios/opt/libepcre.a"  "$ERTS_LIB/"
cp "$OTP_SRC/erts/emulator/ryu/obj/aarch64-apple-ios/opt/libryu.a"     "$ERTS_LIB/"
cp "$OTP_SRC/lib/asn1/priv/lib/aarch64-apple-ios/asn1rt_nif.a"         "$ERTS_LIB/"

# Add required headers
ERTS_INC="$STAGE/erts-16.3/include"
mkdir -p "$ERTS_INC"
cp "$OTP_SRC/erts/emulator/beam/erl_nif.h"                              "$ERTS_INC/"
cp "$OTP_SRC/erts/emulator/beam/erl_nif_api_funcs.h"                    "$ERTS_INC/"
cp "$OTP_SRC/erts/emulator/beam/erl_drv_nif.h"                          "$ERTS_INC/"
cp "$OTP_SRC/erts/include/aarch64-apple-ios/erl_int_sizes_config.h"     "$ERTS_INC/"
cp "$OTP_SRC/erts/include/erl_fixed_size_int_types.h"                   "$ERTS_INC/"

# Add Elixir stdlib (same as sim — bake in for version stability)
ELIXIR_LIB=$(elixir -e "IO.puts(:code.lib_dir(:elixir))" | xargs dirname)
for app in elixir logger eex; do
    mkdir -p "$STAGE/lib/$app/ebin"
    cp "$ELIXIR_LIB/$app/ebin/"* "$STAGE/lib/$app/ebin/"
done
echo "Bundled Elixir $(elixir --version | grep Elixir | awk '{print $2}')"

# ── EPMD source + iOS-arm64 configure output ─────────────────────────────────
# build_device.sh static-links EPMD into the iOS app. The .c sources and the
# arch-specific config.h must be present alongside the install tree. The
# downloader's `valid_otp_dir?/2` checks for these and re-downloads if absent.
mkdir -p "$STAGE/erts/epmd/src"
cp "$OTP_SRC/erts/epmd/src/epmd.c"     "$STAGE/erts/epmd/src/"
cp "$OTP_SRC/erts/epmd/src/epmd_srv.c" "$STAGE/erts/epmd/src/"
cp "$OTP_SRC/erts/epmd/src/epmd_cli.c" "$STAGE/erts/epmd/src/"
# epmd.c → epmd.h, epmd_int.h, both in erts/epmd/src/.
cp "$OTP_SRC/erts/epmd/src/"*.h        "$STAGE/erts/epmd/src/"

mkdir -p "$STAGE/erts/aarch64-apple-ios"
cp -r "$OTP_SRC/erts/aarch64-apple-ios/"* "$STAGE/erts/aarch64-apple-ios/"

mkdir -p "$STAGE/erts/include" "$STAGE/erts/include/internal"
cp -r "$OTP_SRC/erts/include/"*          "$STAGE/erts/include/"
cp -r "$OTP_SRC/erts/include/internal/"* "$STAGE/erts/include/internal/"

# Tar it up — exclude any stray app build dirs left in OTP_RELEASE
BASE=$(basename $STAGE)
tar czf "/tmp/otp-ios-device-$HASH.tar.gz" -C "$(dirname $STAGE)" "$BASE"

# Verify all four schema requirements: erts-*/ install, EPMD source files,
# iOS-arm64 config.h, and Elixir stdlib.
tar tzf "/tmp/otp-ios-device-$HASH.tar.gz" | grep "erts-16"           | head -1
tar tzf "/tmp/otp-ios-device-$HASH.tar.gz" | grep "erts/epmd/src/epmd.c"
tar tzf "/tmp/otp-ios-device-$HASH.tar.gz" | grep "erts/epmd/src/epmd_srv.c"
tar tzf "/tmp/otp-ios-device-$HASH.tar.gz" | grep "erts/epmd/src/epmd_cli.c"
tar tzf "/tmp/otp-ios-device-$HASH.tar.gz" | grep "erts/aarch64-apple-ios/config.h"
tar tzf "/tmp/otp-ios-device-$HASH.tar.gz" | grep "lib/elixir/ebin/elixir.app"
```

---

## Step 4 — Publish the GitHub release

Tag format: `otp-<hash>` (e.g. `otp-7721ab74`).

```bash
HASH=<hash>

# Create the release (or use an existing one)
gh release create "otp-$HASH" \
    --repo GenericJam/mob \
    --title "OTP pre-built runtime $HASH" \
    --notes "Pre-built OTP for Android (aarch64 + arm32), iOS simulator (aarch64-apple-iossimulator), and iOS device (aarch64-apple-ios). OTP source commit: $HASH."

# Upload tarballs (all four)
gh release upload "otp-$HASH" \
    "/tmp/otp-android-$HASH.tar.gz" \
    "/tmp/otp-android-arm32-$HASH.tar.gz" \
    "/tmp/otp-ios-sim-$HASH.tar.gz" \
    "/tmp/otp-ios-device-$HASH.tar.gz" \
    --repo GenericJam/mob

# Verify
gh release view "otp-$HASH" --repo GenericJam/mob --json assets \
    -q '.assets[] | "\(.name) \(.size)"'
```

To replace a bad asset:
```bash
gh release delete-asset "otp-$HASH" otp-ios-sim-$HASH.tar.gz --repo GenericJam/mob --yes
gh release upload "otp-$HASH" /tmp/otp-ios-sim-$HASH.tar.gz --repo GenericJam/mob
```

### Re-uploading without bumping the hash (schema-bump pattern)

When the tarball *contents* change but the underlying OTP commit hasn't (e.g.
adding EPMD source to the iOS device tarball), the canonical move is to
re-upload at the same hash. `MobDev.OtpDownloader.valid_otp_dir?/2` is the
gate that decides whether a cached extracted dir is still acceptable — if you
add a new schema requirement there, existing users' caches fail validation
and the next `mix mob.deploy --native` re-downloads the asset automatically.
No hash bump, no user action needed.

---

## Step 5 — Update OtpDownloader

Edit `lib/mob_dev/otp_downloader.ex` — update the hash and ERTS version:

```elixir
@otp_hash    "7721ab74"    # ← new hash
```

If the ERTS version changed (e.g. from 16.3 to 16.4), update `build_release.md`
to match.

## Step 6 — Update the bundled-versions manifest

Edit [`priv/security/bundled_versions.exs`](priv/security/bundled_versions.exs)
— update `:active_hash` and add (or modify) the `:bundles` entry for the new
hash with the versions baked into the new tarballs:

```elixir
%{
  active_hash: "<new-hash>",
  bundles: %{
    "<new-hash>" => %{
      erts: "16.3",            # erts-* directory in the tarball
      otp_release: "28",
      elixir: "1.19.5",
      openssl: "3.4.0",        # `strings libcrypto.a | grep '^OpenSSL'`
      exqlite_beam: "0.36.0",
      openssl_release_date: "2024-10-22"
    }
  }
}
```

This file is the source of truth that `mix mob.security_scan` checks against.
The bundled-runtime scan layer fingerprints the cached tarball and raises if
the binary disagrees with what's declared here — drift between what we say
shipped and what actually shipped is exactly what this manifest is designed
to catch. Update it in the **same PR** as the OTP hash bump; otherwise the
scan will report a `:high` "manifest mismatch" finding for every Mob app.

---

## Troubleshooting

**"No erts-* directory found in $OTP_ROOT"** — tarball extracted incorrectly.
Check that `--strip-components=1` produces `erts-<vsn>/` at the top level:
```bash
tar tzf otp-ios-sim-<hash>.tar.gz | head -5
tar tzf otp-ios-sim-<hash>.tar.gz | sed 's|[^/]*/||' | head -5   # after strip
```

**iOS build fails with missing header** — confirm `erl_nif.h` is in the tarball:
```bash
tar tzf otp-ios-sim-<hash>.tar.gz | grep "\.h$"
```

**Android BEAM crash `{undef, app:start}`** — OTP runtime not on device. Check
that `ensure_android/0` succeeded and `push_otp_release_android` ran.

**Wrong tarball uploaded** — the iOS tarball once accidentally contained a compiled
`.app` bundle instead of OTP. Always verify with:
```bash
tar tzf otp-ios-sim-<hash>.tar.gz | grep "erts-"
```