Remote inspection and interaction helpers for connected Dala apps.
All functions accept a node atom and operate on the running screen via
Erlang distribution. Connect first with mix dala.connect, then use these
from IEx or from an agent via :rpc.call/4.
Quick reference
node = :"my_app_ios@127.0.0.1"
# Inspection
Dala.Test.screen(node) #=> MyApp.HomeScreen
Dala.Test.assigns(node) #=> %{count: 3, ...}
Dala.Test.tree(node) #=> %{type: :column, ...}
Dala.Test.find(node, "Save") #=> [{[0, 2], %{...}}]
Dala.Test.inspect(node) #=> %{screen: ..., assigns: ..., tree: ...}
# Interaction
Dala.Test.tap(node, :increment) # tap a button by tag
Dala.Test.back(node) # system back gesture
Dala.Test.pop(node) # pop to previous screen (synchronous)
Dala.Test.navigate(node, MyApp.DetailScreen, %{id: 42})
Dala.Test.pop_to(node, MyApp.HomeScreen)
Dala.Test.pop_to_root(node)
# Lists
Dala.Test.select(node, :my_list, 0) # select first row
# Device API simulation
Dala.Test.send_message(node, {:permission, :camera, :granted})
Dala.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}})
Dala.Test.send_message(node, {:location, %{lat: 43.65, lon: -79.38, accuracy: 10.0, altitude: 80.0}})
Dala.Test.send_message(node, {:notification, %{id: "n1", title: "Hi", body: "Hey", data: %{}, source: :push}})Tap vs send_message
tap/2 is for UI interactions that go through handle_event/3 via the native
tap registry. send_message/2 delivers any term directly to handle_info/2.
Use send_message/2 to simulate async results from device APIs (camera, location,
notifications, etc.) without having to trigger the actual hardware.
Synchronous vs fire-and-forget
Navigation functions (pop, navigate, pop_to, pop_to_root) are synchronous —
they block until the navigation and re-render complete. This makes them safe to
follow immediately with screen/1 or assigns/1 to verify the result.
back/1 and send_message/2 are fire-and-forget (they send a message to the
screen process and return immediately). Use :sys.get_state/1 as a sync point
if you need to wait before reading state:
Dala.Test.send_message(node, {:permission, :camera, :granted})
:rpc.call(node, :sys, :get_state, [:dala_screen]) # flush mailbox
Dala.Test.assigns(node)Two layers of inspection: render tree vs native UI
Dala.Test exposes two complementary views of what the app is showing:
| API | Source | When to use |
|---|---|---|
tree/1, find/2 | Dala render tree (logical components) | Dala apps you control. Fast, exact, has on_tap tags, no AX activation needed. |
view_tree/1, find_view/2 | Native view hierarchy via NIF | Native pixel frames; works for any app on iOS UIKit; shallow on SwiftUI/Compose. |
ui_tree/1 | OS accessibility tree | What sighted users read; works on any app if AX is active (iOS: VoiceOver). Strict superset of view_tree for UIKit; the only path to semantics inside SwiftUI/Compose. |
Choose render tree first if your app is Dala-rendered. Reach for view_tree
when you want native frames or geometry. Reach for ui_tree when you need
to inspect non-Dala content (alerts, system overlays, third-party SDK UI),
or to verify the rendered state matches the logical render.
Driving controls beyond plain taps
- Buttons / nav items —
tap/2(by tag, fastest), ordala_nif:tap/1(by accessibility label), ortap_xy/3(by coordinate). - Sliders, steppers, pickers —
adjust_slider/4and the underlyingax_action/3/ax_action_at_xy/4useaccessibilityIncrement/accessibilityDecrement. Synthetic drag gestures don't fire SwiftUI'sDragGesturereliably; AX actions do. - Switches / toggles —
toggle/2finds the switch by nearby label and activates it via the AX path (sendsaccessibilityActivate). - Modals / alerts / sheets —
dismiss_alert/2usesaccessibilityActivateon the named button;ax_action/3with:escapesendsaccessibilityPerformEscape. - Scroll views —
ax_action/3with:scroll_up/:scroll_down/:scroll_left/:scroll_rightsendsaccessibilityScroll:. - System back —
back/1(Dala screens, framework-level) or — for sidecar mode against arbitrary apps — synthetic edge-pan viaswipe/5fromx=0, but iOS owns that gesture above the app process and the synthetic pan won't fire. Useback/1for Dala, document the limitation for sidecar.
Platform support matrix
| Helper | iOS sim | iOS device | Android |
|---|---|---|---|
screen/1, assigns/1 | ✅ | ✅ | ✅ |
tap/2 (by tag) | ✅ | ✅ | ✅ |
back/1, pop/1, navigate | ✅ | ✅ | ✅ |
send_message/2 | ✅ | ✅ | ✅ |
screen_info/1 | ✅ | ✅ | ✅ |
view_tree/1 | ✅ (shallow†) | ✅ (shallow†) | ✅ (root only‡) |
find_view/2 | ✅ | ✅ | ✅ |
ui_tree/1 (legacy AX) | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_loaded |
ax_action/3 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_supported |
ax_action_at_xy/4 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_supported |
toggle/2 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
dismiss_alert/2 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
adjust_slider/4 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
tap_xy/3 | ✅ (AX path) | ✅ (HID inj.) | n/a |
swipe/5 | ⚠️ scroll only | ✅ (HID inj.) | n/a |
- † SwiftUI doesn't expose its content as separate UIView instances —
view_treereaches the SwiftUI hosting view's container and stops. For semantic content on Dala screens usetree/1(render tree); for any other SwiftUI-based content useui_tree/1. - ‡ Android's Dala renderer is Compose. The View walk stops at the
AndroidComposeViewhost. The eventual fix isModifier.onGloballyPositionedin Dala's components writing to a registry the NIF reads (planned). Seeissues.md#11. - § "AX active" means an iOS accessibility client is asking for the
AX tree so SwiftUI materializes it. Today: VoiceOver toggle. Production:
XCAXClient_iOSactivation, debug-only — see WireTap stretch goals infuture_developments.md.
Helpers that depend on AX return clear error tuples on Android instead of
raising. Callers should match on {:error, :not_supported_on_android} and
{:error, :ui_tree_unavailable} and either skip or fall back to
send_message/2 for state mutations.
Known limitations affecting AX automation
Even on iOS with AX active, three Dala component defects keep the natural paths from working today. Workarounds in each helper's docstring:
- Slider —
accessibilityIncrement/Decrementare no-ops because Dala's iOS Slider doesn't attach.accessibilityAdjustableAction. Seeissues.md#7. - Toggle — the
label:prop doesn't reach the AX tree;toggle/2can't find the switch by label name. Useax_action_at_xy/4with coordinates for now. Seeissues.md#8. - Alert OK button —
accessibilityActivateon the AX-tree button doesn't fire the underlyingUIAlertAction. Use DalaAlertwithaction:atoms andsend_message/2to dismiss programmatically. Seeissues.md#9.
System-level gestures iOS owns above the app process (edge-pan back,
swipe-up app switcher, pull-down notification center) are out of reach
for in-process synthetic touches on physical devices. Use back/1 for
Dala screens; for sidecar mode against arbitrary apps, document the
limitation rather than promising the gesture.
Summary
Functions
Step a slider toward a target percentage (0.0..1.0) using accessibility increment/decrement actions. Reliable when synthetic-drag won't fire (SwiftUI Slider's DragGesture ignores in-process touches on iOS).
Return the current screen's assigns map.
Invoke an accessibility action on the first AX element matching match.
Invoke an AX action on whatever element occupies the given screen coordinates.
Simulate the system back gesture (Android hardware back / iOS edge-pan).
Clear all text in the focused input (select-all + delete).
Delete one character behind the cursor (backspace).
Dismiss a modal/alert overlay by tapping its first button labelled with
button_label (e.g. "OK", "Cancel"). Mirrors what a user does when an
alert pops up.
Find all nodes in the current tree whose text contains substring.
Returns a list of {path, node} tuples where path is a list of
indices from the root.
Find elements in the native accessibility tree whose label or value contains text.
Find nodes in the view tree whose label or value contains text.
Flatten an already-fetched view tree. Pure function — useful for tests and for inspecting a captured tree without re-fetching.
Return a map with :screen, :assigns, :nav_history, and :tree
(the raw render tree from calling render/1 on the current screen).
Press a special key on the focused text input.
Locate an element by visible label text or accessibility ID (tag atom name). Returns the element's screen frame.
Long-press at screen coordinates for duration_ms milliseconds (default 800ms).
Push a new screen onto the navigation stack. Synchronous.
Pop the current screen and return to the previous one. Synchronous.
Pop the stack until dest is at the top. Synchronous.
Pop all screens back to the root of the current stack. Synchronous.
Replace the entire navigation stack with a new root screen. Synchronous.
Return the current screen module.
Return screen geometry in logical units (points on iOS, dp on Android).
Select a row in a :list component by index.
Send an arbitrary message to the screen's handle_info/2. Fire-and-forget.
Swipe from (x1, y1) to (x2, y2). Drives UIScrollView contentOffset on simulator; synthesises a drag gesture on real device.
Send a tap event to the current screen by tag atom.
Locate an element and tap it via the simulator's native UI mechanism.
Tap at screen coordinates on the native app. On simulator uses accessibility activation; on real device synthesises a UITouch via IOHIDEvent.
Toggle a switch by a label substring. SwiftUI exposes Toggle as a button
with an empty accessibility label and value "0" or "1" — so we find the
Text element matching label_match, then activate the next button below it.
Return the current rendered tree (calls render/1 on the live assigns).
Type text into the currently focused text field.
Return the live accessibility tree from the running native app.
Return the live UI tree as a nested map, walking native views directly.
Return the view tree flattened to a list of {path, node} tuples.
Wait until predicate returns true when called with the current ui_tree,
polling every interval_ms until timeout_ms elapses.
Wait until an element whose label or value contains text appears in the
accessibility tree.
Clear a WebView input element by CSS selector.
Evaluate JavaScript in the WebView and return the result.
Go forward in the WebView history.
Navigate the WebView to a new URL.
Send a message to the WebView page via window.dala._dispatch().
Reload the current WebView page.
Take a screenshot of the WebView content.
Stop loading the current WebView page.
Tap an element in the WebView by CSS selector.
Type text into a WebView input element by CSS selector.
Functions
Step a slider toward a target percentage (0.0..1.0) using accessibility increment/decrement actions. Reliable when synthetic-drag won't fire (SwiftUI Slider's DragGesture ignores in-process touches on iOS).
match is a substring of the slider's label or value (e.g. "Volume").
target is a fraction 0.0..1.0. max_steps caps the increment loop
(default 30) so a wrong match can't spin forever.
Returns {:ok, final_pct} or {:error, reason}.
Dala.Test.adjust_slider(node, "Volume", 0.30)
#=> {:ok, 0.30}Implementation note: each AX increment/decrement on a SwiftUI slider moves
by the slider's .step value (default 0.10 of the range). The function
re-reads the slider value after each step to converge.
Known limitation (issues.md #7)
Dala's iOS Slider component does not currently attach
.accessibilityAdjustableAction { … }, so accessibilityIncrement and
accessibilityDecrement are silently dropped by SwiftUI even though the
NIF returns :ok. This helper currently returns {:error, :max_steps_exhausted} against an unfixed slider. Until issue #7 lands,
drive sliders via Dala.Test.send_message(node, {:change, :slider_tag, value}).
Return the current screen's assigns map.
Invoke an accessibility action on the first AX element matching match.
Platform support
- iOS: works once AX is active (today: VoiceOver on; future:
XCAXClient_iOSactivation, seefuture_developments.md). - Android: returns
{:error, :not_supported_on_android}. The Compose semantics walker is queued under WireTap (issues.md #11).
Used for controls where synthetic touches don't reach the gesture recognizer (sliders, scrolls, modal dismissal).
match is a string searched in both label and value. action is one of:
:increment, :decrement, :activate, :escape, :scroll_up,
:scroll_down, :scroll_left, :scroll_right.
Dala.Test.ax_action(node, "Volume", :decrement)
Dala.Test.ax_action(node, "Cancel", :activate)
Invoke an AX action on whatever element occupies the given screen coordinates.
Useful when label/value substring matching is ambiguous (e.g. multiple
sliders that all read "50%", a toggle whose accessibility label is empty).
Caller picks coordinates from ui_tree/1 and points at the exact element.
Dala.Test.ax_action_at_xy(node, 187.0, 296.0, :increment)Platform support
- iOS: works once AX is active (VoiceOver on, today).
- Android: returns
{:error, :not_supported_on_android}— seeax_action/3.
@spec back(node()) :: :ok
Simulate the system back gesture (Android hardware back / iOS edge-pan).
Fire-and-forget. The framework pops the navigation stack; if already at the
root, it exits the app. Prefer pop/1 when you need to know that navigation
has finished before reading state.
Clear all text in the focused input (select-all + delete).
Delete one character behind the cursor (backspace).
Dismiss a modal/alert overlay by tapping its first button labelled with
button_label (e.g. "OK", "Cancel"). Mirrors what a user does when an
alert pops up.
Dala.Test.dismiss_alert(node, "OK")Known limitation (issues.md #9)
UIAlertController exposes its buttons twice in the AX tree (visual view +
action target). Activating the visual view doesn't fire the action. This
helper currently reports :ok while the alert stays on screen. Workaround:
define alerts with action: :tag_atom and dismiss via
Dala.Test.send_message(node, {:alert, :tag_atom}).
Find all nodes in the current tree whose text contains substring.
Returns a list of {path, node} tuples where path is a list of
indices from the root.
Dala.Test.find(node, "Device APIs")
#=> [{[0, 1, 8], %{"type" => "button", "props" => %{"text" => "Device APIs →", ...}}}]
Find elements in the native accessibility tree whose label or value contains text.
Dala.Test.find_native(node, "Increment")
#=> [{:button, "Increment", "", {164.0, 400.0, 54.0, 54.0}}]
@spec find_view(node(), String.t()) :: [{[non_neg_integer()], map()}]
Find nodes in the view tree whose label or value contains text.
Returns [{path, node}] for each match. Faster and more accurate than
find_native/2 (no AX dependency, sees all views).
Dala.Test.find_view(node, "Roll Dice")
#=> [{[0, 0, 0, 4], %{type: :button, label: "Roll Dice", ...}}]
@spec flatten_tree(map()) :: [{[non_neg_integer()], map()}]
Flatten an already-fetched view tree. Pure function — useful for tests and for inspecting a captured tree without re-fetching.
tree = Dala.Test.view_tree(node)
flat = Dala.Test.flatten_tree(tree)
Return a map with :screen, :assigns, :nav_history, and :tree
(the raw render tree from calling render/1 on the current screen).
Press a special key on the focused text input.
Keys: :return | :tab | :escape | :space
Dala.Test.key_press(node, :return)
Dala.Test.key_press(node, :escape)
Locate an element by visible label text or accessibility ID (tag atom name). Returns the element's screen frame.
Requires idb (iOS) to be installed.
Dala.Test.locate(:save)
#=> {:ok, %{x: 0.0, y: 412.0, width: 402.0, height: 44.0}}
Dala.Test.locate("Save")
#=> {:ok, %{x: 0.0, y: 412.0, width: 402.0, height: 44.0}}
@spec long_press_xy(node(), number(), number(), non_neg_integer()) :: :ok | {:error, atom()}
Long-press at screen coordinates for duration_ms milliseconds (default 800ms).
Dala.Test.long_press_xy(node, 195.0, 400.0)
Dala.Test.long_press_xy(node, 195.0, 400.0, 1200)
@spec pop(node()) :: :ok
Pop the current screen and return to the previous one. Synchronous.
Returns :ok once the navigation and re-render are complete, so it is safe
to call screen/1 or assigns/1 immediately after.
No-op (returns :ok) if already at the root of the stack.
Pop the stack until dest is at the top. Synchronous.
dest is a screen module or registered name atom. No-op if not in history.
@spec pop_to_root(node()) :: :ok
Pop all screens back to the root of the current stack. Synchronous.
Replace the entire navigation stack with a new root screen. Synchronous.
Use this to simulate auth transitions (e.g. login → home with no back button).
Return the current screen module.
Return screen geometry in logical units (points on iOS, dp on Android).
Dala.Test.screen_info(node)
#=> %{
# width: 393.0, height: 852.0, scale: 3.0,
# safe_area: %{top: 59.0, bottom: 34.0, left: 0.0, right: 0.0}
# }:scale is the device-pixel ratio (UIScreen.scale on iOS, displayMetrics.density
on Android). All other values are already in logical units; no further conversion
needed in the agent.
@spec select(node(), atom(), non_neg_integer()) :: :ok
Select a row in a :list component by index.
list_id must match the :id prop on the type: :list node. index is
zero-based. Delivers {:select, list_id, index} to handle_info/2.
Fire-and-forget.
Dala.Test.select(node, :my_list, 0) # first row
Send an arbitrary message to the screen's handle_info/2. Fire-and-forget.
Use this to simulate results from device APIs without triggering real hardware:
# Permissions
Dala.Test.send_message(node, {:permission, :camera, :granted})
Dala.Test.send_message(node, {:permission, :notifications, :denied})
# Camera
Dala.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}})
Dala.Test.send_message(node, {:camera, :cancelled})
# Location
Dala.Test.send_message(node, {:location, %{lat: 43.6532, lon: -79.3832, accuracy: 10.0, altitude: 80.0}})
Dala.Test.send_message(node, {:location, :error, :denied})
# Photos / Files
Dala.Test.send_message(node, {:photos, :picked, [%{path: "/tmp/photo.jpg", width: 800, height: 600}]})
Dala.Test.send_message(node, {:files, :picked, [%{path: "/tmp/doc.pdf", name: "doc.pdf", size: 4096}]})
# Audio / Motion / Scanner
Dala.Test.send_message(node, {:audio, :recorded, %{path: "/tmp/audio.aac", duration: 12}})
Dala.Test.send_message(node, {:motion, %{ax: 0.1, ay: 9.8, az: 0.0, gx: 0.0, gy: 0.0, gz: 0.0}})
Dala.Test.send_message(node, {:scan, :result, %{type: :qr, value: "https://example.com"}})
# Notifications
Dala.Test.send_message(node, {:notification, %{id: "n1", title: "Hi", body: "Hello", data: %{}, source: :push}})
Dala.Test.send_message(node, {:push_token, :ios, "abc123def456"})
# Biometric
Dala.Test.send_message(node, {:biometric, :success})
Dala.Test.send_message(node, {:biometric, :failure, :user_cancel})
# Custom
Dala.Test.send_message(node, {:my_event, %{key: "value"}})
Swipe from (x1, y1) to (x2, y2). Drives UIScrollView contentOffset on simulator; synthesises a drag gesture on real device.
Dala.Test.swipe(node, 195.0, 500.0, 195.0, 100.0) # scroll down
Send a tap event to the current screen by tag atom.
The tag comes from on_tap: {self(), :tag_atom} in the screen's render/1.
Check the screen's render function to find available tags.
Fire-and-forget — does not wait for the screen to finish processing.
Dala.Test.tap(node, :save)
Dala.Test.tap(node, :open_detail)
Locate an element and tap it via the simulator's native UI mechanism.
Requires idb (iOS) to be installed. Exercises the full native gesture path
rather than sending a BEAM message — useful for testing gesture recognizers
or verifying that the native layer wired up the tap handler correctly.
Prefer tap/2 for testing Elixir logic; use tap_native/1 when you need
the native path.
Dala.Test.tap_native("Save") # by visible text
Dala.Test.tap_native(:save) # by accessibility_id (= tag atom name)
Tap at screen coordinates on the native app. On simulator uses accessibility activation; on real device synthesises a UITouch via IOHIDEvent.
Dala.Test.tap_xy(node, 289.7, 518.8)
Toggle a switch by a label substring. SwiftUI exposes Toggle as a button
with an empty accessibility label and value "0" or "1" — so we find the
Text element matching label_match, then activate the next button below it.
Dala.Test.toggle(node, "Notifications")Known limitation (issues.md #8)
Dala's iOS Toggle component does not currently surface its label: prop as
a separate :text AX element, so find_label_y/2 returns
{:error, :label_not_found}. Workaround: use ax_action_at_xy/4 directly
with the toggle's frame from ui_tree/1 (filter for :button with value
"0" or "1"). Once issue #8 lands, this helper works as documented.
Return the current rendered tree (calls render/1 on the live assigns).
Type text into the currently focused text field.
Tap the field first to give it focus, then call this function.
Dala.Test.tap_xy(node, 195.0, 300.0)
Process.sleep(100)
Dala.Test.type_text(node, "hello@example.com")
Return the live accessibility tree from the running native app.
Each element is a tuple: {type, label, value, {x, y, w, h}}
Dala.Test.ui_tree(node)
#=> [{:button, "Increment", "", {164.0, 400.0, 54.0, 54.0}}, ...]
Return the live UI tree as a nested map, walking native views directly.
Unlike ui_tree/1 (which uses the accessibility subsystem and requires
VoiceOver activation on iOS), this walks UIView/View hierarchies directly:
no AX activation needed.
Coverage caveat
- UIKit apps (sidecar mode): full UIView hierarchy with labels and frames.
- SwiftUI apps (current Dala): shallow — SwiftUI doesn't expose its content
as separate UIView instances under the hosting view. You'll see containers
and scroll views but not individual buttons/text. For Dala apps, prefer
Dala.Test.tree/1(the logical render tree, which has all the semantic info) orDala.Test.ui_tree/1(AX walk, requires VoiceOver activation). - Android (planned): a registry populated via
onGloballyPositionedin Dala's Compose components — seefuture_developments.md"WireTap" section.
Returns a nested map:
%{
type: :root, label: nil, value: nil,
frame: {0.0, 0.0, 393.0, 852.0},
children: [
%{type: :window, ..., children: [
%{type: :scroll, ..., children: [
%{type: :button, label: "Roll Dice",
frame: {24.0, 416.0, 327.0, 53.5}, children: []}
]}
]}
]
}On Android, the JSON returned by dala_nif:ui_view_tree/0 is decoded here.
@spec view_tree_flat(node()) :: [{[non_neg_integer()], map()}]
Return the view tree flattened to a list of {path, node} tuples.
path is the list of child indices from the root — e.g. [0, 2, 1] is
"the second child of the third child of the first child of the root."
Useful for filter/find — see find_view/2.
Dala.Test.view_tree_flat(node)
#=> [
# {[], %{type: :root, ...}},
# {[0], %{type: :window, ...}},
# {[0, 0], %{type: :scroll, ...}},
# ...
# ]
Wait until predicate returns true when called with the current ui_tree,
polling every interval_ms until timeout_ms elapses.
Dala.Test.wait_for(node, fn tree ->
Enum.any?(tree, fn {_, label, _, _} -> label == "Success" end)
end)
Wait until an element whose label or value contains text appears in the
accessibility tree.
Dala.Test.wait_for_text(node, "Welcome")
Dala.Test.wait_for_text(node, "Error", timeout_ms: 2000)
Clear a WebView input element by CSS selector.
Evaluate JavaScript in the WebView and return the result.
Result arrives asynchronously via:
handle_info({:webview, :eval_result, result}, socket)Fire-and-forget.
@spec webview_go_forward(node()) :: :ok
Go forward in the WebView history.
Send a message to the WebView page via window.dala._dispatch().
Fire-and-forget.
@spec webview_reload(node()) :: :ok
Reload the current WebView page.
@spec webview_screenshot(node()) :: :ok
Take a screenshot of the WebView content.
Result arrives via:
handle_info({:webview, :screenshot, png_data}, socket)
@spec webview_stop_loading(node()) :: :ok
Stop loading the current WebView page.
Tap an element in the WebView by CSS selector.
Result arrives via:
handle_info({:webview, :interact_result, %{"action" => "tap", "success" => ...}}, socket)
Type text into a WebView input element by CSS selector.