Manual Qa Flutter Macos

Manual QA driver for the Zekra Flutter macOS app

A repeatable recipe for exercising the live macOS shell of Zekra end-to-end: boot it, see each screen, click through, verify state. The skill assumes you'll combine three tools in concert, not just one:

The combination is intentional: screenshots prove the UI rendered, clicks exercise the actual UX, DDS evaluates verify the underlying state — and when clicks become unreliable, DDS becomes the fallback driver so the test still completes.

Prerequisites

which cliclick screencapture osascript    # all three required
# screencapture/osascript ship with macOS; install cliclick if missing:
brew install cliclick

flutter_gemma's macOS native cache must be populated (see memory reference_zekra_macos_testing for the recipe — the cached SHA256 in hook/build.dart is wrong, you have to pre-extract the tarball yourself).

ObjectBox host dylib + model symlinks must be in place:

ls /Users/amanie/Documents/zekra-ai/flutter/libobjectbox.dylib
ls ~/Documents/model.litertlm        # symlink to gemma4-e4b-base
ls ~/Documents/w600k_r50.onnx        # symlink to flutter/assets/models/

First time on a new Mac: macOS will block screencapture and synthesized mouse events. You'll see "could not create image from display" or silent no-op clicks. Grant Screen Recording AND Accessibility permission to the parent process (Cursor, Terminal, iTerm — whichever owns the bash session you're running from) in System Settings → Privacy & Security. The user will probably need to restart Claude Code after granting.

Tiling window managers (Amethyst, Yabai, Rectangle in auto-tile mode): ask the user to quit them before running. They intercept window focus events and re-layout windows out from under your script — you'll see the macOS "Tall"/"Wide" resize pill flash mid-sequence and your cliclick c:X,Y will hit empty space because Zekra just slid sideways. Amethyst in particular re-tiles on every activate, which races with tell application "zekra" to activate; sleep 0.7. If the user can't quit theirs, fall back to DDS-driven completion (Step 6) where layout doesn't matter.

Step 1: Boot a fresh Flutter instance with an isolated store

main.dart honors two env vars (added 2026-05-09) that let you boot a clean instance without touching the user's real data:

• ZEKRA_OBJECTBOX_DIR=/tmp/zekra-something/objectbox — overrides the

default ~/Documents/zekra-objectbox/ so you don't trash the user's real store. The dir must exist; the app creates data.mdb inside it.

• ZEKRA_NO_GEMMA=1 — skips the slow LLM init. Use when you're testing

the UI / tool layer and the model isn't relevant (boot drops from ~5min to ~30s). You still get full dispatchTool + ZekraStore behavior.

# Pick a unique tag so a sibling Claude session running in parallel doesn't
# step on your store / log / window. `$$` is the bash PID, good enough.
TAG="qa-$$"
STORE_DIR="/tmp/zekra-${TAG}/objectbox"
LOG="/tmp/zekra-${TAG}-run.log"
mkdir -p "$STORE_DIR"

# DO NOT blanket-kill `zekra.app` or `flutter run` — see "Multi-instance
# coexistence" below. Only kill PIDs you spawned yourself.

cd /Users/amanie/Documents/zekra-ai/flutter
ZEKRA_OBJECTBOX_DIR="$STORE_DIR" ZEKRA_NO_GEMMA=1 \
  nohup flutter run -d macos --debug > "$LOG" 2>&1 &
FLUTTER_PID=$!
echo "$FLUTTER_PID" > "/tmp/zekra-${TAG}.pid"
echo "TAG=$TAG FLUTTER_PID=$FLUTTER_PID"

Wait for build + DDS:

# Monitor tool with an until loop is the right pattern here — one wakeup, ~2 min
until grep -qE "Dart VM Service on macOS is available at|BUILD FAILED" /tmp/zekra-qa-run.log; do sleep 2; done
grep "Dart VM Service" /tmp/zekra-qa-run.log | tail -1
# → http://127.0.0.1:NNNN/TOKEN=/

The Dart VM Service URL is what you'll feed into DDS evaluate.

Move Zekra to its own Space (Desktop) for clean screenshots

screencapture -x always captures the display, not a specific window — so any other window overlapping Zekra ends up in your PNG (Cursor's IDE pane is the classic offender). Flutter macOS doesn't expose its window through System Events' Accessibility API ("no windows yet" / "Invalid index"), so the per-window screencapture -l WINDOWID trick that works for native apps doesn't work here without pyobjc for CGWindowList.

The easy fix: ask the user to drag the Zekra window to its own Space (macOS Mission Control → "+", then drag Zekra over). Then osascript -e 'tell application "zekra" to activate' switches to that Space and the fullscreen screencapture only sees Zekra + the menu bar. Verified 2026-05-13 — onboarding/patient-shell screenshots came out clean.

A side-by-side split-screen also works: Cursor on the right half, Zekra on the left, fullscreen capture, crop the left half in post (or trust the multimodal Read to focus on the half that matters — it does).

Multi-instance coexistence

A sibling Claude session may already be running its own `zekra.app`. Verified 2026-05-13: the user runs two Claude Code sessions at once, both testing the macOS app. Killing the "stray" process you didn't spawn will trash their session.

Rules:

1. Track only your PIDs. Save them in /tmp/zekra-${TAG}.pid (above). Never pkill -f zekra.app or pkill -f "flutter run". To shut down your instance: kill "$(cat /tmp/zekra-${TAG}.pid)" 2>/dev/null; pkill -P "$(cat /tmp/zekra-${TAG}.pid)" 2>/dev/null (kills your flutter run plus its zekra.app child).

2. Unique ObjectBox dir per instance. ZEKRA_OBJECTBOX_DIR=/tmp/zekra-${TAG}/objectbox — both sessions can run end-to-end onboarding flows in parallel without colliding. The FlutterPool infrastructure in scripts/flutter_pool.py already uses this pattern for N=4 concurrent instances; copy the convention.

3. Unique log path per instance. /tmp/zekra-${TAG}-run.log. Don't tail /tmp/zekra-onboard-run.log and assume it's yours.

4. Window activation is racy with two instances of the same bundle. osascript -e 'tell application "zekra" to activate' activates whichever zekra.app comes first — possibly the sibling Claude's window. Two options:

• Find your window's CGWindowID by your zekra.app child PID, then

screencapture -x -l "$WINID" /tmp/...png to snapshot only your window. Find the child via pgrep -P "$FLUTTER_PID", then the CGWindowID via a one-liner Python Quartz.CGWindowListCopyWindowInfo loop.

• Or run one instance per Mac user / display and let the user pick

whose terminal owns which window.

5. xcodebuild build.db is process-global. Two concurrent flutter run spawns will race on the same build.db lock. FlutterPool solves this by spawning sequentially and waiting for the Built build/macos/Build/Products/Debug/zekra.app marker before starting the next. If you're starting a single instance and a sibling is already mid-build, wait — don't try to parallel-build.

Step 2: Screenshot + Read = visual confirmation

osascript -e 'tell application "zekra" to activate'
sleep 0.7    # let the activate animation settle
screencapture -x -o /tmp/zekra-qa-01.png

Then use the Read tool: Read /tmp/zekra-qa-01.png. The image renders inline in the conversation — you can describe what's on screen, count fields, spot UI bugs, all directly.

`screencapture -x` = silent (no shutter sound). `-o` = no preview window. For a single-window capture instead of fullscreen: screencapture -x -W /tmp/foo.png (lets the user click the target window — not useful for automation).

Step 3: Convert screenshot pixel coords to logical click coords

Screenshots capture native pixels; cliclick clicks logical (display) coords. The ratio depends on the user's display scaling.

sips -g pixelWidth -g pixelHeight /tmp/zekra-qa-01.png      # native pixels
system_profiler SPDisplaysDataType | grep -E "Resolution|Pixel"    # logical

Compute the scale factor:

scale = native_pixel_width / logical_screen_width

On the dev Mac at time of writing: native 3420×2224, logical 1710×1112 → scale = 2.0. Most Retina Macs are 2.0; scaled displays ("More Space" in System Settings) can be 1.33 or 1.5.

To turn a button visible at screenshot pixel coords (px, py) into a cliclick command: cliclick c:$((px/scale)),$((py/scale)).

When you can't read the screenshot pixel coords precisely, eyeball it from the displayed image and multiply by (logical_width / displayed_width) — Read renders the image at a fixed display size which is often 855 wide on a 1710-logical screen, i.e. a 2x ratio between displayed pixel and logical click coord.

Step 4: Click and type

CC=/opt/homebrew/bin/cliclick

# Click a field at logical (850, 240)
$CC c:850,240
sleep 0.3

# Type text via System Events
osascript -e 'tell application "System Events" to keystroke "Eleanor Whitfield"'
sleep 0.3

# Tab to the next field
osascript -e 'tell application "System Events" to keystroke tab'
sleep 0.3
osascript -e 'tell application "System Events" to keystroke "Nora"'

# Click a button at (850, 742)
$CC c:850,742
sleep 1
screencapture -x -o /tmp/zekra-qa-02.png

cliclick can only type ASCII via t:. For accented characters, emoji, or anything that needs a modifier key, use osascript … keystroke which routes through the OS input layer and handles everything the keyboard can produce.

Always `sleep 0.3+` between input events. Flutter's text input debounces; back-to-back keystrokes can drop characters. After clicks that trigger a route change (Continue → next step), sleep 1+ so the PageController animation finishes before the next screenshot.

The "post-click screenshot shows Cursor" gotcha (it's not what you think)

This is the failure mode you'll hit first, and the diagnosis is counter-intuitive. The symptom is: you click Zekra's Continue button, take a screenshot, and the PNG shows Cursor's IDE pane — not the next Zekra screen. Natural conclusion: "Cursor stole focus, my click missed."

That conclusion is usually wrong. Verified 2026-05-13: the clicks land on Zekra correctly; the screenshots are what fail. A tiling WM (Amethyst confirmed, others likely) re-layouts windows on every focus event, and screencapture taken during the re-layout catches whichever window is front-most at that instant — often Cursor. Activate Zekra again, screenshot again, and you'll see the click landed all along.

Symptoms:

• Screenshot taken right after a click shows Cursor's pane layout, not

Zekra. A subsequent screenshot (after another activate; sleep 0.7) shows Zekra on the next step — proving the click worked.

• The "Tall" or "Wide" macOS resize hint pill flashes briefly in the

middle of the screen — that's the tiling WM in action.

• A field in Zekra stays blank after a keystroke call → this one IS

click/focus related: the click landed on Zekra's window but on a non-input region, so the keystrokes went to whatever held the text caret (usually nothing, sometimes Cursor's editor).

Mitigations:

(0) First, ask the user to quit Amethyst / Yabai / Rectangle's auto-tile

If a tiling WM is running, screenshots will lie to you and you'll waste time chasing a Cursor focus theory that isn't real. Verify with pgrep -fl 'Amethyst|yabai|Rectangle'. Asking the user to quit it for the duration of the session is the fastest fix.

(a) Activate before every click, batch the whole sequence inside one Bash call

osascript -e 'tell application "zekra" to activate' && sleep 0.7
$CC c:X1,Y1 && sleep 0.3
osascript -e 'tell application "System Events" to keystroke "..."' && sleep 0.3
$CC c:X2,Y2 && sleep 1
screencapture -x -o /tmp/final.png

A single Bash invocation reduces the windows where Cursor can intrude. Each && short-circuits if a previous step fails. Still not bulletproof when the user interacts with Cursor mid-sequence.

(b) Tile Zekra to a fixed half-screen pane

osascript -e 'tell application "zekra" to activate'; sleep 0.5
osascript << 'APPLE'
tell application "System Events" to tell process "zekra"
  click menu item "Right" of menu "Move & Resize" of menu item "Move & Resize" of menu "Window" of menu bar 1
end tell
APPLE
sleep 1

After tiling, your click coords are within a known rectangle (right half: x from logical_width/2 to logical_width, y from menu-bar-height to bottom). The downside: any subsequent tell application "zekra" to activate un-tiles it back to fullscreen — you'll need to re-tile after each activate. Best for short bursts.

(c) When (a) and (b) keep failing, switch to DDS-driven completion

The point of testing each step is to prove the underlying machinery works. If clicks are unreliable, drive the state transition via DDS evaluate. You lose the "did the button visually react" check, but you keep the "did the state correctly advance" check, and you can screenshot the destination screen which proves the route landed. See Step 6.

Practical guidance: Ask the user to keep their hands off the Cursor window for the duration of the automated sequence — explain that you need ~30s of focus on Zekra. This is usually faster than building a bulletproof click loop.

Step 5: Inspect state via DDS evaluate

The Flutter app exposes the Dart VM Service. Connect via WebSocket and evaluate Dart expressions against package:zekra/main.dart's library scope, which gives you the top-level globals gemmaService, objectboxStore, and the imported symbols dispatchTool, ZekraStore, PatientProfileRepository, etc.

Reuse `scripts/macos_chat_harness.connect()` instead of writing the WebSocket code yourself — it handles isolate + library lookup, has a Vm.evaluate() method, and skips async event frames correctly:

import sys
sys.path.insert(0, '/Users/amanie/Documents/zekra-ai/scripts')
from macos_chat_harness import connect

# Auto-detects from /tmp/zekra_macos_run.log, OR pass ws_url= explicitly
vm = connect(ws_url="ws://127.0.0.1:NNNN/TOKEN=/ws")    # from flutter run log

# Read box counts
r = vm.evaluate(
    "[ZekraStore(objectboxStore).noteBox.count(),"
    " ZekraStore(objectboxStore).reminderBox.count(),"
    " ZekraStore(objectboxStore).patientProfileBox.count()]"
    ".toString()"
)
print(r["result"]["valueAsString"])    # e.g. "[3, 7, 1]"

Idioms

• Always end your expression with `.toString()` — the VM service

surfaces results as valueAsString on the response. Without it you get a @Instance handle and no readable payload.

• Single expression, no statements. The eval surface rejects block

bodies. Wrap multiple writes in a list literal: [box.put(a), box.put(b)].toString().

• Type literals: `<String,dynamic>{}` not `{}`. Bare {} parses as

Set<dynamic> and breaks Map<String,dynamic>-typed args (e.g. dispatchTool's args parameter).

• Future-returning expressions chain to a file write. .evaluate()

returns synchronously, so a Future arrives as a handle and the completion is invisible. Pattern:

  someAsync().then((r) => File("/tmp/x.log").writeAsStringSync(r.toString()))

File is in scope because main.dart imports dart:io. File writes survive DDS disconnects (which happen often during long calls); print() to flutter run stdout is lost when DDS drops.

Drive a chat turn end-to-end (when LLM is loaded)

gemmaService.send("Hi").then(
  (r) => File("/tmp/zekra_demo.log").writeAsStringSync(
    "kind=${r.runtimeType}\nbody=${r}\n"
  )
).catchError((e, st) => File("/tmp/zekra_demo.log").writeAsStringSync(
  "ERROR: $e\n$st"
))

Drive a tool dispatch directly (when running with ZEKRA_NO_GEMMA=1)

dispatchTool("today_summary", <String,dynamic>{}, ZekraStore(objectboxStore)).then(
  (r) => File("/tmp/zekra_demo.log").writeAsStringSync(r.toString())
)

Read entities you typed via the UI

After clicking through the profile form, verify the entity landed:

ZekraStore(objectboxStore).patientProfileBox.getAll().map(
  (p) => p.name + "|" + p.preferredName + "|" + p.homeAddress + "|" + p.caregiverPhone
).join("\n").toString()

Step 6: DDS as the fallback driver

When (a)+(b) above keep failing because Cursor steals focus, drive the state transition via DDS. You lose the per-step visual check but keep the end-to-end completion + state assertion.

Example: complete onboarding programmatically

If clicks through steps 3-5 are flaking, write the profile + PIN directly and force the relaunch:

// Single-expression form
[
  ZekraStore(objectboxStore).patientProfileBox.put(
    PatientProfileEntity(
      name: 'Eleanor Whitfield',
      preferredName: 'Nora',
      homeAddress: '12 Maple Lane, Beirut, Lebanon',
      caregiverName: 'Amanie',
      caregiverPhone: '+961 70 123 456',
    )
  ),
  // pinHash must be set non-empty; SettingsRepository.setPin does the
  // hashing — call it via a saved repo handle if you have one wired,
  // or write the SettingsEntity directly.
].toString()

Then pkill -f zekra.app and relaunch. AppModeController.build() (flutter/lib/auth/app_mode.dart:18) checks pinHash.isEmpty || profile == null on boot — with both set, it returns AppMode.patient and skips onboarding. Screenshot the landing screen to prove the route landed.

This is a real test, not a shortcut: it asserts that the entity schema, the box write, the settings hash, and the boot-time mode detection all compose correctly. The only thing you skip is the visual click-by-click rendering of intermediate steps, which the UI inspection already covered for steps 1-2.

What works, what doesn't, on this stack

Works

• screencapture + Read PNG round trip (multimodal Read sees the pixels).
• cliclick c:X,Y for clicks at known logical coords.
• osascript … keystroke for arbitrary text including non-ASCII.
• osascript … keystroke tab to advance between TextFields.
• Reading the menu bar via tell application "System Events" to tell process "zekra" (the menu items DO show up in AX).
• DDS evaluate for read + write against ZekraStore and any singleton.
• Pre-populating ObjectBox via DDS before driving a UI flow.

Doesn't work (don't waste time)

• osascript … get bounds of window 1 against the Flutter window → returns -1728 Can't get bounds. Flutter macOS doesn't expose window-level AX bounds.
• tell process "zekra" to get every window returns the window reference but position/size properties throw -1719 Invalid index. Same root cause — Flutter's NSWindow isn't fully AX-exposed.
• Reading the widget tree from outside Flutter. There's no public API. The closest is developer.registerExtension('ext.zekra.x', ...) registered at app start, which lets you call arbitrary Dart from DDS — but that requires a code change.
• Re-tiling via the Window menu after tell application "zekra" to activate — activate un-tiles to fullscreen first.
• cliclick t:"…" with non-ASCII — use osascript keystroke for that.

Test fixture leftovers to know about

• The default ObjectBox dir is ~/Documents/zekra-objectbox/. If you booted with ZEKRA_OBJECTBOX_DIR=/tmp/... once and then run again without the env var, you're back on the real store — be deliberate about isolation.
• The macOS app is non-sandboxed (com.apple.security.app-sandbox = false), so getApplicationDocumentsDirectory() returns ~/Documents/. The model symlinks live at ~/Documents/model.litertlm and ~/Documents/w600k_r50.onnx — if those break, ObjectBox schema mismatch errors will surface as Incoming property ID ... does not match existing UID ....
• where_am_i returns STEP=unknown on macOS — LocationService hard-gates on Platform.isAndroid. Don't try to test location-aware flows on macOS.

Quick-start template

# 1. Boot isolated instance (skip LLM, isolated store)
mkdir -p /tmp/zekra-qa/objectbox
pkill -f zekra.app 2>/dev/null; sleep 1
cd /Users/amanie/Documents/zekra-ai/flutter
ZEKRA_OBJECTBOX_DIR=/tmp/zekra-qa/objectbox ZEKRA_NO_GEMMA=1 \
  nohup flutter run -d macos --debug > /tmp/zekra-qa-run.log 2>&1 &

# 2. Wait for DDS, capture URL
until grep -qE "Dart VM Service|BUILD FAILED" /tmp/zekra-qa-run.log; do sleep 2; done
DDS=$(grep -oE "http://127.0.0.1:[0-9]+/[A-Za-z0-9_=\-]+/" /tmp/zekra-qa-run.log | tail -1)
WS=${DDS/http:/ws:}ws
echo "DDS WebSocket: $WS"

# 3. Screenshot starting state
osascript -e 'tell application "zekra" to activate'; sleep 0.7
screencapture -x -o /tmp/zekra-qa-01.png
# Then: Read /tmp/zekra-qa-01.png

# 4. Click + type one or more steps
CC=/opt/homebrew/bin/cliclick
$CC c:LOGICAL_X,LOGICAL_Y && sleep 0.3
osascript -e 'tell application "System Events" to keystroke "value"' && sleep 0.3
screencapture -x -o /tmp/zekra-qa-02.png

# 5. Verify state via DDS (Python)
.venv/bin/python3 -c "
import sys; sys.path.insert(0, '/Users/amanie/Documents/zekra-ai/scripts')
from macos_chat_harness import connect
vm = connect(ws_url='$WS')
r = vm.evaluate('ZekraStore(objectboxStore).patientProfileBox.getAll().length.toString()')
print(r['result']['valueAsString'])
"

# 6. Teardown
pkill -f zekra.app

Anti-patterns to avoid

• Don't `pkill -f zekra.app` or `pkill -f "flutter run"`. A sibling

Claude session may be using their own instance — see "Multi-instance coexistence" above. Only kill PIDs you saved in /tmp/zekra-${TAG}.pid.

• Don't `flutter install`. It uninstalls the prior bundle, wiping

~/Documents/zekra-objectbox/ and the model symlinks. Use the env var isolation in Step 1 instead.

• **Don't rm -rf /tmp/zekra-onboard/objectbox or any path you didn't

create with your $TAG.** Another agent may be midway through a test run against that store.

• Don't assume "clicking it" === "verified the feature". The UI

binds to ObjectBox boxes via Riverpod StreamProviders. A successful click moves the visible UI; a successful click plus a box.count() diff proves the underlying write the rest of the app sees.

• Don't sleep < 200ms between Flutter input events. Text field

debouncing drops chars. 300ms is the safe floor; 500ms for keystrokes that trigger validation re-runs.

• Don't trust screenshots taken immediately after `activate` — give

it 0.7s+ for the window-server animation. Otherwise you'll snap Cursor.

• **Don't drive the chat tab via clicks if you can call chatAgent.send

via DDS.** The chat-tab UI exists; the production agent loop runs through chatAgent.send → gemmaService.send. DDS evaluate against chatAgent.send("text") exercises the same path with deterministic capture.

When to use this skill vs. alternatives

This skill is the "I want to see and touch the app" path. The HF probe is the "I want to stress-test the agent at scale" path. They share the DDS connect/evaluate primitives.