When to use
Use this skill when you need to drive a running Flutter app as a user would:
- Tap buttons, icons, list items, or any interactive widget.
- Type text into fields, submit forms, clear inputs.
- Scroll or swipe to reveal off-screen content.
- Navigate between routes programmatically (push, pop, popUntil).
- Dismiss dialogs and bottom sheets.
- Give a widget keyboard focus, then press keys (Enter, Escape, Tab, arrows, ASCII chars).
- Hot-reload or hot-restart after editing Dart source files.
- Combine reload + screenshot + semantics in one round-trip for fast iteration.
Selectors
Most interaction tools target a widget by ref — a short string like "s_0" returned by semantic_snapshot. For visible widgets, call semantic_snapshot, scan the returned nodes, find the right ref, then pass it. Narrow the node set with identifierPrefix or subtreeOf (a ref or an identifier), and cut every node down to the keys you read with fields; refs are those of the full tree either way. For off-screen targets with stable semantics text or identifier, use reveal_search; it performs a bounded snapshot → match → scroll loop and returns a fresh ref/snapshotId.
Snapshot node keys to scan:
| Want to find | Scan field | Example value |
|---|---|---|
| By semantics identifier | identifier |
"nav.tasks" |
| By visible label / text | label |
"Login" |
| By value or hint | value / hint |
"user@example.com" |
| By semantic role | type |
"button" |
| By what it accepts | actions |
["tap"] |
Example — find the "Login" button ref:
semantic_snapshot()
→ nodes: [{ref:"s_0", label:"Login", actions:["tap"]}, ...]
tap_widget(ref: "s_0")
Pass snapshotId (from the snapshot response) to any interaction call. If the tree has changed, the call returns stale_snapshot with both IDs so you know to re-snapshot. Refs are only valid against the most recent snapshot.
Recipes
Tap a widget by text
semantic_snapshot()
→ find node where label == "Submit" → ref "s_3"
tap_widget(ref: "s_3", snapshotId: <id>)
Fill a login form
semantic_snapshot() → email ref "s_1", password ref "s_2"
fill_form(fields: [{ref:"s_1", text:"user@example.com"}, {ref:"s_2", text:"secret"}], snapshotId: <id>)
→ one round-trip; stops on first failure
Scroll to find an item
scroll(direction: "down", distance: 300)
semantic_snapshot() → item now visible → ref "s_5"
tap_widget(ref: "s_5")
Reveal an off-screen field by identifier
reveal_search(query: "greeting_input_field", matchBy: "identifier", direction: "down", maxAttempts: 4)
→ returns ref "s_14", snapshotId 2, match, attempts trace
enter_text(ref: "s_14", snapshotId: 2, text: "hello")
Wait for a widget to appear
wait_for(predicate: {kind: "text", text: "Welcome"}, timeoutMs: 8000)
→ returns fresh snapshot when text appears
tap_widget(ref: <ref from wait_for snapshot>)
Navigate to a route
navigate(action: "push", route: "/settings", arguments: {tab: "account"})
semantic_snapshot() → fresh refs in the new screen
Hot reload after a code change
hot_reload_and_capture()
→ screenshot + semantic snapshot + errors in one call
Press the back hardware button
press_key has no Back key. Use navigate(action: "pop") for Navigator pop; handle_dialog(action: "dismiss") for dialogs; press_key(key: "Escape") on desktop.
navigate(action: "pop")
Tool reference
tap_widget
Tap a widget by ref. ref • string • required. snapshotId • integer • optional. connection • object • optional.
{"name": "tap_widget", "arguments": {"ref": "s_3", "snapshotId": 7}}
Returns: {"via": "semantic_action", "ref": "s_3"} — Failures: stale_snapshot, ref_not_found
long_press
Long-press a widget by ref. ref • string • required. snapshotId • integer • optional. connection • object • optional.
{"name": "long_press", "arguments": {"ref": "s_2"}}
Returns: {"via": "semantic_action"} — Failures: stale_snapshot, ref_not_found
enter_text
Enter text into a text field; taps to focus before typing. ref • string • required. text • string • required. snapshotId • integer • optional. connection • object • optional.
{"name": "enter_text", "arguments": {"ref": "s_1", "text": "hello@example.com"}}
Returns: {"via": "editable_state", "verified": true, "appliedText": "..."}. A write the field kept nothing of fails with text_not_applied, appliedText (empty) and restoredText (the value put back). Failures: stale_snapshot, ref_not_found, text_not_applied
reveal_search
Find a semantic target that may be off-screen. query • string • required. matchBy • string • optional (text|identifier|label|value|hint, default text). direction • string • optional (up|down|left|right, default down). maxAttempts • integer • optional • max 10. distance • number • optional. connection • object • optional.
{"name": "reveal_search", "arguments": {"query": "greeting_input_field", "matchBy": "identifier", "direction": "down", "maxAttempts": 4}}
Returns: {"ref": "s_14", "snapshotId": 2, "match": {...}, "attempts": [...]} — Failures: missing_query, target_not_found, scroll_blocked, target_not_actionable. identifier matches whole and case-sensitively; the other modes are case-insensitive substring tests. A miss by identifier carries identifiersSeen (how many the searched screens published) and nearIdentifiers — the closest of them, so panel.tab not found comes back with panel.tab.overview / panel.tab.jobs to pass instead.
fill_form
Batch text entry: fills multiple fields in one call. Stops on first failure. snapshotId validated on first field only. fields • array of {ref, text} • required. snapshotId • integer • optional. connection • object • optional.
{"name": "fill_form", "arguments": {"fields": [{"ref":"s_1","text":"user"},{"ref":"s_2","text":"pass"}], "snapshotId": 5}}
Returns: {"filled": 2} — Failures: stale_snapshot, ref_not_found
scroll
Scroll to reveal content. "down" reveals content below (finger swipes up). direction • string • required (up|down|left|right). ref • string • optional — the list to scroll, or any node inside it; without it the list under the screen centre scrolls. distance • number • optional • default 300, honoured exactly where the list takes an offset, otherwise rounded to a viewport page. snapshotId • integer • optional. connection • object • optional.
{"name": "scroll", "arguments": {"direction": "down", "ref": "s_0", "distance": 500}}
Returns: {"via": "semantic_action", "scrollBefore": 0.0, "scrollAfter": 500.0, "distance": 500.0}. scrollBefore/scrollAfter are forwarded whenever measurable; distance is present on exact-offset and pointer-scroll paths. A node that advertises the scroll action without a scroll position (custom Semantics(onScrollUp: ...)) takes it and returns success: true, verified: false, unmeasured: true. Failures: ref_not_found, stale_snapshot, no_scrollable_at_point, no_scroll_movement, unsupported_scroll_action, semantics_owner_unavailable
swipe
High-velocity fling. Same direction model as scroll. Always Tier 2 pointer events. direction • string • required. ref • string • optional. distance • number • optional • default 300. snapshotId • integer • optional. connection • object • optional.
{"name": "swipe", "arguments": {"direction": "left", "ref": "s_4"}}
Returns: {"via": "pointer_events", "scrollBefore": 0.0, "scrollAfter": 420.0} when movement is measurable. When finite scroll offsets are unavailable, a dispatched gesture returns success: true, verified: false, and measurementReason. Failures: ref_not_found, stale_snapshot, no_scroll_movement, web_gesture_not_supported
drag
Drag from one widget to another. Always Tier 2. fromRef • string • required. toRef • string • required. snapshotId • integer • optional. connection • object • optional.
{"name": "drag", "arguments": {"fromRef": "s_2", "toRef": "s_7"}}
Returns: {"via": "pointer_events"} — Failures: ref_not_found, web_gesture_not_supported
hover
Synthesize a mouse hover. Desktop/web only — no hover concept on mobile. ref • string • required. snapshotId • integer • optional. connection • object • optional.
{"name": "hover", "arguments": {"ref": "s_5"}}
Returns: {"via": "pointer_events"} — Failures: ref_not_found, platform error on mobile
focus_widget
Give keyboard focus to a widget, so the press_key that follows reaches it. Uses the node's semantic focus action when the snapshot lists one, otherwise the focusable widget inside the ref's bounds (the only route on iOS, where Flutter exposes no focus action). Leaves a parked hover in place. ref • string • required. snapshotId • integer • optional. connection • object • optional.
{"name": "focus_widget", "arguments": {"ref": "s_5"}}
Returns: {"via": "semantic_action" | "focus_node", "verified": true, "verifiedBy": "semantics_flag" | "focus_node", "focusMoved": bool} — success: true means the target reports focus after the framework caught up; focusMoved: false means it already held it. On the focus-node route, focusedNow names the control inside the ref that actually took focus. Failures: ref_not_found, stale_ref, stale_snapshot, target_disabled, focus_not_exposed (nothing focusable behind the ref), focus_refused (the request went through but the target does not report focus; focusedNow says where it sits)
press_key
Synthesize key press (down+up). Accepted: Enter Escape Tab Backspace Delete Space ArrowUp ArrowDown ArrowLeft ArrowRight plus single ASCII (a-z 0-9). key • string • required. ctrl/shift/alt/meta • boolean • optional • default false. connection • object • optional.
{"name": "press_key", "arguments": {"key": "Enter"}}
Returns: {"key": "Enter", "handled": bool} — handled says whether either dispatch phase (hardware keyboard handlers or focus chain) claimed the main key-down event; false does not describe the key-up or modifier events. clearedStaleModifiers / releasedStuckModifiers name any modifier the call had to release — a modifier left held by an earlier keystroke makes every plain shortcut stop matching, so the call normalizes it and says so. Failures: unknown_key
wait_for
Wait for a UI predicate; returns fresh semantic snapshot. Predicates: {kind:"text",text} | {kind:"noText",text} | {kind:"time",ms} | {kind:"stable",stableWindowMs}. stable samples once per frame and matches after the semantics tree has remained unchanged for the requested wall time; the match reports stableFor.sampledFrames and stableFor.elapsedMs. stableWindowMs must be less than timeoutMs (default 5000); an impossible budget fails immediately with invalid_predicate. predicate • object • required. timeoutMs • integer • optional • default 5000 • max 30000. connection • object • optional.
{"name": "wait_for", "arguments": {"predicate": {"kind": "text", "text": "Dashboard"}, "timeoutMs": 8000}}
Returns: fresh semantic snapshot — Failures: timeout, invalid_predicate
navigate
Drive the registered Navigator. Requires MCPToolkitBinding.instance.navigatorKey = key in the app. action • string • required (push|pop|popUntil). route • string • required for push/popUntil. arguments • object • optional (for push). connection • object • optional.
{"name": "navigate", "arguments": {"action": "push", "route": "/profile", "arguments": {"userId": "42"}}}
Returns: {"action": "push", "route": "/profile"}. A generated unnamed route returns success: true, verified: false, and via: "stack_changed_unnamed_route" because the stack changed but its requested name cannot be checked. pop reads success off the stack and reports handled from maybePop: a page under PopScope(canPop: false) comes back handled: true, success: false, nothing_popped. Failures: navigator_not_configured, route_not_found
handle_dialog
Dismiss the topmost popup/dialog route. Only action: "dismiss" supported. Requires navigatorKey = key on MCPToolkitBinding.instance in the app. action • string • required (must be "dismiss"). connection • object • optional.
{"name": "handle_dialog", "arguments": {"action": "dismiss"}}
Returns: {"success": true, "handled": true, "routeType": "DialogRoute<void>"}. success is read off the route stack; handled is what maybePop answered, so a dialog under PopScope(canPop: false) comes back handled: true, success: false, dialog_declined_pop. Failures: navigator_not_registered, no_popup_route, dialog_declined_pop
hot_reload_flutter
Hot reload the app. Preserves state. force • boolean • optional • default false (reload even without source changes). connection • object • optional.
{"name": "hot_reload_flutter", "arguments": {}}
Returns: "Hot reload completed" + report JSON — Failures: vm_not_connected, compilation_error
hot_restart_flutter
Full restart. App state not preserved. No required params. connection • object • optional.
{"name": "hot_restart_flutter", "arguments": {}}
Returns: {"report": {"type": "Success", "success": true}} — Failures: vm_not_connected
hot_reload_and_capture
Hot reload then capture screenshot + semantics + errors in one call. compress • boolean • default true. includeSemantics • boolean • default true. includeErrors • boolean • default true. errorsCount • integer • default 4. connection • object • optional.
{"name": "hot_reload_and_capture", "arguments": {"includeErrors": true}}
Returns: screenshot (base64) + semantic snapshot + errors — Failures: vm_not_connected, compilation_error
Patterns
Always wait_for before tap_widget after navigation
After navigate(action: "push") the new route's widgets are not in the tree yet. Use wait_for with a text predicate to confirm the destination has rendered, then snapshot and act.
navigate(action: "push", route: "/checkout")
wait_for(predicate: {kind: "text", text: "Order Summary"}, timeoutMs: 5000)
semantic_snapshot() → tap target widgets
Prefer fill_form over multiple enter_text calls
Each enter_text is a separate VM round-trip. fill_form sends all field/text pairs in one call; snapshotId is checked once (on the first field). For any form with 2+ fields, always prefer fill_form.
# Avoid: 2 round-trips
enter_text(ref: "s_1", text: "Alice")
enter_text(ref: "s_2", text: "secret")
# Prefer: 1 round-trip
fill_form(fields: [{ref: "s_1", text: "Alice"}, {ref: "s_2", text: "secret"}])
After hot_reload_*, wait for the new tree before continuing
Hot reload completes asynchronously. Use wait_for(predicate: {kind:"stable", stableWindowMs:300}) to confirm the tree has settled before calling semantic_snapshot. Or use hot_reload_and_capture which returns a post-reload snapshot directly.
hot_reload_flutter()
wait_for(predicate: {kind: "stable", stableWindowMs: 300})
semantic_snapshot() → interact with reloaded widgets
# Or in one call (preferred):
hot_reload_and_capture() → screenshot + semantics + errors already post-reload