[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-1615":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":14,"starSnapshotCount":14,"syncStatus":25,"lastSyncTime":26,"discoverSource":27},1615,"agent-simulator","jasonkneen\u002Fagent-simulator","jasonkneen","An iOS simulator in a browser you can inspect",null,"TypeScript",193,11,1,0,5,3.24,"GNU Affero General Public License v3.0",false,"main",true,[],"2026-06-12 02:00:30","# Agent Simulator\n\n**Live iOS Simulator preview with React Native inspector, accessibility-tree driven tools, and an MCP bridge.** Drive iOS apps from your browser or an AI agent — see the live screen, click any element to get its real source code, tap \u002F swipe \u002F type without moving the macOS cursor.\n\nBuilt for three audiences at once:\n\n1. **RN \u002F Expo devs** who want a Figma-style \"layers panel\" for their running app, with real `App.tsx:42` source lookups.\n2. **QA & demo flows** that want a scriptable sim without the pain of XCUITest — accessibility-label taps, swipes, keyboard input, screenshots.\n3. **AI agents** (Claude Desktop, OpenAI Agents SDK, Cursor, …) that need a first-class tool surface to *see* what's on the screen and *act* on it. Everything the browser can do is exposed over MCP.\n\n```\n┌──────────────────────────┐    ┌─────────────────────────────┐\n│ Browser UI               │    │ MCP client                  │\n│ \u002F   (agent-simulator)    │    │ (Claude Desktop, Codex, …)  │\n│  - live MJPEG stream     │    └────────────────┬────────────┘\n│  - iOS a11y layer tree   │                     │ stdio\n│  - React component tree  │                     ↓\n│  - props + source window │    ┌─────────────────────────────┐\n│  - tap\u002Fswipe\u002Fdrive\u002Ftype  │    │ mcp\u002Fserver.mjs              │\n└─────────────┬────────────┘    │ 15 MCP tools over WS        │\n              │ WS + HTTP       └────────────────┬────────────┘\n              ↓                                  │ WS\n┌──────────────────────────────────────────────────┐\n│ server.js (Node)                                 │\n│ static · \u002Fapi\u002Fconfig · \u002Fapi\u002Ftree · \u002Fapi\u002Fsource   │\n│ WS bridge · Metro \u002Fsymbolicate proxy             │\n└──┬────────┬──────────────────────────────────────┘\n   │ stdin  │ WS (opt.)\n   ↓        ↓\n┌──────────┐ ┌─────────────────────┐\n│sim-server│ │ Expo \u002F RN app with  │\n│ (Rust)   │ │ agent-simulator     │\n│ MJPEG +  │ │ metro plugin        │\n│ axe HID  │ │ (inspector bridge)  │\n└─────┬────┘ └─────────────────────┘\n      │\n  iOS Simulator (axe + simctl)\n```\n\n---\n\n## Features\n\n| | |\n|---|---|\n| **Cursor-free input** | Every tap \u002F swipe \u002F key goes through [`axe`](https:\u002F\u002Fgithub.com\u002Fcameroncooke\u002FAXe) → `FBSimulatorHID` → `CoreSimulator`. No macOS cursor movement, no Simulator.app focus requirement, pixel-exact at any zoom. |\n| **Drive any iOS app** | Taps, drags, mouse-wheel scroll, multitask, home, lock, keyboard passthrough. Works on Settings, Maps, Messages — not just RN apps. |\n| **iOS accessibility tree** | Populated from `axe describe-ui` on boot. Every on-screen element with its label, role, and frame — before you click anything. |\n| **React component tree** | For RN apps running the metro plugin, inspect any rendered component and get the full fiber stack with real source paths (React 19 `_debugStack` + Metro `\u002Fsymbolicate`). |\n| **Real source code panel** | Select any component and the Properties panel fetches the actual file and renders a code window around the target line. Works for your code, RN's Libraries, and Expo shims. |\n| **MCP bridge** | 15 tools: `sim_tree`, `sim_tap_by_label`, `sim_swipe`, `sim_type`, `sim_inspect`, `sim_source`, `sim_screenshot`, and more. Every browser capability exposed to AI agents. |\n\n---\n\n## Requirements\n\n- **macOS** with Xcode + at least one iOS simulator installed.\n- **Node ≥ 18** (or Bun — scripts are bun-friendly).\n- **Rust toolchain** (for building `sim-server` once).\n- **[axe](https:\u002F\u002Fgithub.com\u002Fcameroncooke\u002FAXe)** — the cursor-free input driver. One-line install:\n\n  ```bash\n  brew install cameroncooke\u002Faxe\u002Faxe\n  ```\n\n- **A React Native \u002F Expo app** with the agent-simulator metro plugin, *only if* you want React component inspection. Driving, screenshots, and the iOS accessibility tree all work against any iOS app.\n\n---\n\n## Quick start (1 minute)\n\n```bash\n# 1. Clone & install\ngit clone https:\u002F\u002Fgithub.com\u002Fjkneen\u002Fagent-simulator\ncd agent-simulator\nbun install\n\n# 2. Build the Rust sim-server (once)\n(cd sim-server && cargo build --release)\n\n# 3. Build the web UI (once, or after UI changes)\n(cd web-app && bun install && bun run build)\n\n# 4. Boot any simulator yourself, then start the server\nxcrun simctl boot \"iPhone 17 Pro\" || true\nopen -a Simulator\nbun start\n\n# 5. Open the UI\nopen http:\u002F\u002Flocalhost:3200\n```\n\nThe UI will attach to whatever iOS simulator is booted. Open Settings, Maps, Messages, or any app — you can drive it immediately. The iOS layer tree on the left is populated from the accessibility API; the React tree is empty until you load an RN app with the inspector bridge (see below).\n\n---\n\n## One-command demo (the fun one)\n\nWe ship an Expo demo app and a launcher that starts everything for you.\n\n```bash\nbun demo\n```\n\nThis will:\n\n1. Boot the first available iPhone simulator (or reuse the booted one).\n2. Start Metro for `examples\u002Fexpo-demo` on port 8081.\n3. Launch the demo via `exp:\u002F\u002F127.0.0.1:8081` in Expo Go.\n4. Start the agent-simulator Node server on :3200.\n5. Open `http:\u002F\u002Flocalhost:3200` in your browser.\n\nCtrl-C cleans up both processes. Once it's running you can:\n\n- Click anywhere in the preview to drive the app (the tap animates with a pulse ring, and the Layers panel expands to the nearest accessibility element).\n- Hit `I` to enter **Inspect mode**, hover \u002F click a React component, and watch the Properties panel fill with the component name, source file, and actual JSX code.\n- Tap the `+` on the counter — the Code panel opens `examples\u002Fexpo-demo\u002FApp.tsx:42` with the `\u003CText style={styles.pillText}>+\u003C\u002FText>` line highlighted.\n\nFlags:\n\n```bash\nbun demo --no-open             # skip launching the browser\nbun demo --port=3300           # use a different agent-simulator port\nbun demo --metro-port=8082     # use a different Metro port\nbun demo --device=\u003CUDID>       # target a specific simulator\n```\n\n---\n\n## Adding agent-simulator to your own RN \u002F Expo app\n\nIf you already have an app, add the metro plugin so the inspector bridge injects at boot:\n\n```js\n\u002F\u002F metro.config.js\nconst { getDefaultConfig } = require(\"expo\u002Fmetro-config\");\nconst { withAgentSimulator } = require(\"agent-simulator\u002Fruntime\u002Fmetro-plugin\");\n\nmodule.exports = withAgentSimulator(getDefaultConfig(__dirname));\n```\n\n…then install this package alongside:\n\n```bash\n# Note: `agent-simulator` is not published to npm yet.\n# Install directly from GitHub:\nbun add -D github:jasonkneen\u002Fagent-simulator\n```\n\nOr, if you don't want to touch Metro config, import the bridge at your app's entry point *before* `registerRootComponent`:\n\n```ts\n\u002F\u002F index.ts\nimport \"agent-simulator\u002Fruntime\u002Finspector-bridge\";\nimport { registerRootComponent } from \"expo\";\nimport App from \".\u002FApp\";\nregisterRootComponent(App);\n```\n\nWhen your app launches with either setup, the **BRIDGE** pill in the toolbar turns green and inspect clicks produce full component stacks with real source locations.\n\n---\n\n## The UI\n\n| Region | What it shows |\n|---|---|\n| Header | Device chip, stream + bridge status pills, Inspect toggle, Home \u002F Multitask \u002F Lock \u002F Rotate, panel toggles, theme menu. |\n| Layers panel (left) | Toggles between the **iOS accessibility tree** (populated automatically) and the **React component tree** (populated on inspect). Click any layer to select, hover to highlight, the tree auto-expands ancestors when you tap inside the sim. |\n| Simulator canvas | Live MJPEG. In drive mode, the native cursor is hidden and replaced by a round in-preview pointer; taps fire a ping ring, drags become swipes, the mouse wheel scrolls, the keyboard types into the focused field. Inspect mode replaces the pointer with a crosshair and draws selection overlays. |\n| Properties panel (right) | Selected layer's accessibility properties, React component name + source, and a highlighted code window around the JSX call site or component definition. |\n| Status bar | UDID, mode flags, last-inspect frame count, keyboard shortcut hints. |\n\n### Keyboard shortcuts\n\n- `I` — toggle Inspect (disabled when no RN bridge)\n- `Esc` — cancel Inspect \u002F unpin selection\n- Any other keystroke while the preview is focused → forwarded to the sim (works for any text field)\n\n---\n\n## MCP bridge\n\n`mcp\u002Fserver.mjs` is a standalone [Model Context Protocol](https:\u002F\u002Fmodelcontextprotocol.io) server. It speaks stdio to any MCP host and bridges to the running agent-simulator server over WebSocket. All 15 tools:\n\n| Tool | Purpose |\n|---|---|\n| `sim_info` | Device UDID \u002F name \u002F stream URL \u002F device-point size. |\n| `sim_tree` | Full iOS accessibility tree. `flat: true` returns a labelled-element list for cheap LLM context. |\n| `sim_tap` | Tap at `(x, y)` sim-ratio. |\n| `sim_tap_by_label` | Look up an AX element by label \u002F value (optional type filter) and tap its centre — robust to layout changes. |\n| `sim_swipe` | Single-call native swipe with `durationMs`. |\n| `sim_type` | Type printable text into the focused TextField. |\n| `sim_key` | Press one USB-HID keycode (Return=40, Backspace=42, Esc=41, arrows 80–83…). |\n| `sim_button` | `home` \u002F `lock` \u002F `power` \u002F `side-button` \u002F `siri` \u002F `apple-pay`. |\n| `sim_multitask` | Open the app switcher. |\n| `sim_screenshot` | Returns the current frame as an MCP `image` content. |\n| `sim_inspect` | Inspect the RN component at `(x, y)` — returns a source-symbolicated component stack. |\n| `sim_source` | Read a code window around `file:line` for any absolute path in the workspace. |\n| `sim_select_by_source` | Given `fileName` + `line`, probes a grid until it finds a rendered component whose source matches. |\n| `sim_subscribe_selections` | Streams `inspectResult` events back as MCP logging notifications. |\n| `sim_unsubscribe_selections` | Stops the above. |\n\n### Configure an MCP client\n\nFor **Claude Desktop**, add to `~\u002FLibrary\u002FApplication Support\u002FClaude\u002Fclaude_desktop_config.json`:\n\n```jsonc\n{\n  \"mcpServers\": {\n    \"agent-simulator\": {\n      \"command\": \"node\",\n      \"args\": [\"\u002Fabsolute\u002Fpath\u002Fto\u002Fagent-simulator\u002Fmcp\u002Fserver.mjs\"],\n      \"env\": { \"SIM_PREVIEW_URL\": \"http:\u002F\u002Flocalhost:3200\" }\n    }\n  }\n}\n```\n\nThen start the agent-simulator server (`bun start` or `bun demo`). The MCP server connects on demand.\n\nFor any other MCP host (OpenAI Agents SDK, Cursor, Continue, Codex, custom harnesses): spawn `node \u002Fpath\u002Fto\u002Fmcp\u002Fserver.mjs` with `SIM_PREVIEW_URL` pointing at your running server.\n\n### Example agent flow\n\n```\n1. sim_tree flat:true                       → see every element on screen\n2. sim_tap_by_label \"+\"                     → tap the counter's plus\n3. sim_tap_by_label \"Your name\" type:\"TextField\"\n4. sim_type \"hello agent\"\n5. sim_key 40                               → Return\n6. sim_inspect x:0.86 y:0.24                → returns App.tsx:42\n7. sim_source file:\"...\u002FApp.tsx\" line:42    → read the JSX\n```\n\n---\n\n## Architecture\n\nThree binaries, one simulator:\n\n1. **sim-server (Rust)** — captures the simulator screen via `simctl io screenshot`, encodes MJPEG, serves `\u002Fstream.mjpeg`, `\u002Fsnapshot.jpg`, `\u002Fapi\u002Ftree`, and `\u002Fhealth`. Reads tap \u002F swipe \u002F type commands from stdin and shells out to `axe`.\n2. **server.js (Node)** — orchestrates sim-server, serves the Vite-built UI, exposes `\u002Fapi\u002Fconfig` \u002F `\u002Fapi\u002Ftree` \u002F `\u002Fapi\u002Fsource`, bridges browser and MCP WebSocket clients, proxies stack frames through Metro's `\u002Fsymbolicate` for React inspection.\n3. **web-app (React + shadcn + Tailwind)** — the UI. Dark \u002F light themes, two trees, resizable panels, keyboard capture, drag-to-swipe, wheel-to-scroll.\n\n### Touch injection\n\nClicks from the UI (or MCP) travel:\n\n```\nbrowser ── WS ──> server.js ── stdin ──> sim-server ── exec ──> axe tap -x … -y … --udid …\n                                                                       │\n                                                                       └──> FBSimulatorHID → CoreSimulator\n                                                                           (device-point coords, cursor-free)\n```\n\nCoordinates are carried as `[0, 1]` ratios end-to-end; `sim-server` converts to device points using the root `AXFrame` from `axe describe-ui` (cached with a 5 s TTL, invalidated on rotate). Every gesture is one subprocess call — no per-step stdin traffic.\n\n### `\u002Fapi\u002Ftree` and `\u002Fapi\u002Fsource`\n\n- **`GET \u002Fapi\u002Ftree`** — proxies `axe describe-ui --udid \u003Cudid>` via sim-server and returns the raw JSON. The UI pulls this on boot and on refresh; the MCP `sim_tree` tool wraps it.\n- **`GET \u002Fapi\u002Fsource?file=ABS&line=N&context=M`** — reads a window of lines around `line` from an absolute path, returns `{file, lines, startLine0, endLine0, targetLine0, language}`. Powers the Code panel in the UI and the `sim_source` MCP tool.\n\n### React source resolution\n\nReact 19 removed `fiber._debugSource`. Sources now live inside `fiber._debugStack`, an `Error` object whose stack string points at bundle-URL line\u002Fcol pairs. On every `inspectResult`:\n\n1. The inspector-bridge (in the RN app) parses `_debugStack`, collects up to 12 candidate frames per fiber, attaches them as `bundleFrames: […]`.\n2. `server.js` batch-POSTs every bundle frame to `http:\u002F\u002Flocalhost:8081\u002Fsymbolicate` in one round trip and picks the first candidate per fiber that isn't inside React's own `createElement` \u002F `jsxDEV` plumbing.\n3. The UI receives a regular `stack[i].source` with a real absolute file path.\n\nThis means *every* user component, RN internal component (RCTView, ScrollView, …), and Expo shim resolves to actual code on disk.\n\n---\n\n## Stream quality and the FPS ceiling\n\nClick the **Quality** dropdown in the toolbar. Five presets plus fine-tune\nsliders for FPS, JPEG quality, scale, and capture mode.\n\n| Preset | Target FPS | Resolution | Mode | Notes |\n|---|---|---|---|---|\n| **Eco** | 2 | 300×650 | MJPEG | battery-friendly, background monitoring |\n| **Balanced** | 3 | 400×870 | MJPEG | default — matches the CSS size the browser paints |\n| **Smooth** | up to 10 | 400×870 | MJPEG | good for scrolling |\n| **Max** | up to 30 | 400×870 | MJPEG | upper limit of the MJPEG path |\n| **Fluid** | up to 60 | 400×870 | BGRA + keepalive | experimental, see caveats below |\n\nChanging a preset fires a WebSocket `setCapture`, which makes the server\nrespawn `sim-server` with the new arguments and broadcast a\n`configChanged` event. The browser `\u003Cimg>` re-subscribes to the new\nstream URL automatically. Total switch time is about half a second.\n\n### Click accuracy is decoupled from resolution\n\nEvery tap \u002F swipe carries `[0, 1]` ratios. They get converted to\ndevice points server-side using the root `AXFrame` from\n`axe describe-ui`, so a 300×650 Eco stream has exactly the same click\nprecision as a 1206×2622 native-retina stream. Lower resolutions only\ncost you pixels on screen, not targeting.\n\n### Honest FPS numbers\n\nMeasured on an iPhone 17 Pro simulator on an M-series Mac:\n\n| Path | Requested | Actual |\n|---|---|---|\n| `axe screenshot` (one-shot) | — | 280–370 ms per call — **~3 fps ceiling** |\n| `axe stream-video --format raw --fps 30` | 30 | ~5–10 fps (axe's screenshot loop is the bottleneck) |\n| `axe stream-video --format bgra` | 60 | First frame, then stalls — `FBVideoStreamConfiguration` is hard-coded to `framesPerSecond: nil`, IOSurface callback stays silent on current iOS sims |\n\nSo the \"up to 60 fps\" on the Fluid preset is aspirational: the\narchitecture is in place, but **axe as shipped cannot deliver 60 fps today**. To unblock real 60 fps you'd need one of:\n\n1. Fork [axe](https:\u002F\u002Fgithub.com\u002Fcameroncooke\u002FAXe) and pass a concrete `framesPerSecond` value into `FBVideoStreamConfiguration` on the BGRA path.\n2. Replace the `axe stream-video` subprocess with a Swift helper that links `FBSimulatorControl.xcframework` directly and drives `SimDevice` \u002F `SimDeviceIOClient` ourselves.\n\nBoth are real half-day projects. The current wire protocol is already\na drop-in replacement — swapping in a faster backend is one file.\n\n### Keepalive in Fluid mode\n\nBecause BGRA otherwise delivers a single frame and then nothing,\nsim-server runs a **screenshot keepalive** alongside the BGRA reader\nin that mode. If no frame arrives for 300 ms, it fires one\n`axe screenshot`, decodes the PNG, re-encodes as JPEG, and broadcasts\nit — so the preview is never blank. When BGRA does fire (animation-heavy\napps, scrolling), the keepalive stays silent.\n\nNet effect in practice:\n\n- Idle screen → ~3 fps from keepalive.\n- Active scrolling → keepalive silent, BGRA may or may not contribute.\n- Worst case is never 0 fps.\n\n### Env-var overrides\n\nThe server accepts the same knobs as environment variables so you can\npick defaults without the UI:\n\n```bash\nSP_FPS=10 SP_QUALITY=60 SP_SCALE=0.33 SP_CAPTURE=mjpeg bun start\nSP_CAPTURE=bgra bun start   # force BGRA path from boot\n```\n\n## Tips\n\n- Use **`axe`'s simulator conventions** (device points, not Mac pixels) if you're writing agent scripts directly. The UI handles the ratio ↔ point conversion.\n- **`axe describe-ui`** is a great reference: `axe describe-ui --udid \u003CUDID> | jq '.[0]'`.\n- The Metro cache is invalidated when `inspector-bridge.js` changes. If the bridge version log doesn't advance after an edit, restart Metro with `expo start --clear`.\n- `SP_FPS` and `SP_QUALITY` env vars tune MJPEG performance (defaults: `fps=3 q=55`).\n\n---\n\n## Development\n\n```bash\n# Re-build the Rust server after Rust edits\n(cd sim-server && cargo build --release)\n\n# Re-build the UI after web-app edits\n(cd web-app && bun run build)\n\n# Run the MCP server standalone\nbun mcp\n\n# Tail MCP traffic with the official inspector\nnpx @modelcontextprotocol\u002Finspector node mcp\u002Fserver.mjs\n```\n\nProject layout:\n\n```\nagent-simulator\u002F\n├── server.js              Node orchestrator, WS bridge, HTTP\n├── scripts\u002Fdemo.mjs       One-command demo launcher\n├── sim-server\u002F            Rust MJPEG + axe driver\n├── runtime\u002F               RN metro plugin + inspector bridge\n├── web-app\u002F               Vite + React + shadcn UI\n├── web\u002F                   Legacy single-file UI (served at \u002Fclassic)\n├── mcp\u002F                   MCP server (15 tools)\n├── examples\u002Fexpo-demo\u002F    Expo app with the plugin pre-configured\n└── LICENSE                AGPL-3.0-or-later\n```\n\n---\n\n## License\n\n**AGPL-3.0-or-later.** See [`LICENSE`](.\u002FLICENSE).\n\nIf you're embedding agent-simulator inside a commercial product, or you need a different license, open an issue — I'm happy to talk about dual-licensing.\n","Agent Simulator 是一个可以在浏览器中运行并进行检查的iOS模拟器。它通过React Native inspector、基于可访问性树的工具以及MCP桥接技术，支持用户在浏览器或AI代理中操作iOS应用，并实时查看屏幕、点击元素获取实际源代码、无需移动macOS光标即可实现点击\u002F滑动\u002F输入等交互。该项目使用TypeScript编写，具备无光标输入、驱动任意iOS应用、显示iOS可访问性树和React组件树等功能特点。非常适合React Native或Expo开发者用于调试应用结构与源码定位；QA团队及演示流程中自动化测试需求；以及需要可视化界面来观察和控制iOS屏幕内容的人工智能代理。",2,"2026-06-11 02:45:02","CREATED_QUERY"]