SKILL.md
3,855 tokens · o200k_base · 16,976 bytes
Source excerpt starting at line 1.---name: cmux-cuadisable-model-invocation: truedescription: "Use only after the user explicitly asks for Computer Use: drive real macOS apps from a cmux agent session via the bundled engine (accessibility tree + screenshots, click/type/scroll/drag, branded cursor), or explain its user-directed permission setup. Reading or discovering this skill is not consent."--- # cmux-cua cmux bundles a local computer-use engine (packaged as `cmux Computer Use` withthe MCP proxy named `cmux-cua`, from a pinned build ofthe `manaflow-ai/cmux-cua` fork) and attaches it as an MCP tool server named`cmux-cua` to every agent session cmux launches (Claude Code, Codex).The agent can then perceive and operate real macOS apps: read the accessibilitytree, take screenshots, and click / type / scroll / drag. Everything runs locally through the bundled **cmux Computer Use** helper. Thehelper has its own TCC identity, so Accessibility and Screen Recording neverbelong to the main cmux app and granting Screen Recording never requiresrestarting cmux. Upstream telemetry and update checks are disabled at runtime. Do not invoke this skill, start its helper, request permissions, or perform aGUI action when the user is only reading, asking about, quoting, or mentioningComputer Use. Wait for a direct user request to use Computer Use; missing toolsor permissions are not a reason to begin setup automatically. ## How it attaches - The `cmux-claude-wrapper` and `cmux-codex-wrapper` inject `cmux-cua` as an MCP proxy using `mcp --socket <cmux-owned socket>` plus cursor-branding and state-dir env. Codex launches the exact tag-installed helper executable as its authenticated approval broker; Claude uses the bundled native-profile proxy client. The Codex wrapper additionally passes `--codex-computer-use-compat`; the Claude wrapper deliberately does not. Attachment availability is not user consent: merely starting an agent or discovering this skill is not a request to use it, and it must not open a permission window or perform GUI work.- `ComputerUseRuntimeService` is the only helper lifecycle owner. It installs the nested helper under the tag-scoped `~/Library/Application Support/cmux/cmux-cua/helper/<scope>/` directory and launches that explicit app URL through LaunchServices.- The native daemon uses the tag-scoped `/tmp/cmux-cua-<uid>/<scope>/cmux-cua.sock`; the Codex compatibility daemon uses `cmux-cua-codex.sock` beside it. Both fit Darwin's Unix-socket path limit and share the tag-scoped cmux Application Support state directory.- Neither wrapper installs global skills or adds skill directories by default. Use the agent's normal skill installer for a persistent user-owned installation. Alternatively, `CMUX_COMPUTER_USE_INSTALL_GLOBAL_SKILL=1` requests an app-managed global link for that launch (`~/.agents/skills/cmux-cua` for Codex or `~/.claude/skills/cmux-cua` for Claude). Export the flag to retain these links on future launches; they may also appear outside cmux. Unset or `=0` removes only verified app-managed links. Project and user-owned skills take precedence, with no competing global install or automatic fallback. Codex 0.153 does not discover new paths from `skills.config`; no Claude projection or plugin fallback is used. Migration of the canonical and legacy `cmux-computer-use` / `codex-cua` links requires an existing cmux bundle ID, known install/build root, and root/current-user ownership. Unknown or dangling targets, real skill directories, unrelated symlinks, and project paths are preserved. `CMUX_CUA_DIAGNOSTICS=1` identifies preserved paths blocking explicit install and verified managed links retired by the per-launch policy. Historical app-created and manually-created symlinks with identical verified targets cannot be distinguished retroactively; recognized app-bundle links are treated as cmux-managed, while unknown and dangling links are preserved.- While Computer Use is enabled, the helper daemon starts quietly at cmux startup with its internal permission gate disabled. Starting cmux or an agent never requests access or shows onboarding.- Wrappers are pure forced proxies. They never copy or launch the helper and never fall back to in-process computer use. cmux owns the onboarding window and opens the permanent macOS permission panes directly; it does not ask the helper to raise an intermediate native prompt. The proxy keeps its external-flow flag on so the first driving call waits for both helper grants before it is forwarded.- Kill switch: set `CMUX_COMPUTER_USE_MCP_DISABLED=1`, or toggle it off in Settings → Computer Use (persists to `~/.config/cmux/cmux.json` and is exported to spawned terminals).- Attaches only on cmux-launched, live-socket sessions (same authority bar as cmux hooks); hooks-disabled and stale-socket sessions do not attach. ## Permissions (one-time, granted to the helper) Two macOS permissions are required and are owned by **cmux Computer Use**, notthe main cmux app: - **Accessibility** — inspect and drive app UI (`AXIsProcessTrusted`).- **Screen Recording** — screenshots / vision (`CGPreflightScreenCaptureAccess`). Onboarding is opened only by a deliberate user action in Settings → ComputerUse (the **Grant…** or **Open System Settings** permission controls), not by atool call, skill load, prompt text, MCP discovery, cmux startup, or agentresume. Settings → Computer Use always shows the two authoritativepermission states; choosing **Grant…** for an ungranted permission opens thatsame permission step and its draggable helper-app recovery path. Each **Allow**action opens the matching permanent System Settings pane in one step and stayslabeled **Allow** until the helper reports the grant; pressing it again simplyreopens the same pane. If macOS has not listed the helper yet, drag or add the**cmux Computer Use** app tile to the list, then turn it on. cmux reads statusfrom the helper over its Unix socket, advances beside System Settings to thenext missing permission, and shows completion in place once both are granted.On macOS Tahoe a third confirmation follows Screen Recording: the system'sdirect-capture consent, an alert that says **cmux Computer Use** "is attemptingto bypass the system private window picker". That alert is expected — it comesfrom onboarding's host-authenticated capture probe, onboarding explains it inplace, and the user must allow it before setup completes. Never "fix" it bysuppressing the probe; without that consent, agent screenshots on Tahoe fail.The consent follows the helper's code signature, so every rebuilt (ad-hocsigned) dev helper can require the direct-capture step again: cmux invalidatesits cached direct-capture-ready flag whenever it replaces the installed helperbuild. This remains quiet until the user deliberately re-enters Settings;helper replacement never presents onboarding on its own.Do not invoke `check_permissions {prompt:true}` or any standalone helper whilethis flow is active: that creates the stray native permission dialogs thisonboarding deliberately avoids. The main cmux process never calls a TCC API orexecutes the cmux-cua binary. If an already-attached proxy is unconfigured, its protected call remains quietand returns the helper's setup-required response after its bounded readinesswait: **“Computer Use onboarding is still in progress. Finish setup in cmux,then retry.”** Re-enter Settings deliberately to start setup; do not try togrant consent by calling a setup/status tool. A TCC prompt naming **Codex Computer Use** (`com.openai.sky.CUAService`) isnot from cmux. The `codex` CLI ships its own computer-use helper; when codexruns inside a cmux terminal and pokes that helper with an Apple Event, macOSattributes the request to the responsible parent — the cmux app — so thedialog reads as cmux asking to control "Codex Computer Use". Nothing in cmuxor the cmux-cua engine references that service; denying the prompt does not affectcmux computer use. If actions fail with a permission error, grant Accessibility to cmux ComputerUse. If screenshots come back blank, grant Screen Recording to cmux ComputerUse. The helper daemon refreshes/restarts to pick up the grant while cmux staysopen. Retry the tool call after onboarding reports both grants. ## Using the tools (agent-facing) cmux already owns the MCP connection's session identity. Do **not** call`start_session` / `end_session`, and do not pass a custom `session` argument.The proxy binds every call to the originating cmux surface so the menu-baritem, cursor, recording cleanup, and background/focus controls stay attachedto the right agent. ### Codex profile Codex gets the exact ten-tool Computer Use roster, in order: `list_apps`, `get_app_state`, `click`, `perform_secondary_action`, `set_value`,`select_text`, `scroll`, `drag`, `press_key`, `type_text`. Use it like the built-in Computer Use connector: 1. Call `get_app_state` with the app name, full path, or unambiguous bundle id before acting. It launches the app if needed and returns the logical-size JPEG screenshot plus the compact accessibility tree.2. Prefer the current snapshot's string `element_index`; use screenshot-local x/y coordinates only as fallback.3. Use xdotool-style key strings such as `super+l` with `press_key`.4. Operate controls with visible pointer clicks by default: the branded cursor gliding to each button and clicking is the product experience, so requests like “click 100 + 105” mean literal button-by-button pointer interaction on Calculator's buttons. Reserve `type_text` for entering text into text fields (search boxes, forms, editors) — not as a shortcut around clicking on-screen controls. Only fall back to a single keyboard sequence when the user explicitly asks for speed over visibility or a control has no clickable element.5. Actions return a compact dispatch acknowledgement, not a screenshot or accessibility tree. After one or more actions, call `get_app_state` before deciding what to do next. The returned screenshot/tree is the authoritative verification surface, matching the built-in Computer Use connector.6. Numeric `element_index` values belong only to the state that displayed them. Re-snapshot before using an index that may have been renumbered by a layout change (Calculator's **All Clear** removes display nodes, for example). Multiple coordinate actions against a stable layout may be issued in one host turn; do not batch element-index actions across a state-changing step. Do not expect native cmux extensions such as `get_window_state`, tokens,`perform_actions`, cursor controls, diagnostics, recordings, or browser/CDP inthis profile. Their absence is required for Codex schema parity. ### Claude/native cmux profile Perceive, act in logical groups, then verify: 1. `get_window_state` (pid + window_id) returns the accessibility tree **and** a screenshot. Ground on both. Prefer element addressing.2. Act by element: `click` with `element_token` (or `element_index` + pid + window_id) is the robust path. Pixel addressing (`x`,`y`) is the fallback.3. For a stable, already-snapshotted control set, call `perform_actions` once with the ordered `click` / `type_text` / `press_key` / other input steps. This reuses the existing element-token cache and visible cursor inside the persistent proxy instead of paying one model/MCP round trip and AX scan per click. Do not put navigation, modal-opening, or layout-changing actions before later control references in the same group; re-snapshot immediately after any action that can invalidate those controls.4. Verify the completed group by re-snapshotting and reading the element `value` / screenshot — do not assume actions landed (clicks are never helper-verified). Operate on-screen controls with visible pointer clicks by default — the gliding branded cursor is the product experience. Use `type_text` for entering text into text fields, and fall back to a pure keyboard sequence only when the user explicitly prefers speed over visibility. Notes:- In the native profile, use `list_apps` / `launch_app` / `list_windows` to find targets; `get_window_state` needs a `window_id` from `list_windows`.- Catalyst apps (e.g. Calculator) can expose an empty AX tree briefly after launch and return spurious AX error codes (-25204) even when the action landed — re-snapshot and check the result rather than trusting the code.- Pixel input is obstruction-checked: if another window covers the target point cmux-cua refuses with `background_occluded` naming the occluder instead of clicking the wrong window. Retry with `delivery_mode:"foreground"` or front the target. ## The branded agent cursor The agent's pointer shows as the cmux logo gradient (`#12c7f5 → #2d8cff →#6c5cff`) with a `cmux` label, so it is visually distinct from the user'scursor. It is configured by env the wrapper injects(`CMUX_CUA_CURSOR_GRADIENT` / `_BLOOM` / `_LABEL`) and is auto-active whilethe helper daemon is driving. It remains visible across normal reasoning gapsand is removed when the driving session ends or the proxy control connectioncloses. Each later action reasserts the cursor directly above the driventarget. If no cursor appears during an action, confirm the MCP config uses thehelper socket, has a stable `CMUX_CUA_DEFAULT_SESSION`, and uses the pinned cmux-cua build. ## Finding and focusing the driving session While an agent is driving, the **cmux Computer Use** menu-bar item projects onlythe most recently active live agent session and offers two presentation modes: - **Focus Computer Use** — bring forward the app the agent is driving and resume automatically following new targets.- **Focus Calling Terminal** — return to the terminal that invoked Computer Use and reveal the exact workspace + surface running that agent while automation continues in the background. The helper pins its cursor window directly above the driven target window atthe normal application window level. That keeps the cursor visible on thetarget while allowing any app the user places in front of that target to coverthe cursor naturally; presentation mode never promotes it to an always-on-toplayer. The active target and session ordering come from cmux-cua's per-session statefiles under `~/Library/Application Support/cmux/cmux-cua/runtime/<scope>/state/`. The item hides when there is no live or recent session. Toggle visibility inSettings → Computer Use. ## Troubleshooting - **Agent has no computer-use tools** — Settings → Computer Use must be on; start a *new* session (tools attach at launch).- **Clicks do nothing / not permitted** — grant Accessibility to cmux Computer Use.- **Black/empty screenshots** — grant Screen Recording to cmux Computer Use; restart only the helper if its automatic refresh has not completed yet.- **No menu-bar icon** — needs a live/recent session; check the visibility toggle.- **Prompts name the main cmux app** — a non-cmux fallback executed the helper directly. Stop there and report the failure; the bundled path must use the tag-scoped socket with `CMUX_CUA_MCP_FORCE_PROXY=1`.- **Prompts name CmuxCua** — a stale `/Applications/CmuxCua.app` daemon or a standalone helper launch is active. Stop it and reset/remove its TCC entry; the bundled path never uses that identity. ## Development - Engine source: `manaflow-ai/cmux-cua` (`libs/cmux-cua/rust`). cmux consumes it via `CMUX_CUA_PINNED_SHA` in `scripts/build-cmux-cua.sh`, which builds, lipos, and codesigns the binary plus the nested helper into the app bundle.- The helper daemon's `CMUX_CUA_EXTERNAL_PERMISSION_FLOW=1` prevents agent-supplied `check_permissions {prompt:true}` from bypassing cmux onboarding. The wrappers set `CMUX_CUA_MCP_FORCE_PROXY=1` and preserve the external-flow contract in the proxy process so protected calls wait for cmux's post-verification readiness signal. There is no ambient executable override. Both profiles resolve the app-bundled `cmux-cua` executable and the tag-scoped helper only; missing ownership fails closed instead of running a user-supplied executable.- If the cmux-owned daemon is unavailable, do **not** invoke `cmux-cua` directly through Bash and do not start its default socket. Tell the user to open Settings → Computer Use or restart the tagged cmux build, then retry the MCP tool after the helper runtime is healthy.- Never hand-edit `docs/.../cmux-cua/mcp-tools.mdx` in the fork — it is generated from the Rust tool descriptions.- cmux-side UX lives in `Sources/App/ComputerUse*.swift`, `Packages/macOS/CmuxSettingsUI/.../Sections/ComputerUseSection.swift`, and the two wrappers under `Resources/bin/`.
Discovery context
Discovered by repository scan. No exact path reference found in the snapshot’s root CLAUDE.md.