Codesk Control: Text-First macOS Desktop Control for Low-Latency AI Agents

Introduction

AI agents that operate desktop software need a reliable observation-action loop. The most universal loop is visual: capture the screen, infer state from pixels, choose coordinates, click or type, and repeat. This mirrors a human remote-desktop workflow, but it makes a machine agent pay for perception even when the operating system can already expose the relevant state as text. It also encourages coordinate coupling: an action that depends on a pixel location can fail when a window moves, a button shifts, or the display scale changes.

Codesk Control explores a different default. On macOS, the front application, focused window, focused accessibility element, visible labels, menus, and keyboard input path are available through native frameworks such as Accessibility and Core Graphics. The central hypothesis is simple: for many desktop tasks, an agent should inspect text and accessibility state first, move through keyboard shortcuts second, use accessibility actions third, and use screenshots only as a fallback.

This paper presents Codesk Control, a command-line and MCP-based control layer for macOS. It makes desktop state available as JSON, provides app-aware shortcut aliases, supports paste and typing, selects menus, presses accessibility elements, and captures screenshots only when needed. The implementation is intentionally CLI-first so it can be used both directly by engineers and indirectly by tool-using agents.

Background

Desktop control bottlenecks

Screenshot-first desktop automation has three important costs. First, each step must capture or retrieve an image. Second, an agent or vision model must interpret the image before action. Third, the result is often spatial rather than semantic: a click target is a coordinate, not a named button or field. For highly graphical applications, this generality is valuable. For common tasks such as focusing a browser address bar, opening a file, saving a document, selecting a menu item, or reading the current window title, it is often wasteful.

AppleScript and osascript provide another path, but each shell invocation incurs process startup and Apple Event overhead. AppleScript is also an awkward fit for fast iterative control when the desired primitive is ``tell me the front app and visible labels, then press a shortcut.''

macOS primitives

Codesk Control uses three families of macOS primitives. The Accessibility API exposes top-level application accessibility objects, attributes, focused elements, and actions such as AXPress. Apple documents AXUIElementCreateApplication as creating the top-level accessibility object for an application process ID, and related APIs expose attributes and actions. Core Graphics exposes low-level Quartz events; Apple documents CGEvent as representing low-level hardware events and provides a keyboard-event initializer for virtual key codes and key-up/key-down state. Launch Services and AppKit are used for opening paths and activating applications.

MCP integration

The Model Context Protocol provides a structured way for host applications and models to discover and call external tools. Codesk Control exposes desktop-control operations as MCP tools such as codesk_state, codesk_quick, codesk_paste, codesk_press, and codesk_menu. The MCP server uses JSON-RPC over standard input and output, matching MCP's tool-listing and tool-call model.

Design

Codesk Control follows a four-step control ladder:

1. Inspect with text and accessibility state.
2. Move with application activation, file/URL opening, paste, and keyboard shortcuts.
3. Act on named accessibility targets such as buttons and menus.
4. Fall back to screenshots when state is visual or inaccessible.

The command-line interface is deliberately small:

codesk state –json codesk text codesk app Safari codesk open https://example.com codesk q address codesk key enter codesk paste "query text" codesk press Save codesk menu "File > Save" codesk screenshot codesk mcp

The corresponding MCP tools preserve the same vocabulary. A host can call codesk_state to retrieve front app, bundle identifier, process ID, focused role, focused values, selected text, window title, permission state, and visible text lines. The host can then call codesk_quick with an alias such as address in a browser, or codesk_menu with a menu path such as File > Save.

Implementation

The native CLI is written in Swift. It uses AppKit for application activation and workspace operations, Core Graphics for keyboard event posting, Accessibility APIs for reading and acting on UI elements, and standard macOS command-line utilities only where appropriate, such as screencapture for screenshot fallback.

The first plugin implementation used a dependency-free Node.js MCP server that translated MCP calls into subprocess invocations of the Swift CLI. This made integration simple, but every MCP tool call paid process-spawn overhead. The current implementation adds codesk mcp, a persistent native Swift MCP server. The Codex plugin now launches a small shell wrapper that resolves the repository path and executes:

.build/release/codesk mcp

The native server reads newline-delimited JSON-RPC messages from standard input, handles initialize, tools/list, and tools/call, and executes the same internal Swift controllers used by the CLI. The result is that a repeated codesk_state MCP call no longer spawns another codesk process.

Evaluation

Method

Benchmarks were run on a live macOS desktop running Codex, Computer Use MCP processes, Webtool MCP processes, a Telecodex bridge, and Chrome remote-debugging processes. Each benchmark used three warmup iterations and ten measured iterations. The benchmark harness records both timing and behavioral coverage summaries. It does not claim full semantic accuracy against a manually labeled ground-truth corpus; instead, it reports whether each path returned structured state, text, or pixels.

Two benchmark runs are central:

• Pre-change: Codesk 0.1.0, Node MCP wrapper spawning the CLI per MCP call.
• Post-change: Codesk 0.2.0, persistent native Swift MCP server.

Latency results

Table shows median latency before and after the native MCP implementation.

The major change is isolated to the MCP path. The CLI state path, AppleScript probes, and screenshot probe remain in the same timing range. This indicates that the speedup comes from removing repeated process spawning from the plugin path rather than from changes in the underlying accessibility traversal.

Post-change comparison

Table compares post-change state-call latency against representative legacy probes.

Behavioral coverage

Codesk returns structured UI state. In the benchmark environment, codesk_state_json and codesk_mcp_state returned the front app, bundle identifier, process ID presence, window-title presence, accessibility trust status, and visible text count. The AppleScript front-app probe returned only one text field. The AppleScript window-title probe exited successfully but returned no text in these specific runs. The screenshot probe produced a PNG but no structured text. Thus, Codesk provides richer structured observations for this scenario. A future evaluation should measure end-to-end task success against labeled workflows rather than treating structured coverage as semantic accuracy.

Use Cases

Browser navigation.

An agent can activate Safari or Chrome, focus the address bar with codesk_quick(address), paste a URL, press enter, and then verify title or visible text with codesk_state. This avoids screenshot interpretation for common browsing actions.

Editor and IDE control.

An agent can open Visual Studio Code, invoke quick-open or the command palette with named aliases, paste a file path or command, and confirm visible state through accessibility text.

Document workflows.

For save, export, print, or menu-driven operations, codesk_menu can select stable menu paths while codesk_press can activate labeled buttons in dialogs.

Desktop state monitoring.

Because codesk_mcp_state is fast after the native MCP change, a host can poll state opportunistically without paying the latency of repeated screenshots or subprocess launches.

Human-in-the-loop automation.

Text-first state is easier for a human to audit than a coordinate-click trace. A log entry such as ``pressed AXButton title=Save'' is more interpretable than ``clicked x=844, y=613.''

Implications

Text-first control shifts desktop agents toward semantic interaction. The result is faster in the measured path, more inspectable in logs, and less dependent on display geometry. It also changes the safety model. A semantic action such as codesk_menu("File > Save") is easier to review, deny, or replay than an opaque pixel action.

However, richer control also creates risk. Keyboard events and accessibility actions can affect arbitrary applications once permissions are granted. A Codesk-style tool should therefore be paired with permission prompts, explicit user consent for destructive actions, audit logs, allowlists for high-risk shortcuts, and clear fallback behavior. MCP tools should expose narrow, typed operations rather than raw shell command execution whenever possible.

Limitations

Codesk Control depends on macOS Accessibility permissions and on applications exposing useful accessibility attributes. Some canvas-heavy, game-like, remote, or custom-rendered interfaces may not provide meaningful text state. The benchmark in this paper is a live-system microbenchmark, not a broad corpus of desktop tasks. It measures latency and structured coverage, but not full task-completion accuracy. The screenshot path is measured only as capture time, not as capture plus model vision inference, so it understates the full end-to-end cost of visual control.

Future Work

Future work should add a ground-truth task suite with labeled expected state and success criteria; persistent caching for stable accessibility subtrees; per-application shortcut profiles distributed as data; a policy layer for dangerous actions; and comparative studies across macOS applications, display configurations, and permissions states. A valuable next benchmark would measure complete workflows, such as opening a URL, finding text, saving a file, or dismissing a dialog, with success rate, latency, and number of observations required.

Conclusion

Codesk Control demonstrates that desktop agents can be made faster and more structured by preferring text state and keyboard semantics over visual coordinate loops. The native MCP implementation reduced repeated state-call latency from 34.69 ms to 1.37 ms in a live Codex desktop environment while preserving the same structured observation output. The broader lesson is not that screenshots are unnecessary, but that they should be the fallback rather than the default when the operating system can already describe the interface.