MACOS.md

macOS backend, in depth

Everything specific to reading the macOS Accessibility ("AX") tree lives in canopy/backends/macos.py. This document explains the API, the permission model, and the handful of macOS realities that trip everyone up the first time — each of which is handled in the code, but worth understanding when something goes wrong.

The AX API in one paragraph

macOS exposes every app's UI as a tree of AXUIElement handles. You start from an application element (AXUIElementCreateApplication(pid)), read attributes off an element (AXUIElementCopyAttributeValue), and follow the AXChildren attribute downward. Attributes are strings like AXRole, AXTitle, AXValue, AXPosition. Some elements advertise actions (AXPress) you can perform — that is the road to v2 invocation. Canopy reaches all of this through pyobjc, the Python↔Objective-C bridge.

Permission (TCC) — the thing everyone gets stuck on {#permission}

Reading another app's AX tree is gated by macOS's privacy system (TCC). The calling process must be granted System Settings → Privacy & Security → Accessibility.

The trap: the grant attaches to the responsible process — the actual binary that launched your Python — not to the .py file and not to the python executable in the abstract. Concretely:

• Run canopy from Terminal → you must authorize Terminal.
• Run it from iTerm → authorize iTerm.
• Run it from an IDE or an agent runner → authorize that app.

AXIsProcessTrusted() returns whether the current responsible process is trusted. canopy diagnose calls it; canopy diagnose --prompt calls AXIsProcessTrustedWithOptions({kAXTrustedCheckOptionPrompt: True}), which makes macOS pop the dialog that deep-links to the right settings pane.

Two things people miss even after granting:

1. Relaunch the host. After you toggle the permission on, fully quit and reopen the terminal/IDE so the new grant takes effect for new processes.
2. It's the host, not Canopy. You will never see "canopy" or "python" in the Accessibility list per se — you grant the app that runs them.

If AXIsProcessTrusted() is false, Canopy refuses the capture early with the exact remediation steps rather than returning an empty tree.

The (error, value) out-parameter convention

This is the single most common pyobjc stumbling block. The C function

AXError AXUIElementCopyAttributeValue(AXUIElementRef element,
                                      CFStringRef attribute,
                                      CFTypeRef *value);   // out-parameter

becomes, in pyobjc, a function you call with None for the out-parameter and which returns a tuple:

err, value = AXUIElementCopyAttributeValue(element, "AXRole", None)
if err != 0:
    value = None

Every read in the backend goes through _AXVisitor._copy_attr, which applies exactly this pattern, so the convention is in one place. err == 0 (kAXErrorSuccess) means success; anything else (attribute unsupported, no value, cannot-complete/timeout, API-disabled) is treated as "no value".

Bounding time: the messaging timeout

Because every read can block on a busy app, the backend calls

AXUIElementSetMessagingTimeout(ax_app, timeout_seconds)

on the application element right after creating it. This bounds every subsequent message to that app's element subtree. --timeout controls it (default 2.0s). A wedged app therefore costs you at most the timeout per stuck read, never an indefinite hang.

Read-only v1 uses only synchronous attribute reads, so no CFRunLoop is needed. (AX notifications/observers would require a run loop; Canopy's watch is poll-based — re-capture + diff — precisely to avoid that complexity.)

Electron and Chromium apps look empty until you ask nicely

A large fraction of modern Mac apps (VS Code, Slack, Discord, anything Electron; Chrome itself) ship with accessibility off by default for performance. Their AX tree comes back nearly empty — you'll see a window and almost nothing inside.

The fix is to set a special attribute on the application element to opt the app into exposing its tree:

AXUIElementSetAttributeValue(ax_app, "AXManualAccessibility", True)

Canopy does this whenever enhance=True (the default; disable with --no-enhance). It is best-effort — native apps ignore it, and some apps reject the set — so failures are swallowed into a warning, never fatal. (Historically Chrome also responded to AXEnhancedUserInterface, but that attribute has had side effects like VoiceOver-related window resizing, so Canopy prefers the modern AXManualAccessibility.)

If an Electron app is still sparse, give it a moment after enabling — some apps build their accessibility tree lazily on first request.

Retina: points are not pixels

When --geometry is on, AXPosition and AXSize come back as AXValue wrappers around CGPoint/CGSize, which Canopy unwraps via AXValueGetValue. These are in points, not pixels. On a Retina (2×) display, 1 point = 2 physical pixels, so geometry is not a pixel coordinate you can hand to a raw screenshot grid. This mismatch is one reason geometry is off by default and why v2 invocation will prefer native AXPress over synthesizing clicks at coordinates.

One more wrinkle the unwrap code guards: the AXValue type constants were renamed across SDK versions (kAXValueCGPointType → kAXValueTypeCGPoint). The backend imports whichever exists.

Virtualized lists and tables

A long AXList/AXTable/AXOutline can advertise thousands of rows through AXChildren, most of them not even on screen. Pulling them all is slow and pointless. For these roles the visitor prefers AXVisibleChildren, falling back to AXChildren when it is unavailable. This is the macOS analogue of pruning virtualized content — the biggest single speedup on row-heavy UIs.

Accessible name, and a known simplification

Canopy computes a node's name as the first non-empty of AXTitle then AXDescription. For static text, the displayed string is in AXValue, so it is carried in value and (for adjacent runs) coalesced.

Known simplification: this does not yet follow AXTitleUIElement, the attribute that links a control to a separate label element elsewhere in the tree. So a field whose visible label is a sibling AXStaticText may show an empty name in v1. The hook to improve this is localized to make_node.

Redaction

If an element's AXSubrole is AXSecureTextField (a password field), its value is never serialized — it renders = <redacted> and the capture's redaction count is incremented. This is a hard rule in make_node, not a configurable option. More broadly: a capture pulls real on-screen text into a model's context, so treat the output as sensitive, and remember that on-screen text is a prompt-injection surface exactly as web page text is.

Role mapping

The backend maps native AX roles (refined by subrole) onto Canopy's normalized Role. Unrecognized roles map to unknown — never dropped, because an unknown control is still a control. Selected mappings:

AX role	Normalized	Notes
AXApplication	application
AXWindow	window
AXSheet	dialog	modal sheet
AXGroup, AXDrawer	group	candidates for single-child collapse
AXToolbar	toolbar
AXMenuBar / AXMenuBarItem	menubar / menuitem
AXMenu / AXMenuItem	menu / menuitem
AXButton, AXMenuButton	button
AXPopUpButton, AXComboBox	combobox
AXStaticText	text	value carries the string
AXTextField	textfield
AXTextArea	textarea
AXCheckBox / AXRadioButton	checkbox / radio	AXValue 0/1 → checked/unchecked state
AXList	list	uses AXVisibleChildren
AXTable, AXOutline	table	uses AXVisibleChildren
AXRow / AXColumn / AXCell	row / column / cell
AXTabGroup	tabgroup
AXImage	image	decorative empties pruned
AXLink	link
AXSlider, AXIncrementor	slider
AXProgressIndicator	progress
AXScrollArea / AXScrollBar	scrollarea / scrollbar
AXSplitGroup	splitter

Subrole refinements: AXSecureTextField/AXSearchField → textfield; AXCloseButton/AXMinimizeButton/AXZoomButton/AXFullScreenButton/ AXToolbarButton → button.

Window selection

--window chooses the root element:

• auto (default) — AXFocusedWindow, else AXMainWindow, else the first of AXWindows, else the whole app element.
• focused / main — that specific window (with the same fallbacks).
• all — the application element itself, so the capture includes every window and the menu bar. Use this to read menus; expect a larger tree.

Troubleshooting

Symptom	Likely cause	Fix
canopy diagnose says permission NOT granted	The responsible (host) process isn't trusted	Authorize the terminal/IDE that runs Canopy, then relaunch it. canopy diagnose --prompt opens the dialog.
Capture returns just a window with little inside	Electron/Chromium app with a11y off	Ensure --no-enhance is not set; give the app a moment; re-capture.
pyobjc is not importable	macOS extra not installed	pip install 'canopy-a11y[macos]' (or the two pyobjc-framework-* packages).
Capture is very slow or TRUNCATED	Huge/virtualized tree, or a busy app	Lower --max-nodes/--depth, scope with --window focused, lower --timeout. Row-heavy views already use AXVisibleChildren.
A labeled field shows an empty name	Label is a separate element via AXTitleUIElement (not yet followed)	Known v1 simplification; the value/identifier still address it.
Geometry numbers don't match screenshot pixels	Retina points vs pixels	Expected; points ≠ pixels on 2× displays. Avoid coordinate-based actions.
Nothing happens / cannot-complete errors	App is not responding to AX messages in time	Increase --timeout; confirm the app is foregrounded and responsive.
An admin/elevated tool isn't fully readable	Cross-privilege AX restrictions	Run from a host with appropriate privileges.

When in doubt, run canopy diagnose first: it pinpoints whether the problem is platform, dependency, permission, or the end-to-end smoke read.