Getting background operation into someone else's Mac agent
On 8 September 2026 I opened pull request #65 against open-computer-use, an open-source computer-use server for Mac, Windows and Linux. I maintain mac-cua, which is built on the same idea: drive apps through accessibility and screenshots without taking the person’s keyboard, focus or pointer.
The project had part of that already. Another contributor had added sky_click, a click delivered to a window in the background, on 23 July. Typing was the gap. Keys posted to a background Chromium or Electron window arrived and no text appeared. A covered window also dropped its page from the accessibility tree, so the agent could click in the background and then read nothing back.
Since then I have opened 44 pull requests there. 20 are merged, 3 were closed and 21 are open. On 14 November I was made a maintainer of the project.
| Opened | Pull requests | Merged | Closed | Open |
|---|---|---|---|---|
| 8 to 22 Sep | 6 | 3 | 3 | 0 |
| 22 to 23 Oct | 18 | 6 | 0 | 12 |
| 8 Nov | 20 | 11 | 0 | 9 |
One pull request, 4,054 lines
#65 carried three things. As merged it added 4,054 lines across 51 files.
Background keys. Chromium inserts text only while its NSWindow is the key window, and a window in a background app is never key. key_method=sky_key posts an activation record to the target process, then a pair of key-window records, then the ordinary postToPid keys, then a record that undoes the state. The frontmost app does not change. Command chords like cmd+a are pressed through the target’s own menu bar with AXPress, because AppKit dispatches menu shortcuts only for the active app. The code is in SkyKeyboardSimulation.swift and SkyLightSPI.swift, and the reference note lists what I tried that did not work.
Covered windows. An app learns that its window is covered from window-server notifications, and Chromium, Electron and WebKit respond by hiding the page. WindowOcclusionKeepAlive.swift turns those notifications off for one window after a snapshot, while it is still visible.
Windows that were already hidden. For those, AgentDisplay.swift creates a 1920 by 1080 virtual display and moves the window onto it.
I measured all of it on one machine, an M5 on macOS 27. A fully covered Chrome window took 50 of 50 background keys with the frontmost app unchanged. On that machine the keys went through with no settle delay before or after them, so my commits set the two sky_key delays to 0. The numbers are in the benchmark report.
Review arrived as commits
There are no inline review comments on #65. The review is in the commit list. On 9 September, the day after I opened it, the maintainer pushed a commit onto my branch. He had rerun the benchmark on macOS 26.6.2.
| macOS | sky_key delay before and after the keys |
Keys delivered |
|---|---|---|
| 27.0 | 0 ms | 50 of 50 |
| 26.6.2 | 0 ms | 18 of 20, then 49 of 50 |
| 26.6.2 | 10 ms each | 50 of 50 |
His commit made 10 ms the default on both sides:
static let keyWindowFallbackSettleDefaultMilliseconds = 10.0
static let releaseSettleDefaultMilliseconds = 10.0
It also restored the gaps for ordinary input that my timing commit had set to 0, 20 ms between type_text chunks and 100 ms after press_key. His note in the repository says a zero-delay result verified on one system version should not become the default for ordinary input. The same commit changed cleanup. The occlusion state put back is the one the API returned when it was turned off, and a parked window’s record is kept until the restore is confirmed.
Then the pull request sat. I asked on 11 September. On 23 September it conflicted with main and I offered to split it into three: the key path, window visibility and Spaces, and the agent display. On 24 September the maintainer pushed a second commit instead. It tied the background state to a session. The end of a turn, a stdio shutdown and a closed connection now all release the occlusion pins and return parked windows. restore returns every window parked for that app, where mine had used the window the snapshot happened to select. get_app_state lost its read-only annotation, because it now changes window-server state. The virtual display shim catches exceptions from the private classes and reports itself unavailable.
On 2 October he wrote that he had been busy. #65 merged on 10 October, 32 days after I opened it. On 30 October he went one step further in #133: the virtual display is off in the configuration by default, and while it is off the parameter is not in the tool schema at all. I had made it opt-in per call.
What did not land
Three of my September pull requests were closed.
#75 added a js tool so a model could compose several actions in one call, running on JavaScriptCore. On 22 September the maintainer merged his own persistent JavaScript REPL on Node, and I closed mine as superseded the same day. I then opened #79 against his REPL to surface thrown errors. He found an edge case in it within three hours: a finished call that throws later from a timer could reject the next call. I confirmed it, found that the same bug let one call’s output land in another’s, fixed both and added the regression test he asked for. It merged the same day.
#72 switched screenshots to JPEG and removed the byte cap. He closed it on 10 October without a comment. The same day he added screenshot settings to main, with PNG as the default and JPEG as an option.
#74 ran an app’s App Intents. The maintainer spent several days on it and pushed three commits of his own. His conclusion was that it needed a generated shortcut the user has to confirm adding, which was too much setup, and he closed it on 16 October.
Two follow-ups to the agent display, #114 and #128, have been open since 22 October.
Password fields
On 8 November I opened 20 pull requests, each for one fault. #164 was the one that mattered most. Nothing in the kit checked for a password field before reading AXValue. AppKit’s NSSecureTextField answers AXValue with an empty string whatever it holds. With a field holding hunter2, type_text "X" took the empty string for the field’s text, spliced X into it and wrote the result back. The password was replaced by X.
My patch put a subrole check in front of each single AXValue read. The maintainer’s follow-up, written into the pull request the next day, points at what that left. The batched scan still requested AXValue for every node and dropped it afterwards for a password field, and AXSelectedText was not covered. His commit took the value out of the batch and widened the guard, which now reads:
if (attribute == kAXValueAttribute as String || attribute == kAXSelectedTextAttribute as String),
isSecureTextField(subrole: attributeValue(of: element, attribute: kAXSubroleAttribute) as? String) {
return nil
}
His test fixture offers the password in clear and counts the reads, and the count has to be zero. Typing into a password field now goes through the keyboard path, so the field ends up holding hunter2X.
#178 was the Linux half. GTK 4’s GtkPasswordEntry answered with the plaintext, and it went into the tree line the model reads. My patch withheld it for the AT-SPI role password text. The maintainer added Windows to the same pull request, keyed on UI Automation’s IsPassword, and made an unknown status withhold the value. Both merged on 9 November. The rules are written down in password-fields.md.
The limits are plain. All of this depends on the app labelling the control, so a GTK 4 GtkEntry with visibility off is still read. Screenshots are not redacted. sky_key rests on private window-server calls, and it refuses an app hidden with cmd+h.
What I would do differently
Run the benchmark on the oldest macOS I can reach before choosing a default. The dropped keys on 26.6.2 were found by the maintainer, a day after I had written zero into the code.
Send #65 as three pull requests from the start. I offered the split 15 days in. The one-fault pull requests of 8 November were merged, 11 of them, the next day.
Put a privacy check before every read, the batched ones included. My password patch still asked for the value in one path and threw it away.