A new system called ControlAnything lets a mixed‑reality headset connect to a physical device only when the wearer says a command. The headset shows an outline of the target device, the user speaks a command, and an AI agent writes a tiny program on the wearer’s laptop. That program tests the connection with the real device using hand‑written transports such as MIDI, Bluetooth, or keyboard keystrokes, then returns either AR controls that live on the headset or a hand‑mapping program that runs every frame.

In a test with a single wearer and a single laptop, nine of ten requests reached a “ready” state. The median time from command to ready was 32.5 seconds, ranging from 24.9 to 57.8 seconds. Most of that time (about 90 %) came from calling the AI model; the system cached most input tokens, speeding subsequent calls.

The evaluation examined three transport methods. For example, MIDI sent commands to an OP‑1 synthesizer, Bluetooth to a gimbal, and keystrokes to a web page. All three transports were exercised, and the system could bind the headset’s hand motions to the correct device action.

However, the study used only one wearer, one laptop, and non‑independent requests. Consequently, the authors did not test safety, scalability, or user experience, and the system’s ability to handle devices with no transport remains unknown.

These results show that AI‑generated connectors can be written on‑the‑fly, but further work is needed before the approach can be broadly applied.