First-hand information for everyone
A new streaming video assistant, AURA 2.0, can start conversations on its own, ask for extra information by calling tools, and point out parts of the video that it talks about. These three upgrades let the model “listen” to a live feed, decide when something worth discussing appears, and act without being prompted by a user.
The team trained the system in three phases. In the first phase the model learned general video‑questioning skills and how to use external tools when a user asks. The second phase added the new abilities: letting the model pick up new topics (Model‑Initiated Conversation) and find things the user highlighted (Interactive Visual Grounding). The final phase fine‑tuned everything together with data that mixed all three skills, using a special loss trick that helps the model decide whether to speak.
When tested on the IPIBench benchmark, AURA 2.0 led all open‑source streaming models in two key areas. Its “Proactive Monitoring” score was 49.30 %, higher than every competitor, and its “Proactive Task Management” average reached 24.10 %, again topping the rest. (The benchmark tests whether a model can detect and react to events in a video stream that require action.)
There are limits. When the model calls an external tool, the information that the tool returns is added to its internal context, but the model does not say it out loud in the audio output. Users will have to listen for other clues or read the tool’s reply on a screen.
These advances show how large‑language models can move beyond answering questions to proactively engaging users and pointing out relevant moments in a video. However, the system still does not voice tool results, so the experience is partly text‑based until further improvements are made.