OmniAgent — ICML 2026 Booth Talk

ICML 2026 · Seoul · Qwen Booth #B400 · Jul 8, 13:20 KST

OmniAgent

Native Active Perception as Reasoning for Omni-Modal Understanding

7B > 72B

beats a 10× larger passive model on LVBench

−73%

frames ingested per video — 203 vs 768

10/10

all 10 benchmarks improve — up to 7× on temporal grounding

Zhenghao Xing · CUHK Ph.D. Candidate · Multimodal Agent & Audio-Visual Reasoning

Zhenghao Xing* · Ruiyang Xu* · Yuxuan Wang* · Jinzheng He · Ziyang Ma · Qize Yang · Yunfei Chu · Jin Xu† · Junyang Lin · Chi-Wing Fu · Pheng-Ann Heng†

Qwen · CUHK · SJTU · NTU · *equal contribution · †corresponding author · built on Qwen2.5-Omni-7B

Motivation

Passive models watch everything. Cost grows with the video.

“Why are the vlogger family members so happy at 22:03?”

2,448 s vlog · audio: yes

Passivewatch it all

0:0040:48

768 uniform frame probes — dense across the whole video (illustrative)

What the query needsquery-driven

0:00▲ 22:0340:48

4 targeted observations — frames + audio around 22:03

today: pay for video length wanted: pay for question difficulty

OmniAgent · ICML 2026

Our idea

See. Hear. Reason. Act. Inside one native model.

OmniAgent — a native omni-modal agent for active perception, built on Qwen2.5-Omni-7B. One model that sees, hears, and decides what to look at next — no external tools, no pipeline.

see

Request frames on demand

Broad scans locate candidates; dense scans refine boundaries.

hear

Switch to audio when needed

Audio confirms what frames alone can't.

reason

Remember notes, not pixels

Raw media is deleted each turn; short notes persist.

act

Decide what to do next

Frames, audio, clip, or answer — stopping is a decision too.

not this

external captioner / ASR toolchain · exhaustive caption-everything pre-scan

this

native active perception — train the looking, not just the answering

result

small + active > big + passive — exact margins in the results

OmniAgent · ICML 2026

Method — active perception as a POMDP

Look, Think, Act. Repeat.

Persistent memory

Query + short notes only

Starts as the question plus video metadata. Grows one note per turn — never raw media.

write a note

⟵

read every turn

⟶

OmniAgent — one native model

Observationwhat did I just see / hear?

Thoughtwhat is still missing?

Actionget_frames · get_audio · get_clip · answer

action

⟶

transient percept — purged after the turn

⟵

Video environment Ω

A dumb media extractor

Returns the requested frames, audio, or A/V clip. No captioner, no ASR, no retrieval — all perception happens inside the model.

purge — raw media deleted each turn; only notes persist

cost — grows with the reasoning, not the video length

stop — the answer action ends the loop once evidence suffices

Worked example · temporal grounding

Scan. Verify. Refine.

real demo trajectory

Find all time ranges where “Linda Grimbeck discusses tourism security measures in an interview.”

Video: 260.10 s · audio + visual news clip

Task: temporal grounding, no explanation in final answer

First action: get_frames(0.0, 260.1, 50)

Opening frame from the full-video scan — not the evidence

First candidate interview segment with Linda Grimbeck — candidate A

Second candidate interview segment with Linda Grimbeck — candidate B

End frame from the full-video scan — end card

1 · quick glance — 50 frames, the model's own choice (not a 768-frame pre-scan)

0s 260s A · 22-76s B · 175-226s

Two promising islands. Both need verifying — and their edges are still fuzzy.

2 · switch to audio

smart cameras
intelligence network
tourism victim support helpline
tourism monitors

Frames found the candidates. Audio confirms the topic: tourism security.

3 · refine cuts

range A

Frame before the first Linda Grimbeck segment

Frame inside the first Linda Grimbeck segment

Frame after the first Linda Grimbeck segment

range B

Frame before the second Linda Grimbeck segment

Frame inside the second Linda Grimbeck segment

Frame after the second Linda Grimbeck segment

Worked example · OTA trace

Only the text memory persists

Transient percepts

Used once, then deleted

50-frame global scan

finds candidates, not the answer

purge

Audio on candidate windows

verifies the security topic

purge

Dense boundary frames

before / inside / after the cuts

purge

Persistent text memory

What remains reads like a lab notebook

01 Candidate A around 22-76 s; candidate B around 175-226 s.

02 Audio says smart cameras, intelligence network, victim support helpline, tourism monitors.

03 Both candidates are Linda Grimbeck discussing tourism security.

04 Remaining work: refine start/end boundaries, then answer.

answer action · turn 9

[22.00, 76.00]

[175.04, 225.96]

OmniAgent · ICML 2026

Method — trajectory supervision

Train the looking, not the answering

ordinary video-QA SFT

Supervise the answer

sampled video tokens

→

answer label

More answer labels just reinforced the passive habit — the score dropped.

LVBench

43.0 → 41.6

agentic SFT

Supervise the trajectory

Observation

→

Thought

→

Action

Best-of-N exploration with self-correction; a path is kept only if the answer is right and the reasoning matches what was seen.

LVBench

43.0 → 48.7

58K

accepted trajectories

2 filters

right answer + supported reasoning

learns when

to scan, listen, zoom in, and stop

OmniAgent · ICML 2026

Method — turn-level credit

Not every turn deserves the same credit

Problem — advantage homogenization

GRPO pays every turn the same

Every turn gets the same credit — the turn that found the evidence earns no more than filler.

79.2%

of decision-point turns show above-average entropy — uncertainty marks the forks

Fix — entropy-rescaled turn credit

Weight each turn by its uncertainty

Pay each turn by its uncertainty. Discovery earns more; routine earns less; confident mistakes are punished harder.

discovery turn

Illustrative — routine turns get down-weighted; the turn that finds the evidence earns amplified credit.

TAURA — Turn-aware Adaptive Uncertainty Rescaled Advantage

Â_i,k = A_itrajectory advantage · w_i,kturn-aware weight

w_i,k = H_i,k 1N_G ^GΣ_j=1 ^K_jΣ_m=1 H_j,m ← this turn's entropy
← mean entropy over the group

Normalized so E[w_i,k] = 1; credit shifts to high-entropy discovery turns without changing gradient scale.

Every stage earns its keep — LVBench: SFT 48.7 → GRPO 49.8 → TAURA 50.5 · DailyOmni: GRPO dips to 62.2, TAURA 64.8

Headline result — LVBench · hour-plus videos

Small model. Sharper eyes.

50.5%

OmniAgent-7Bactive perception

47.3%

Qwen2.5-VL-72Bpassive, 10× the parameters

frames per video: 203 vs 768 — ~73% fewer

OmniAgent (active) agentic baselines passive models

Results — benchmark sweep

Broad gains. Strongest on search-heavy tasks.

10/10

all paper benchmarks improve over Qwen2.5-Omni-7B

Paper Tables 1-3: video understanding, audio-visual understanding, and temporal grounding. VUE-TR reports Vision+Audio / Vision.

Video understanding Table 1

VideoMME overall / long

64.8 → 67.8
54.8 → 59.6

+3.0 / +4.8

VSI-Bench reasoning · avg 1.6 min

35.5 → 48.4

+12.9

MLVU long · 3-120 min

65.2 → 71.1

+5.9

Minerva reasoning · 2-90 min

33.4 → 41.4

+8.0

LVBench long · avg 68 min

43.0 → 50.5

+7.5

Audio-visual understanding Table 2

DailyOmni generic · avg 0.7 min

60.1 → 64.8

+4.7

WorldSense generic · avg 2.4 min

45.4 → 47.2

+1.8

OmniVideoBench reasoning · avg 6.4 min

29.3 → 37.1

+7.8

Temporal grounding Table 3 · IoU

LongVALE avg 3.9 min

5.7 → 39.1

+33.4

VUE-TR V+A / vision

3.5 → 36.5
8.0 → 46.1

+33.0 / +38.1

OmniAgent · ICML 2026

Results — temporal grounding

Finding the moment is a search problem.

LongVALE · IoU — vs. base model

5.7

→

39.1

+33.4 absolute — a near-7× jump

scan

find candidate windows

verify

use audio or vision evidence

refine

tighten start and end cuts

VUE-TR (vision + audio) · IoU — frontier comparison

OmniAgent-7B36.5

GPT-4o11.1

Gemini-2.5-Pro12.1

Closed-source generalists see the video; OmniAgent also learns where to look next.

OmniAgent · ICML 2026

Analysis — test-time scaling

It thinks more only when it has to

+6.2%

VideoMME-Long accuracy, as the max turn budget K grows 6 → 52

~11.7

turns actually used even at K = 52 — it answers once the evidence is enough

16.9 → 5.7

turns per video-hour as videos grow 20–40 → 120–140 min (8.5 → 12.5 turns) — compute follows the question, not the clock

accuracy — VideoMME-Long turns actually used

OmniAgent · ICML 2026

Release — web demo

The same loop, visible in the browser

inspectable interface

Choose or uploadBuilt-in examples or your own video.
Watch the loopObservation → Thought → Action, every turn.
Inspect evidenceFrames, audio, exported trajectory JSON.

Fully open-source — one launch script and a single A100 80GB gets you this exact interface.

Recorded walkthrough: the active-perception loop exposed as a web interface for real inspection.

Takeaways

Three things to remember

Perception can be reasoning

It turns hours of video into a few notes — cost tracks the question, not the video.

Train the looking, not just the answering

Passive SFT made things worse (43.0 → 41.6). Agentic SFT + TAURA fix that.

Active beats bigger

7B > 72B on LVBench, ~73% fewer frames — and it keeps improving the more it thinks (+6.2%).

One honest caveat — the sequential loop adds latency vs. a single forward pass; parallelized exploration is next.

github.com/HarryHsing/OmniAgent

Released — code · env · RL-7B / SFT-7B · demo · Apache-2.0
Paper — arXiv:2606.19341 · Booth — Jul 8 · 13:20 · Qwen #B400
Poster — Wed Jul 8 · 2:30–4:15 PM KST · Hall A #1200
“Native Active Perception as Reasoning for Omni-Modal Understanding”

GitHub

Thread