WGO-Bench results, run on 2026-11-16
Nua Sapphire on WGO-Bench
Sapphire splits egocentric and robot video into steps, timing each one and captioning what happened. On Macrodata’s WGO-Bench it scores 55.9 step timing F1, the highest of 12 systems (next: NVIDIA Flora, 50.6). When the captions must also be right it scores 34.9, the second highest, behind GPT-6 Astra’s 39.0. Every Sapphire and Flora output, score and judge verdict behind these numbers is public, and this page shows how to check them.
| Model | Step timing F1 | Steps + captions F1 |
|---|---|---|
| Nua Sapphire | 55.9 | 34.9 |
| NVIDIA Flora | 50.6 | 31.2 |
| GPT-6 Astra (reported in the Flora paper) | 46.0 | 39.0 |
| GPT-5.6 Sol (reported in the Flora paper) | 33.7 | 24.6 |
| GPT-5.6 Luna (reported in the Flora paper) | 30.2 | 19.9 |
| Gemini 3.8 Flash (reported in the Flora paper) | 26.4 | 21.7 |
| Kimi K3 (reported in the Flora paper) | 26.1 | 16.7 |
| Gemini 3.1 Pro (reported in the Flora paper) | 25.7 | 17.3 |
| Claude Opus 5 (reported in the Flora paper) | 25.5 | 15.4 |
| Qwen3.5-27B (reported in the Flora paper) | 22.4 | 13.7 |
| Claude Sonnet 5 (reported in the Flora paper) | 20.8 | 11.0 |
| Qwen3.5-397B-A17B (reported in the Flora paper) | 19.5 | 11.2 |
Every subset, both metrics
WGO mixes robot and human footage. The headline averages these three subsets with equal weight; here they are one by one, first Sapphire against Flora (both our runs), then every system. Flora is ahead in HomER steps + captions (15.0 vs 14.3); Sapphire leads Flora everywhere else.
- Nua Sapphire79.7
- NVIDIA Flora71.8
Nua Sapphire ahead by 7.9
- Nua Sapphire54.2
- NVIDIA Flora51.0
Nua Sapphire ahead by 3.2
- Nua Sapphire26.5
- NVIDIA Flora24.6
Nua Sapphire ahead by 1.9
- Nua Sapphire14.3
- NVIDIA Flora15.0
NVIDIA Flora ahead by 0.7
- Nua Sapphire61.5
- NVIDIA Flora55.4
Nua Sapphire ahead by 6.1
- Nua Sapphire36.0
- NVIDIA Flora27.7
Nua Sapphire ahead by 8.3
All 12 systems
Sorted by step timing average; the best score in each column is bold. † As reported in NVIDIA’s Flora paper (Table 2).
| Model | Step timing F1 | Steps + captions F1 | ||||||
|---|---|---|---|---|---|---|---|---|
| Galaxea | HomER | DROID-derived | Avg | Galaxea | HomER | DROID-derived | Avg | |
| Nua Sapphire | 79.7 | 26.5 | 61.5 | 55.9 | 54.2 | 14.3 | 36.0 | 34.9 |
| NVIDIA Flora | 71.8 | 24.6 | 55.4 | 50.6 | 51.0 | 15.0 | 27.7 | 31.2 |
| GPT-6 Astra† | 47.2 | 25.2 | 65.5 | 46.0 | 43.1 | 19.3 | 54.5 | 39.0 |
| GPT-5.6 Sol† | 33.5 | 23.5 | 44.0 | 33.7 | 23.4 | 16.9 | 33.6 | 24.6 |
| GPT-5.6 Luna† | 37.8 | 20.6 | 32.3 | 30.2 | 24.0 | 13.3 | 22.3 | 19.9 |
| Gemini 3.8 Flash† | 24.5 | 23.9 | 30.8 | 26.4 | 20.9 | 20.6 | 23.7 | 21.7 |
| Kimi K3† | 31.1 | 18.9 | 28.2 | 26.1 | 19.6 | 12.7 | 17.9 | 16.7 |
| Gemini 3.1 Pro† | 38.1 | 12.0 | 27.0 | 25.7 | 29.4 | 7.5 | 14.9 | 17.3 |
| Claude Opus 5† | 27.2 | 10.6 | 38.8 | 25.5 | 18.7 | 7.4 | 20.1 | 15.4 |
| Qwen3.5-27B† | 25.7 | 12.7 | 28.8 | 22.4 | 20.0 | 5.4 | 15.8 | 13.7 |
| Claude Sonnet 5† | 29.6 | 3.2 | 29.5 | 20.8 | 19.4 | 2.6 | 10.9 | 11.0 |
| Qwen3.5-397B-A17B† | 19.4 | 10.5 | 28.7 | 19.5 | 11.5 | 8.2 | 14.0 | 11.2 |
Show every count, precision and recall
| Subset | Metric | Model | Matched | Predicted | Reference | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|---|
| Galaxea (robot) | Step timing F1 | Nua Sapphire | 94 | 113 | 123 | 83.2 | 76.4 | 79.7 |
| NVIDIA Flora | 93 | 136 | 123 | 68.4 | 75.6 | 71.8 | ||
| Galaxea (robot) | Steps + captions F1 | Nua Sapphire | 64 | 113 | 123 | 56.6 | 52.0 | 54.2 |
| NVIDIA Flora | 66 | 136 | 123 | 48.5 | 53.7 | 51.0 | ||
| HomER (human, first-person) | Step timing F1 | Nua Sapphire | 111 | 369 | 470 | 30.1 | 23.6 | 26.5 |
| NVIDIA Flora | 133 | 612 | 470 | 21.7 | 28.3 | 24.6 | ||
| HomER (human, first-person) | Steps + captions F1 | Nua Sapphire | 60 | 369 | 470 | 16.3 | 12.8 | 14.3 |
| NVIDIA Flora | 81 | 612 | 470 | 13.2 | 17.2 | 15.0 | ||
| DROID-derived (robot) | Step timing F1 | Nua Sapphire | 87 | 133 | 150 | 65.4 | 58.0 | 61.5 |
| NVIDIA Flora | 80 | 139 | 150 | 57.6 | 53.3 | 55.4 | ||
| DROID-derived (robot) | Steps + captions F1 | Nua Sapphire | 51 | 133 | 150 | 38.3 | 34.0 | 36.0 |
| NVIDIA Flora | 40 | 139 | 150 | 28.8 | 26.7 | 27.7 | ||
| All 100 clips, pooled | Step timing F1 | Nua Sapphire | 292 | 615 | 743 | 47.5 | 39.3 | 43.0 |
| NVIDIA Flora | 306 | 887 | 743 | 34.5 | 41.2 | 37.5 | ||
| All 100 clips, pooled | Step timing F1, edges not snapped | Nua Sapphire | 232 | 615 | 743 | 37.7 | 31.2 | 34.2 |
| NVIDIA Flora | 277 | 887 | 743 | 31.2 | 37.3 | 34.0 | ||
| All 100 clips, pooled | Steps + captions F1 | Nua Sapphire | 175 | 615 | 743 | 28.5 | 23.6 | 25.8 |
| NVIDIA Flora | 187 | 887 | 743 | 21.1 | 25.2 | 22.9 |
Precision is matched ÷ predicted steps, recall is matched ÷ reference steps, and F1 is their harmonic mean. Pooled rows score all 100 clips as one set, so the large HomER subset counts for more.
How the numbers are made
From the benchmark itself to the final average, in the order the scoring happens.
The benchmark
WGO-Bench is a public benchmark from Macrodata for splitting video into steps. Its clips cover robots seen from a head camera (Galaxea), people seen first-person (HomER) and robots seen from outside (DROID-derived).
We scored against one pinned revision of it. We don’t redistribute the videos or the reference labels; get them from Hugging Face under their licence.
- Clips
- 100
- Reference steps
- 743
- Subsets
- 25 Galaxea, 25 HomER, 50 DROID-derived
- Revision
- 1c86a62
- File
- data/annotations.parquet
- SHA-256
- 24c6c23447753385b9cea0d17f17d7a8aa74131d800d9bc9ab74d7669d7404ab
- Licence
- CC BY-NC-SA 4.0
What counts as a step match
We follow the benchmark authors’ grading definition and their reference implementation at commit 5f76982.
A predicted step matches a reference step when their overlap in time, measured as intersection over union (IoU), is at least 0.75. Each step can match at most once. Captions are ignored for this metric.
Snapping the outer edges
Before scoring, the first predicted start and the last predicted end are snapped to the reference’s outer boundaries, as the protocol specifies. Nothing else moves.
Scores without the snap are in the full counts table above.
How captions are judged
For steps + captions F1, a timing match must also pass a semantic check: the judge decides whether the predicted and reference captions describe the same thing. Different wording is allowed.
Every verdict is published, and the clip explorer below marks each one. An LLM judge varies slightly between runs, typically by under one point.
- Judge
- gemini-3.5-flash
- Rubric
- Official WGO rubric
- Checks
- Same event, object and end state
- Verdicts
- judgments/
Averaging across subsets
Both scores are computed per subset, then averaged across the three subsets with equal weight, the way NVIDIA’s Flora paper reports WGO.
HomER holds 470 of the 743 reference steps, so pooling every step instead would weight it most heavily. Both views are shown here.
| Subset average | Pooled | |
|---|---|---|
| Nua Sapphire, timing | 55.9 | 43.0 |
| NVIDIA Flora, timing | 50.6 | 37.5 |
| Nua Sapphire, captions | 34.9 | 25.8 |
| NVIDIA Flora, captions | 31.2 | 22.9 |
How each model was run
Nua Sapphire: run by Nua on the 100 benchmark videos on 2026-11-16.
NVIDIA Flora: the publicly released model, run by Nua on the same 100 videos.
Neither model saw the reference steps. Both models’ outputs are published exactly as each produced them.
The other models
Numbers for the other 10 models are as reported by NVIDIA. They were run under the same matching protocol: outer edges snapped, one-to-one matching at IoU ≥ 0.75, a Gemini 3.5 Flash judge with the public rubric, and equal-weight subset averages.
Their outputs are not in our bucket, so they can’t be checked here the way Sapphire and Flora can. For Flora we show our own run; the difference from NVIDIA’s figure is in judging, and every verdict behind ours is in judgments/flora.json.
- Source
- FLORA: An Open Model Suite for Scaling Robotics Data Engines
- Table
- Table 2
- Run by
- NVIDIA, not rerun by us
- Flora, as NVIDIA reports
- 50.3 timing, 34.5 captions
- Flora, our run
- 50.6 timing, 31.2 captions
Every clip, step by step
Pick a clip to see both models’ steps on one timeline. Each step that cleared the timing threshold shows the judge’s verdict on its caption. Hover or tap a step to find it in the list.
Loading 100 clips from the bucket…
Check our work in four steps
Everything runs from a terminal. Steps 1 and 3 involve no judge, so they should reproduce our files and timing scores.
- 1
Check the files
Download this version’s folder and confirm every checksum.
Terminalgsutil -m cp -r gs://nua-sapphire-benchmarks/wgo/v2026-11-16 .cd v2026-11-16shasum -a 256 -c SHA256SUMSEvery file should report
OK. - 2
Get WGO-Bench at the pinned revision
Download the reference labels from Hugging Face at the revision we scored against, then confirm the file’s checksum.
Terminalhf download macrodata/WGO-Bench data/annotations.parquet \ --repo-type dataset \ --revision 1c86a62e53029ff85cea3a238bc58af3a40055b6 \ --local-dir wgo-benchshasum -a 256 wgo-bench/data/annotations.parquetThe checksum should be
24c6c23447753385b9cea0d17f17d7a8aa74131d800d9bc9ab74d7669d7404ab. - 3
Rescore step timing
Check out Macrodata’s reference implementation at the pinned commit and score
outputs/*.jsonlagainst the references.Terminalgit clone https://github.com/macrodata-labs/refinercd refinergit checkout 5f7698228074f0c1163cdb830185c20dfecb3f41This part has no judge, so it should reproduce our step timing scores in
scores.json. - 4
Re-judge or audit the captions
Re-run the semantic judge on the matched pairs, or audit our verdicts in
judgments/line by line. An LLM judge varies slightly between runs, typically by under one point.Terminal# Count judged pairs and accepted captionsjq '[.decisions[][]] | {judged_pairs: length, accepted: map(select(.)) | length}' \ judgments/sapphire.jsonFor Sapphire this prints 292 judged pairs and 175 accepted, its timing and joint match counts in
scores.json.
Files
Version v2026-11-16. Versions are never overwritten; a new run gets a new folder.
- README.mdMethodology and verification stepssha256 15df57a93dfa965d05e00cbcc818d7c1609b5e8719f8a22e66ad989f9f06b2ec
- manifest.jsonVersions, sources and file checksumssha256 c36c45a6c763d66950eead707fe267c762c46f1b460a7b33c72db2610cd1b1a5
- SHA256SUMSChecksums for every file in the folder
- scores.jsonSubset-average, pooled and per-subset scoressha256 c7c2c83c40dcc486910ff50db33b644ee75bb24bb03e8c128ecaf33f441bb75a
- outputs/sapphire.jsonlNua Sapphire’s steps, one line per clipsha256 975832698eb0693e8728531de2949123935c9e48985e9ec3abc3d9bd60cd7b83
- outputs/flora.jsonlNVIDIA Flora’s steps, one line per clipsha256 8c23fdbb7219dbb1e71437861766f49fb0ea8ce49c5ccf5b4824d7afda827350
- judgments/sapphire.jsonJudge verdicts on Sapphire’s timing matchessha256 cd1ead1a24a4cc1e6f40882c1e89d5a9d633fdfb850e4a7a6e6fe413d4c02b87
- judgments/flora.jsonJudge verdicts on Flora’s timing matchessha256 851f0dbbb5bf50d732186f1475fe86e21d6347474fcf05a2323456c5095acdbf
Questions, or a different result? Contact us through platform.usenua.com.
$1.50 per video-hour
Run Sapphire on your own first-person or robot video and get timed, captioned steps back. For large volumes or custom terms, talk to us.