WGO-Bench results, run on 2026-11-16

Nua Sapphire on WGO-Bench

Sapphire splits egocentric and robot video into steps, timing each one and captioning what happened. On Macrodata’s WGO-Bench it scores 55.9 step timing F1, the highest of 12 systems (next: NVIDIA Flora, 50.6). When the captions must also be right it scores 34.9, the second highest, behind GPT-6 Astra’s 39.0. Every Sapphire and Flora output, score and judge verdict behind these numbers is public, and this page shows how to check them.

Model
Average F1 over WGO’s three subsets, the way NVIDIA’s Flora paper reports WGO. Click a column to re-rank. Nua Sapphire and NVIDIA Flora are our runs, with every output public. † As reported in NVIDIA’s Flora paper (Table 2); not run by us.
WGO-Bench, average F1 over three subsets
ModelStep timing F1Steps + captions F1
Nua Sapphire55.934.9
NVIDIA Flora50.631.2
GPT-6 Astra (reported in the Flora paper)46.039.0
GPT-5.6 Sol (reported in the Flora paper)33.724.6
GPT-5.6 Luna (reported in the Flora paper)30.219.9
Gemini 3.8 Flash (reported in the Flora paper)26.421.7
Kimi K3 (reported in the Flora paper)26.116.7
Gemini 3.1 Pro (reported in the Flora paper)25.717.3
Claude Opus 5 (reported in the Flora paper)25.515.4
Qwen3.5-27B (reported in the Flora paper)22.413.7
Claude Sonnet 5 (reported in the Flora paper)20.811.0
Qwen3.5-397B-A17B (reported in the Flora paper)19.511.2

Every subset, both metrics

WGO mixes robot and human footage. The headline averages these three subsets with equal weight; here they are one by one, first Sapphire against Flora (both our runs), then every system. Flora is ahead in HomER steps + captions (15.0 vs 14.3); Sapphire leads Flora everywhere else.

Step timing F1
Steps + captions F1
Galaxea (robot)
25 clips · 123 reference steps
Step timing F1
  • Nua Sapphire79.7
  • NVIDIA Flora71.8

Nua Sapphire ahead by 7.9

Steps + captions F1
  • Nua Sapphire54.2
  • NVIDIA Flora51.0

Nua Sapphire ahead by 3.2

HomER (human, first-person)
25 clips · 470 reference steps
Step timing F1
  • Nua Sapphire26.5
  • NVIDIA Flora24.6

Nua Sapphire ahead by 1.9

Steps + captions F1
  • Nua Sapphire14.3
  • NVIDIA Flora15.0

NVIDIA Flora ahead by 0.7

DROID-derived (robot)
50 clips · 150 reference steps
Step timing F1
  • Nua Sapphire61.5
  • NVIDIA Flora55.4

Nua Sapphire ahead by 6.1

Steps + captions F1
  • Nua Sapphire36.0
  • NVIDIA Flora27.7

Nua Sapphire ahead by 8.3

All 12 systems

Sorted by step timing average; the best score in each column is bold. † As reported in NVIDIA’s Flora paper (Table 2).

ModelStep timing F1Steps + captions F1
GalaxeaHomERDROID-derivedAvgGalaxeaHomERDROID-derivedAvg
Nua Sapphire79.726.561.555.954.214.336.034.9
NVIDIA Flora71.824.655.450.651.015.027.731.2
GPT-6 Astra†47.225.265.546.043.119.354.539.0
GPT-5.6 Sol†33.523.544.033.723.416.933.624.6
GPT-5.6 Luna†37.820.632.330.224.013.322.319.9
Gemini 3.8 Flash†24.523.930.826.420.920.623.721.7
Kimi K3†31.118.928.226.119.612.717.916.7
Gemini 3.1 Pro†38.112.027.025.729.47.514.917.3
Claude Opus 5†27.210.638.825.518.77.420.115.4
Qwen3.5-27B†25.712.728.822.420.05.415.813.7
Claude Sonnet 5†29.63.229.520.819.42.610.911.0
Qwen3.5-397B-A17B†19.410.528.719.511.58.214.011.2
Show every count, precision and recall
SubsetMetricModelMatchedPredictedReferencePrecisionRecallF1
Galaxea (robot)Step timing F1Nua Sapphire9411312383.276.479.7
NVIDIA Flora9313612368.475.671.8
Galaxea (robot)Steps + captions F1Nua Sapphire6411312356.652.054.2
NVIDIA Flora6613612348.553.751.0
HomER (human, first-person)Step timing F1Nua Sapphire11136947030.123.626.5
NVIDIA Flora13361247021.728.324.6
HomER (human, first-person)Steps + captions F1Nua Sapphire6036947016.312.814.3
NVIDIA Flora8161247013.217.215.0
DROID-derived (robot)Step timing F1Nua Sapphire8713315065.458.061.5
NVIDIA Flora8013915057.653.355.4
DROID-derived (robot)Steps + captions F1Nua Sapphire5113315038.334.036.0
NVIDIA Flora4013915028.826.727.7
All 100 clips, pooledStep timing F1Nua Sapphire29261574347.539.343.0
NVIDIA Flora30688774334.541.237.5
All 100 clips, pooledStep timing F1, edges not snappedNua Sapphire23261574337.731.234.2
NVIDIA Flora27788774331.237.334.0
All 100 clips, pooledSteps + captions F1Nua Sapphire17561574328.523.625.8
NVIDIA Flora18788774321.125.222.9

Precision is matched ÷ predicted steps, recall is matched ÷ reference steps, and F1 is their harmonic mean. Pooled rows score all 100 clips as one set, so the large HomER subset counts for more.

How the numbers are made

From the benchmark itself to the final average, in the order the scoring happens.

The benchmark

WGO-Bench is a public benchmark from Macrodata for splitting video into steps. Its clips cover robots seen from a head camera (Galaxea), people seen first-person (HomER) and robots seen from outside (DROID-derived).

We scored against one pinned revision of it. We don’t redistribute the videos or the reference labels; get them from Hugging Face under their licence.

Clips
100
Reference steps
743
Subsets
25 Galaxea, 25 HomER, 50 DROID-derived
Revision
1c86a62
File
data/annotations.parquet
SHA-256
24c6c23447753385b9cea0d17f17d7a8aa74131d800d9bc9ab74d7669d7404ab
Licence
CC BY-NC-SA 4.0

What counts as a step match

We follow the benchmark authors’ grading definition and their reference implementation at commit 5f76982.

A predicted step matches a reference step when their overlap in time, measured as intersection over union (IoU), is at least 0.75. Each step can match at most once. Captions are ignored for this metric.

Reference
2–9s
Predicted
3–10s
Overlap
6s
Union
8s
0s2s4s6s8s10s12s
IoU = overlap ÷ union = 6s ÷ 8s = 0.75. That is exactly the threshold, so this pair counts as a timing match. Shift the prediction one second later (4–11s) and it would not: 5s ÷ 9s ≈ 0.56.

Snapping the outer edges

Before scoring, the first predicted start and the last predicted end are snapped to the reference’s outer boundaries, as the protocol specifies. Nothing else moves.

Scores without the snap are in the full counts table above.

Reference
As predicted
Scored
0s2s4s6s8s10s12s
Only the outermost edges move: the first predicted start goes to the first reference start and the last predicted end goes to the last reference end (lighter shading). Every boundary in between is scored exactly as predicted.

How captions are judged

For steps + captions F1, a timing match must also pass a semantic check: the judge decides whether the predicted and reference captions describe the same thing. Different wording is allowed.

Every verdict is published, and the clip explorer below marks each one. An LLM judge varies slightly between runs, typically by under one point.

Judge
gemini-3.5-flash
Rubric
Official WGO rubric
Checks
Same event, object and end state
Verdicts
judgments/

Averaging across subsets

Both scores are computed per subset, then averaged across the three subsets with equal weight, the way NVIDIA’s Flora paper reports WGO.

HomER holds 470 of the 743 reference steps, so pooling every step instead would weight it most heavily. Both views are shown here.

Subset averagePooled
Nua Sapphire, timing55.943.0
NVIDIA Flora, timing50.637.5
Nua Sapphire, captions34.925.8
NVIDIA Flora, captions31.222.9

How each model was run

Nua Sapphire: run by Nua on the 100 benchmark videos on 2026-11-16.

NVIDIA Flora: the publicly released model, run by Nua on the same 100 videos.

Neither model saw the reference steps. Both models’ outputs are published exactly as each produced them.

The other models

Numbers for the other 10 models are as reported by NVIDIA. They were run under the same matching protocol: outer edges snapped, one-to-one matching at IoU ≥ 0.75, a Gemini 3.5 Flash judge with the public rubric, and equal-weight subset averages.

Their outputs are not in our bucket, so they can’t be checked here the way Sapphire and Flora can. For Flora we show our own run; the difference from NVIDIA’s figure is in judging, and every verdict behind ours is in judgments/flora.json.

Source
FLORA: An Open Model Suite for Scaling Robotics Data Engines
Table
Table 2
Run by
NVIDIA, not rerun by us
Flora, as NVIDIA reports
50.3 timing, 34.5 captions
Flora, our run
50.6 timing, 31.2 captions

Every clip, step by step

Pick a clip to see both models’ steps on one timeline. Each step that cleared the timing threshold shows the judge’s verdict on its caption. Hover or tap a step to find it in the list.

Loading 100 clips from the bucket…

Check our work in four steps

Everything runs from a terminal. Steps 1 and 3 involve no judge, so they should reproduce our files and timing scores.

  1. 1

    Check the files

    Download this version’s folder and confirm every checksum.

    Terminal
    gsutil -m cp -r gs://nua-sapphire-benchmarks/wgo/v2026-11-16 .cd v2026-11-16shasum -a 256 -c SHA256SUMS

    Every file should report OK.

  2. 2

    Get WGO-Bench at the pinned revision

    Download the reference labels from Hugging Face at the revision we scored against, then confirm the file’s checksum.

    Terminal
    hf download macrodata/WGO-Bench data/annotations.parquet \  --repo-type dataset \  --revision 1c86a62e53029ff85cea3a238bc58af3a40055b6 \  --local-dir wgo-benchshasum -a 256 wgo-bench/data/annotations.parquet

    The checksum should be 24c6c23447753385b9cea0d17f17d7a8aa74131d800d9bc9ab74d7669d7404ab.

  3. 3

    Rescore step timing

    Check out Macrodata’s reference implementation at the pinned commit and score outputs/*.jsonl against the references.

    Terminal
    git clone https://github.com/macrodata-labs/refinercd refinergit checkout 5f7698228074f0c1163cdb830185c20dfecb3f41

    This part has no judge, so it should reproduce our step timing scores in scores.json.

  4. 4

    Re-judge or audit the captions

    Re-run the semantic judge on the matched pairs, or audit our verdicts in judgments/ line by line. An LLM judge varies slightly between runs, typically by under one point.

    Terminal
    # Count judged pairs and accepted captionsjq '[.decisions[][]] | {judged_pairs: length, accepted: map(select(.)) | length}' \  judgments/sapphire.json

    For Sapphire this prints 292 judged pairs and 175 accepted, its timing and joint match counts in scores.json.

Files

Version v2026-11-16. Versions are never overwritten; a new run gets a new folder.

  • README.md
    Methodology and verification steps
    sha256 15df57a93dfa965d05e00cbcc818d7c1609b5e8719f8a22e66ad989f9f06b2ec
  • manifest.json
    Versions, sources and file checksums
    sha256 c36c45a6c763d66950eead707fe267c762c46f1b460a7b33c72db2610cd1b1a5
  • SHA256SUMS
    Checksums for every file in the folder
  • scores.json
    Subset-average, pooled and per-subset scores
    sha256 c7c2c83c40dcc486910ff50db33b644ee75bb24bb03e8c128ecaf33f441bb75a
  • outputs/sapphire.jsonl
    Nua Sapphire’s steps, one line per clip
    sha256 975832698eb0693e8728531de2949123935c9e48985e9ec3abc3d9bd60cd7b83
  • outputs/flora.jsonl
    NVIDIA Flora’s steps, one line per clip
    sha256 8c23fdbb7219dbb1e71437861766f49fb0ea8ce49c5ccf5b4824d7afda827350
  • judgments/sapphire.json
    Judge verdicts on Sapphire’s timing matches
    sha256 cd1ead1a24a4cc1e6f40882c1e89d5a9d633fdfb850e4a7a6e6fe413d4c02b87
  • judgments/flora.json
    Judge verdicts on Flora’s timing matches
    sha256 851f0dbbb5bf50d732186f1475fe86e21d6347474fcf05a2323456c5095acdbf

Questions, or a different result? Contact us through platform.usenua.com.

Pricing

$1.50 per video-hour

Run Sapphire on your own first-person or robot video and get timed, captioned steps back. For large volumes or custom terms, talk to us.