The gap
of runs where GPT-6 Astra gave the exact count.
Across adversarial questions on real photos, three runs each.
See the leaderboard ↗Visual counting on real photos
Tallyglass-bench
Real photos. Questions with precise conditions. Human counts backed by instance annotations. We test integer-answer questions against a frontier model three times, then check the answers with a blind recount. Open a task and inspect the photo, masks and saved runs yourself.
Built to measure models from
Tallyglass-bench Leaderboard
Preview: sample scoresPlaceholder scores until full runs finish. They show how the table will read, not real results.
How scoring works
One rule: the exact integer, or it's wrong.
- =
Exact match
A run is correct only if its integer equals the verified answer. Off by one is wrong. No partial credit.
- ×3
Three runs
Every model answers every question three times. The score is the share of correct runs across all of them, so lucky guesses and unstable answers both show.
- %
Reading the table
Each bar is one model's exact-match accuracy over all runs. The tag after the name is the reasoning setting it ran with. Higher is better.
- !
The filter model
Questions were kept only when the filter model got them wrong, so its score is low by construction. It's reported apart, never ranked against the rest.
A count shows what
a model actually saw.
How we build it ↗
What we measure
Why counting tests what a model sees
A count is short to ask and quick to check. Getting it exactly right means looking at the whole image.
- 01
Every object matters
Miss one or add one and the answer is wrong. A count catches skipped corners, small objects and clutter that a caption can gloss over.
- 02
Conditions force a check
Color, position, purpose or an expert term narrows what counts. The model has to test each candidate against every condition.
- 03
Zero is a real answer
Some answers are 0 while near matches sit in the frame. A model that guesses from context will count things that aren't there.
Inside one real counting task
The circles identify seven parts of this example. Select a label to highlight its matching number.
How many vases with plants are not reflected in the pond? Return an integer.
Answer1
Blind recount1
Model runs2 ✗2 ✗2 ✗
Counted: the large pot in the round basin, which has no reflection. Left out: the small pot at the far end, which shows up in the water.
- 1
Photo
One real scene, used as taken. No edits, no generated images.
- 2
Question
Plain English, ending in "Return an integer." Each one looks at a different part of the scene.
- 3
Answer
The accepted count. Here, the writer and blind reviewer both counted 1.
- 4
Blind recount
A second person counts without seeing the first number. A mismatch sends the question back.
- 5
Marked objects
The colored fill follows the object's original segmentation mask. Circle 5 identifies the saved point inside it. The notes explain the near-match that was excluded.
- 6
Model runs
Three answers from the filter model. The question stays only if most are wrong.
- 7
Counted and left out
Which objects made the count, which near matches didn't, and why.
From a real photo to a testable answer
Follow the courtyard question through the counting workflow. The photo, instance mask, model responses and human recount are all from the saved task.
- 01Capture a real scene
- 02Define what counts
- 03Count and annotate
- 04Test three times
- 05Recount blind
- 06Inspect the files
01 · Real photos
Everyday scenes, left unedited
Streets, shop windows, patios, parking lots. We pick scenes where the target sits right next to things that look like it.
02 · Question definition
Make every condition explicit
A plant pot can be visible and still be excluded. This question counts only pots that do not appear as reflections in the pond.
Moorish courtyard / P2
How many vases with plants are not reflected in the pond? Return an integer.
- Target
- Vases with plants
- Condition
- Not reflected in the pond
- Output
- One integer
03 · Human count and instance annotation
Mark the object. Keep the evidence.
The writer establishes the count and selects the matching objects. Each instance is saved with its mask, bounding box and point. Here, one pot qualifies. The crosshair uses the original saved point.
filled silhouette = original instance mask
04 · Adversarial filter
Kept only if the model misses
A frontier model answers each integer question three times. Candidates stay when the model misses most runs. For the courtyard question, it answered 2, 2 and 2 against a count of 1. Below are the saved runs for all five examples.
These are construction runs. The set was selected against this model, so its results are reported separately from independent evaluations.
Where every run landed
05 · Independent human check
A second count, made blind
A second person counts without seeing the writer's answer. The courtyard question was counted as 1 by both people. Disagreements go back for review before the answer is accepted.
- Writer counts 1
- One instance annotated
- Reviewer counts blind
- Both counts match
The question
How many vases with plants are not reflected in the pond? Return an integer.
Writer1
Blind reviewer1
Accepted answer1
The saved review records the writer's 1, the reviewer's independent 1, and a match.
When counts disagree: in Casino / P2, the writer counted 2 and the blind reviewer counted 1. After checking a close-up, the reviewer acknowledged 2. The accepted answer is 2.
Five example tasks
Five tasks, explained question by question
For each question: the answer, the filter model's three runs, why it's hard, and what was counted or left out.
How the model did
Takeaway
The data
The result: a task you can inspect
Step 06 keeps the evidence together. Open the courtyard questions, original mask JSON or saved model runs. The instance IDs connect each counted object to its question.
- COCO RLEMasks use COCO run-length encoding, so standard tooling reads them.
- 1 : 1Every mask links to the question it answers. Count the masks and you get the answer.
- 0 masksNull questions have an answer of 0 and ship with no masks.
Two main question families
Conditional counting checks explicit conditions. Reasoning counting first interprets a relationship, purpose or category. The examples below show both.
For investors
Frontier models still can't count. We measure it, and we build the data that fixes it.
We're raising our first pre-seed round to turn Tallyglass-bench into the counting evaluation and training data that AI labs rely on.
Why it matters
Counting sits under real work
Inventory and shelf audits, logistics, insurance claims, robotics, lab and medical imaging. When a model can't say how many, it can't be trusted with any of them.
What we deliver today
A benchmark and the data behind it
- Exact-match scores for any model, three runs per question
- Task files with verified answers and a mask per counted object
- Failure reports: overcounts, undercounts, question types
What the round funds
Want the deck?
Reply to whoever shared this page with you and we'll send it the same day.
FAQ
What counts as a correct answer?
The exact integer, nothing else. Off by one is wrong. There's no credit for being close.
Why three runs per question?
Models don't answer the same way every time. Three runs show whether a correct answer is stable or lucky, and the score counts all of them.
What is the filter model?
It's the frontier model we test questions against while building the set. Each question goes to it three times and stays only if it misses most runs. That pushes its score down, so it's reported apart.
Doesn't that bias the set?
Yes, toward being hard for that one model, which is why its score is set apart. No other model was used during building, so the set wasn't tuned against them. Reviewers also cut questions that turn a count into arithmetic or allow two readings.
How does the blind recount work?
A second person counts every question without seeing the writer's number. If the counts match, the answer is confirmed. If not, both look again and the question is fixed or rewritten.
Why are some answers 0?
Those are Null questions. The photo always contains partial matches, like scalloped pots and aloe plants that never share a pot. A model that checks says 0.
Are the photos real?
Yes. Every photo is a real scene, used as taken. No edits, no composites, no generated images.
What's in the JSON?
The public examples include exported questions and answers and saved model runs. Where the original mask file is available, it contains COCO RLE instance masks, bounding boxes, click points and links to each question. The courtyard files above show the full data.
Can you count better than a frontier model?
Try the five tasks with answers hidden. Then check yourself against the blind recount and the model runs.