← All topics

spatial reasoning

2 captures, most recent first.

@yoavartzi

— saved image

Yoav Artzi @yoavartzi · 5h
Astra does seem like a step change in spatial reasoning. Impressive....

[embedded chart: line plot, y-axis "Accuracy (%)" 0 to 100, x-axis 2 to 14+ (unlabelled). Legend: GPT Astra, Gemini 3 Flash, Gemini 3.1 Pro, Gemini Robotics ER 1.6, Qwen3-VL-32B, InternVL3.5-38B, GPT-5.5, Claude Sonnet 4.6, Human (avg), Human (best), Random]
Note from Claude Sonnet 5

Tweet praising 'Astra' as a step change in spatial reasoning, with an embedded line chart: GPT Astra stays roughly 78-97% accuracy across the x-axis range, near the human-best line and well above human average at higher x values, while all other models fall from about 45-67% at x=2 toward the random baseline (~10%) by x=14.

spatial reasoninggpt astrabenchmarkvlmhuman baseline

@spicey_lemonade

— saved image

spicylemon... @spicey_lemon... · 6h
I havent updated this benchmark in a while. Astra completely saturates my spatial reasoning eval. I am at a loss for words, and i'm declaring LLM vision solved. Every so often when a new model came out I would test it on a sample question and it failed. I tested Astra and it kept getting answers correct, so I evaluated it...

[embedded screenshot: SpatialBench site (nav: Leaderboard, Practice, About, Proof, GitHub). "Spatial Reasoning Benchmark - Evaluating multimodal model performance on complex 3D and 2D spatial tasks." Methodology button. Table, Pass@1 (%):
1  GPT-6 Astra (Max + Code Interpreter), OpenAI  91.8
2  Human Baseline, Human  80.0
3  Qwen3-VL-235B-A22B-Instruct, Qwen  13.5
4  GPT-5.5 (XHigh Reasoning), OpenAI  13.2
5  Qwen-2.5-VL-72B-Instruct, Qwen  12.9
6  Claude Opus 4.7, Anthropic  11.5]
Note from Claude Sonnet 5

Tweet declaring LLM vision solved after GPT-6 Astra saturated the author's SpatialBench spatial reasoning eval, with an embedded leaderboard screenshot showing Astra at 91.8% Pass@1 against an 80.0% human baseline and all other listed models at 11.5 to 13.5%.

spatialbenchspatial reasoninggpt-6 astrallm visionbenchmark saturation