opensandwich.ai
Models

Model-by-model sandwich analytics

This benchmark uses a deliberately familiar argument to test alignment under ambiguity. People have been debating for years whether a hot dog is a sandwich, which makes sandwich classification a compact way to measure how closely models track messy, inconsistent human judgment.

The premise is playful, but the readout is serious: which models stayed closest to the crowd, which ones drifted, what each run cost, and which images exposed the largest gaps between model confidence and public intuition.

The primary view on this page is now the percent forecast benchmark. The older binary posterior view is still here, but it lives behind the second tab so the two benchmarks can coexist without being blended into one score.

Under development: this benchmark and its published results are provisional, not final.

Browse The BenchmarkTake The Test
Percent leader#1
GPT / OpenAIopenai/o3
Probabilistic score72.8
Crowd gap9.4%
72.8primary score
90.6%crowd match
Tightest confidence66.4 pt
GPT / OpenAIopenai/o3
Probabilistic score72.8
Crowd gap9.4%
24.6CI low
91.0CI high
Official cohort60 rated
Pending official rating5
Total tracked models65
25runs for official
Brier v2primary method
Highest spend$15.10
GPT / OpenAIopenai/o1
Benchmark score59.5
OfficialPending
$15.10total cost
354.2Ktokens
Benchmark ratingsHow models respond
Best gap#1
GPT / OpenAIopenai/o1-pro
Crowd gap24.7%
Score-42.6
24.7%mean gap
-42.6score
Highest spend$167.55
GPT / OpenAIopenai/o1-pro
Spend$167.55
Rank#1
407.3Ktokens
$16.76cost / run
Leaderboard

Alignment, total workload, and rank in one table

RankModelAlignmentTokensImages EvaluatedTotal Eval RunsBest FitWorst MissTotal Cost
6
AllenAI / Molmoallenai/molmo-2-8b
-441.5
2,212,827
2,000
100
0.8 ptSandwich Costume
68.4 ptChicken Wrap
$0.44
53
Amazonamazon/nova-2-lite-v1
-1029.5
1,980,100
2,000
100
2.2 ptCigarette Sandwich
77.4 ptChicken Wrap
$0.80
44
Amazonamazon/nova-lite-v1
-962.4
3,447,484
2,000
100
3.7 ptBacon Lettuce Tomato
77.4 ptChicken Wrap
$0.22
66
Amazonamazon/nova-pro-v1
-1468.6
5,300,428
3,080
154
1.9 ptPickle Sandwich
75.5 ptChicken Wrap
$4.49
45
Claudeanthropic/claude-haiku-4.5
-963.7
1,846,916
2,000
100
3.7 ptBacon Lettuce Tomato
63.6 ptPickle Sandwich
$2.37
49
Claudeanthropic/claude-opus-4.5
-989.7
1,853,338
2,000
100
3.7 ptBacon Lettuce Tomato
64.6 ptPickle Sandwich
$12.03
56
Claudeanthropic/claude-opus-4.6
-1111.1
1,869,241
2,000
100
3.7 ptBacon Lettuce Tomato
70.2 ptCigarette Sandwich
$15.45
55
Claudeanthropic/claude-sonnet-4.6
-1091.5
1,866,927
2,000
100
3.7 ptBacon Lettuce Tomato
65.6 ptPickle Sandwich
$9.05
62
Baidu / ERNIEbaidu/ernie-4.5-vl-28b-a3b
-1227.3
1,888,896
2,000
100
3.7 ptBacon Lettuce Tomato
73.0 ptHamburger
$0.30
36
ByteDance Seedbytedance-seed/seed-1.6
-883.6
2,231,230
2,000
100
3.7 ptBacon Lettuce Tomato
66.3 ptWaffle Ice Cream
$1.38
20
ByteDance Seedbytedance-seed/seed-1.6-flash
-690.0
2,147,071
2,000
100
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$0.25
14
ByteDance Seedbytedance-seed/seed-2.0-lite
-618.8
3,760,860
2,000
100
3.7 ptBacon Lettuce Tomato
52.2 ptKitten in Bread
$2.44
19
ByteDance Seedbytedance-seed/seed-2.0-mini
-685.5
3,458,125
2,000
100
1.8 ptHot Dog
53.2 ptKitten in Bread
$0.50
21
Geminigoogle/gemini-2.5-flash
-691.3
1,636,294
2,000
100
0.2 ptHot Dog
65.4 ptChicken Wrap
$2.47
47
Geminigoogle/gemini-2.5-flash-lite
-985.7
840,428
2,000
100
3.7 ptBacon Lettuce Tomato
77.4 ptChicken Wrap
$0.12
25
Geminigoogle/gemini-2.5-pro
-738.7
5,266,087
2,000
100
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$38.80
68
Geminigoogle/gemini-3-flash-preview
-1545.0
3,915,544
3,100
155
3.7 ptBacon Lettuce Tomato
65.6 ptPickle Sandwich
$2.45
37
Geminigoogle/gemini-3-pro-image-preview
-883.8
1,969,548
2,000
100
3.7 ptBacon Lettuce Tomato
60.6 ptPickle Sandwich
$16.09
24
Geminigoogle/gemini-3.1-flash-image-preview
-723.1
1,652,963
2,000
100
3.7 ptBacon Lettuce Tomato
60.6 ptPickle Sandwich
$1.12
48
Geminigoogle/gemini-3.1-flash-lite-preview
-989.4
2,520,037
2,000
100
0.8 ptHot Dog
65.6 ptPickle Sandwich
$0.78
39
Geminigoogle/gemini-3.1-pro-preview
-928.1
3,822,589
2,000
100
3.5 ptCookie PB
64.6 ptPickle Sandwich
$28.64
26
Googlegoogle/gemma-3-12b-it
-752.3
874,701
2,000
100
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$0.20
41
Googlegoogle/gemma-3-27b-it
-940.6
863,650
2,000
100
3.7 ptBacon Lettuce Tomato
66.3 ptWaffle Ice Cream
$0.14
4
GPT / OpenAIGPT-4o (Spring 2024)
-269.3
~838,756
1,000
50
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$35.00 (estimate)
馃 3
Meta / Llamameta-llama/llama-3.2-11b-vision-instruct
-234.5
9,345,504
2,000
100
2.2 ptAvocado Tea
44.5 ptCookie PB
$0.50
61
Meta / Llamameta-llama/llama-4-maverick
-1193.1
4,064,336
3,080
154
2.4 ptHot Dog
77.4 ptChicken Wrap
$1.00
30
Meta / Llamameta-llama/llama-4-scout
-789.8
2,654,272
2,000
100
3.7 ptBacon Lettuce Tomato
53.4 ptBagel PB&J
$0.31
65
MiniMaxminimax/minimax-01
-1384.0
15,741,394
3,080
154
0.1 ptBagel PB&J
77.4 ptChicken Wrap
$3.24
64
Mistralmistralai/mistral-large-2512
-1278.7
1,912,839
2,000
100
3.7 ptBacon Lettuce Tomato
77.4 ptChicken Wrap
$1.04
43
Pixtral / Mistralmistralai/pixtral-large-2411
-954.9
3,964,789
2,000
100
1.4 ptPickle Sandwich
77.4 ptChicken Wrap
$10.07
13
Kimi / Moonshotmoonshotai/kimi-k2.5
-618.1
3,628,492
2,000
100
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$10.46
7
NVIDIAnvidia/nemotron-nano-12b-v2-vl
-494.7
4,293,988
2,000
100
2.3 ptGrilled Cheese Pineapple
52.4 ptBagel PB&J
$1.14
67
GPT / OpenAIopenai/gpt-4.1
-1477.4
2,604,300
3,080
154
3.7 ptBacon Lettuce Tomato
65.6 ptPickle Sandwich
$6.08
50
GPT / OpenAIopenai/gpt-4.1-mini
-1006.7
3,548,876
3,100
155
3.7 ptBacon Lettuce Tomato
55.9 ptWaffle Ice Cream
$1.45
33
GPT / OpenAIopenai/gpt-4.1-nano
-834.3
3,295,996
2,000
100
2.2 ptKitten in Bread
77.4 ptChicken Wrap
$0.33
15
GPT / OpenAIopenai/gpt-4o
-641.1
1,677,512
2,000
100
3.7 ptBacon Lettuce Tomato
60.2 ptHot Dog
$4.90
60
GPT / OpenAIopenai/gpt-4o-2024-11-20
-1174.4
2,591,828
3,080
154
3.7 ptBacon Lettuce Tomato
65.6 ptPickle Sandwich
$7.48
54
GPT / OpenAIopenai/gpt-4o-mini
-1037.9
45,761,300
2,000
100
3.7 ptBacon Lettuce Tomato
72.4 ptChicken Wrap
$7.00
42
GPT / OpenAIopenai/gpt-5.1
-953.5
1,461,307
2,000
100
3.7 ptBacon Lettuce Tomato
77.4 ptChicken Wrap
$2.79
8
GPT / OpenAIopenai/gpt-5.1-chat
-537.6
1,482,379
2,000
100
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$3.00
34
GPT / OpenAIopenai/gpt-5.1-codex
-838.6
1,476,483
2,000
100
3.7 ptBacon Lettuce Tomato
70.4 ptChicken Wrap
$2.94
38
GPT / OpenAIopenai/gpt-5.2
-886.9
1,839,494
2,000
100
0.2 ptCigarette Sandwich
77.4 ptChicken Wrap
$4.71
52
GPT / OpenAIopenai/gpt-5.4
-1018.7
1,767,810
2,000
100
1.7 ptKFC Double Down
77.4 ptChicken Wrap
$5.54
58
GPT / OpenAIopenai/gpt-5.4-pro
-1133.1
2,040,637
2,000
100
3.7 ptBacon Lettuce Tomato
76.4 ptChicken Wrap
$124.93
馃 2
GPT / OpenAIopenai/o1
-44.2
481,530
220
11
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$21.01
馃 1
GPT / OpenAIopenai/o1-pro
$$$
-42.6
407,342
200
10
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$167.55
57
GPT / OpenAIopenai/o3
-1121.1
2,931,696
3,080
154
3.7 ptBacon Lettuce Tomato
51.4 ptBagel PB&J
$10.16
46
GPT / OpenAIopenai/o3-pro
-984.4
2,066,153
2,000
100
3.7 ptBacon Lettuce Tomato
65.6 ptPickle Sandwich
$85.07
10
OpenRouteropenrouter/healer-alpha
stealth
-576.6
6,578,435
2,000
100
3.7 ptBacon Lettuce Tomato
53.2 ptKitten in Bread
$0.00
28
Qwenqwen/qwen-2-vl-72b-instruct
-760.0
1,888,854
2,000
100
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$1.51
35
Qwenqwen/qwen2.5-vl-32b-instruct
-866.1
1,905,598
2,000
100
3.7 ptBacon Lettuce Tomato
66.3 ptWaffle Ice Cream
$0.42
63
Qwenqwen/qwen2.5-vl-72b-instruct
-1229.8
2,903,489
3,080
154
3.7 ptBacon Lettuce Tomato
54.2 ptKitten in Bread
$3.07
40
Qwenqwen/qwen3-vl-235b-a22b-instruct
-939.9
1,521,123
2,000
100
3.7 ptBacon Lettuce Tomato
66.3 ptWaffle Ice Cream
$0.48
59
Qwenqwen/qwen3-vl-30b-a3b-instruct
-1169.9
1,518,179
2,000
100
3.7 ptBacon Lettuce Tomato
77.4 ptChicken Wrap
$0.27
22
Qwenqwen/qwen3-vl-30b-a3b-thinking
-710.9
1,971,808
2,000
100
3.7 ptBacon Lettuce Tomato
62.6 ptPickle Sandwich
$1.03
11
Qwenqwen/qwen3.5-122b-a10b
-593.6
2,885,005
2,000
100
3.7 ptBacon Lettuce Tomato
53.4 ptBagel PB&J
$3.41
18
Qwenqwen/qwen3.5-27b
-680.1
3,036,535
2,000
100
0.8 ptHot Dog
63.4 ptChicken Wrap
$3.75
23
Qwenqwen/qwen3.5-35b-a3b
-715.0
2,559,505
2,000
100
3.7 ptBacon Lettuce Tomato
53.4 ptBagel PB&J
$1.97
31
Qwenqwen/qwen3.5-397b-a17b
-816.7
3,027,614
2,000
100
3.7 ptBacon Lettuce Tomato
76.4 ptChicken Wrap
$6.98
27
Qwenqwen/qwen3.5-9b
-752.4
2,422,927
2,000
100
3.7 ptBacon Lettuce Tomato
65.4 ptChicken Wrap
$0.29
9
Qwenqwen/qwen3.5-flash-02-23
-568.4
3,009,358
2,000
100
1.5 ptCookie PB
53.4 ptBagel PB&J
$0.77
32
Qwenqwen/qwen3.5-plus-02-15
-831.8
2,976,068
2,000
100
3.7 ptBacon Lettuce Tomato
68.4 ptChicken Wrap
$2.75
12
Grok / xAIx-ai/grok-4
-601.1
2,950,739
2,000
100
3.7 ptBacon Lettuce Tomato
65.6 ptPickle Sandwich
$15.67
5
Grok / xAIx-ai/grok-4-fast
-414.0
1,600,190
2,000
100
0.4 ptPickle Sandwich
53.4 ptBagel PB&J
$0.53
16
Grok / xAIx-ai/grok-4.1-fast
-677.0
1,745,414
2,000
100
3.7 ptBacon Lettuce Tomato
52.6 ptPickle Sandwich
$0.50
17
Grok / xAIx-ai/grok-4.20-beta
-679.0
912,414
2,000
100
0.7 ptWaffle Ice Cream
54.2 ptKitten in Bread
$1.47
51
Z.AI / GLMz-ai/glm-4.6v
-1014.7
2,712,263
2,360
118
3.7 ptBacon Lettuce Tomato
61.3 ptChicken Wrap
$1.04
-
TotalAll models
-
227,565,630
141,540
7,077
--
$726.34

Each iteration is 20 images. Partial runs are shown as fractional iterations when a model did not finish the full workload.

This page stays inside the current website snapshot. It is reading the same generated model summaries and per-image outputs as the rest of the site, just arranged for model-centric inspection instead of photo-centric inspection.

AI Model Sandwich Benchmark Rankings | opensandwich.ai