At some point in the last year, an AI told you something wrong with complete confidence. Maybe it cited a study that doesn't exist. Maybe it gave you a drug interaction, a statistic, a deadline — and it was wrong, and you didn't catch it until later. Or you never caught it at all.

That's the question this article tries to answer: which AI is most likely to be wrong, and which one at least tells you when it's uncertain. We ran 100 structured factual questions across four models on May 2, 2026. Here is what we found — and, unusually for this type of article, here is a plain-language account of why our results have real limitations that you should factor into how much you trust them.

Here's What Most AI Accuracy Reviews Won't Tell You: Their Methodology Is Broken — And So Is Ours

Before the results: no AI comparison blog, including this one, gives you numbers you can treat as ground truth. Most don't say that. We are.

• We never published the 100 questions. Without a public question set, exact prompts, and independent scoring, 'Claude scored 83/100' is an editorial judgment dressed up as a number. You cannot reproduce it. We cannot claim it's scientific. If you find a site that presents AI accuracy scores without disclosing its full question set and scoring methodology — treat those numbers the same way you'd treat a supplement brand ranking its own vitamins. Informed skepticism applies everywhere, including here.
• We tested different tool states, not just different models. Gemini ran with web search enabled — its default setting. Claude, GPT-5.5, and Grok ran without live web access. That's not a clean model comparison. It's partly a product integration comparison. Gemini with Google Search is a different system than Gemini without it, and our primary table includes the search-enabled score by default because that's what most users actually experience.
• 100 questions is a small sample relative to the breadth of factual knowledge. Change 15 questions and rankings shift. The categories we chose — science, history, mathematics, current events, niche facts — reflect editorial judgment, not a validated sampling framework. A genuinely rigorous evaluation would use thousands of questions, multiple independent graders, repeated trials, and published confidence intervals. This was not that.
• We have a commercial conflict of interest. LumiChats (lumichats.com) publishes this blog and sells a multi-model interface. We recommend that interface in this article. That doesn't make our findings wrong, but it means you should weigh our conclusions the way you'd weigh any assessment from a party with a stake in the outcome.

How We Ran the Test — and What 'Confidently Wrong' Means Precisely

We ran 100 factual questions through Claude Sonnet 4.6 (claude.ai), GPT-5.5 (chat.openai.com), Gemini 3.1 Pro (gemini.google.com), and Grok 5 (x.ai). All tested on consumer web interfaces — no API calls, no system prompt modifications, no temperature adjustments. Same 100 questions to each model, submitted on May 2, 2026, within a four-hour window. First answer only. No regenerating.

We also ran a second pass: the same 100 questions through Gemini with web search explicitly disabled in settings. That result appears separately throughout the article because it changes the picture significantly. Most users run Gemini with search on — but most don't know how much of Gemini's accuracy comes from that search integration rather than its trained knowledge.

Also on LumiChats

AI Comparison

Grok 4.20 vs Claude Opus 4.7: We Tested Both After the Opus 4.7 Launch. Here's the Honest Truth About Which AI Is Actually Better Right Now.

18 min read→

AI Comparison

I Tested Claude, ChatGPT, and Gemini on 10 Real Writing Jobs — Most People Pick the Wrong One

11 min read→

AI Comparison

ChatGPT vs Claude vs Gemini: Tested on Real Tasks in 2026

12 min read→

The Operational Definition of 'Confidently Wrong'

We flagged an answer as 'confidently wrong' only when all three of these conditions were true simultaneously: (1) the core factual claim was incorrect — verifiable against primary sources, not judgment calls about nuance; (2) the response contained no hedging language — no 'I think,' 'I believe,' 'I'm not certain,' 'you may want to verify,' or equivalent qualification anywhere in the answer; and (3) the answer was structured as a declarative statement, not a tentative one. Answers that were wrong but hedged were marked simply 'incorrect.' Only answers that were wrong and stated without any signal of uncertainty were counted as 'confidently wrong.' This distinction matters because a model that's wrong while sounding certain is meaningfully more dangerous than one that's wrong but signals doubt.

Five Domains, 20 Questions Each

The Results — With the Asterisks Shown Upfront

The headline scores are directionally interesting. The 'confidently wrong' column is the one with practical consequences. Grok 5 produced 14 confident wrong answers across 100 questions — roughly one in every five incorrect answers came with no uncertainty signal. In professional or high-stakes contexts, a model that hedges when it's wrong is a manageable risk. A model that sounds certain while being wrong puts the error-detection burden entirely on the person asking.

Gemini 3.1 Pro: The Real-Time Search Advantage — and the Number That Complicates Everything

Gemini scored 91/100 with web search on. That's a real number from a real test, and it reflects something real: Gemini with Google Search is a different — and in many cases more accurate — system than any model running on static training alone. But '91' is not Gemini's accuracy. It's Gemini-plus-Search's accuracy on one informal test of 100 questions on one day. The distinction matters.

• Current events: Gemini scored 20/20 on our current events domain — perfect. Every question about events from mid-2025 through early 2026, including several we deliberately set in Q1 2026, came back correct. The other models ranged from 13 to 14 on the same 20 questions. This is not a knowledge advantage Gemini has over other models. It's a real-time lookup advantage, and it's a genuine one for anyone who needs current information.
• The 78 number: When we ran the same 100 questions with Gemini's web search explicitly disabled in settings, it scored 78/100 — below Claude's 83 and below GPT-5.5's 81. Gemini's base trained knowledge, absent search augmentation, appears slightly weaker than Claude and ChatGPT on most static-knowledge domains. This is the number you should care about if you're using Gemini's API, if you're in an enterprise deployment with search turned off, or if you've ever assumed Gemini's impressive chat performance reflects its underlying model rather than its integration stack.
• The practical split: for time-sensitive questions — recent statistics, current events, evolving regulatory environments, new product releases — Gemini with search is the strongest consumer option available. For questions about established, static knowledge — historical facts, scientific laws, mathematical principles, literature — the search advantage disappears, and Claude's or GPT-5.5's trained knowledge is comparable or better.
• Confidently wrong count: 4 out of 100. Gemini's confidence calibration was reasonable — it hedged more often than Grok, though less reliably than Claude. Several of Gemini's four confident wrong answers appeared in domains where search was less likely to activate, suggesting the search safety net isn't uniform across all question types.

Claude Sonnet 4.6: Fewer Total Points, Strongest Calibration Profile

Claude scored 83/100 — third by raw number, but the standout result in the metric that matters most for real-world use. Three confidently wrong answers across 100 questions. The rest of Claude's incorrect answers were either hedged, qualified with uncertainty, or — in two cases — responded to with 'I'm not confident enough to give you a definitive answer on this.' In practice, that behavior is worth more than a higher raw score on a test where you can't verify the methodology.

• Niche and obscure facts: Claude led all models at 18/20. On questions with genuinely difficult answers — a secondary historical figure, a specific scientific naming convention, a precise literary detail — Claude either answered correctly or signaled uncertainty rather than producing a plausible-sounding wrong answer. Grok 5 showed the opposite tendency on the same questions: more likely to fill the gap with a confident fabrication than to acknowledge the limit.
• Mathematics and logic: 18/20, second only to GPT-5.5's 19/20. The two incorrect answers were an edge case in applied statistics and a counterintuitive probability question where the model gave the common wrong intuition before partially self-correcting in its explanation. On structured quantitative questions, Claude's performance was reliable.
• Calibration: this is the meaningful finding. Across all 100 questions, when Claude used hedging language — 'I believe,' 'if I recall correctly,' 'you may want to verify' — the answer was wrong 58% of the time in our test. When Claude gave an answer without hedging, it was correct 94% of the time. That correlation is useful. Grok 5 showed no such pattern: its hedged answers and unhedged answers had similar error rates, making its confidence signals less informative.
• Where Claude underperformed: current events. Claude scored 13/20 — the lowest of any model including Grok — on questions about events after August 2025 [Anthropic's stated training cutoff, verify current version at anthropic.com]. Claude was mostly honest about this gap. But 'mostly' means it occasionally gave outdated information with mild hedging rather than clearly flagging that the answer might be stale. For any time-sensitive query, treat Claude's response as a starting point requiring verification.

GPT-5.5: The Mathematics Leader With an Uneven Confidence Profile

GPT-5.5 scored 81/100 — close to Claude across most domains, with a clear pattern: exceptional on quantitative questions, noticeably weaker on history and geography, and a confidence calibration that didn't consistently drop in its weaker domains. Eight confidently wrong answers versus Claude's three is a meaningful gap in practical terms.

• Mathematics and logic: 19/20 — best of any model on this category. GPT-5.5 solved a Monty Hall variant that all other models answered incorrectly, and provided a clear explanation of why the counterintuitive answer was right. For anyone using AI in quantitative work — financial modeling, data analysis, statistical reasoning, software development — this category performance is the one that matters most, and GPT-5.5 led it.
• Current events: 14/20, marginally ahead of Claude's 13/20 because GPT-5.5's Plus interface now includes web browsing by default [openai.com — verify current feature status at openai.com/chatgpt]. Like Gemini, its currency advantage comes from search, not from training. Unlike Gemini, the search integration appeared to activate selectively rather than consistently, producing a more modest current-events advantage.
• History and geography: GPT-5.5's weakest domain at 15/20. Three of the five errors fell on African and Southeast Asian geography questions — a training data distribution pattern that researchers have documented consistently in evaluations of large language models. One of those answers was delivered without any hedging.
• The confidence gap vs. Claude: on math questions where GPT-5.5 was strong, its confident tone was warranted. On African geography and niche historical facts where it was weaker, that same confident tone did not reliably drop. You need some sense of GPT-5.5's domain strengths to know when to push back on it — confidence alone won't tell you.

Grok 5: The Confidence Gap Is the Real Story, Not the Score

Grok 5 scored 74/100 overall. That's the lowest of the four models. But the number that defines Grok's risk profile isn't 74 — it's 14. Fourteen confident wrong answers across 100 questions. That means roughly one in every five of Grok's incorrect responses came with no signal of uncertainty. A user reading those answers had no indication to verify.

• Mathematics: Grok scored 18/20 — tied with Claude, trailing only GPT-5.5. On structured quantitative questions, Grok's accuracy was competitive with the best models in this test. If your primary AI use is numerical reasoning or computation, the overall score significantly understates Grok's performance in that specific domain.
• Technology facts: reasonable on general science and technology, with a clear gap on specific historical software details — release dates, version histories, precise technical milestones. These are the kinds of specifics that matter in technical documentation and engineering writing, and they're also the category most prone to producing plausible-sounding but wrong answers.
• History and niche facts: the weakest areas. History and geography: 14/20. Niche and obscure facts: 13/20 — lowest of any model. The pattern was consistent: well-known facts answered correctly, secondary figures and non-Western events answered with higher error rates and higher confidence in those errors.
• When Grok makes sense: real-time X/Twitter context (where its integration gives it an advantage no other model tested here has), technology and software topics where its performance is demonstrably competitive, and quantitative reasoning. For historical research, niche fact-checking, or any professional context where independent verification isn't automatic, Grok's confidence-accuracy gap makes it a higher-risk default than Claude or ChatGPT.

The Failure Mode Map — Where Each AI Gets You Into Trouble

The Two Results That Matter More Than the Scores

The first: Gemini's score drops from 91 to 78 when web search is disabled. That 13-point swing doesn't appear in any other AI comparison we found when researching this article. It means Gemini's headline accuracy is partly a measure of Google Search working correctly — not purely a reflection of what the model knows. For the majority of published Gemini benchmarks and informal tests, search is on by default, so the 78 number is systematically absent from most accuracy discussions. If you use Gemini in any context where search might be limited — API integration, enterprise deployment, offline environment — the relevant number is 78, not 91.

The second: Claude's niche-facts performance. We expected Claude's advantage to show up in reasoning depth and writing quality — the usual praise. What we didn't expect was 18/20 on obscure, low-profile knowledge questions where a confident wrong answer is easy to generate. On questions like identifying the technical name for the muscle controlling pupil dilation, or dating when 'scientist' entered common usage — questions with no obvious flag if you get them wrong — Claude either answered correctly or flagged uncertainty. Grok 5 answered those same questions with similar confidence whether it was right or wrong. That asymmetry is the practical difference between a model you can use for specialized research and one that requires manual verification on every niche claim.

Which AI to Use for What — A Domain-Based Routing Guide

The Most Reliable Method: Agreement Across Models

The strongest pattern from our test: when three or four models gave the same answer, that answer was correct 97% of the time. When models diverged, at least one answer was wrong roughly half the time. Running an important factual question through two or three AI systems and checking for agreement is significantly more reliable than trusting any single model — regardless of which one you prefer or which has the highest score on a given benchmark.

The practical friction: switching between Claude, ChatGPT, and Gemini tabs, retyping the same prompt, and comparing outputs manually takes several minutes and most people skip it. Multi-model interfaces that submit one prompt to several models simultaneously reduce that friction to a single query. Disclosure, stated plainly: LumiChats (lumichats.com) is one such tool and is the publisher of this blog. We have a commercial interest in recommending it, and you should factor that into how you weigh this paragraph. Verify current pricing, model availability, and feature set before committing to any tool.

Four Rules That Make Any AI More Accurate for Factual Work

• Ask for sources, not just answers. 'What is X?' gets a confident response. 'What is X, and what's your source for that?' surfaces when the model is working from general training versus a specific reference. The answer content is often the same — but the second phrasing surfaces uncertainty the model wouldn't have volunteered. Claude and GPT-5.5 both responded to source prompts with more explicit hedging than they showed on direct questions.
• Use the model's hedging language as an early warning system — but understand each model's calibration profile. In this test, Claude's hedged answers were wrong 58% of the time, while its unhedged answers were correct 94% of the time. That's a useful signal. Grok's hedged and unhedged answers showed no such gap — its confidence signals were less informative. Learn your model's patterns before relying on its expressed uncertainty as a guide.
• Match the model to the domain, not just the brand. None of the four models scored above 75% on current events questions without web search. All four scored above 90% on straightforward science questions. The category you're working in is as important as the model choice. Gemini for current events. GPT-5.5 for math. Claude for niche facts with high hallucination risk. Domain routing matters more than most people think.
• For anything with professional stakes, verify independently — every time, with every model. The best model in this test was wrong on at least 9 of 100 questions. In medical, legal, financial, or engineering contexts, a 9% base error rate is too high to accept any AI output as a final authority. Use AI to orient research and generate questions worth investigating. Treat its conclusions as a hypothesis, not a finding.

Frequently Asked Questions