Skip to content

LLM Performance

Hallucinations

  • Language models cannot reliably distinguish belief from knowledge and fact and/or fake components for a real sourceWe evaluate 24 cutting-edge LMs using a new benchmark of 13,000 questions across 13 epistemic tasks. Our findings reveal crucial limitations. In particular, all models tested systematically fail to acknowledge first-person false beliefs, with GPT-4o dropping from 98.2% to 64.4% accuracy and DeepSeek R1 plummeting from over 90% to 14.4% (Nature Machine Intelligence, 2025).
  • OpenAI admits AI hallucinations are mathematically inevitable, not just engineering flaws In a landmark study, OpenAI researchers reveal that large language models will always produce plausible but false outputs, even with perfect data, due to fundamental statistical and computational limits (September 2025).

  • AI Hallucination Cases This database tracks legal decisions in cases where generative AI produced hallucinated content – typically fake citations, but also other types of AI-generated arguments.

    • (Tribunale di Milano, 5/6/2026): Questa limitazione è giustificata dalla presenza nella comparsa di costituzione di tre massime giurisprudenziali apparentemente estrapolate da pronunce della Corte di Cassazione (cfr. pag. 13: Cass. 21432/2021, Cass. 7024/2017, Cass. 24058/2004), che però, ad una più attenta lettura, risultano inesistenti: o meglio, le sentenze citate esistono, ma non hanno nulla a che vedere con le citazioni riportate dalla parte convenuta. Ciò denota verosimilmente l’utilizzo di sistemi di intelligenza artificiale nella redazione degli scritti difensivi, senza controllo da parte del difensore sui noti rischi di “allucinazione”.

  • Chasing the Hallucinations: KPMG's AI-Powered Attempt at "Redefining Excellence" This investigation analyzes a KPMG report from October 2025 on customer experience and agentic AI. The report, titled Total Experience: Redefining Excellence in the Age of Agentic AI, summarizes the results of an annual study on consumer experience around the world. Of the 45 citations in the report, only five accurately point to real sources. Another 28 citations provide paraphrased titles The final 12 are too vague or flawed to accurately determine if a source exists. Additionally, around half of the claims evidenced by the 45 citations appear to be fake or misattributed (June 2026).

  • Deloitte refunds Aussie gov after AI fabrications slip into $440K welfare report Deloitte has agreed to refund part of an Australian government contract after admitting it used generative AI to produce a report riddled with fake citations, phantom footnotes, and even a made-up quote from a Federal Court judgment (October 2025).

A personal experience

Summarization and Research

Accuracy

  • TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks We introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. … We find that the most competitive agent can complete 30% of tasks autonomously (September 2025, Carnegie-Mellon, Duke). Leaderboard.
  • CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions We introduce CRMArena-Pro, a novel benchmark for holistic and realistic assessment of LLM agents in diverse professional settings. Experiments show leading LLM agents achieve approximately solely 58% single-turn success rate on CRMArena-Pro, with significant performance drops in multi-turn settings to 35% (May 2025, Salesforce).

  • SpreadsheetBench evaluates large language model agents' capabilities in manipulating complex real-world spreadsheets and business spreadsheet workflows. Leaderboard March 2026: 35%.

  • Error by AI scribe during medical appointment leaves patient devastated When Rebecca Green arrived for her first appointment with a urologist, she was asked whether she consented to having the appointment transcribed by artificial intelligence (AI). She said yes without hesitation….But that decision would cause her a lot of stress, with the AI scribe making up the serious but false claim that Ms Green was taking illegal drugs (August 2026).

  • Chinese farmer kills 25 acres of crops after following AI-generated weed and pest control advice Wu asked the AI for advice on weed and pest control, after which it … recommended that they use “high-efficiency flupyrimethalin” and “flusulfasulfaether” to kill weeds and mixed it with “thiamethoxazine” and “methyl salt.” The farmer followed the AI’s advice to the letter without confirming the information via agricultural technicians (or even just counter-checking its answers online). The result of the AI’s mistake was quickly apparent, as, just the following day, the weeds and the sesame seedlings died en masse (August 2026).

Analyses

  • AI Democratization and Its Discontents - On the feedback loop that AI’s democratization left behind Roberto Pieraccini (June 2026)

    • Let me stay with speech recognition, because the contrast is sharp. When we deployed ASR into call centers, the failures were visible, local, and bounded. The system either understood the caller or it did not, and everyone involved could tell which had happened. The error was legible: it could be measured, reported, and routed back to someone who knew what to do with it....it was tractable mostly because the domain, customer care, was small and manageable.

    • AI’s failure mode is different...LLM-based products can fail at some low but irreducible rate, producing outputs that are subtly wrong rather than obviously broken: a legal summary that quietly omits a liability clause, a medical assistant that states the wrong drug interaction with complete confidence, a financial forecast that gets the number wrong and nobody notices. None of these announce themselves the way a crashed program or a garbled phone call would. The people on the receiving end usually have no way of knowing it is happening... And the domain, unlike call centers, is practically unbounded: document compliance, sales, attrition, and an endless list of others. It is virtually impossible to make a single AI work equally well across such a myriad of uses.

    • There is also a deeper reason these failures stay buried. To catch an error you need a way to verify the output, and for most uses of AI no such verification exists; someone has to know the domain well enough to judge, and often no one checks at all. Code is the great exception: a program compiles or it does not, the tests pass or they fail, and the verdict comes back in seconds. That, I think, is the real reason AI has been so transformative for software, not because writing code is easier than writing prose, but because code carries its own feedback loop, built in. Where the output can be checked, the loop closes on its own; where it cannot, the error has no way of making itself known.

Personal experiences

  • The failure mode is hidden errors (full inline with the above comments) (September 2026). I asked Claude "Is there a GUI for configuring and managing ASR?" (ASR stands for Attack Surface Reduction, one of the many security technologies in Windows). Claude answered confidently: img_asr.png I then continued the chat and at some point I wanted to see the GUI but could not find it. Thus, I asked "I am using Windows 11 pro but I do not find the GUI you mentioned in Manage Settings". The answer was: img_asr2.png
  • A subtle hallucination by the “AI” I play with Gemini for extracting MITRE ATT&CK techniques from cybersecurity incident reports automatically (MITRE ATT&CK is a powerful framework for reasoning about attacks and I use this framework intensively in my Cybersecurity course): you give Gemini the URL of a report and will immediately obtain the attack techniques used in that attack campaign…. I experienced subtle extraction errors, that I detected only because I compared the answer to the original document. (essential but often overlooked requirements of AI applications: the cost of a mistake must be small)…

Reliability

  • Towards a science of AI agent reliability AI agents are evaluated based on a single number, the average success rate at the task. That number has been going up quickly on many tasks over the last two years, which is why there’s so much excitement about deploying agents… When we consider a coworker to be reliable, we don’t just mean that they get things right most of the time. We mean something richer:

    • They get it right consistently, not right today and wrong tomorrow on the same thing (Consistency);
    • They don’t fall apart when conditions aren’t perfect (Robustness);
    • They tell you when they’re unsure rather than confidently guessing (Calibration);
    • When they do mess up, their mistakes are more likely to be fixable than catastrophic (Safety)

We found that reliability has improved only modestly over 18 months, while accuracy improved substantially. All three major providers cluster together, so this appears to be an industry-wide limitation (February 2026, Princeton). Reliability Dashboard.

  • LLMs Corrupt Your Documents When You Delegate Our large-scale experiment with 19 LLMs reveals that current models degrade documents during delegation: even frontier models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupt an average of 25% of document content by the end of long workflows, with other models failing more severely. Additional experiments reveal that…degradation severity is exacerbated by document size, length of interaction, or presence of distractor files. Our analysis shows that current LLMs are unreliable delegates: they introduce sparse but severe errors that silently corrupt documents, compounding over long interaction (April 2026).
  • Meta's Head of AI Safety Just Made a Mistake That May Cause You a Certain Amount of Alarm Yue admitted that she screwed up by letting OpenClaw take control of her computer, after which it proceeded to unintentionally hold her “important” emails hostage (February 2026).
  • Agents of Chaos We report an exploratory red-teaming study of autonomous LLM–powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution. Over a two-week period, twenty AI researchers interacted with the agents under benign and adversarial conditions. Observed behaviors include unauthorized compliance with non-owners, disclosure of sensitive information, execution of destructive system-level actions, denial-of-service conditions, uncontrolled resource consumption, identity spoofing vulnerabilities, cross-agent propagation of unsafe practices, and partial system takeover (February 2026, Stanford, MIT, Carnegie-Mellon and others).