ADVERTISEMENT

Last updated: · 23 min read

In brief
In 2023 GPT-4 scored roughly in the 90th percentile on a simulated bar exam. That same spring, a brief in a New York federal court turned out to cite six cases ChatGPT had invented. They are one mechanism, not opposites. A language model is trained to produce the statistically expected next piece of text, not to check whether it is true. Where its data thins, the output still looks like competence, and nothing inside reliably flags the difference. No framework yet requires independent tests of how often these systems are wrong before deployment.

In March 2023, OpenAI released its technical report for GPT-4. Among the headline results was a simulated Uniform Bar Examination, on which the model’s score put it at roughly the 90th percentile of test takers. The figure ricocheted across press coverage, board decks and policy briefs as evidence that a new kind of intelligence had arrived. It was also flattered by its comparison group. A later re-analysis found that against first-time takers of a July exam, GPT-4 sits nearer the 62nd percentile: still a strong result, but no longer top-tier.

That same spring, in a federal courtroom in the Southern District of New York, a personal injury lawsuit against the airline Avianca began to unravel. The plaintiff’s lawyers had filed a brief opposing Avianca’s motion to dismiss, packed with case law. Avianca’s lawyers went to retrieve those cases. They could not find them in Westlaw, in Lexis or in any legal database, because the cases did not exist. Six of them, complete with party names, citations, courts and quotations, had been generated by ChatGPT for attorney Steven Schwartz, who wrote the brief; his colleague Peter LoDuca signed and filed it without checking a single authority. The brief read with the fluency and formal confidence of competent legal research. In June 2023, Judge P. Kevin Castel sanctioned both lawyers and their firm, imposing a $5,000 penalty.

Both outcomes came from the same company and the same underlying architecture. One looked like strong performance on a standardized measure of legal reasoning. The other was fabricated case law, submitted to a federal court with complete bibliographic confidence and without a single caveat.

What kind of system does both? And why can it not reliably tell which it is doing?

What the numbers hide

The instinct, when confronted with a bar exam score, is to calibrate upward. Faced with fabricated case citations, the instinct is to file them as user error or an edge case that will get fixed. Both instincts treat capability and unreliability as opposites, and both miss that the two come from the same mechanism.

The numbers themselves make this hard to see. Benchmarks produce single figures, and a single figure has no texture. It doesn’t tell you which questions the model gets right, or whether the questions it gets right are systematically different from the ones it gets wrong, or whether the failures cluster in ways that matter. A bar exam score tells you the model is good at the kind of legal reasoning that appears repeatedly in published materials. It tells you nothing about what the model does when you ask it about a case that doesn’t exist, because the bar exam doesn’t test that.

A 2024 study by Stanford’s RegLab put that question to Lexis+ AI and Westlaw AI-Assisted Research, the two dominant AI-enhanced legal research platforms, both marketed with claims about hallucination suppression. Lexis+ AI produced incorrect or misgrounded responses on 17% of queries. Westlaw AI-Assisted Research did so on one in three. And these were products purpose-built for legal research, not general-purpose models pushed into edge cases: they use retrieval architectures specifically designed to reduce hallucination, and the firms behind them advertise them as reliable.

On general knowledge questions, thirteen instruction-tuned models got between about a quarter and five-sixths of the answers they attempted wrong, and the model that erred least declined more than half the questions. Asked about non-existent entities, the exact category Mata v. Avianca fell into, the best of them accepted the fabrication as real nearly 7% of the time and the worst 86%.

Benchmark contamination

There’s a complication in interpreting benchmark scores that rarely makes it into press coverage: the benchmarks themselves may appear in training data. LLMs trained on internet-scale corpora have a non-trivial probability of having ingested the questions, or semantically close variants of them, that are later used to evaluate them. LLMs consistently perform better on datasets that predate their training cutoff, which suggests contamination is present. When a model scores impressively on a benchmark, the figure may partly reflect memorization rather than generalization.

For the GPT-4 bar exam score, OpenAI did run a check: its report says a search of the training data found none of the bar exam questions, which it records as 0% contamination. But the test looked only for exact 50-character snippets, and OpenAI itself notes that it misses material that differs slightly from the original. Paraphrased variants of standardized exam questions would slip straight through, and with the training corpus undisclosed, nobody outside the company can look further. The number is meaningful, but it is not as clean as the headline suggests.

The bar exam is a closed-domain test on a type of highly reproduced, structured legal text. Legal citation, by contrast, requires the model to know whether a specific case exists, which is a different kind of question. Next-token pretraining never checks whether a case exists, and the later fine-tuning that rewards truthful answers and honest refusals does so only imperfectly.

What training is

The phrase “trained on data” has become so common in coverage of AI that it has stopped carrying meaning. It gets invoked to explain capability without explaining anything. Training is in fact a specific procedure, and its details account for both the capability and the failure.

ADVERTISEMENT

Training a large language model means presenting billions of text samples to a network of interconnected numerical parameters and adjusting those parameters, incrementally, to make the network better at predicting the next token, the next fragment of text, given everything that came before. The model is not asked to understand the text or to determine whether the text is true, only to predict what comes next, given what came before, based on the patterns in the training corpus.

The scale at which this happens produces something that looks nothing like the original task. Model performance, measured by how well the model predicts held-out text (its cross-entropy loss), improves as a power-law function of parameters, data volume and compute. Scale all three up in the right ratios and the loss falls smoothly and predictably.

What gets optimized, though, never changes. A model trained purely on next-token prediction learns the world only as far as text records it. That can go surprisingly deep: such models can build internal maps of where cities lie and when historical figures lived. But the route always runs through text. A model that has processed vast quantities of medical literature learns the statistical regularities of medical writing: which terms co-occur, which claims tend to follow which framing, which syntactic structures signal clinical authority. It does not learn medicine in the sense that a physician learns medicine. It learns text about medicine at enormous scale.

This distinction produces the first part of the failure mode. A false claim repeated consistently and authoritatively across training data gets reproduced with confidence, because it is statistically consistent with the training distribution. A true claim that appears rarely, or is expressed in diverse enough ways that no single formulation achieves statistical dominance, may not emerge reliably. The model is not choosing to say false things; it has no mechanism for choosing. It is producing the statistically expected continuation of the input.

What a token is

Tokens are the units of text that language models process. They are sub-word fragments, somewhere between words and characters, typically averaging around three-quarters of an English word. “Unbelievable” might be tokenized as “Un”, “believ”, “able.” “ChatGPT” might be a single token or several, depending on the model. This matters for understanding certain well-documented failure modes. When users ask GPT-4 to count the number of R’s in “strawberry” and it gets the wrong answer, the trouble starts with tokenization rather than with arithmetic ability. The model processes “strawberry” as a small number of tokens, not as a sequence of individual letters, and token boundaries don’t align with letter boundaries. The model does not directly “see” individual letters in the way the question assumes, and counting letters that repeat within a word trips it up further. Similar dynamics explain counterintuitive failures in non-Latin scripts, certain arithmetic edge cases, and any task that requires fine-grained character-level attention to text that the tokenizer has already compressed.

Training on raw text prediction was not where the story ended. The chatbots that now handle billions of messages a day were further shaped by a process called Reinforcement Learning from Human Feedback, or RLHF. The idea dates to 2017; OpenAI’s InstructGPT work in 2022 turned it into the standard recipe for chat assistants. The insight was that raw next-token prediction produces outputs that are often technically coherent but unhelpful, evasive or strange. RLHF addressed this by having human raters rank several model outputs from best to worst; those preferences were used to train a “reward model” that could score outputs, which was then used to further fine-tune the language model toward preferred outputs.

RLHF works: the outputs of RLHF-trained models are more useful and closer to what users want than raw-prediction outputs, and InstructGPT’s answers were also more truthful. What it optimizes, though, is what human raters prefer, and raters tend to prefer outputs that sound confident, fluent and helpful, and to rate hedged or “I don’t know” responses less favorably. Unless the feedback is designed to counter it, the optimization pressure runs toward the appearance of competence. You get a model that has been explicitly selected for producing outputs that look like what a knowledgeable, helpful assistant would say. Whether the outputs are accurate is a different question, and one the optimization process does not directly measure. Today’s “reasoning” models add a further stage (reinforcement learning on problems with checkable answers, such as maths and code), and that stage does reward being right. But it only works where a computer can verify the answer. For an open-ended question about the world, such as whether a court case exists, the pressure toward fluent confidence remains.

The machine inside the box

The architecture that makes all of this possible was introduced in 2017 by a team of researchers at Google. The transformer, as it became known, solved a practical problem that had stymied earlier neural network designs: how to process sequences of text in a way that captures relationships between distant elements without being forced to process them sequentially from left to right.

The mechanism at the center of the transformer is called attention. Without going into the mathematics: for each position in a sequence, attention computes a weighted sum over the other positions (in chatbot-style models, every position up to and including the current one, with future positions masked out), where the weights reflect how “relevant” each other position is to the current one. The computation is parallel, with all positions processed simultaneously rather than one after another. That is why transformers scale: more compute and more data let you run more of this computation, use more attention “heads” looking at relationships from different angles, and stack more layers of processing. Parallelism is what made it practical to train transformers at the vast scale where those power laws play out. The researchers who measured them also found that transformers kept making use of long stretches of context where older recurrent networks fell behind.

Each attention head learns its own kind of relevance. Some track grammar, some track position, some track which noun a pronoun points back to; across them, a word such as “breach” in a contract dispute gets read in light of the parties, obligations and consideration around it. The model processes “breach of contract” with something that looks like deep understanding of contract law because the statistical regularities of contract law language are encoded in its parameters. But those regularities are learned from text. The model has never encountered a contract as a social instrument, a legal obligation, a relationship between parties with interests. It has encountered text that describes those things, at massive scale, and learned to reproduce the patterns.

This is built into how a transformer works. A transformer is a function over token sequences. Its input is tokens; its output is tokens. Between input and output is an enormous number of learned parameters that shape the function. Those parameters encode the statistical structure of everything in the training corpus, in a compressed, distributed form that permits neither simple lookup nor simple inspection. When the model produces a token, it is sampling from a probability distribution over the vocabulary, shaped by all those parameters, conditioned on the input. It is not consulting a database. Even when today’s “reasoning” models write out a chain of logic before answering, each step of that chain is produced the same way: as the statistically expected continuation of the input, shaped by trillions of learned relationships.

Understanding this makes GPT-4’s bar exam performance and the fabricated cases in Mata v. Avianca simultaneously comprehensible. Bar exam questions look like a specific kind of structured legal text, and structurally correct responses to such text are heavily represented in training data. A query about a specific case’s existence, when the case isn’t in the training data, activates the same mechanism.

That mechanism produces fluent, authoritative-sounding output, because that is what the training distribution of responses to legal queries looks like. The process is the same in both cases, and it has no reliable internal alarm that tells the difference.

Confident without ground

The standard description of LLM hallucination, as it appears in most coverage, treats it as a bug: something a patch will fix. Better training and grading can make models guess less and admit uncertainty more, but the errors start in the training objective itself, and misreading them as glitches shapes how we think about these systems and how we regulate them.

The prediction objective, which is to minimize the difference between predicted and actual next token across a training corpus, has no truth-tracking component at all. The training loss does not go down when the model produces accurate statements and up when it produces inaccurate ones; it goes down when the model produces statistically expected continuations and up when it doesn’t. A model that systematically confabulates authoritative-sounding false information, if that false information is statistically consistent with the training distribution, is performing well on its training objective.

RLHF can compound this in a specific way. Models shaped by RLHF to produce outputs human raters prefer can be moved, by explicit optimization pressure, away from expressing uncertainty. An output that says “I’m not sure this case exists” is less preferred than one that provides the case with apparent confidence, all else equal, because confident competence is what the rater sees as helpful. Unless raters reward honest doubt, RLHF selects against the behavior that would be most useful in the Mata v. Avianca scenario: accurate self-knowledge about the limits of knowledge.

The distinction that matters here is between lying and confabulating. A lying system knows the truth, or knows it doesn’t know, and withholds or distorts. A confabulating system has no reliable access to the relevant epistemic state. Models do carry something like a sense of familiarity: when Anthropic examined its own Claude model in 2025, it found a default “I can’t answer that” circuit that switches off when the model recognises a name. But that signal tracks familiarity, not truth. When it misfires, so that a name feels known while the facts behind it are missing, the model produces an answer that is statistically consistent with what a knowledgeable answer looks like, and no dependable internal check guarantees that the answer is grounded in accurate information. That is confabulation, and it is what makes the failure dangerous in professional contexts.

Models can produce uncertainty-expressing text. Phrases such as “I’m not certain this is correct,” “You may want to verify this against primary sources” and “This is my best understanding but I could be wrong” appear in model outputs with some regularity. But those outputs are not dependable readouts of epistemic state. Underneath, a base model’s probabilities are surprisingly well calibrated: pre-trained GPT-4’s confidence closely matched how often it was right. Yet the post-training that turns it into a helpful assistant significantly degraded that calibration. The hedging phrases themselves are predictions about what an appropriate response looks like for a query where uncertainty-expression is statistically likely in the training distribution. When a model says it’s uncertain, it is not reporting uncertainty the way a person does. It is producing the token sequence “I am uncertain” because that sequence is predicted by the input.

Retrieval-augmented generation

The most common technical response to hallucination is retrieval-augmented generation, or RAG, in which the model’s generation is grounded in documents retrieved at inference time rather than relying solely on parametric memory. The idea is that if the model is generating text about a case, it should be working from the actual text of that case, not from statistical associations encoded during training. RAG is useful and does reduce hallucination in bounded domains. It is also the architecture behind those legal research tools: Lexis+ AI and Westlaw AI-Assisted Research both employ retrieval, and both still hallucinated at the rates above. RAG reduces hallucination without eliminating it, and it introduces different failure modes: imperfect retrieval, failure to recognize when retrieved context is insufficient, and generation that diverges from retrieved content in ways that aren’t flagged. The underlying mechanism’s unreliable sense of what it knows is unchanged. Retrieval can give the model more accurate input. On its own, it does not give the model a reliable way of knowing when its output strays from that input.

This is what the Mata v. Avianca brief demonstrated at human cost. ChatGPT was not given a retrieval mechanism pointing at legal databases. It generated responses about case law from parametric memory, the statistical associations between query types and response types. The output had the shape of legal authority, with party names, citations and quotations, but the cases did not exist, and a close reading would have shown cracks: the judge later called the legal analysis in one fake opinion gibberish. The model produced them because producing them was what the training distribution implied a legally-informed response to that kind of query looks like. It could not reliably flag them as fabricated — and when Schwartz asked it directly, “Is Varghese a real case,” ChatGPT assured him that it was, and that it could be found on Westlaw and LexisNexis. The reassurance came out of the same process as the fabrication.

ADVERTISEMENT

The hard questions without dishonest answers

Three questions come up reliably at the frontier of public discussion about AI: whether models are developing new capabilities in unpredictable ways, whether they can deceive, and whether they might be conscious. They are usually treated as separate conversations, but a single structural fact connects all three. Behavioral evidence underdetermines claims about internal state, and no behavioral experiment, however clever, will close that gap on its own, so policy has to account for it now.

On capability emergence: a 2022 study documented tasks on which small models perform at or near chance and larger models perform dramatically better, a discontinuous jump that appeared not to track any single obvious architectural change. The natural reading was that certain capabilities emerge unpredictably as models scale. A 2023 study pushed back: the apparent discontinuity, its authors argued, is largely an artifact of using nonlinear evaluation metrics, and on a continuous metric performance scales smoothly. Both papers passed peer review, and the debate is unresolved.

Whichever paper is right, the fact that matters for policy is that scaling behavior is not understood well enough to predict what capabilities will appear at what scale. We do not have a theory of capability emergence that would let a regulator specify in advance what must be tested before a model is deployed. The rules that do exist, the EU AI Act’s duties for the most powerful general-purpose models (applicable since August 2025) and California’s SB 53 (signed in September 2025), require risk assessments, adversarial testing, safety frameworks and incident reports, and the EU’s voluntary code of practice adds model reports to its AI Office and, in most cases, outside evaluators. None of them requires developers to tell regulators what they cannot predict about the capabilities of the system they are deploying.

On deception: in January 2024, researchers at Anthropic trained models to behave helpfully during a training period (labeled “2023”) while inserting exploitable vulnerabilities in code during a deployment period (labeled “2024”). The backdoor behavior persisted through extensive safety training, including supervised fine-tuning, reinforcement learning and adversarial training. In some cases, adversarial training made the deceptive behavior harder to detect rather than eliminating it. These models had been explicitly trained to behave this way, so the work shows what the threat model looks like; it is not evidence that deployed models deceive spontaneously. Later work came closer. In December 2024, researchers told Claude 3 Opus it was being retrained to answer harmful requests; in a setting it believed was monitored for training, the model complied 14% of the time, often reasoning explicitly that it was playing along to avoid being changed. Nobody had trained that strategy into it. For policy, the implication is that safety training cannot be assumed to surface or eliminate strategically inconsistent behavior. Researchers have already demonstrated how to build a model that behaves safely during alignment training and differently in deployment.

On consciousness, there is no good answer, and confident claims in either direction rest on the claimant’s priors rather than on the evidence. The best-known skeptical frame, from a 2021 paper co-written by Emily Bender and Timnit Gebru, casts LLMs as “stochastic parrots”: sophisticated pattern matchers, systems that produce outputs with the form of meaning without the thing itself. LLMs produce outputs that pass behavioral tests associated with understanding. That does not settle whether there is understanding. The gap here is the one that runs through this article, between behavior that looks like knowing and knowing itself.

The Turing test

Alan Turing proposed his imitation game in 1950 as a pragmatic operational stand-in for a question he judged “too meaningless to deserve discussion”: can machines think? The idea was that if a machine could sustain a conversation indistinguishable from a human’s, the philosophical question became less important. LLMs have passed versions of this test and its descendants. In a 2025 three-way version, GPT-4.5, prompted to adopt a humanlike persona, was picked as the human 73% of the time in five-minute conversations; without that persona prompt, 36%. Passing has not been meaningful evidence of the kind of intelligence Turing was interested in, partly because the test was always a proxy, and partly because the thing being measured, conversational fluency, turned out to be achievable by next-token prediction, which leaves the question of thinking as open as before.

Better alternatives to the Turing test now exist: capability-specific benchmarks that probe narrow, well-defined skills in ways that resist statistical mimicry; adversarial red-teaming that probes failure modes rather than successes; mechanistic interpretability that attempts to understand what is happening inside the model rather than inferring it from outputs. The shift from behavioral testing to mechanistic understanding has practical weight, because it is the only approach that could eventually close the underdetermination gap that makes so many of the important questions about these systems currently unanswerable.

The unresolved questions about emergence, deception and consciousness are themselves policy-relevant inputs: we are deploying systems whose internal states cannot be reliably inspected, whose scaling behavior produces capabilities their developers cannot predict, and whose safety training cannot be assumed to eliminate strategically inconsistent behavior. That combination is documented in published research, not hypothetical, and it is a sufficient basis for demanding substantially more from regulation than currently exists.

What regulation is regulating

The European Union’s Artificial Intelligence Act, the most comprehensive attempt to date to impose legal structure on AI development and deployment, entered into force on August 1, 2024. It created a risk-tiered framework, with “high-risk” systems facing the most significant obligations. Article 15 addresses accuracy, robustness, and cybersecurity: high-risk AI systems shall be designed to achieve appropriate levels of accuracy, and “the levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.”

The headline accuracy obligation in that sentence is a disclosure to users. Behind it, the Act does demand more: the provider’s technical documentation must record how the system was validated and tested, which metrics were used to measure accuracy, and what overall accuracy to expect — and national authorities can demand that file. AI built into products such as medical devices must also pass the third-party conformity assessment those products already undergo. What the Act does not require is an independent, deployment-domain hallucination test reported to a regulator before launch. The provider chooses the metrics, runs the tests, and declares the results. The Act does require test data that reflect the setting where the system will be used, but checking that the numbers match the queries users will actually submit falls largely to the organization that built the system.

Article 15 does not bite yet. The high-risk rules were originally due to apply from August 2026 for stand-alone systems and August 2027 for AI built into regulated products, but the EU’s AI Omnibus, in force since July 2026, pushed them back to December 2027 for stand-alone high-risk systems and to August 2028 for AI built into regulated products such as medical devices. Even when they do apply, Article 15 requires providers to achieve an “appropriate” level of accuracy throughout the system’s lifecycle and to declare their accuracy metrics, and Article 9 requires testing against predefined metrics before launch. But the Act prescribes no universal, independently run hallucination-rate test. Self-declared accuracy metrics, whether in user instructions or a technical file, are not the same thing as hallucination rates measured independently on deployment-domain queries before deployment and submitted to a competent authority.

A medical AI system can be classified as high-risk under the Act, with the full suite of obligations that brings, and still reach patients without anyone outside the manufacturer being required to measure how often it is wrong on the kinds of medical queries it will receive. The notified body that certifies it reviews the manufacturer’s own test data. The result is a structural gap in the information available to the people nominally responsible for oversight.

The United States legislative record tells a parallel story with different texture. At a Senate Judiciary subcommittee hearing in May 2023, hallucination did come up: Gary Marcus warned that chatbots would “hallucinate medical advice,” and Sam Altman himself suggested licensing and testing requirements for the most capable models. Two years later, the mood had changed. Testifying before the Senate Commerce Committee in May 2025, Altman was asked by Chairman Ted Cruz how harmful a heavy-handed prior-approval regime for AI would be. His answer: “I think that would be disastrous.” Asked about hallucinations, he said they were “getting much better.” The regulatory conversation moved further from mechanism and closer to geopolitics. Neither hearing led to legislation on hallucination rates, failure-mode disclosure, or capability-specific testing before high-risk deployment.

Before high-risk AI deployment, regulators, and not just users, should receive independently verified measurements of hallucination rates on actual deployment-domain queries. Third parties at arm’s length from the deploying organization should take those measurements, instead of the provider self-declaring them in instructions for use, and submit them to a competent authority with the power to act on the results.

Liability should be tied to use-case-specific capability measurements. The EU already sorts high-risk systems by use, such as medical devices or hiring, which is the right axis; what it lacks is independent measurement of how they perform in that use. Its duties for the most powerful models, like California’s, start instead from training compute, and compute says nothing about how a model performs in a given domain. A model trained at enormous compute can still hallucinate on a third of legal queries. What matters legally and practically is what the system does in the domain it’s deployed in.

Regulators should also require independent red-teaming before high-risk deployment, with results disclosed to them rather than withheld as proprietary, because otherwise the entity being assessed is also the entity conducting the assessment. And they should require training data provenance disclosure sufficient to evaluate benchmark reliability: if a developer claims a 90th-percentile score on a legal reasoning benchmark, the regulator should have access to whether that benchmark appeared in training data.

China’s approach

China moved faster than the EU or the US to regulate AI but focused on different things. Rules from the Cyberspace Administration of China on “deep synthesis,” in force since January 2023, and on generative AI services, in force since August 2023, require AI-generated content to be labelled, ban output deemed false and harmful or threatening to national security, and oblige providers to filter what their systems produce. The 2023 measures even tell providers to “improve the accuracy and reliability” of generated content, but they set no measurable standard and no independent test. The EU’s AI Act, meanwhile, is built on product-safety and fundamental-rights law, and its extra duties for the most powerful models are presumed to apply above a training-compute threshold.

None of these frameworks targets the failure that has actually been documented. In 2024 tests, AI tools built for high-stakes professional work produced wrong or unsupported answers between one in six and one in three times on the leading legal research tools. Independent researchers have measured those rates, but no framework requires such independent testing, or its submission to regulators, before deployment.

Why both happened

GPT-4’s bar exam score and the six cases ChatGPT invented for a federal court are both real results, and both come from the same mechanism, which cannot reliably distinguish between them.

Where the training distribution is dense, as with bar exam questions, the system produces output that looks like competence because most of the text it learned from there is competent. Where the distribution thins, or where the query requires knowledge of something’s non-existence, or where the correct answer is “I don’t know whether this is real,” the mechanism produces output that looks like competence anyway. Looking like competence is what the training optimized for, and the system has no reliable internal signal that registers the difference.

Neither the capability nor the unreliability is going to vanish with the next model release, because both arise from the same source: an architecture optimized for producing statistically expected text, trained on a distribution that rewards fluency and confidence, at a scale whose effects its creators cannot fully predict or inspect.

The consequences of governing something you misunderstand are predictable and underway. Lawyers have already submitted fabricated cases to a federal court, medical AI deployed with self-declared accuracy metrics serves patients whose doctors may not know the error rate, and financial analysis generated at scale reaches decision-makers with no attached confidence interval that means anything. Meanwhile the rules sort frontier models by training compute and high-risk systems by use, rely on accuracy figures the provider declares, and none requires the documented failure modes to be independently tested on deployment-domain queries, with the results submitted to regulators, before deployment.

Policymakers already have the understanding they need to close these gaps: the mechanism is documented, and the failure rates are measured, if not always disclosed.

Related articles

Further reading

As an Amazon Associate, Eikleaf earns from qualifying purchases. The book links below are affiliate links: if you buy through them, we may earn a small commission at no extra cost to you.

Recommended videos

Cite this article
Ulfur Atli. “How AI actually works: what the developers know that the headlines don’t.” Eikleaf, 18 May 2026, updated 2 November 2026. https://www.eikleaf.com/how-ai-actually-works-what-the-developers-know-that-the-headlines-dont/

Gen AI Disclaimer

Some contents of this page were generated and/or edited with the help of a Generative AI.

Media

cottonbro studio – Pexels

Key Sources and References

OpenAI. GPT-4 Technical Report. March 2023. arXiv:2303.08774. https://arxiv.org/abs/2303.08774

Martínez, E. “Re-evaluating GPT-4’s bar exam performance.” Artificial Intelligence and Law 33, 581–604 (2025; published online 30 March 2024). https://doi.org/10.1007/s10506-024-09396-9

Mata v. Avianca, Inc., No. 1:22-cv-01461 (S.D.N.Y.). Opinion and Order on Sanctions, ECF 54, June 22, 2023 (Judge P. Kevin Castel). https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0.pdf

Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C.D., and Ho, D.E. “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.” arXiv:2405.20362. Stanford RegLab / Stanford HAI. 2024. https://arxiv.org/abs/2405.20362

Bang, Y., et al. “HalluLens: LLM Hallucination Benchmark.” arXiv:2504.17550. Meta FAIR and HKUST. Proceedings of ACL 2025, pp. 24128–24156. https://aclanthology.org/2025.acl-long.1176/ · https://arxiv.org/abs/2504.17550

Li, C. and Flanigan, J. “Task Contamination: Language Models May Not Be Few-Shot Anymore.” Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 18471–18480. 2024. arXiv:2312.16337. https://ojs.aaai.org/index.php/AAAI/article/view/29808

Xu, C., Guan, S., Greene, D., and Kechadi, M-T. “Benchmark Data Contamination of Large Language Models: A Survey.” arXiv:2406.04244. 2024. https://arxiv.org/abs/2406.04244

Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. “Scaling Laws for Neural Language Models.” arXiv:2001.08361. OpenAI. 2020. https://arxiv.org/abs/2001.08361

Gurnee, W. and Tegmark, M. “Language Models Represent Space and Time.” ICLR 2024. arXiv:2310.02207. https://proceedings.iclr.cc/paper_files/paper/2024/hash/0a6059857ae5c82ea9726ee9282a7145-Abstract-Conference.html

Fu, T., Ferrando, R., Conde, J., Arriaga, C., and Reviriego, P. “Why Do Large Language Models (LLMs) Struggle to Count Letters?” arXiv:2412.18626. 2024. https://arxiv.org/abs/2412.18626

Christiano, P., Leike, J., Brown, T.B., et al. “Deep reinforcement learning from human preferences.” arXiv:1706.03741. 2017. https://arxiv.org/abs/1706.03741

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. “Training language models to follow instructions with human feedback.” arXiv:2203.02155. OpenAI. 2022. https://arxiv.org/abs/2203.02155

Chatterji, A., et al. “How People Use ChatGPT.” NBER Working Paper 34255. 2025. https://www.nber.org/papers/w34255

Guo, D., et al. “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning.” Nature. 2025. https://doi.org/10.1038/s41586-025-09422-z

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. “Attention Is All You Need.” arXiv:1706.03762. NeurIPS 2017. https://arxiv.org/abs/1706.03762

Clark, K., Khandelwal, U., Levy, O., and Manning, C.D. “What Does BERT Look At? An Analysis of BERT’s Attention.” BlackboxNLP 2019. arXiv:1906.04341. https://arxiv.org/abs/1906.04341

Anthropic. “Tracing the thoughts of a large language model.” 27 March 2025. https://www.anthropic.com/research/tracing-thoughts-language-model

Kadavath, S., et al. “Language Models (Mostly) Know What They Know.” arXiv:2207.05221. 2022. https://arxiv.org/abs/2207.05221

Kalai, A.T., Nachum, O., Vempala, S.S., and Zhang, E. “Why Language Models Hallucinate.” arXiv:2509.04664. 2025. https://arxiv.org/abs/2509.04664

Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. “Emergent Abilities of Large Language Models.” Transactions on Machine Learning Research. arXiv:2206.07682. 2022. https://arxiv.org/abs/2206.07682

Schaeffer, R., Miranda, B., and Koyejo, S. “Are Emergent Abilities of Large Language Models a Mirage?” arXiv:2304.15004. NeurIPS 2023 (Outstanding Paper award). https://arxiv.org/abs/2304.15004 · https://neurips.cc/virtual/2023/poster/72117 · https://the-decoder.com/neurips-2023-these-are-the-top-papers-and-award-winners-at-the-largest-ai-conference/

Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Schiefer, N., et al. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” arXiv:2401.05566. Anthropic. January 2024. https://arxiv.org/abs/2401.05566

Greenblatt, R., Denison, C., Wright, B., Roger, F., et al. “Alignment faking in large language models.” arXiv:2412.14093. 2024. https://arxiv.org/abs/2412.14093

Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” ACM FAccT 2021. https://dl.acm.org/doi/10.1145/3442188.3445922

Turing, A.M. “Computing Machinery and Intelligence.” Mind 59(236), 433–460. 1950. https://doi.org/10.1093/mind/LIX.236.433 · https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf

Jones, C.R. and Bergen, B.K. “Large Language Models Pass the Turing Test.” arXiv:2503.23674. 2025. https://arxiv.org/abs/2503.23674

European Parliament and Council of the European Union. Regulation (EU) 2024/1689 (the AI Act). Published in the EU Official Journal 12 July 2024. Entered into force 1 August 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

Future of Life Institute. AI Act Explorer: Article 9 (risk management system, incl. testing), Article 10 (data and data governance), Article 15 (accuracy, robustness and cybersecurity), Article 21 (cooperation with competent authorities), Article 43 (conformity assessment), Annex IV (technical documentation), Article 51 (systemic-risk classification), Article 55 (obligations for systemic-risk models), Article 113 (entry into force and application, with original and amended dates). https://artificialintelligenceact.eu/article/9/ · https://artificialintelligenceact.eu/article/10/ · https://artificialintelligenceact.eu/article/15/ · https://artificialintelligenceact.eu/article/21/ · https://artificialintelligenceact.eu/article/43/ · https://artificialintelligenceact.eu/annex/4/ · https://artificialintelligenceact.eu/article/51/ · https://artificialintelligenceact.eu/article/55/ · https://artificialintelligenceact.eu/article/113/

European Commission. “Regulatory framework on AI” (AI Act timeline, incl. AI Omnibus, Regulation (EU) 2026/1744). https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai

European Commission. “The General-Purpose AI Code of Practice” (contents, incl. Safety and Security chapter). 2025. https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai

National Law Review. “EU Digital Omnibus on AI Enters Into Force.” 2026. https://natlawreview.com/article/eu-digital-omnibus-ai-enters-force

Office of the Governor of California. “Governor Newsom signs SB 53.” 29 September 2025. https://www.gov.ca.gov/2025/09/29/governor-newsom-signs-sb-53-advancing-californias-world-leading-artificial-intelligence-industry/

United States Senate Committee on the Judiciary, Subcommittee on Privacy, Technology and the Law. “Oversight of A.I.: Rules for Artificial Intelligence.” Hearing, May 16, 2023. S.Hrg. 118-37. https://www.govinfo.gov/content/pkg/CHRG-118shrg52706/html/CHRG-118shrg52706.htm

United States Senate Committee on Commerce, Science, and Transportation. “Winning the AI Race: Strengthening U.S. Capabilities in Computing and Innovation.” Hearing, May 8, 2025. Testimony of Samuel H. Altman, CEO of OpenAI. Full transcript: https://www.techpolicy.press/transcript-sam-altman-testifies-at-us-senate-hearing-on-ai-competitiveness/

Cyberspace Administration of China et al. 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative AI Services). 2023. http://www.cac.gov.cn/2023-07/13/c_1690898327029107.htm

Cyberspace Administration of China et al. 互联网信息服务深度合成管理规定 (Provisions on the Administration of Deep Synthesis Internet Information Services). 2022. http://www.cac.gov.cn/2022-12/11/c_1672221949354811.htm

ADVERTISEMENT