A Sophisticated Assumption-Making Machine

Designing Trustworthy AI Experiences

---

I’ve been developing the ideas in this piece for years, and it’s only a slice of a larger set I now teach in a course called Designing Trustworthy AI Experiences. Sharing this part because it’s the foundation the rest is built on — if it resonates, there’s a lot more where it came from.

---

You’ll hear LLMs (the true name of what we think of as artificial intelligence) called answer machines, reasoning machines, creativity machines. They’re none of those things, and the names matter, because your mental model of a tool determines how you use it.

An LLM is a very sophisticated assumption-making machine. That’s the whole invention. Given incomplete human communication — and all human communication is incomplete — it infers the missing parts the way another human would. The essays, the code, the login screen: all downstream of that one capability.

To see why that’s remarkable, remember what computers were for the previous sixty years: machines that assumed nothing. Every compiler, every stylesheet, every command line gave you exactly what you specified and not one pixel more. We built entire disciplines around that fact — programming languages are, at bottom, grammars for leaving nothing to assumptions, because there was nothing on the other end that could make assumptions. The craft was explicitness. If the output was wrong, you had left something out.

Roughly three years ago, that stopped being the kind of machine on your desk. And most people — including most designers — have not updated their mental model to match.

This is measured, not metaphor

The claim that these machines assume “the way another human might” is now an empirical finding, not a slogan.

The formal account of human gap-filling comes from the philosopher Paul Grice: conversation runs on a cooperative principle, and listeners routinely recover meanings the speaker never said — what he called implicature — by assuming the speaker is relevant, truthful, and exactly as informative as needed. Most of what we call “understanding each other” is this. Communication is compression; the listener’s assumptions are the decompression algorithm.

For sixty years, no machine could run that algorithm. Now: researchers at MIT ran humans and language models through the same battery of pragmatic tests — irony, indirect speech, deceits, implicature — and found the models not only recover unstated meaning at rates approaching humans, they make the same kinds of mistakes humans make.A 2025 study found GPT-4 exceeding average human performance on Gricean implicature tasks outright.

And the assumptions never stop, because they can’t. Real-world instructions are almost never fully specified — the research community calls this task ambiguity — so the model must infer which task you meant from the several tasks consistent with your words.Your five-word login prompt was consistent with a banking app, a children’s game, and a hospital records system. It picked one. It always picks one. But just one.

The feature is the failure mode

The mechanism that makes these machines useful is the same mechanism that makes them dangerous.

The machine that fills in details helpfully is the machine that fills in details wrongly — silently, and with total confidence. Hallucination is not a bug bolted onto the side of these systems. Researchers have proven a statistical version of this: a language model calibrated to generate plausible text must fabricate certain kinds of facts at a mathematically inevitable rate, regardless of architecture or training data. A 2025 follow-up showed why the problem persists: training and evaluation reward confident guessing over admitting uncertainty — the machine is graded like a student who’s learned that a blank answer scores zero but a plausible one might pass.[

Communication is compression;
the listener’s assumptions are the decompression algorithm.

Hallucination is the feature, pointed in the wrong direction. The machine fabricates for the same reason it understands you: it is always completing the pattern. It always has an “opinion”.

You’ve heard the dismissive framings — “fancy autocomplete,” “stochastic parrots,” Andrej Karpathy’s “dream machines.” Are they right?

True: there’s no desire behind the output and no appreciation of it afterward. The philosopher Martha Nussbaum argued that emotions aren’t fuel for reasoning but complex parts of reasoning itself — and the machine has none. It does not care about your login screen.

False: “just making assumptions” is not a small thing. Making good assumptions is most of what humans do when we communicate. Dismissing a machine that can finally do it is like dismissing the printing press as fancy handwriting. But notice what kind of statements the dismissals are: descriptions. They tell you what the machine is and leave you standing there. “Assumption-making machine” is an instruction. If the machine’s fundamental act is assuming, your fundamental act is auditing.

Why this is a trust problem

Trust — between you and an AI tool, and eventually between your actors and the AI experiences you design — is not a feeling. It’s a state of knowing which assumptions are in play. You cannot trust output when you don’t know what it assumed getting there; you can only approve it or not, which is how confident, fluent, wrong work gets shipped. The assumption made silently is a landmine. The same assumption said out loud is a gift — it’s the machine showing you its mental model so you can correct it before anything gets built on top of it.

There’s empirical support that saying-out-loud works: when language models actively elicit intent — asking informative questions instead of silently inferring — their outputs align with what people actually wanted significantly better than even carefully engineered prompts.Asking beats guessing. Every design researcher already knew that about humans. It turns out to be true of the machine, too.

Which raises the obvious question: if the machine won’t ask on its own, whose job is it to close the gap?

Yours. Starting now.

---

This is one idea from a much bigger picture. In the class, it becomes the starting point for the harder questions: how trust between you and an AI actually works, and how to do UX research without letting AI quietly invent your evidence — because a tool built entirely from what people said is the last thing you want closing the gap between what people say and what they do. If that’s the kind of thing you want to spend six weeks on, come join me.

→ maven.com/cranford-teague/trustworthy-ai-xd

---