A confident, fluent answer that happens to be wrong is the most dangerous thing an AI model will hand you. In Trust & Safety and fraud work, that answer can shape an enforcement call or a fraud decision, and the cost lands on a real user's account or livelihood. Learning to catch those answers is the heart of AI literacy, and it's a skill you can build without writing a line of code. What it takes is a working understanding of how these systems behave, where they break, how they mislead you while sounding sure of themselves, and what you can do to steer them. Each of those changes how much you can trust the AI you're already using in your work.
At Musubi we use LLMs for content moderation and fraud analysis every day, and we think these tools are some of the best things to happen to this field in years. Used well, they review content faster than any human team, bring consistency to decisions, and extend coverage you could never staff for. The catch is in those two words, used well. Almost anyone can get a usable-looking answer out of an LLM, but getting one you can trust takes skill and knowledge.
Some of what follows is about using AI well day to day, and some is about building the systems behind it; even if you never touch the latter, knowing how it works lets you steer the tools you depend on and ask sharper questions of the people who build them.
A note on scope
This piece focuses on generative AI and large language models, the systems that produce text in response to prompts. It's where most T&S and fraud teams are spending their attention right now, and where the failure modes are least understood.
This isn't the whole landscape, though. The classification models that have powered fraud scoring and content moderation for years behave and fail in different ways than LLMs do. They produce calibrated probabilities, they're trained on your labeled data, and their failure modes are things like drift and class imbalance. Agentic systems that plan and take their own actions introduce yet another category of risk. Agents come up briefly below, but most of what follows is about LLMs specifically.
How LLMs behave
LLMs are probabilistic, not deterministic. At each step, the model breaks your text into tokens (chunks of words) and produces a probability distribution over what the next token should be, then picks one and repeats. That's the core mechanism, applied billions of times over. The Center for Security and Emerging Technology has a clear walkthrough of how next-word prediction produces all this behavior.
Because the model samples from a probability distribution, the same prompt can produce different answers on different runs. The amount of variation is partly controlled by a setting called temperature: lower temperature makes the model more repetitive and predictable, higher temperature makes it more varied and creative. For enforcement work this matters directly, because if you score the same borderline case twice and get two different calls, that's the model doing exactly what it's designed to do. Consistency in moderation and fraud decisions is something you engineer for, through fixed settings, clear instructions, and validation, and it's very achievable once you know to aim for it.
This same probabilistic nature is why models sound confident even when they're wrong. The model isn't weighing whether it knows something and then reporting its certainty; it's generating fluent, plausible text. Fluency has no inherent relationship to accuracy. When you ask an LLM for a "confidence score" or a "risk level," the number it gives back is generated text shaped to look like a confidence score, not a calibrated probability the way the output of your fraud classifier is. Unless the system has been specifically built and validated to produce calibrated numbers, treat an LLM's "92% likely fraud" as a fluent guess wearing the costume of a statistic.
A trustworthy number is one of the clearest payoffs of doing this well. At Musubi we produce a risk score that runs from 0 for confidently safe to 10 for confidently unsafe, with the middle of the range reserved for real grey areas. The score looks simple, but making it reliable took data science and ML expertise and a lot of validation against labeled data. A score is only useful if it's calibrated, meaning a 2 really does correspond to lower risk than a 7 across thousands of cases. You don't get that by typing "give me a confidence score from 1 to 10" into a prompt and trusting whatever comes back. Certainty and severity are also different things, which people routinely blur: a model can be highly certain that something is safe and highly certain that something else is egregiously unsafe.
Once a score is calibrated, you can act on it by setting thresholds that auto-clear the confident-safe cases, auto-action the confident-unsafe ones, and route the grey middle to a human or to a larger, slower model for a closer look. That kind of layered triage pays off most as content gets longer. Many teams are now seeing a shift toward long-form AI-generated text that's slow for a human to read end to end, and a model that summarizes it and flags the few passages that need eyes can cut review time sharply.
An LLM's context window is the total amount of text the model can consider at once, including your instructions, the case or content you've pasted in, and the conversation so far. Anything outside that window is invisible to the model. Less obviously, information buried in the middle of a very long context tends to get under-weighted even when it technically fits, a pattern researchers describe as models getting lost in the middle. If you paste a fifty-page policy followed by a long case history and the model seems to ignore a key rule from page twenty-five, this is often why. To avoid it, put the instructions that matter most at the very start or very end of what you send, and keep context focused rather than dumping everything in and hoping.
These models also keep getting better, fast. If you formed your impression of AI even six months ago, it's probably out of date, and the gap between a current frontier model and an older or free one is often larger than people expect. Recent gains include models that reason through a problem step by step before answering, much larger context windows, noticeably better instruction-following, and stronger handling of images and audio. This is good news, because tasks that produced garbage a year ago may work cleanly now, and a model you wrote off for nuanced policy calls might be worth another look. It also means you should re-test your assumptions when models change. You don't necessarily have to rewrite everything for each new release: a well-written, general policy often carries over, while a prompt hand-tuned to one model's quirks is more likely to need adjustment. Either way, the best practice is to run a solid evaluation on a new model before you trust it in production, so you're measuring the upgrade rather than guessing at it.
How LLMs fail
When a model gives you a bad output, the easy conclusion is that "AI doesn't work for this." The more useful response is to diagnose why it failed, because the fix is different in each case, and diagnosis is the skill that separates teams that improve their systems over time from teams that stay stuck. Almost every failure has a clear cause and a clear fix once you know where to look.
Always ask the model for its reasoning, because that's where the diagnostic signal lives. An answer on its own tells you only what the model concluded. The reasoning tells you how it got there, which usually reveals whether the problem is something you can fix or a limitation you have to design around. The reasoning a model writes out, though, is not always a faithful record of how it actually reached the answer. Sometimes it's a plausible story assembled after the fact, so treat "explain your reasoning" as a lead to investigate rather than a lie detector.
From there, most failures fall into a few buckets:
Policy and instruction gaps. Sometimes the model fails because the guidance you gave it is vague. If your policy leans on an undefined qualifier like "excessive," "obvious," or "high risk" without saying what that means in practice, the model has to fill in the blank, and it will fill it in inconsistently from run to run. Other times the problem is a pure gap: the policy simply doesn't address the situation in front of it, so there's nothing to apply. Both of these surface as model errors when the real issue is in the specification, which is good news, because a specification is something you control. This is why, when one of our scores or decisions starts behaving oddly, the first move is usually to sharpen the policy the model is working from rather than to keep rephrasing the prompt around it. The policy is the real specification, and many problems that look like the model misbehaving are problems with what we asked it to do. Our guide to the top challenges of using LLMs for content moderation touches on several of these.
Prompt gaps. Your prompt can contradict itself, instructing the model to err toward leniency in one place and toward strict enforcement in another. It can also lean too hard on emphatic instructions, where stacking "ALWAYS," "NEVER," and "this is critical" on top of each other muddies the priorities instead of sharpening them, because the model has no reliable way to rank ten things that are all marked as the most important. Negations are a related weak spot: instructions phrased as "do not" are handled less reliably than the same rule phrased as what the model should do instead. When results are erratic, reread your prompt as if you were a literal-minded reader trying to satisfy every line at once. Two more come up constantly in our own work. Examples in a prompt are powerful, but a model can overfit to them, latching onto surface features of your samples instead of the rule they're meant to illustrate, so choose examples that teach the principle. And small inconsistencies do outsized damage: we've watched a single stray line saying a scale runs "1 to 10," while the rest of the prompt says "0 to 10," skew an entire scoring distribution, because the model follows your contradiction faithfully.
Model limitations. Some failures are simply the model you happen to be using. A fast way to test for this is to run the identical prompt through a different LLM. If a second model handles it cleanly, the issue wasn't your policy or your prompt, and the fix might be as small as switching models or upgrading to a newer version. Different models also have different strengths, so the one that's best at terse policy classification may not be the one that's best at summarizing a long investigation.
Agentic failure. When a model takes a sequence of actions rather than answering one question, errors compound. A small misread in step two becomes a wrong assumption in step five, and by the time you see the final output the original mistake is buried under everything built on top of it. These systems need validation at each step rather than a single check at the end. For fraud teams experimenting with agents that pull records, cross-reference accounts, and recommend actions, this is the difference between a useful assistant and an automated way to act confidently on a wrong premise.
How LLMs mislead
Failing is one problem. Misleading is the more dangerous one, because the output looks completely fine. Here are the main patterns to watch for, and what to do about each.
Hallucination. The model states something false with exactly the same fluency it uses for true things, including fabricated sources: citations, studies, internal-sounding policy references, and quotes that look real and don't exist. This isn't random malfunction. OpenAI's research argues that hallucination is partly a predictable result of how models are trained and evaluated, because standard scoring rewards confident guessing over admitting uncertainty, the same way a test-taker is better off guessing than leaving a question blank. So if a model gives you a reference, a precedent, or a specific number, verify it exists before relying on it, because the model has a built-in incentive to produce one whether or not it's real.
Sycophancy. Models tend to tell you what you seem to want to hear. Anthropic's research found that this is a consistent behavior across major AI assistants, and that it likely comes from the training process itself, since the human feedback used to shape these models tends to reward answers that agree with the user. In practice this means if you imply the conclusion you're hoping for, or push back on a correct answer, many models will cave and revise. When you're working a case or analyzing data, this is a real risk: the model's agreement is not independent confirmation that you're right, and a model that flips the moment you challenge it is showing you it was never anchored to the evidence in the first place. The fix is cheap: ask neutrally instead of signaling the answer you want, and have the model argue against its own conclusion before you accept it.
Bias. Models absorb the patterns of their training data, including biases that show up in moderation and fraud contexts in ways that matter for fairness, consistency, and legal exposure. This can mean uneven treatment of dialects, names, regions, or writing styles in ways that correlate with protected groups. We go deep on this in our piece on understanding and addressing bias in content moderation. The practical defense is to measure performance across the groups and content types you care about rather than trusting an aggregate accuracy number, because bias hides inside the average.
These tendencies also vary from model to model. One may hallucinate more, another may be more eager to agree with you, another may carry a different bias profile, and any of these can shift meaningfully between versions of the same model. Knowing the specific model you're working with, and re-checking it when it updates, is part of the literacy.
Using AI to analyze data
This is where it's easiest to get burned, because the errors are invisible. When you point a model at an appeals queue, an enforcement log, a set of fraud reports, or a pile of flagged transactions and ask "what patterns do you see?", it feels like a fast win. The risk is that the model flattens your messy, unstructured, context-heavy data into clean, confident-sounding patterns that may not be there at all.
A handful of failure modes recur in this kind of analysis:
- Fabricated or paraphrased evidence. The model attributes claims, quotes, or trends to your data that aren't actually in it.
- Generic themes. It surfaces observations like "users are frustrated with enforcement" or "fraud is rising in high-value transactions" that are vaguely true and useless for making a decision.
- Signal without context. It flags an apparent spike that's really a known seasonal pattern, a single prolific bad actor, or an artifact of how the data was logged.
- Averaged-away contradictions. It smooths two directly opposing findings into one bland middle that represents neither and hides the tension you most needed to see.
None of this makes AI useless for analysis. Done well, it's one of the fastest ways to make sense of a messy queue. The fix is to treat any pattern the model reports as a hypothesis to check against the underlying data, rather than a finding you can act on. That habit matters here more than almost anywhere. The dangerous output is the credible one you never think to question, and that's exactly the kind these failures produce. For a deeper and very practical treatment, Caitlin Sullivan's piece in Lenny's Newsletter on how to do AI analysis you can actually trust is worth reading, though it's paywalled after the first couple of sections.
It's tempting to treat a labeled dataset or a log of past moderator decisions as ground truth, but human labels carry real errors, and inconsistent ones. So before you lean on a golden dataset, go over it with a fine-toothed comb: have several reviewers check the labels until you're confident the set is as close to perfectly labeled as you can get, because everything you measure against it inherits its mistakes. We wrote a full guide on how to build golden datasets for content moderation if you want a process to follow. LLMs can actually help here: techniques from the ML world like confident learning flag the cases where a model and its labels most strongly disagree, which points your reviewers straight at the examples most likely to be wrong. We've used this on our own data to find and fix mislabeled examples, and it measurably improved the model we trained afterward.
How to QA your outputs
A team that gets durable value from AI is a team that builds healthy habits:
- Verify sources before trusting them. Confirm any cited study, policy, precedent, or number against something real before it informs a decision.
- Make the model show its reasoning. Ask how it reached an answer so you can interrogate the path, not just accept the conclusion.
- Push back, but watch for sycophancy. Challenge answers that feel off, while remembering that a model which instantly reverses itself is signaling low confidence rather than confirming you were right.
- Spot-check independently. Pull two or three specific claims and check them against the source data yourself. Accuracy on the parts you can verify earns a little trust on the parts you can't.
- Insert your own judgment last. The model doesn't carry your platform's history, your regulatory exposure, or your accountability for the decision, and those are exactly the things that should decide a close call.
Calibration: pressure-test where decisions disagree
Calibration means systematically comparing decisions to find where they diverge, and it's one of the most useful things we do. There are three versions worth running:
- LLM against human ground truth. See where the model and your policy experts part ways, so you know whether to fix the policy or the labels.
- Humans against the LLM. Measure agreement rates across your moderators. This surfaces inconsistency in your team as much as in the model.
- Models against each other. Use one LLM to judge another to quickly find the hard cases, edge cases, and grey areas that most deserve a second look.
Each comparison turns a vague sense that something is off into a specific list of disagreements you can work through. Numbering the lines in your policy or having very granular policy sub-categories helps with this a lot, because then any disagreement can be traced to the exact clause it hangs on.
The core idea: AI is steerable
AI is steerable, and most mistakes happen when you're passive with it. Every failure described above has a move that answers it:
- A model that hallucinates can be caught by verifying its sources.
- A vague policy that produces inconsistent calls can be sharpened with concrete definitions. A contradictory or overloaded prompt can be rewritten so the model can actually follow it.
- A sycophantic answer can be stress-tested by pushing back and watching whether it holds.
- An invisible error in a data analysis can be surfaced by treating the output as a hypothesis and checking it against the source.
Once you can see the failure modes, they stop being mysterious and start being manageable, and what's left is a tool that does real work for you: faster reviews, more consistent decisions, and coverage no human team could staff. The teams getting the most out of AI treat the model's output as a draft to interrogate, steer, and verify rather than an answer to accept. None of that requires you to become a data scientist. It just takes curiosity and skepticism — two traits that T&S and fraud professionals have in abundance.
Exercise: stress-test a real output
This takes about thirty minutes and works best on something real rather than a toy example.
- Pick a recent output. Choose an AI result you actually used or were about to act on, such as a case summary, a policy draft, or a data analysis.
- Ask the model to critique itself. Paste the output back in and ask it to list the three weakest or least supported claims in its own answer, and to explain why each one is shaky.
- Verify independently. Take two or three specific claims and check them against the source yourself, whether that's the original content, the underlying data, or the actual policy text. Don't take the model's word for any of them.
- Write down what held up and what didn't. Note which claims survived verification and which fell apart, and look for a pattern in the failures.
- Ask how to prevent it. Ask the model what change to the prompt, the policy, or your process would have caught these issues earlier, then decide whether that change is worth making.
You'll come away knowing where this tool is reliable and where it needs a human check.
For T&S and fraud leaders
Everything above applies to you personally, but as a leader you also have to manage AI literacy across a team, and that brings its own failure modes that have nothing to do with the models themselves:
- People hiding that they're using AI. When team members use AI without saying so, you lose your feedback loop, QA, and any visibility into where AI is helping and where it's introducing risk into enforcement or fraud decisions.
- Over-trusting outputs, where people take AI results at face value because nothing in the culture tells them to check.
- People not using AI at all and slowly falling behind on capability, because nothing visibly breaks when a team is simply slower and less effective than it could be.
Part of the job is building a shared mental model across a team that starts from very different places. Some people have been deep in these tools for a year, while others tried a chatbot once, hit a bad answer, and wrote the whole thing off. You can't assume a common baseline, and the kind of literacy described here has to be taught and reinforced rather than presumed. This is doubly true on mixed teams where fraud analysts, content moderators, and policy people bring different instincts about what "trusting a model" even means.
You'll also have to decide when an AI literacy gap is a development opportunity and when it's become a performance issue. Early on, gaps are normal, and the right response is teaching, examples, and time. It tips toward a performance question when someone has had real support and clear expectations and the work still isn't meeting the bar, particularly when careless AI use is creating downstream risk in decisions that affect users or money. Telling those two situations apart honestly, and not defaulting to either blanket leniency or blanket blame, is a real part of managing this well.
Exercise for leaders: make failure visible
The goal is to make catching AI mistakes a normal, low-stakes part of how your team works, so people surface their misses instead of hiding them.
- Find a real example. Pick one AI output that looked credible but turned out to be wrong or incomplete, ideally one someone on your team caught during real work. If you don't have one yet, run the individual exercise above on a few outputs until you find a good case.
- Reconstruct how it looked trustworthy. With the team, walk through what made the output convincing, such as a confident tone, plausible detail, or a fabricated source that looked real.
- Show how it broke and how it surfaced. Trace where the error actually was and what caught it, whether that was a verification step, a spot-check, or someone's sense that something was off.
- Turn it into a habit. Agree on one lightweight check the team will run on AI outputs going forward, and make sharing a caught mistake something you praise rather than something people feel they should hide.
Run this every so often with a fresh example, and catching AI errors becomes part of the team's muscle memory rather than a one-time lesson.