The Silence in Ten Thousand Answers: An Audit of the AI Money Test

Somewhere inside a test of 121 financial questions — repeated five times each across a stable of AI models, more than ten thousand answers in total — one answer about a pension sat quietly in the data. If a real person had acted on that single sentence, the penalty could have reached £17,500 from HMRC. That detail is buried beneath a cleaner headline number: 57% failure. But Alpha hides in the silence of the audit, and the silence that matters here is not the 57%. It is the fact that the test measured answers, not accountability. No one in that study is on the hook when the pension advice ruins a retirement. The model is not liable. The platform is not liable. The person asking is. That gap — between who answers and who pays — is the real subject of the study, even though the study never once names it. The test in question came from Saturn, a UK fintech, reported by BeInCrypto. The design deserves more credit than most AI press releases ever earn: 121 questions, each asked five times, over ten thousand answers, scored against a rubric that counted arithmetic errors, missing tax-law updates, fabricated rules, and — tellingly — the absence of adequate disclaimers. The results read like a confession. 57% overall failure. 88% on the hardest questions. Free models failing 63% against paid models at 49%. And the best performer of the lot — labeled "Claude Opus 5," a version designation I would immediately flag as unverified, and which you should treat as a soft signal about the study's own care with technical detail — still failing 39%. Why does this belong in a crypto publication, and why does it belong on your desk if you hold tokens, run a vault, or let an agent trade for you? Because the exact failure modes in this study are the ones arriving on-chain first. Autonomous agents are already rebalancing DeFi positions, managing yield vaults, and executing treasury policy. The vertical that failed this test — high-stakes, fact-constrained, computationally exact — is precisely the vertical that blockchain finance claims to automate. The chatbot with your money is not a hypothetical future. It is a smart contract with a language model for a brain, and the audit is already overdue. Start with the three failure types, because they map cleanly onto three known weaknesses of large models — and each one has a different fix. Arithmetic errors reflect the absence of a reliable symbolic-computation module; a model predicting the next token is not a calculator, and the remedy is tool-calling, not more parameters. Missing tax changes reflect the knowledge cutoff — static pre-training cannot see a rule that changed last quarter, and the remedy is retrieval from authoritative live sources. Fabricated rules are hallucination in its purest form: plausible, confident, false. The remedy is verification against primary documents. Notice that none of these three fixes is "a bigger model." That is the structural insight the headline hides. This is not a scaling problem. It is an architecture-of-trust problem, and architecture is something a fund, a regulator, or a determined developer can actually redesign. Then look at the gradient between free and paid. 63% versus 49% is a fourteen-point improvement for money — real, but nowhere near the cliff you would need to call financial advice safe. On the hardest questions, the free tier fails 93% of the time. If you are an investor, this is the uncomfortable part: paying more buys reliability on a curve that flattens well before it reaches trustworthy. In my own due diligence work, I have learned to discount any vendor claim whose safety margin lives in the same sentence as its pricing tier. A 49% failure rate sold at a premium is not a product. It is a liability with a receipt. The detail I keep returning to, though, is the repetition. Five runs per question. The researchers chose that for a reason — they were measuring variance, not just error rate. And then, in the reporting, the variance vanished. We got a single failure percentage and nothing more. But for a financial decision, inconsistency can be more dangerous than inaccuracy. If a model is wrong the same way every time, at least you can learn its bias and correct around it. If it gives you five different answers to the same pension question, you have no basis for choosing which one to trust. The number we should fear is not 57%. It is the standard deviation nobody printed. Now widen the frame, because this is where the stakes compound in a way the original study, sitting comfortably inside regulated UK finance, never has to confront. In traditional finance, a wrong chatbot answer meets a slow, litigious, human world — an advisor, a regulator, a paper trail. On-chain, the answer meets execution. An AI agent that misreads a yield parameter does not get corrected by a compliance officer at the next meeting; it drains a vault before anyone finishes reading the incident report. And the same trust mismatch the study exposes in the UK general public has a crypto-native twin. Retail users assume "audited" means "safe" and "decentralized" means "fair," when neither word has ever guaranteed either. I watched this assumption fail in 2022, counseling 150 distressed investors in Rome after the FTX collapse. They had trusted systems they believed were supervised and fair. Most of them had never read a single governance document. That is the pattern I now refuse to let slide. Which brings me to the part of the study the study would rather not foreground. Saturn ran the test, and Saturn's CEO used it to call for FCA action — while declining to disclose Saturn's own failure rate. That is not a neutral research posture. It is a positioning move. If AI financial advice is brought under regulation, general-purpose model providers absorb the compliance cost, and licensed fintechs like Saturn inherit a moat built from other people's failures. I am not accusing anyone of bad faith. I am reading the incentive structure the way I read a token's vesting schedule: the loudest warning about a market's danger often comes from the party best positioned to profit from the fear. Note, too, the absence of any disclosure about Saturn's own product — the studied silence of a stakeholder with something to sell. Read the docs. Question the whisper. And the whisper, here, is the number 56% — the share of respondents who told the FCA they trust AI financial information more than they trust TV and radio. Alongside it sits 44% who wrongly believe that information is regulated. High trust, low literacy. That combination is not a maturity signal; it is a fragility signal — the same one I scan for when a protocol's community votes on a proposal without reading it, driven by sentiment rather than comprehension. EY found 49% of users already lean on AI for savings and investment decisions. PensionBee found nearly 60% would act on advice they have not verified. Demand is here. Reliability is not. The two curves have not met, and the market is pricing them as if they already had. Here is the counter-intuitive angle, and it is not the one the headline wants you to hold. The instinct is to read this study as "AI is not ready for finance." I read it as something worse: AI is ready enough to be dangerous. A tool that is obviously broken gets ignored. A tool that is right 43% of the time, fluent 100% of the time, and confident always gets adopted — and that adoption is already happening, at scale, in thousands of small decisions nobody will ever file a complaint about. The failure mode is not malfunction. It is almost-working, which is the most seductive and least auditable state a system can occupy. There is also a critique I owe the researchers on their own terms. Counting "missing disclaimers" as a failure inflates the number. A conversational model that answers conversationally is doing what it was built to do; folding that into a 57% failure rate blends "wrong" with "insufficiently hedged." That does not rescue the models — the arithmetic errors and the hallucinations stand on their own, and they are the ones that cost money — but it should make you treat the headline figure as an upper bound shaped by a subjective rubric, not a measured constant. The precision of 57% is the least trustworthy thing in a study about untrustworthy precision. Hold the number loosely. Hold the failure modes tightly. So where does this leave the person deciding whether to let an agent touch their money? Not at "never." At "not without a human in the loop and a document you have actually read." I built a Human-in-the-Loop consensus framework for an AI-crypto protocol in 2026 precisely because the efficiency and the safety curves do not converge on their own — they converge when someone designs the override, and someone keeps watching after launch. The signal to track is not the next model release. It is whether FCA moves, and whether the next version of this test publishes its methodology, its variance, and its conflicts of interest. Until a failure rate arrives with those three things attached, treat any AI number — mine included — as a starting question, not a verdict.