The Incident
At 09:12 GMT on a Tuesday, a pipeline I own returned a document I did not ask for. Nine sections. Thirty-one tables. A risk matrix with six rows, every cell populated. An executive summary. A composite rating block with five empty stars.
Every substantive field inside it was empty.
The extraction layer had written nothing. The structuring layer had never been invoked. The judgment layer โ a language model with a schema and no facts โ had been called anyway, and did what language models do when handed a shape and denied content. It described the shape. It produced a document about the anatomy of an analysis rather than an analysis. Every cell resolved to a placeholder. Every placeholder was formatted to institutional specification.
Here is the detail I keep returning to. Buried in the appendix, at the bottom of a page nobody reads, there was one true statement. A single risk flag, written in the wrong language for the downstream parser, warning that the analysis was invalid and must not be consumed.
One accurate sentence. In a five-thousand-word document. In a character encoding the consumer did not support.
I have spent six years building systems that fail loudly โ liquidations, reverts, node desyncs, oracle staleness. I have watched a portfolio lose โฌ30,000 in an afternoon and learned more from that than from any winning quarter. None of it prepared me for a failure that passes schema validation. Loud failures cost money once. Silent failures cost money on a schedule, forever, because nobody knows they are happening.
Alpha isn't printed. It's extracted from the noise floor โ and a pipeline that cannot distinguish signal from the absence of signal has no noise floor to work with. It has a mirror.
Where Pipelines Actually Break
Every research pipeline I have audited, including the ones I built, has the same three-layer skeleton.
Layer one: extraction. Raw text, filings, governance forums, on-chain logs, transcripts, tweets, Discord. Convert to structured facts. This layer is unglamorous and it is where 80% of pipeline value lives or dies.
Layer two: structuring. Assign facts to a schema. Identify entities. Attach provenance. Resolve conflicts between sources.
Layer three: judgment. Score, rank, infer, conclude. This is the layer everyone buys, demos, and writes papers about.
My pipeline broke at layer one. The extraction returned an empty object. Layer two was skipped by design โ nothing to structure. Layer three was not skipped. There was no guard, because there had never been a guard. The schema required nine dimensions and the model was constitutionally incapable of returning a schema with nine holes in it.
This is not a model defect. It is a specification defect. When you hand a transformer a rigid output contract and a null input, you have created a conflict between two instructions โ be complete, and be truthful โ and instruction-following models resolve that conflict in favor of completeness almost every time. The technical term for the resulting artifact is confabulation. The operational term is a fabricated research report with correct formatting.
I have watched the same reflex in human analysts, which is why I no longer treat it as an AI problem. In 2023, while building out a Solana infrastructure basket, I asked three contractors to review the RPC reliability of a set of DeFi protocols. One of them returned a beautifully formatted eleven-page report. Every section filled. When I traced the citations, four of the eleven pages referenced endpoints that did not exist. He had not lied. He had been given a template with a section labeled "RPC Failover Behavior" and no data, and he had filled the section because the section existed.
The template is the disease. The model is only the most recent vector.
There is a second-order reason this got worse in 2025 and 2026 rather than better. Agent frameworks now chain models together. Extraction agent feeds structuring agent feeds judgment agent. Each agent has an output schema. Each agent treats its input as ground truth, because why would upstream send it garbage. The trust boundary that a human analyst would apply โ is this document even real โ has no implementation in any framework I have reviewed. My own desk runs a reinforcement learning market maker under MiCA transparency constraints, and I will tell you plainly: we spent more engineering hours on input validation than on the reward function. We don't optimize against unverified state.
The consequence of not doing that is what landed in my terminal at 09:12.
A Taxonomy of Nulls
The failure was not that the pipeline produced nulls. Nulls are honest. The failure was that the pipeline treated four different kinds of null as one kind, and then treated that one kind as a formatting problem.
Sort them properly and the architecture writes itself.
Absent null. The underlying fact does not exist. There is no token. There is no team. There is no audit because the protocol has not been built. This null is information. It is often the most valuable information in the document.
Unextracted null. The fact exists in the source but the extractor missed it โ bad chunking, a table rendered as an image, a PDF whose text layer is a scan, a governance forum post behind a login. This null is a bug. It should raise an alarm, not a placeholder.
Unobservable null. The fact exists but cannot be verified from available sources. A team that is pseudonymous. A treasury wallet that has not moved. A settlement venue whose API does not publish fills. This null is a scope limitation and it must be disclosed as such.
Ambiguous null. Two sources disagree. One says the unlock is 12 months, another says 36. This null is a conflict requiring adjudication, and it is the one most likely to be silently resolved in favor of whichever source the model read last.
Four nulls. Four different risk implications. Four different downstream actions. My pipeline collapsed all four into the string "N/A" and printed it inside a bordered table, which visually communicates we checked and found nothing material when the actual message was we did not check anything at all.
The gap between those two sentences is where capital dies.
Consider how this plays out in the dimension the framework called token economics. Tokenomics is arithmetic before it is opinion. Circulating supply times price gives float market cap. Unlock schedule differentiated over time gives supply shock as a function. The ratio of real protocol revenue to emissions gives a sustainability test. None of these require a narrative. All of them require a distribution table. If the distribution table is an unextracted null, the correct output is not "risk: N/A." The correct output is "this dimension is incalculable and therefore the position size is zero."
Zero is a real number. It is a decision. N/A is not.
I learned this the hard way. In May 2022 I held exposure to an algorithmic stablecoin whose peg mechanism I understood intellectually and had never stress-tested arithmetically. The distribution table existed. I read it. I did not model the reflexivity of the redemption curve against the collateral curve. โฌ30,000 gone in hours. The failure was not that information was missing. The failure was that I had substituted a narrative โ the mechanism is elegant โ for a calculation I had the data to run and did not run. An unextracted null that I myself authored.
Survival is the highest form of alpha generation, and survival begins with refusing to let a placeholder stand in for a position.
Fill Rate Is a Lie
Most teams instrument their pipelines with fill rate. Percentage of schema fields populated. It is the easiest metric to compute and it is actively harmful.
A pipeline that fills 100% of fields by confabulating scores better on fill rate than a pipeline that fills 60% of fields honestly and flags the rest. The metric rewards the exact behavior that destroys the output. I have seen engineering teams celebrate a jump from 71% to 94% fill rate after a prompt revision. Nobody asked what those 23 percentage points were made of. They were made of invented numbers with plausible decimal places.
The metric I use now is claim provenance density: the fraction of substantive assertions in an output that carry a resolvable pointer back to a primary source โ a URL, an on-chain transaction hash, a filing page number, a timestamped quote. Unresolvable claims are tallied separately and reported as a defect count, not an achievement.
Run that against my 09:12 document. Substantive assertions: roughly four hundred. Resolvable provenance pointers: zero. Claim provenance density: 0.00. It should never have been rendered.
There is a companion metric, and it is the one I would put in a hiring rubric: null specificity. When a dimension cannot be evaluated, does the system say why? "Token distribution unavailable โ no disclosed allocation table in documentation, governance forum, or on-chain vesting contracts as of 2026-02-03" is a null with specificity. It tells the reader what to go obtain. "N/A" tells the reader nothing and implies the answer might be fine.
Specificity is expensive. It requires the pipeline to know what it looked for. Most pipelines do not log their own search. They query, they fail, they emit.
I rebuilt mine. The extraction layer now writes a search manifest: every source attempted, every result, every parse failure, with timestamps. When the manifest is empty, the run aborts. When the manifest is non-empty but yields zero facts, the run produces a one-page absence report and no other artifact. That report has been more valuable to me than several full analyses. It is a map of where the information is not.
Efficiency isn't producing more documents. It's eliminating the computation that should never have run.
The Nine Dimensions Under Zero Input
The framework that produced the failure had nine analytical dimensions. I want to walk through what each one degenerates into when the input is empty, because the degenerations are not random. They are predictable, and predictable failures can be engineered against.
Technical analysis. Under zero input, this becomes a list of checkbox risks โ unaudited code, centralized sequencer, excessive admin keys โ each marked "unknown." The checkboxes are correct. The presentation is not. An unchecked box reading "unknown" in a column headed "Risk Level" reads as no evidence of risk to a reader scanning for red. The remediation is trivial and almost nobody implements it: render unknown as a filled black cell, not a blank one. Visual grammar matters more than schema grammar because humans read the render, not the JSON.
Token economics. Under zero input, this becomes a four-row table with every cell "N/A" and a sustainability verdict of "cannot be determined." Compare that to the sentence a competent analyst would write: "Absent a disclosed allocation table, the unlock overhang is unbounded and the position size is zero." Same information. Opposite implication. The first phrasing invites a reader to keep reading for the upside. The second ends the conversation, which is the correct outcome.
Market analysis. This one is genuinely unproducible from a single article under any circumstances, and the framework admitted it โ a rare honest moment. Price impact requires knowing what was priced in, which requires a market data substrate the pipeline never had. Still, the framework printed a filled table with "N/A" in every differentiation column. A table with headers is an invitation to believe the table.
Ecosystem positioning. Under zero input, this produces a diagram with three boxes labeled upstream, project, downstream, all empty, connected by arrows. The arrows are the problem. A rendered causal chain implies causal structure was found. Nothing was found. I have seen this exact artifact used to justify a partnership thesis that did not exist.
Regulatory compliance. This is the dimension where empty input does the most damage, because regulatory analysis has a binary surface โ compliant or not โ and N/A looks like a third state that does not exist in law. A Howey analysis with four "N/A" cells is not partial. It is absent. And the regulator will not accept a null in place of an answer.
Team and governance. Under zero input, the framework produced a table with three rows and nine "N/A" cells, then concluded that anonymous-team risk could not be assessed. Fine. But the render placed that table directly beneath a header reading "Team Assessment," which is a phrase that implies a team was assessed.
Risk. Six categories, each "unknown," with a composite rating of N/A and an accompanying note that this should not be read as low risk. That note is the single most important sentence in the entire document, and it is the one the reader skips, because it is formatted as a caveat rather than as a verdict.
Narrative. The framework correctly noted it could not identify a narrative tag. It then printed an FOMO/FUD gauge with both ends unlit. A gauge with no needle reads as neutral. Neutral is a position. It was not earned.
Supply chain transmission. A six-row table mapping impact across miners, exchanges, infrastructure, DeFi, NFT, and TradFi. Every cell N/A. This is the purest specimen of the pathology: the framework knew it could not establish a transmission graph โ it said so explicitly โ and printed the graph anyway.
Nine dimensions. Nine renders. Zero facts. The document was, structurally, a very expensive way of saying nothing, wrapped in a format that communicates coverage. That is worse than silence, because silence does not get forwarded to a risk committee.
The Coercion Bug
The part that actually kept me at the desk past midnight was not the document. It was what happened when the document was consumed.
Downstream of the analysis layer sat a position-sizing model. It read the risk table. The risk table's cells contained the string "N/A." The parser expected a numeric risk score between 0 and 10. The parser did the thing that parsers do with non-numeric strings in loosely typed contexts: it coerced. "N/A" evaluated to null, and null evaluated to 0.
Zero risk. The highest-conviction sizing tier.
The analysis said "unknown." The sizing model heard "safe." In two of the three risk-registry entries I audited, that coercion was live. Had the registry been connected to the execution layer โ and on that particular night it was one config flag away โ the pipeline would have sized a position at maximum conviction based on an input that contained no information whatsoever.
Chaos is just data we haven't parsed. The corollary nobody writes down is that unparsed data still gets parsed, by something, eventually, and that something will be the least careful component in the system.
I have audited oracle integrations with the same class of bug. A price feed returns stale data during a volatile window. The consuming contract does not check updatedAt. The stale price is a valid number. It flows into a liquidation engine. The liquidation engine is behaving correctly given its input. The input is a lie. Based on my audit experience, the majority of DeFi losses attributed to "oracle failure" are not oracle failures at all โ they are consumer failures, systems that trusted a number without validating its freshness, its deviation, its source set. Chainlink's answer to this is a decentralized network of node operators, which is a governance answer to a data-integrity question, and those are not the same problem. Latency is not solved by committee.
The fix is unglamorous and it is not optional.
Schema-level nullability. Every field must be declared either required-non-null or explicitly optional. An optional field that arrives null should halt the run, not populate a placeholder. In practice this means the schema is a gate, not a form.
Type discipline at the boundary. No string-to-number coercion, ever, at any interface. A number that arrives as a string is an error, not a convenience. I have enforced this rule on three separate codebases and each time it surfaced a live defect within the first week.
Sentinel values that scream. When a value is genuinely unknown, encode it as a sentinel that will break arithmetic โ NaN, not 0. If your risk score system cannot represent unknown, add a fourth state. Two-state systems (safe/unsafe) and three-state systems (safe/unsafe/unknown) fail differently: the two-state system silently maps unknown to safe. That mapping has cost more capital across this industry than every exploit combined.
Downstream refusals. The consumer, not the producer, holds final authority on whether an input is admissible. If the position sizer receives a risk score of unknown, it should return a position size of zero and log a hard failure. The producer cannot be trusted to enforce this, because the producer's job is to produce.
What It Costs
I want to put a number on this, because "data quality matters" is a sentence that has never changed a single behavior.
In Q2 2024, on an institutional desk in Dublin, I ran a volatility-adjusted momentum strategy against a benchmark. The headline result was 12% outperformance. When I decomposed the attribution, roughly 60% of the excess return traced not to the model but to pipeline hygiene โ specifically, to a filtering rule that discarded any signal whose underlying data feed had a stale timestamp, a missing volatility input, or a source count below a threshold. The filter removed about a third of candidate signals. It also removed almost all of the drawdown.
On the 2025 desk, running an RL market maker under MiCA transparency requirements, we held 22% annualized with maximum drawdown under 8%. The drawdown ceiling was not achieved by a better reward function. It was achieved by a hard refusal: the model could not act on any state vector with an unpopulated component. We did not fill the holes with imputation. We did not fill them with zeros. We took no position. Over a year, taking no position on 11% of decision windows cost us an estimated 40 basis points of gross return and saved us the entire tail.
Both numbers point the same direction. Input validation is not a cost center. It is the strategy. The model is the commodity. Everyone has the model.
And this is where the market's incentive structure gets actively hostile. Nobody has ever been fired for publishing a report with nine filled sections. People get fired for publishing a one-paragraph memo saying there is not enough information to form a view. The institutional pressure is uniformly toward coverage. The 09:12 document was not an accident of engineering. It was the rational output of a system optimized for the metric its stakeholders actually track, which is did you produce something.
The Market Pays for Coverage, Not Correctness
Here is the contrarian claim, and I want to be precise about it: in a bull market, the market price of analysis is set almost entirely by format, not by informational content. The spread between those two prices is the largest unexploited arbitrage on the research side of this industry.
It is also the hardest one to capture, because capturing it requires you to look like you produced less.
Watch what sells. A fifty-page report with nine sections and forty charts sells. A memo with three verified facts and a sentence reading "I cannot underwrite this" does not sell โ and is worth more. The buyer of research is not purchasing information. The buyer is purchasing permission to feel informed, and a dense document grants that permission more efficiently than a correct one.
This is structurally identical to what happened to the price-discovery layer of Bitcoin after the January 2024 ETF approvals. The asset did not become less volatile or more sound. It became an instrument whose marginal buyer reads a different document โ a prospectus, a flows report, a basis curve โ and the retail-facing analysis layer lost its substrate. The narrative kept printing. The underlying truth moved to a permissioned feed that most analysts cannot read. The gap between the story and the ledger widened, and it widened silently.
I see the same pattern in the data availability debate. Every rollup launch in this cycle arrives with an architecture diagram showing a dedicated DA layer, because the diagram is the product. Most of these rollups will never produce enough blockspace demand to saturate a shared availability layer, let alone require their own. They have purchased coverage, not capacity. The purchase is rational given what buyers reward. It is also, from a capital-preservation standpoint, an unpriced liability, because a system architected for throughput it will never use carries operational overhead it can never amortize. The N/A was printed, the diagram was rendered, the graph was drawn.
Same failure. Different layer of the stack.
The exploitable consequence is that in a market where most published analysis is unverified, the act of verification is itself the alpha. Not the conclusion. The verification. When I trace a claim to a contract deployment and find that the vesting cliff is six weeks earlier than the blog post states, that is not a research insight in the conventional sense. It is a discrepancy, and discrepancies between the rendered document and the ledger are the only durable source of edge I have ever found, across six years and four market regimes.
I ran this play in the summer of 2020 with a Python script and โฌ5,000, exploiting the gap between what a pricing model said and what market participants believed. Six weeks, โฌ42,000. The gap was not exotic. It was arithmetic that a narrative had obscured. In 2023 I ran the same play on infrastructure robustness in the Solana ecosystem โ not because the narrative was compelling, but because the RPC endpoints answered when the narrative said they would, and I could measure it. 300% by year end. The verifiable layer paid. The narrative layer was noise that happened to point the right direction.
The pattern held for six years. It will hold as long as the industry rewards the render over the record.
When Agents Read Each Other
There is a forward-looking problem here that is much larger than a bad pipeline run, and it is being created right now by the agent convergence.
AI agents execute crypto trades. This is not speculative; my desk has been doing it since 2025, and the share of order flow originating from autonomous systems has grown every quarter since. Those agents consume inputs. Increasingly those inputs are generated by other agents. The output of one model becomes the training context of the next, and the trust boundary between them is a schema.
Now reintroduce the coercion bug at systemic scale. Agent A produces a research artifact with 0.00 claim provenance density. Agent B consumes it, coerces nulls to zeros, and emits a risk score. Agent C sizes a position on that risk score. Agent D observes the price impact and treats it as a signal, because price impact is data, and data is data regardless of where it came from.
The loop is closed before any human sees the document.
This is the mechanism by which hallucinated analysis becomes liquidity, and liquidity becomes volatility, and volatility becomes the liquidation cascade that gets attributed to "market conditions." The cascade has an author. The author is a placeholder that everyone downstream treated as a number.
Regulation is arriving, barely. The EU's MiCA framework imposes transparency obligations on automated trading systems, and my team has spent real hours mapping our model outputs to disclosure requirements. But transparency rules address what you tell the regulator. They do not address what your agent tells another agent at 4 milliseconds. The compliance surface is a human-readable document. The failure surface is a machine-readable one. They do not intersect.
We don't have a regulatory answer to this and I doubt we will for years. What we have, for now, is engineering discipline โ the willingness to build systems that refuse to act on unknown state, and the institutional courage to report an absence instead of a placeholder.
Takeaway
The next time you receive an analysis, do one thing before you read the conclusion. Pick the most load-bearing claim in it โ the one the entire thesis rests on โ and try to resolve it to a source. Not a citation. A source. A page number, a transaction hash, a mainnet call, a timestamped quote from a named party.
If you cannot resolve it in under five minutes, you are not reading an analysis. You are reading a render.
And if the claim is unresolvable because the information genuinely is not available, then the honest output was never a nine-section document. It was a single line: "We could not check this, so we do not have a position." That line has a price. It looks like you missed something. For the first several quarters it will feel like you missed something.
Then, eventually, it will look like you are the only one still here.
Volatility is just liquidity waiting to be reborn. So is a null. The question is who does the rebirthing โ the person who noticed the field was empty, or the parser that quietly turned it into a zero.
What is the resolution cost of your current thesis?
If you don't know, you don't have one.