The pipeline ran to completion. Nine sections. Roughly four thousand words of formatted output โ a technical assessment, a token supply table, a competitive landscape grid, a Howey-test breakdown with four elements and four verdicts, a six-row risk matrix with probability, impact, and mitigation columns. Every substantive cell read N/A.
It wasn't an error page. No stack trace, no timeout, no red banner. It was a finished document with correct headings and confident formatting, and underneath each heading the same sentence: information insufficient, cannot assess.
The risk matrix is the part I keep coming back to. Six categories โ technical, market, operational, regulatory, competitive, narrative. Probability: unknown. Impact: unknown. Mitigation: N/A. Composite rating: cannot be rated, classified as information-deficiency risk. The document even flagged its own condition, noting that unknown in a due-diligence context should be treated as negative until proven otherwise.
Then it did the responsible thing and refused to fabricate. It listed what it needed: a title, a source, a timestamp, five or more extractable information points, a project name.
Here's what bothered me. If that document had landed in a human inbox, it would have been read correctly โ as a failure. If it had landed in a scoring function, unknown becomes a float. Probability 0.5, impact 0.5, product 0.25. Below the 0.4 threshold. Cleared for sizing.
A report that refuses to answer is safe. A pipeline that converts refusal into a passing grade is a loaded gun with the safety painted on. I've spent enough time on the wrong side of that sentence to know which failure costs more.
Context: two-stage extraction and where the floor gives out
The architecture in question is standard for crypto research tooling in 2025 and 2026. Stage one ingests a source โ an article, a governance post, an audit summary, a Telegram leak โ and decomposes it into information points. An information point is the atomic unit: subject, action, number, timestamp. Protocol X raised $100M at a $1.2B valuation on March 4. That's one. It's independently verifiable, and it's the only thing downstream analysis is allowed to cite.
Stage two takes those points and runs them through a fixed analytical frame: technical, tokenomics, market, ecosystem, regulatory, team and governance, risk, narrative, supply-chain transmission. Every conclusion in stage two must be traceable to a numbered point from stage one. That constraint is the whole design. It exists because the alternative โ a language model free-associating about a token's prospects โ is worse than useless. It's confident, fluent, and unfalsifiable.
The design is correct. The failure was at the seam.
Stage one returned zero points. Stage two ran anyway. And stage two, being well-constructed, produced a technically accurate document about nothing: nine dimensions of cannot assess, each with a citation line reading information point list is empty. The output is a monument to good schema discipline and a total failure of process control.
I've built versions of this. In 2019 I ran a Python MEV bot against a Uniswap V2 / Kyber spread โ 4,000 fills a month, $12,000 gross. The strategy logic was fine. What killed it was a variable I hadn't put in the model: gas price volatility during a network spike. One hour in January 2020, net negative $3,500. The code didn't crash. It executed exactly as written against an input distribution I'd assumed was stationary. The bot didn't fail; the market changed rules โ and my model had no field for rules changed.
This research pipeline has the same shape. Stage two executed exactly as written against an input it had no way to reject.
The fix in 2020 was dynamic gas estimation and slippage guards. The fix here is a validator gate. But before we get to the fix, we should be precise about what actually broke, because the input was empty is the symptom, not the diagnosis.
Core: three failure classes, one shared root
Failure class one: the null-rate blindness
Any pipeline that ingests third-party text has a null rate. Percentage of fields that came back empty. In a healthy system it fluctuates โ 2% on a good day, 8% when a source publishes in images, 30% when you point it at a paywalled research note.
The null rate is a metric, and like every metric, it only works if someone is watching it. In this case the null rate hit 100% and the pipeline advanced to stage two. That means one of two things: either nobody monitors the null rate, or the monitor exists and its threshold is set above 100%, which is a polite way of saying it's decoration.
I've seen this pattern in oracle design, which is where I first learned to distrust it. A price feed that stops updating doesn't return an error. It returns its last value. Chainlink's heartbeat model means a feed can be stale by design โ the deviation threshold hasn't tripped, so the last round carries forward. A lending protocol reading that feed sees a number. The number is well-formed, correctly typed, and wrong.
That's the same bug in a different costume. A stale feed and an empty analysis look identical to the consumer, because both arrive as well-typed values. The consumer has no way to distinguish price is $2,400 from price was $2,400 forty minutes ago unless the feed carries a timestamp and the consumer checks it.
Which is why every serious integration reads updatedAt before it reads answer. And why the L2 oracle pattern matters more than people give it credit for: Chainlink ships a sequencer uptime feed alongside the price feed on Arbitrum and Optimism, precisely because a sequencer outage produces a price that is fresh-looking and unactionable. You subscribe to the uptime feed, you gate on it, and if the sequencer has been down longer than the grace period you treat the price as UNKNOWN rather than valid.
That is a production system admitting, in code, that a well-typed number can be a lie. Research pipelines haven't internalized the same lesson. They should. Every serious analysis tool needs the equivalent of an uptime feed: an explicit count of valid information points, surfaced to stage two, with a hard floor.
Failure class two: UNKNOWN coerced to NEUTRAL
This is the expensive one. In data governance there's a distinction that gets lost the moment a report becomes a table: UNKNOWN means no data was retrieved. NEUTRAL means data was retrieved and it indicates no directional signal. They are not the same state, and conflating them is the single most common control defect I've encountered in risk systems.
The mechanics of the conflation are boringly simple. A risk matrix has six rows. Each row needs a probability and an impact to compute a score. If a row has no data, the engineer has three options: leave it null and break the aggregation, assign a neutral midpoint, or fail the whole computation.
Option one breaks things, so it gets deprioritized. Option three requires a kill switch, so it gets deferred to v2. Option two ships.
And option two is correct exactly once: when missing data is genuinely random. It is never correct when missing data is correlated with the thing you're measuring. In crypto research, missing data is almost perfectly correlated with risk โ the projects that publish the least are the ones with the most to hide. An unaudited contract doesn't announce that it's unaudited; the audit section simply comes back blank.
So the neutral midpoint doesn't average out. It systematically scores opaque projects as safer than transparent ones, because transparent projects produce disclosures that can be scored as risky and opaque projects produce nothing that can be scored at all.
That's not a rounding error. That's an inverted risk function, and it runs silently.
I watched a version of this play out in May 2022. I was holding $15,000 in UST, bought during the 2021 run. When the peg started slipping, the interesting signal wasn't the price โ it was the supply mechanics. LUNA's mint-burn arbitrage had a hard capacity limit and the curve was bending. I staged out over several hours and took a 40% hit, preserving 60%. The people who took full losses weren't reading worse data than me. They were reading data that had been coerced into a neutral frame: peg is $1.00, feed says $0.98, within tolerance. The tolerance was the bug.
Same lesson from DeFi Summer, earlier and cheaper. I had $50,000 of personal capital in a leveraged ETH yield position across Compound and SushiSwap, pulling 140% APR. The number that made me pull everything wasn't my own APR โ it was a $2 million exploit in a structurally similar third-party vault in July 2020. I had no audit data on my own position. What I had was the absence of a disclosed audit on an adjacent one, and I treated that absence as negative rather than neutral. Competitors who scored the blank as fine gave back 60%. Absence of evidence is not evidence of absence, but in a risk function it should still be priced.
Failure class three: schema drift, the slow version
The third failure is the one that doesn't produce a dramatic empty document. It produces a document that's 94% full and quietly worse than it was last quarter.
Extraction schemas drift because sources drift. A news site changes its markup. A protocol moves its docs behind a JS-rendered shell. A team starts publishing metrics as images instead of tables. None of these break the pipeline. All of them shave the information point count โ 11 points becomes 8, becomes 5, becomes 4 with one of them being a duplicate.
The report still generates. The citations still resolve. The conclusions are just built on a thinner floor, and nothing in the output signals that the floor moved.
This is where I'd put monitoring effort if I could only pick one place. Not on the model, not on the prompt, not on the analysis frame โ on the extraction yield per source, tracked over time, with an alert on the derivative rather than the level. A source that used to yield 12 points and now yields 4 has a story. A source that yields 4 and always yielded 4 is just a thin source. The level tells you nothing; the rate of change tells you everything.
Alpha decays faster than the code that finds it, and pipeline decay is quieter than both.
The cost model: what a false clean actually costs
Let me put numbers on this, because it's a control issue is the kind of phrase that gets nodded at and ignored.
Suppose the risk score feeds position sizing. A common form: size = base_size ร (1 โ risk_score). Risk score of 0.25 means you take 75% of base. Risk score of 0.9 means 10%.
Now run the empty-input case. Six categories, all unknown, all coerced to 0.5. Composite 0.25. Sizing at 75% of base on a position whose actual risk profile is entirely unmeasured.
If the true risk distribution of unscored assets is bimodal โ and in crypto it is, you get audited-and-boring or unaudited-and-lethal, very little in between โ then the expected value of that 75% position is negative before you even look at the trade thesis. You've built a system that upweights the tail.
Scale it. A $500,000 book, 20 positions, average $25,000 each. A neutral-coerced sizing error of 25% on five positions where the true risk was high isn't a rounding issue; it's a $31,000 misallocation into the part of the distribution you'd have avoided with any data at all.
In April 2024 I ran ETF arbitrage against the spot approval โ backtested a 0.3% inefficiency in the first hour, executed $2 million notional, captured $6,000 risk-free. That trade worked because the preparation was specific: I knew the inefficiency existed, I knew its half-life, and I knew what would invalidate it. Six thousand dollars on two million is thin. It's only worth doing if the model is exact.
A research pipeline doesn't need to be exact. It needs to be honest about when it isn't. We optimize for edges, not comfort โ and the comfortable output here was the one that said cleared.
Case study: the unlock table that came back empty
The tokenomics section of the failing report is worth pausing on, because it's the clearest illustration of why blank is not benign.
Four rows: team, early investors, community and liquidity, treasury and ecosystem fund. Every row: N/A. No supply figure, no allocation percentage, no vesting curve, no TGE date, no inflation rate, no burn mechanism. The report correctly noted it could not assess unlock pressure.
Now imagine that document getting skimmed. The reader sees a tokenomics section that exists, is formatted, and contains no red flags. There are no red flags because there is no content. The absence of a cliff schedule reads, to a fast reader, as the absence of a cliff.
In practice an empty unlock table is the single most dangerous output in the entire report. The three-to-six month window after TGE is where supply mechanics do their damage, and it's entirely knowable in advance โ allocation percentages are in the docs, vesting terms are in the token contract, the emission curve is usually in a Medium post nobody read. If the pipeline can't extract any of it, the correct conclusion is not neutral. It's that the project either doesn't disclose or the extraction failed, and both of those are things you price.
I'd rather see a table that says team 22%, 12-month cliff, 36-month linear, than a table that says N/A. A bad schedule is a tradeable fact. A blank one is a position you're taking without knowing it.
The gate
The fix is fifteen lines. I've written it in three languages and it's fifteen lines in all of them.