Data-quality risk: what a dashboard number can hide

Data-quality risk is the risk of reaching a wrong conclusion from a correctly displayed number. Very little in this field is measured the way a thermometer measures temperature; most figures are the output of a definition, an inclusion rule and a piece of code that someone wrote and can change. The failure is quiet, because a dashboard renders an estimate and an assertion in exactly the same typeface.

Some figures are measured and some are asserted

Circulating supply is the clearest case. A chain records the total units in existence, but circulating supply requires a judgment about which units are excluded as locked, reserved, burned or held by the issuing entity, and there is no universal standard for that judgment. In practice much of it is self-reported by the project and accepted by data providers with varying verification. Because supply is the multiplier in market capitalization, a supply assumption propagates into every ratio that uses it, and a difference in exclusion policy is enough to make two sites disagree about size without either being wrong by its own definition. Fully diluted valuation avoids that particular judgment by using total or maximum supply, at the cost of pricing units that may not exist for years, which is why FDV to market cap is worth reading as a measure of how much of the disagreement is about future supply. The related figure circulating supply is presented on this site with its source stated for exactly this reason.

Fee and revenue figures depend on an adapter

There is no general ledger for a protocol. Fee and revenue figures are produced by an adapter: code that reads specific events from specific contracts and classifies the value flowing through them. Every property of that adapter is a decision. Which contracts are included, whether new versions were added, whether value routed through an aggregator is attributed here or elsewhere, whether payments to liquidity providers count, and whether value extracted through transaction ordering is counted at all. When an adapter is corrected, the entire history changes, which is why a chart can appear to revise the past.

Value locked, volume and the ways each is inflated

Total value locked has three well-known distortions. It double counts when an asset deposited in one protocol is represented by a receipt token deposited in another, so the same underlying value is counted at each layer. It is denominated in fiat terms, so it falls when prices fall even if not a single unit has been withdrawn, which means a declining TVL chart may show a price move rather than an exit. And its boundary is a definition: whether staked assets, bridged assets or borrowed collateral count varies by source. Volume has a different problem, which is incentive. Wash trading costs little on venues that rebate fees or list tokens by reported activity, so unaudited venue volume is the least reliable widely published figure in the sector, and on-chain volume routed through aggregators can be counted more than once as it passes through several pools.

Revisions, backfills and the bias in every backtest

Historical series in this field are not stable. Data is backfilled when a new source is integrated, revised when a definition changes, and reconstructed when an indexing pipeline is rebuilt, so a figure recorded today may differ from the figure that the same query returned last month. Two specific analytical hazards follow. Look-ahead bias occurs when a study uses a value that was not yet available or not yet in its final revised form at the date being tested, which flatters any strategy built on it. Survivorship bias is more severe here than in most asset classes, because a large number of assets have gone to effectively zero and been removed from listings entirely, so a universe assembled from currently listed assets has quietly deleted its failures. Any historical claim about this asset class is worth checking against how the universe was assembled and whether the values were point-in-time.

Ratio hygiene, and the equity words that do not transfer

Ratios such as market cap to fees resemble a price-to-earnings ratio and differ in ways that change their meaning. Fees are not earnings: no operating costs, no compensation and no tax have been deducted, and there is no accrual accounting or audited statement behind them. The denominator is often an annualization of a short window, so a period containing unusual activity is projected across a year. Most importantly, a token holder generally has no residual claim on those flows, whereas a shareholder holds a legal claim on the residual value of the enterprise. A low ratio therefore does not carry the meaning it carries in equity analysis, and the same is true in reverse. The safe reading is that these ratios compare usage to price, which is useful, and that they are not valuations of a claim on cash flows, because there is usually no such claim.

Personal security closes this track. Methodology states how each figure on this site is constructed, data sources names where each comes from, and the metric catalog gives the definition and the caveats for every figure named above.