When the Audit Disagrees with the Source

The lesson

A confident audit can produce a non-fix. The source of truth — the actual artifact, output, or behavior — is more reliable than any audit’s confidence in describing that artifact. When the audit and the source disagree, the source wins. Always.

Analogy — RAILWAY

A railway dispatcher (the audit) tells the engineer “block clear, proceed.” The track-circuit (the source) reports red. Standing rule across every railway since 1900: track-circuit wins. Always.

The dispatcher can be wrong about block status — a train they don’t know about, a fractured rail, a wire broken in the dispatcher’s office, a shift handover that forgot to mention the work crew. The track-circuit measures the actual condition (block voltage continuity); it doesn’t care about the dispatcher’s belief. Engineers who throttle past a red signal because the dispatcher cleared them are the ones who get fired or killed in the after-action report. Engineers who stop at red despite the dispatcher’s clearance are the ones who survive long careers.

The audit is one frame removed from the rails. The rails are the truth.

Historical anchor — Theranos, 2003–2018

Theranos was founded in 2003 by Elizabeth Holmes and raised over $700M, peaking at a $9B valuation in 2014. Multiple confident audits cleared the company over a decade:

Walgreens diligence team (2013): approved a 40+ store rollout of Theranos blood-testing services — without ever testing the device against a reference lab.

Theranos board: Henry Kissinger, James Mattis, George Shultz, Bill Frist, Sam Nunn — bipartisan establishment credibility, none with diagnostic-device expertise.

Cleveland Clinic (2015): explored partnership; preliminary audits did not surface device failures.

Major media: Fortune cover (June 2014), Forbes “youngest self-made female billionaire” (2014), Inc. cover, Glamour Woman of the Year. Each story audited the founder narrative, not the device output.

Each audit measured proxies — corporate process, demo polish, financial filings, board credentials, founder presentation. The source — the actual blood-test results coming off the Edison device — went unaudited until Tyler Shultz, a 23-year-old research engineer at Theranos and grandson of board member George Shultz, source-checked the lab’s quality-control data in 2014. He saw the device’s results were wildly inaccurate. He filed an internal complaint and was attacked by the company’s lawyers. He was the source-check.

WSJ reporter John Carreyrou broke the story October 16, 2015, working from internal documents that Tyler Shultz and other whistleblowers (Erika Cheung) had risked their careers to source-check.

By 2018: SEC fraud charges against Holmes and Sunny Balwani (March 2018); $9B valuation evaporated; Walgreens settled lawsuit for ~$30M; Theranos formally dissolved (September 2018). Holmes convicted on 4 counts of fraud (January 2022) and sentenced to 11 years 3 months (November 2022). Balwani convicted on 12 counts (July 2022), sentenced to 12 years 11 months (December 2022).

Cost of trusting audits: ~$700M lost capital + an estimated tens of thousands of inaccurate blood tests run on real patients over years.

Cost of source-checking: a 23-year-old’s afternoon reading the QC data.

Sources

• Theranos — Wikipedia
• Tyler Shultz — Wikipedia
• Bad Blood: Secrets and Lies in a Silicon Valley Startup — John Carreyrou (2018)

Plain-language explanation

The voice journal upload was hanging deterministically. A confident audit said the cause was a missing filename in the form-data upload. When we opened the actual code, the filename was already there. The audit was wrong — not because the auditor was careless, but because the auditor was one frame removed from the source. We almost shipped a non-fix that would have changed nothing.

The lesson: when someone hands you a confident diagnosis, look at the actual thing they’re describing — before you act. The source of truth always beats the audit’s confidence. This is true at every scale: a 30-second source-read inside a build session, or a 23-year-old reading Theranos QC data in 2014.

Cross-references

This lesson extends three prior concepts in the program:

• “Measure the Thing, Not Its Shadow” — track-circuit measures block integrity, not dispatcher belief about block status. Same family — measurement vs. proxy. New mechanic in this lesson: the audit itself is the proxy; the source is the underlying truth.
• “Audit Across Frames” — the author cannot see assumptions baked into the artifact at the level a stranger can. The cross-frame audit is the right tool; the failure mode is acting on the audit’s hypothesis without verifying its premise.
• “Layered Defects (Iceberg Under the Fix)” — a fix only verifies the layer it actually exercised. The audit is itself a layer — and it inherits the same iceberg risk. Two layers of source-check needed: the bug’s existence AND the bug’s described cause.

This is the fourth instance of the source-of-truth-vs-proxy family in the program.

Reader prompt

Where in your current life are you trusting an audit instead of source-checking?

A research summary you didn’t read the underlying paper for. A market metric you didn’t decompose. A self-report on training compliance you didn’t cross-check against the hours actually logged. A recovery score from a wearable you didn’t compare to how you actually feel walking up stairs. A coach’s assessment, a doctor’s read of a referral note, an analyst’s summary, a board’s diligence — which audit are you trusting that you’ve never actually source-checked?

Pick one. Source-check it this week.

---

This is a Forge Finding — a business lesson distilled from one Forge build session. Forge sessions ship code; Forge Findings ship the principle behind the code.