Kahneman, Sibony & Sunstein — Noise: A Flaw in Human Judgment

The Source

Daniel Kahneman's 2021 book, co-written with legal scholar Cass Sunstein and decision-science consultant Olivier Sibony. Its central move is separating error into two distinct components: bias (systematic deviation in one direction, the subject of Kahneman's earlier work) and noise (random, unwanted variability between judgments that should be identical). The book's core claim: organizations obsess over bias and almost completely ignore noise, even though noise is often the larger and more fixable source of error.

1. Finding Noise: Bias vs. Noise

• The target/shooting-range metaphor — a biased team's shots cluster off-center in a consistent direction; a noisy team's shots scatter randomly around the target with no consistent direction at all. A team can be biased without being noisy, noisy without being biased, or (most commonly in real organizations) both at once.
• The key insight: noise is measurable without knowing the right answer — if you only see the backs of the targets (no bull's-eye marked), you still can't tell who's biased, but you can immediately see who's noisy just from how spread out the shots are. This matters because most real-world professional judgments (a diagnosis, a sentence, a hiring decision) don't have a knowable "correct" answer to compare against — but the scatter between different professionals judging the same case can still be measured directly.
• The sentencing origin story — Judge Marvin Frankel's 1970s campaign against sentencing disparity is the book's founding case study. Controlled studies giving different federal judges identical hypothetical cases found "astounding" variation: a heroin dealer's sentence ranging from one to ten years depending purely on which judge got the case; in a 208-judge study, unanimous agreement on whether prison was warranted at all occurred in only 3 of 16 cases.
• Noise from genuinely irrelevant factors, documented directly — judges have been shown to be measurably harsher after the local football team loses over the weekend (an effect that fell disproportionately on Black defendants in one large study); French judges granted more leniency to defendants sentenced on their own birthday; immigration judges granted asylum less often on hot days.
• Noise is everywhere professional judgment is used — the book documents substantial, measured noise in medical diagnosis (including radiology and pathology, not just "soft" fields like psychiatry), child-custody placement, forecasting, asylum decisions, hiring and performance ratings, bail decisions, and even fingerprint and forensic analysis, which is popularly assumed to be near-infallible.

2. The Anatomy of System Noise

• The noise audit — the book's core diagnostic tool: have many professionals independently judge the same set of cases, then measure how much their judgments differ from each other. Run at a large insurance company, a noise audit on underwriters (who were expected to set nearly identical premiums for the same risk) found variation the company's own leadership found shocking and expensive — described in the book as "shattering the illusion of agreement."
• System noise splits into level noise and pattern noise — level noise is the variability in how harsh or lenient a given judge is on average across all cases (functioning almost like a stable personality trait). Pattern noise is the variability in how a judge responds to a particular kind of case relative to their own average — one judge harsher than usual on repeat offenders, another more lenient toward accomplices, in idiosyncratic patterns that don't cancel out even though they're individually consistent. In the original judicial noise audit, level noise and pattern noise contributed roughly equally to total system noise.
• Occasion noise is the noise within a single person — the same professional, given the identical case on a different day, in a different mood, or after a different sequence of prior cases, will not reliably reproduce their own earlier judgment. Documented directly in the book: wine judges at a major competition matched their own earlier scores for the same wine only 18% of the time; software developers estimating the same task's completion time on two different days differed by 71% on average.
• Free throws as the intuitive analogy — even the best professional basketball players don't make every free throw despite identical conditions each time; the book uses this to argue that variability in professional judgment isn't a sign of incompetence, it's an expected feature of any repeated human performance — the question is how much of it there is and whether it can be reduced.

3. How Groups Amplify Noise

• Group discussion often makes noise worse, not better — the book leans on the well-documented finding of group polarization: after people with a shared initial leaning discuss an issue together, they typically end up more extreme in that same direction than any individual member started out, largely because of one-sided information exchange (each person hears mostly arguments supporting the group's initial lean) and social pressure to align with the group.
• The practical implication for meetings and juries — a group's collective judgment can depend heavily on essentially arbitrary factors — who happens to speak first, what the initial anchor in the room is, which members are more forceful personalities — making the group's final number or verdict itself a noisy, hard-to-predict outcome, contrary to the common assumption that group deliberation reliably averages out individual error.
• The book's proposed fix connects directly to decision hygiene — gathering independent judgments before group discussion (so people aren't anchored by whoever spoke first) and only then convening to discuss, rather than starting with open-ended group deliberation, is presented as the single most effective structural fix for group-amplified noise.

4. Objective Ignorance & the Valley of the Normal

• Objective ignorance is a separate concept from noise or bias — it's the hard limit on how accurate any prediction could possibly be, given how much of the future is genuinely unknowable (not just unknown to the particular judge). The book's example: a new hire's career outcome depends on countless future events — a supportive manager, a lucky break, an unrelated personal setback — that no evaluator, however skilled, could have known about at hiring time.
• People reliably underestimate their own objective ignorance — when executives were informally asked how often their own past predictive judgments (e.g., ranking two job candidates) turned out correct, typical answers clustered at 75–85% confidence; the book cites real personnel-selection research putting actual predictive accuracy closer to 59%. The gap between felt and actual accuracy is itself the book's central point about overconfidence in prediction.
• Tetlock's pundit research, cited directly as corroboration — political forecasting research spanning two decades found expert pundits' predictions about major political events barely beat chance, with the most theoretically confident pundits performing worst of all — consistent with the book's argument that confidence in a prediction and the prediction's actual accuracy are almost unrelated once you get past a certain point.
• The valley of the normal — most of daily life unfolds within a narrow, expected range where nothing feels surprising even though very little of it was actually predictable in advance; it only feels predictable in hindsight because it stayed within familiar bounds. This is offered as the explanation for why people systematically overrate how much they could have foreseen ordinary events, right up until something falls outside the valley and gets labeled a shock.

5. Decision Hygiene

• The handwashing analogy, stated directly in the book — unlike debiasing (which targets a specific, identifiable, directional error), decision hygiene techniques reduce an unspecified range of possible errors without ever telling you which particular mistake they prevented — the same way handwashing prevents illness without anyone ever knowing which specific germ was avoided. The book is explicit that this makes decision hygiene a thankless, low-visibility discipline that's chronically under-practiced even by people who intellectually accept its value.
• Named decision hygiene strategies from across the book: sequencing information so early impressions don't contaminate later, independent judgment (structured process using assessments made independently before discussion), aggregating multiple independent judgments (the "wisdom of the crowd" effect, but only when judgments are genuinely made independently first), using a common outside-view frame of reference (comparing a case to a relevant reference class rather than judging it in isolation), and structuring complex judgments into separate mediating assessments rather than one global gut call.
• A real forensic case as the stakes — the 2004 Madrid train bombing fingerprint misidentification of Brandon Mayfield (an Oregon lawyer wrongly linked to the attack, detained, and later paid a $2 million settlement) is used as a concrete illustration of how even supposedly "exact" forensic disciplines are vulnerable to noise and bias when examiners aren't shielded from irrelevant contextual information before making a judgment.

6. The Mediating Assessments Protocol

The book's own synthesis of its decision-hygiene techniques into one practical, general-purpose procedure — useful as a genuinely adoptable checklist rather than just a set of separate principles.

• The six-step protocol, stated directly in the book: (1) structure the decision into separate mediating assessments up front, rather than one holistic judgment; (2) ground each assessment in an outside view/relative comparison wherever possible, rather than an isolated absolute rating; (3) keep the assessments genuinely independent of one another during analysis, so an early impression on one factor doesn't bleed into the next; (4) review each assessment separately in the decision meeting, rather than discussing the whole case at once; (5) have each participant give their individual judgment on each assessment before any group discussion, then use estimate-talk-estimate (individual estimate, then discussion, then re-estimate) rather than open-ended deliberation from the start; (6) delay the final intuitive, holistic judgment until after all the assessments are in — not banned, just moved to the end of the process instead of the start.
• Why this order matters — the book's argument is that intuition isn't the enemy, premature intuition is; letting a holistic gut read form before the structured assessments are gathered anchors and distorts everything that follows, but the same intuitive judgment, applied last as a final check after the structured process, adds real value rather than noise.

7. Optimal Noise: Dignity & Rules vs. Standards

• The book explicitly does not argue for zero noise — it's clear that eliminating noise entirely is sometimes infeasible, sometimes too costly, and sometimes in real tension with other values worth protecting.
• The dignity objection, taken seriously rather than dismissed — people often want an individualized hearing from an actual human being who considers their specific circumstances, even knowing that discretion produces noise; a rigid rule that eliminates a person's chance to be heard can feel like being treated, in the book's own phrase describing the Supreme Court's language, as part of "a faceless, undifferentiated mass." The book uses a 1974 case (a school's mandatory unpaid leave for pregnant teachers, struck down for denying any individualized medical assessment) to show this isn't a fringe concern — it has real constitutional and moral weight.
• The book's own resolution, stated plainly — the interest in individualized treatment is real and shouldn't be waved away, but if that treatment reliably produces serious, documented unfairness (as noisy sentencing and noisy hiring demonstrably do), the burden should be on defenders of open-ended discretion to show the dignity benefit outweighs the measured cost, not the reverse. Where a noise-reduction rule is "crude" (ignoring genuinely relevant individual factors), the right fix is a better, more nuanced rule or guideline — not abandoning structure altogether and returning to unconstrained discretion.
• Rules vs. standards, as a genuine trade-off rather than a simple answer — rigid rules eliminate noise most completely but can't adapt as values and circumstances change (the book's example: a firm rule defined today may look wrong once social norms shift, with no built-in mechanism to update it case by case). Standards preserve flexibility and the capacity for evolving judgment but reopen the door to noise. The book frames most real institutional design as a search for the right point on this spectrum, not a binary choice.

8. Recognition-Guide Connections

• A genuinely new addition, not just a mechanism restatement: the book gives a clean, honest way to evaluate one's own snap judgments about a person or situation — asking "how much of my read on this is level noise (my own general mood/disposition today) versus pattern noise (something genuinely specific to this case) versus signal (something real)?" is a sharper self-check than the existing "notice shifts, don't diagnose" language, because it names the specific sources of unreliability rather than just flagging that confidence can be wrong.
• Directly useful for evaluating claims made by others, including manipulative ones: a persuader's confident, detailed read of a situation (a cold reader's "I can tell you're going through something") trades on the listener not distinguishing real signal from occasion noise in the reader's own perception — the valley of the normal material explains why vague, broadly-true statements feel eerily specific after the fact.
• Directly relevant to the existing coercive-questioning material: the finding that judges are measurably harsher after their local team loses, or more lenient on a defendant's birthday, or affected by outdoor temperature, is a strong, concrete addition to material about how even trained professionals in high-stakes settings are influenced by factors that have nothing to do with the case in front of them — worth citing directly alongside the existing interrogation-context material.

9. Coaching-Curriculum Connections

• Module 2 (Understanding the Other Person): the level noise / pattern noise / occasion noise breakdown is a genuinely useful upgrade to how a coach evaluates their own read of a client session — was today's harsher or softer assessment of a client's progress about the client, or about the coach's own occasion noise (mood, what happened right before the session, which client was seen right before this one)? Worth adding as a direct, concrete self-check alongside the Kahneman page's WYSIATI material.
• Directly applicable to the business's own client outcome tracking: the mediating assessments protocol's six steps are a clean, adoptable structure for any recurring evaluative decision the business makes about clients or programs (e.g., assessing whether a coaching approach is working) — structure the assessment into separate factors, judge each independently before discussing, and hold the holistic gut call until last.
• Module 1 (Self-Command): the dignity chapter's core tension — structure reduces error but can feel impersonal, while individualized attention feels respectful but is noisier and less reliable — maps directly onto BUSINESS-GUARDRAILS.md's lifestyle-over-scale stance. A small, high-trust client roster is explicitly the version of this trade-off the business has already chosen: fewer clients, more individualized (and yes, less standardized) attention, deliberately accepting some "noise" in exchange for the kind of person-to-person quality that a scaled, rule-bound operation couldn't offer. Worth naming that connection directly in the guardrails document as an intentional, examined choice rather than an unexamined default.

My Notes

Open Questions