Motivated Reasoning As Mis-applied Reinforcement Learning
Read the original on Astral Codex Ten →
Follow-up reading
↳ Highlights From The Comments On Motivated Reasoning And Reinforcement Learning — Highlights (companion) · Feb 2022
Summary
A short model (from the Yudkowsky-Ngo dialogue): brain regions are either 'behavioral' (improved by hedonic reinforcement learning) or 'epistemic' (which RL would ruin -- e.g. the visual cortex must NOT learn to stop recognizing lions). The distinction is fuzzy: some behaviors are 'epistemic behaviors' (head-turning to check for a lion; opening your budgeting app) that carry RL-reinforceable dread -- the 'ugh field.' Motivated reasoning = running epistemics on partly-reinforceable architecture, so checking whether your political program worked gets down-weighted like a lion-check.
Why this score
Quality 68 · Strong. Strong: a clean, elegant, genuinely original short model (behavioral-vs-epistemic brain architecture) that reframes motivated reasoning memorably; kept mid-Strong by its brevity and explicitly conjectural framing (Scott calls it 'almost tautological').
Claude’s paradigm shift 52 · Moderate. Moderate: framing motivated reasoning as mis-applied reinforcement learning on the wrong brain architecture is a fresh, non-obvious model.
Real-world impact 1 · Negligible. Negligible/within-community: a conjectural neuroscience-of-cognition note; no material footprint.