Against The Generalized Anti-Caution Argument
Read the original on Astral Codex Ten →
Summary
Dissects the 'generalized anti-caution argument' — the pattern 'they warned X would happen at level N and it didn't; they warned at N+1 and it didn't; therefore caution is discredited.' The toy example is a doctor warning about escalating experimental-drug doses (safe at 100, 250, 500 mg... then 1000 mg kills): 'maybe the thing that happens eventually will happen now' is not a failed prediction. He applies it to Putin escalation (each weapon a ~2% straw-that-breaks-the-camel's-back, so three safe deliveries barely update), Biden's dementia (~4%/year, so false alarms shouldn't lull you), and AI risk (SB1047's 10^25 FLOPs threshold). The crucial distinction: for hazards that must arrive with rising probability (death, drug toxicity, dementia, AI capability), caution is justified and repeated 'false alarms' don't discredit it; for fixed-hazard claims (Republicans-overthrow-democracy each term), you hold a fixed prior and update only on evidence, losing trust in an alarmist in proper Bayesian proportion to how extreme and wrong their claims were.
Why this score
Quality 75 · Excellent. Excellent band, low. A clean, important epistemics contribution that names a common fallacy and supplies the precise Bayesian conditions under which repeated false alarms should or shouldn't discredit a warning; broadly applicable (AI risk, geopolitics, health, politics). In the tier of his strong rationality essays.
Claude’s paradigm shift 48 · Moderate. Moderate. Extends his 'argument against updating on dramatic events'; the named 'generalized anti-caution argument' framing and the rising-vs-fixed-hazard distinction are a fresh, sharp contribution.
Real-world impact 2 · Minor. A clean, important epistemics contribution that names the 'generalized anti-caution' fallacy and supplies the precise Bayesian conditions under which repeated false alarms should or shouldn't discredit a warning — broadly applicable (AI risk, geopolitics, health). Conceptual influence within epistemics discourse, no material change — low RWI.