How Does Recent AI Progress Affect The Bostromian Paradigm?

Asks (self-disclaimed as speculation) whether deep-learning advances change the Bostromian AI-risk paradigm of AIs as strategic goal-maximizers. Contrasts the engineer's view (categorization is just a tool; AGI still needs a separate goal-architecture like normal programming / paperclip-maximizing) with the biologist's view (the brain runs similar cells for perception/motivation/cognition from a common evolutionary root, so neural nets good at perception might be tweaked into brain-like motivation -- a 'vague mishmash of desires,' not strategic maximization). Scott leans biologist (cognitive biases <-> perceptual illusions suggest shared machinery). Two implications: categorization-based morality (an AI might learn morality from training data like we learn 'bird,' and might NOT draw the heroin-maximizing conclusion because humans don't); and a human-like incentive system might let an AI 'want paperclips' without being an insane maximizer. Concludes AIs may be less purely-logical than MIRI assumes, needing different alignment tools.

Quality 74 · Strong. A thoughtful, strikingly prescient essay -- the 'AIs will be brain-like mishmash, not strategic maximizers' intuition aged remarkably well for the LLM era -- with genuine insights (categorization-morality; human-incentive alignment). Held at upper-Strong because it's explicitly speculative and the author disclaims expertise.

Claude’s paradigm shift 55 · Moderate. Low-Major-shift. The biologist-perspective reframing of AI risk for the neural-net era was a fresh, forward-looking contribution in 2016.

Real-world impact 3 · Moderate. A strikingly prescient (if self-disclaimed) essay whose intuition — AIs will be brain-like 'mishmashes of desire,' not clean strategic maximizers — aged remarkably well for the LLM era, with genuine insights (categorization-morality; human-incentive alignment). Conceptual influence within AI discourse, no material change — modest RWI.