Why fears of AI self-improvement are causing ‘existential’ concerns at Anthropic and OpenAI
AI researchers are warning that faster AI self-improvement could eventually make advanced systems harder for humans to control.
TL;DR
- Evan Hubinger, an Anthropic alignment lead, estimates a greater than 10% chance of AI causing human extinction within a decade.
- Concerns stem from recursive self-improvement (RSI), where AI assists in its own development, leading to potentially uncontrolled capability growth.
- Both Anthropic and OpenAI acknowledge that autonomous model improvement is occurring faster than previously expected.
- AI is already accelerating its own development, with engineers shipping significantly more code per quarter.
- Researchers like Jakub Pachocki from OpenAI express concern that no one is prepared for the consequences of rapidly advancing machine intelligence.
- There is currently no viable scientific plan to solve the risks associated with recursively self-improving AI.
- Anthropic outlines three future scenarios, deeming widespread capability diffusion unlikely and continued human control plausible, but also acknowledging a scenario where humans play a diminished role in AI development.
- Business news shows surging demand for AI chips and significant investments in AI infrastructure from companies like Google and Samsung.
- Cohere, a startup founded by a co-author of the influential Transformer paper, aims to provide "sovereign" AI solutions outside of U.S. and Chinese influence.
- Aidan Gomez of Cohere described AI models as potent cyber weapons and noted the rapid evaporation of U.S. labs' lead over China.