
Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied , “Jacob is correct here—we really do earnestly b
Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied , “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN , “I agree with Jacob much more than I disagree with him.”
These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.
These ideas have prompted varied reactions. Some argue that the concept of ASI is being used for marketing or to instigate regulation that will help the incumbent labs squash new competitors. Others focus on the plausibility of such systems killing us all, debating the finer points of specific scenarios such as AI-engineered superviruses. Still others claim that aligning ASI to human interests is feasible and so we should rush ahead to usher in this digital utopia.
Amid these conversations, one question has received comparatively little attention: From an engineering perspective, is superintelligence even possible? The answer is important because it can help us calibrate our anxieties and develop the most effective possible responses to this emerging technology.
I recently explored this topic in a computer science doctoral seminar that I taught at Georgetown University, in Washington, D.C., in which my students and I examined the technical and cultural roots of the superintelligence story. (For those who are interested, we published an annotated bibliography of our class readings.) I came away with several key objections to the feasibility of ASI, which I’ll detail below, as well as a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm.
RSI is an old concept. In a 1965 paper , statistician I.J. Good speculated that a rapid cycle of AI self-improvement could lead to an “intelligence explosion.” The idea gained traction in the 1990s among the Extropians , a Bay Area futurist group, where it offered a way to discuss ASI without having to address the engineering details of how one would build such a system. The concept was taken up by such notable figures as Eliezer Yudkowsky (who later founded the rationalism movement) and Nick Bostrom (who helped steer effective altruism to focus more on existential risks to humanity).
Recently, however, this topic has shifted from thought experiment to practical engineering within both Anthropic (makers of Claude) and OpenAI (makers of ChatGPT). In a June blog post titled “When AI Builds Itself,” Anthropic described how LLM-powered agents are speeding up the writing, debugging, and optimization of code, and also automating parts of research such as running experiments and logging results. The company’s internal data, the report concluded, “is showing that AI is already accelerating the development of AI systems.” OpenAI has been similarly explicit about its intentions; in a September report, the company announced a goal of developing an automated researcher by March 2028.
But do these efforts put the companies on a trajectory toward uncontrollable RSI? Several obstacles undermine this claim. Consider the availability of the data needed to improve AI systems. Through 2023, major advances in LLMs came from scaling up the size of the models. The original GPT-1 had 117 million parameters (values learned during training that shape how the neural network processes information). This quantity jumped to 1.5 billion for GPT-2, to 175 billion for GPT-3 , to (likely) more than a trillion for GPT-4’s release three years ago. After GPT-4 , however, continued scaling began yielding diminishing returns . This led engineers to increasingly turn their attention toward “post-training” phases, where techniques from reinforcement learning are applied to improve an already trained model’s performance on specific tasks. This approach rewards a model for producing successful responses, helping it better produce similar answers in the future. It requires, however, lots of examples of relevant inputs paired with correct outputs.
IEEE Spectrum AI
Original Reporting Source
Explore features, ratings, and alternatives from our 19,000+ AI tools catalog.

OpenAI's flagship conversational AI platform featuring GPT-4o, advanced voice mode, web browsing, DALL-E 3 image generation, and custom GPTs.