Anthropic Partners with NSA Amid Concerns Over AI Self-Improvement

Anthropic's recent actions have sparked significant debate within the AI community and beyond. The company has reportedly assigned around six engineers to the National Security Agency (NSA) to help deploy its Mythos AI model for possible offensive cyber operations, targeting networks in nations including China and Iran. This partnership arises as Anthropic also calls for a global halt on AI development, citing concerns over the technology's swift evolution towards recursive self-improvement.

The NSA Partnership

The Financial Times reports that the engineers embedded within the NSA are responsible for tailoring Mythos for specific uses, though the degree of their involvement in active operations remains unclear. Anthropic has opted not to publicly release this model due to risks of misuse. It is part of a restricted coalition known as Project Glasswing, which includes major tech firms like Microsoft, Apple, and Amazon, all of which have been vetted for access to Mythos.

Anthropic's collaboration with the NSA occurs amid tensions with the Pentagon, following a $200 million contract that fell through after the Defense Department labeled the company as a supply-chain risk. This designation, typically applied to foreign adversaries, reflects the Pentagon's concerns regarding Anthropic's refusal to permit its AI, Claude, for fully autonomous weapons or domestic surveillance. While the NSA contract is exempt from these restrictions, the implications of such partnerships raise ethical questions about the use of advanced AI in military operations.

A Call for Caution

On the same day as the NSA report, Anthropic's internal research institute released a study titled "When AI Builds Itself." The report stresses the urgency of establishing a global moratorium on AI development, likening it to Cold War-era nuclear treaties. Researchers, led by Marina Favaro and co-founder Jack Clark, warn that AI development is progressing toward a scenario where AI systems autonomously design and train their successors, significantly diminishing human involvement.

The report highlights a notable statistic: Claude now produces over 80% of the code in Anthropic's production codebase, a sharp rise from the low single digits before the launch of Claude Code in early 2025. This shift underscores the rapid evolution of AI, raising concerns about a future where AI could function without human oversight. In one instance, Claude agents were assigned to tackle a complex AI safety issue and showed impressive capabilities, recovering 97% of the performance gap between models—far exceeding the 23% recovery achieved by human researchers.

The Implications of Self-Improvement

Anthropic's concern centers on reaching a point where AI not only performs tasks but also evaluates the validity of its own experiments. This situation could lead to compounded misalignments across generations of self-improving AI, creating risks that may be irreversible. To address this, the company is advocating for a verifiable global pause in AI development, stressing that a coordinated effort among leading AI labs is crucial for ensuring safety and accountability.

Despite these warnings, the appeal of AI as one of the most lucrative sectors of the decade poses a significant challenge to calls for restraint. The tension between innovation and the ethical implications of AI technology continues to escalate, as evidenced by previous open letters from the AI research community urging a halt to rapid advancements.

As Anthropic prepares for its initial public offering, expected to value the company above $1 trillion, the interplay between its military partnerships and ethical concerns creates a complex narrative for the future of AI. The Pentagon's impending deadline regarding the use of Claude in its systems adds further pressure, as stakeholders navigate the implications of AI in both civilian and military contexts.

Conclusion

The developments surrounding Anthropic and its collaboration with the NSA highlight broader issues within the AI field, where rapid advancements coexist with significant ethical dilemmas. As the company charts its future, balancing innovation and responsibility will be essential in shaping the trajectory of AI technology and its impact on society.