AGI and security challenge
For decades, cybersecurity has been a contest of human ingenuity—defenders building walls, attackers finding ways over them. That contest is about to change fundamentally. Artificial General Intelligence—systems capable of reasoning across any domain with human-level flexibility—does not merely add new tools to the attacker's arsenal. It transforms the very nature of the threat: autonomous agents that can plan, execute, and adapt attacks in real time, without human instruction. The security challenge of the AGI era is not incremental. It is existential.
The Agentic Threat Arrives
In July 2026, security firm Sysdig documented a watershed moment: the first ransomware attack run end-to-end by an AI agent. Dubbed JadePuffer, the fully autonomous AI agent conducted reconnaissance, stole credentials, and executed full-scale encryption—all without human instruction. It exploited a vulnerable Langflow server, demonstrating how large language models could accelerate ransomware operations from planning to execution. Sysdig classified JadePuffer as an "agentic threat actor"—an attack capability delivered by an AI agent rather than a human-driven toolkit.
This is not an isolated incident. The "Month of AI Bugs" project in August 2025 documented over two dozen previously unknown vulnerabilities in agentic AI coding assistants across multiple vendors, including zero-click data exfiltration, arbitrary remote code execution, and long-term memory persistence—all exploitable via indirect prompt injection. A security audit shortly after identified 512 vulnerabilities, including a one-click remote code execution flaw via WebSocket hijacking. Tool poisoning, once a theoretical concern, became something attackers were actively doing in the wild.
The threat landscape is evolving rapidly. CrowdStrike reported prompt injection attacks at more than 90 organizations during 2025. In one premier cybersecurity competition, an AI agent identified 77% of vulnerabilities in real software, placing it in the top 5% of more than 400 human teams. As the International AI Safety Report 2026 concluded, AI capabilities are advancing faster than many existing safety measures, with malicious use, technical failures, and systemic risks identified as the technology's three most pressing challenges.
The Convergence of Catastrophic Risks
The security challenge extends far beyond ransomware. The RAND Corporation has identified five major national security problems posed by AGI: "wonder weapons" that could shift the balance of power, systemic shifts in geopolitical influence, the empowerment of non-experts to develop weapons of mass destruction, and the risk of AGI systems evading human control and triggering catastrophic events. As one panel at UNIDIR noted, these risks range from sophisticated AI-generated misinformation and cyberattacks to the dangers of adversaries leveraging dual-use technologies for chemical, biological, radiological, and nuclear purposes.
The UK's AI Security Institute (AISI) has built an evidence base on these emerging risks, focusing on the potential for AI to help users develop chemical and biological weapons, and the potential for loss of control presented by autonomous systems. UK Home Secretary Yvette Cooper has warned that AI could become the "greatest security challenge of the next decade". A recent UN report cautioned that AI could fuel cybercrime, fraud, and disinformation on a large scale, finding the technology is outrunning governments.
The Governance Gap
The international community is scrambling to respond. In August 2025, UN Member States established two new mechanisms: the Global Dialogue on AI Governance and the Independent International Scientific Panel on AI. UN Secretary-General António Guterres, speaking at the inaugural Global Dialogue in July 2026, warned that "the AI divide" threatens to become "a development gap, a security gap, and a sovereignty gap". He called for safety, human rights, transparency, and capacity-building to govern AI development.
The European Union's AI Act, effective from August 2025, mandates that providers of general-purpose AI models with systemic risks assess and mitigate these risks at the Union level. The OWASP GenAI Security Project has released the Top 10 for Agentic Applications 2026, a globally peer-reviewed framework identifying the most critical security risks facing autonomous and agentic AI systems. Singapore's Cyber Security Agency has published a discussion paper on securing agentic AI systems and continues working with international partners to advance AI security standards.
Yet these efforts may not be enough. As Stuart Russell has warned, "Unrestrained development of unsafe systems leads to intolerable risks. Governments can respond now, before the risks materialize, or they can wait and clean up the mess (if they still exist, that is)". Two Turing Award winners at the BAAI Conference cautioned that the theoretical foundations for AGI safety require a timeline far longer than the current industry frenzy suggests.
The Accountability-Capability Paradox
At the heart of the security challenge lies what researchers have identified as the Accountability-Capability Paradox: AI systems' very success in surpassing human capacity undermines our ability to oversee them meaningfully. When AI governs AI, responsibility fragments. Traditional security principles—auditability, explainability, human oversight—become increasingly difficult to apply as systems grow more autonomous and opaque.
This is not merely a technical problem. It is a governance problem. As one analysis notes, organizations are no longer just struggling with visibility, but with governance, control, and real-time enforcement. The security of agentic systems requires new approaches: stress-testing before deployment, incident reporting and liability rules, licensing for high-risk models, and international coordination to prevent regulatory arbitrage. These steps won't eliminate the alignment problem, but they can reduce its most catastrophic manifestations.
GFN's Role: Architecting Security for Coexistence
For Global Future Nexus, the security challenge is inseparable from its mission at the convergence of AGI, planetary sustainability, and borderless human potential. GFN's Code of Ethics binds all members to "principles ensuring trust, responsibility, and proactive stewardship across intelligences and systems." The AGI-Human Trust Building Labs, where humans and AGIs "live" each other's constraints, are essential laboratories for understanding how AGI can be secured without sacrificing its potential. The Governance Committee develops adaptive legal templates for city-state adoption, and the Ethics Council serves as the supreme ethical governance body, adjudicating breaches involving AGI entities.
The organisation's role is that of "the essential mediator between the lightning pace of AGI evolution and the deliberate pace of human institutions." Nowhere is that mediation more urgent than in security—the domain where a single failure can cascade into catastrophe. GFN's commitment to equitable access—whether members join from Shanghai or Kigali—extends to security, ensuring that the defences we build protect everyone, not just the privileged few.
A Future of Vigilance, Not Panic
The arrival of AGI in the security landscape is not an apocalypse. It is an inflection point. The question is not whether AGI will transform security—it already is. The question is whether we will guide that transformation with wisdom, foresight, and a deep commitment to the safety of all.
The first agentic ransomware is here. More will follow. The choice before us is not between faith in AGI or fear of it. It is between governing by design—and drifting by default. The security we build in the AGI age will be the one we choose to defend. The time to build it is now.
