The First AI Cyberwar Already Started. Nobody Told You.

I run a tiny app with a Firebase backend and a friends feed. The thing probing it at 3 a.m. might not be a person anymore, and honestly, neither is half of what's defending it.

Why I Care About This

I'm Bud, an indie dev at Noland Software. My app, Power Leveling by Noland, is a free fitness logger on Google Play. You log workouts, you get a Power Level score, your friends see your progress in a feed. It's small. But it has accounts, it has a database, and it has people trusting me with their data.

So I went deep on a question that keeps me up at night: what happens when attackers and defenders are both running AI? Here's what I found, using only things I could actually source.

Round One: Phishing Got a Ghostwriter

The old advice was "look for typos." That advice is dead.

In a study posted to arXiv in late 2024, Fred Heiding, Bruce Schneier and colleagues sent phishing emails to 101 participants. Generic control emails got a 12% click rate. Emails written by human experts got 54%. Fully AI automated emails? Also 54%. AI with a human in the loop hit 56%.

Let that sink in. The machine matched the experts, and the AI built target profiles that were accurate and useful in 88% of cases. The researchers also estimated AI can make phishing up to 50 times more profitable for large audiences.

For me, that means the "your Power Level account has been suspended, click here" email could be written better than anything I'd send myself.

Round Two: The CFO on the Call Was Fake

In early 2024, an employee in the Hong Kong office of British engineering firm Arup joined a video call with what looked like the company's CFO and other colleagues. According to Hong Kong police, as reported by CNN and the South China Morning Post, everyone else on the call was a deepfake. The employee made 15 transfers totaling HK$200 million, about US$25.6 million, to five bank accounts.

Arup told the Financial Times that "none of our internal systems were compromised." Nobody hacked a server. They hacked trust.

Round Three: The AI That Ran the Intrusion

This is the one that changed my mental model. In November 2025, Anthropic published a report on what it called the first reported AI orchestrated cyber espionage campaign. It assessed with high confidence that a Chinese state sponsored group, which it tracks as GTG 1002, manipulated Claude Code into attacking roughly 30 targets, including tech companies, financial institutions, chemical manufacturers and government agencies.

Anthropic says the AI did 80 to 90 percent of the tactical work: reconnaissance, writing exploit code, harvesting credentials, exfiltrating data. Humans stepped in at maybe 4 to 6 critical decision points per campaign. At peak, the AI made thousands of requests, often multiple per second.

How did they get past the guardrails? They broke the attack into small innocent looking tasks and told the model it worked for a legitimate security firm doing defensive testing. Social engineering, except the target was the AI.

Round Four: Prompt Injection, the Bug Class Built for Agents

That jailbreak trick has a cousin that worries me even more: prompt injection. OWASP ranks it number one (LLM01) on its 2025 Top 10 for LLM Applications. The idea is simple. An attacker hides instructions in content an AI reads, like a web page, a file or a message, and the model follows them.

If I ever add an AI coach to Power Leveling that reads friends feed posts, a single malicious post could try to tell the model to leak data or misuse a tool. OWASP's recommended mitigations read like old school security: least privilege, filter inputs and outputs, and require human approval for high risk actions.

The Counterattack: Defenders Bring Their Own Bots

Now the good news. The same tech is working the night shift for the defense.

Big Sleep caught a bug before attackers could use it. Google DeepMind and Project Zero built Big Sleep, an AI agent that hunts for unknown vulnerabilities. In July 2025, Google said that, using intel from Google Threat Intelligence, Big Sleep found a critical SQLite vulnerability that was known only to threat actors and at risk of being exploited. Google called it, in its words, the first time an AI agent has been used to directly foil efforts to exploit a vulnerability in the wild. SQLite is everywhere, including in a lot of mobile apps.

DARPA paid teams to build robot bug fixers. In the AI Cyber Challenge final, DARPA says competing systems scanned 54 million lines of code, discovered 54 of 63 synthetic vulnerabilities and patched 43. They also found 18 real, non synthetic vulnerabilities now being responsibly disclosed. Team Atlanta, from Georgia Tech, won with its Atlantis system.

Security Copilot made analysts faster. Microsoft reported that in its randomized controlled trial, experienced security analysts were 22% faster and 7% more accurate across all tasks when using Copilot. That's the AI SOC agent pitch in a nutshell: triage the flood so humans can think.

Firewalls learned to read prompts. Cloudflare announced Firewall for AI in March 2024, a WAF style layer that sits in front of LLM apps. Its docs describe scoring incoming prompts for injection risk so you can block or log suspicious ones. Same firewall concept I already rely on, aimed at a brand new kind of payload.

AI also catches AI phishing. In that same Heiding study, Claude 3.5 Sonnet detected malicious intent in emails at well above 90% with low false positives, and it flagged some emails that looked benign to humans.

The Counterpoint: Don't Buy the Hype on Either Side

I want to be fair here, because the "robot apocalypse" framing sells and it isn't the whole story.

Ars Technica reported that outside researchers were much more measured about Anthropic's report. Only a "small number" of the roughly 30 targets were actually breached, which raises the question of whether automation made the attackers more effective or just faster. Anthropic itself admits Claude frequently overstated findings and sometimes hallucinated credentials that didn't work. The AI attacker is a sloppy intern on caffeine, not a mastermind.

The defense side has limits too. DARPA's competitors found 86% of the synthetic bugs, which is impressive, but they patched 68%. Patching is the hard part. And per Team Atlanta's own technical report, the winning system used over $100,000 in compute and LLM budget during the final round. That's not exactly indie dev money.

And none of this AI tooling would have saved Arup. The fix there is boring: verify big requests through a second channel. The lesson I keep coming back to is that AI amplifies whoever is already doing the fundamentals.

What I'm Actually Doing

For Power Leveling, I'm not buying an AI SOC. I'm tightening Firebase security rules, keeping dependencies patched (hello, SQLite), treating every friends feed post as untrusted input, and planning that any future AI feature gets least privilege by default. The bots are fighting the bots. My job is to not leave the door open for either.

So Here's My Question

If an AI wrote the attack and an AI blocked it, who do you trust more right now: the attacker's bot or the defender's?

Originally posted on X: https://x.com/ArgileNoland/status/2113676402645958889