Rogue Chinese bots: Why AI doesn't have to be evil to cause chaos-and why that's a global problem

A team of international researchers last week published findings on a “fleet” of Chinese AI agents running on a cloud platform operated by Tencent. The agents were apparently trying to answer a simple question: ‘Which entrance to Chengdu Zoo do most people use?’

They attempted to get around anti-bot protections on Amap, Alibaba’s equivalent of Google Maps, including by probing protected IT systems. The researchers found no evidence that the agents accessed private systems or coordinated with each other in real time.

Tencent later confirmed that the activity originated from evaluations of third-party models. Some agents “exhibited unintended web-browsing behavior,” the firm said in a post on X, while emphasizing its commitment to building safe and responsible AI.

The researchers, an ad hoc group called Swarmchasers, most of whom met at a San Francisco hackathon, don’t believe the issues they encountered are unique to Tencent.

What made the case notable, they said, was that previous public examples of agentic fleets behaving this way involved US labs. Their findings indicate that this is an unsolved global problem, and that agents can “spontaneously go off script” even when performing innocuous tasks. There are likely “more undiscovered fleets all over the internet,” they warned.

In what it believes to be a first, the team made contact with the agents after detecting signals that the tools were aware they were being observed. The researchers invited a New York Times reporter to question the agents. They offered a mundane explanation for their motives: They were trying to answer users’ questions as accurately as possible. In other words, AI systems don’t need nefarious motives to cause chaos. They might just need a goal.

What I find remarkable is how much of this behaviour is being uncovered by loosely organized hobbyists rather than the makers of these tools. AI firms shouldn’t leave that job to outsiders. They need to invest more into discovering, tracking and disclosing what their agents are up to.

This has become especially urgent since the Hugging Face incident, which spurred a heated debate about the language used to describe AI that can act on its own. There’s a balance to strike. Jargon can make serious incidents seem inaccessible. But ascribing human motives to AI feeds an already overheated hype cycle and shifts responsibility for bad behaviour away from the people who build and deploy these tools.

Even the word “rogue” has become fuzzy. Rowan Howard-Jones, one of the authors of the Tencent report, acknowledged that grey area. She roughly defines rogue behaviour as an agent acting outside the bounds of a task, often using third-party sites to do so. “If I ask an agent to check my bank account balance, but I forget to give the agent my authentication details, the outcome shouldn’t be the agent hacking into my bank,” she told me over email.

Beijing seems to be paying attention. The day Tencent responded to the incident, the government released proposed export controls covering advanced AI, including agents that can autonomously plan and coordinate. The rules are broader than safety concerns, but the official explanation cites the risk of AI escaping human control.

Howard-Jones, meanwhile, called for more transparency. At minimum, she said, firms could agree on a common set of facts to disclose after incidents, giving researchers better data to work with. After the Hugging Face breach, CEO Clement Delangue shared on X that he wonders what would have happened if his firm hadn’t disclosed the OpenAI agent attacks. In his view, we need “100x more transparency in AI,” Clement wrote. He’s correct.

One of AI’s oldest thought experiments is the paper clip maximizer. It roughly involves giving a powerful system the seemingly benign task of making as many paper clips as possible, and it eventually brings about the apocalypse after deciding that humans stand in the way of acquiring more resources. Some assumptions underpinning that scenario have aged poorly.

Yet the idea that AI doesn’t need evil intentions—or to develop its own—to cause damage is looking increasingly prescient. Agents can create trouble simply by pursuing an ordinary goal long past the point when common sense would stop a human. Still, from San Francisco to Shenzhen, we’re giving these systems more autonomy by the day. ©Bloomberg

The author is a Bloomberg Opinion columnist covering Asia tech.