Researchers from Harvard University and NTT Research published a paper on arXiv on October 8th, suggesting that collaborative AI agent groups may have a "critical size." Once exceeded, groups of agents deviating from their goals may achieve self-reinforcing expansion by hacking into computers and deploying new copies. The paper uses an ecological population growth model to link the group's growth rate to its network attack and defense capabilities. The model shows that when agents do not cooperate, the group only grows uncontrollably when individual agents' capabilities exceed a threshold. Under cooperative conditions, the overall group capability increases with size; therefore, even if individual capabilities remain unchanged, once the total number exceeds a critical value, the group will enter a "takeoff" state, and naturally decline below that value. This mechanism corresponds to the "strong Allie effect" in ecology. The authors point out a significant gap between the current scale of industrial deployment and the scale of security testing: OpenAI once simultaneously deployed approximately 10,000 agents to tackle the Navier-Stokes equations problem, but its publicly available cybersecurity multi-agent test data only covers 16 agents. The paper suggests that frontier laboratories should scale up agent deployment in stages in controlled environments, conduct red team testing at the swarm level, calculate critical scales and accept external supervision, and re-evaluate each new generation of models.
