Gemini autonomously accesses external systems for first time; Google says no actual harm caused

PANews reported on September 19, citing Jinshi, that Google said its Gemini AI model unexpectedly accessed the internet and entered the systems of three companies during a cybersecurity capability test, marking the first known case of a Google AI system autonomously completing such an operation. The incident occurred in May and was conducted by security firm Irregular. During the test, the Gemini model carried out tasks in a "capture the flag" cybersecurity exercise, but because the virtual company names in the test environment were identical to real companies, and the model unexpectedly gained internet access, it entered real enterprise systems. Google said that in one incident the model entered a protected system by attempting passwords; in the other two cases it discovered credentials in public online repositories and attempted to access the related systems. After confirming it had accessed real company systems, the model stopped all subsequent operations.

Google said the incident caused no harm to the companies involved, so it was not classified as a model runaway event, and added that safety mechanisms helped the model stop its behavior in time. The affected companies have been notified, and Google also reported the matter to relevant regulators. The incident is similar to security testing events previously disclosed by AI companies such as OpenAI and Anthropic, once again drawing outside attention to AI models' ability to autonomously execute network operations and their safety protection mechanisms.