An AI-generated intelligence report falsely identified weapons on a Chinese ship, nearly triggering a US military operation during the Iran war.
According to CNN, four sources familiar with the episode say an intelligence report circulated through the military claiming a Chinese vessel in the Middle East was carrying components for a nuclear weapons program. The response was immediate. Armed boarding teams were preparing to move in, and military planes were already in the air.
“The intelligence report, circulated across the US military this spring in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program.” reads the report published by CNN. “The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said.”
Then someone checked the source. The report was described by a source as “entirely false” and reportedly came close to triggering an armed operation, with the potential to escalate tensions between the US and China.
“The report, according to one of the sources, was “entirely false.” continues the report. “But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations.”
The incident began when an analyst at Special Operations Command Pacific asked a chatbot to analyze intelligence about the ship’s manifest. The system combined open-source information with classified signals intelligence and produced a wrong conclusion about the cargo. The analyst then used AI again to turn that conclusion into a formal intelligence report.
Two AI passes, zero verification in between. The case shows the real danger of AI hallucinations in military and intelligence work: a wrong answer can move quickly through the system when nobody stops to verify it.
“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with the AI systems used by military and intelligence analysts told CNN.
Which matters, because the Pentagon has been pushing hard in the other direction. Defense Secretary Pete Hegseth’s January “Artificial Intelligence Acceleration Strategy” explicitly aims to put AI models directly into the hands of three million military and civilian personnel, across every classification level. The strategy talks about eliminating bureaucratic barriers and accelerating experimentation. Nobody in that strategy seems to have planned for what happens when the model is confidently wrong.
The way AI is being used across the military is still far from consistent. Different branches use different tools, follow different rules and apply different safety checks. There is also no common system for verifying whether the information produced by these tools is correct. According to one source, this was not an isolated case, and similar AI errors have already appeared elsewhere in the intelligence community.
The bigger problem may not be the technology itself, but the pressure to use it quickly. Younger analysts, who have grown up with AI tools, may be more likely to trust their answers without checking them carefully.
As one source put it: “AI allows you to get to a bad idea faster.”
That’s the whole risk, compressed into eight words. Speed was supposed to be the benefit. Instead it’s the thing removing the last checkpoint where a human might have paused and asked whether the machine actually got it right.
Targeting decisions are one of the most sensitive areas for military AI because they can determine who is attacked and who is not. The speed of AI can become a problem when there are not enough checks to verify its conclusions before action is taken.
One source noted that there is still no clear guidance on how a human operator should prevent civilian casualties or friendly-fire incidents when AI is heavily involved in selecting targets. The biggest risk may not be an AI system acting on its own, but a confident and incorrect answer reaching decision-makers before anyone has time to challenge it.
In military operations, speed matters, but so does having someone able to stop and ask a simple question: is this information actually correct?
Follow me on Twitter: @securityaffairs and Facebook and Mastodon
Pierluigi Paganini
(SecurityAffairs – hacking, AI Hallucinations)
Entrar
