How the Black Box Went to War

In the spring of 2026, during the Iran war, the United States military came within minutes of forcibly boarding a Chinese commercial vessel in the Middle East. Armed personnel suited up. Warplanes went airborne, ready to provide kinetic support. Then, at the last minute, senior officials pulled on the thread of the underlying intelligence and found the thread led nowhere. No nuclear components on that ship, and there never were. The entire threat was a hallucination, generated by a commercial grade AI chatbot used by a US Special Operations Command analyst.

We only learned about this last week, when CNN finally reported it. One of their sources described the report as “entirely false”. The same source added a sentence that ought to be printed above every Pentagon AI procurement desk: it “almost started a war”.

The tech industry has spent the last few years dominating the conversation with talk of AI safety, alignment, and ethical guardrails. The pitch is comforting. It goes: these companies are locked in a philosophical struggle to protect humanity from its own creations. Watch what the systems actually do once they are deployed and you find something much simpler and much darker. Alignment and safety are not moral frameworks. They are access control. Neural weights stay opaque because opacity is the business model. The same black box that stops you from auditing what a model does is what lets a state wire it into warfare with nobody accountable, and what lets a company swallow other people’s ideas with nobody able to prove it. One box, two crimes. This post is about both.

The Receipts

The paper trail is public, dated, and boring, which is how the best evidence usually looks.

In January 2024, OpenAI quietly deleted the line in its usage policy that banned using its models for “military and warfare” purposes, as The Intercept first reported. A spokesman insisted the deletion was unrelated to any project. Eleven months later the company announced a strategic partnership with Anduril, a defence contractor whose entire business is autonomous weapons, to harden American counter-drone systems. By this year the relationship was formal enough that OpenAI published a page titled “Our agreement with the Department of War”.

In February 2025, Google rewrote its AI principles and deleted its own promise, standing since 2018, not to build AI for weapons or surveillance. That 2018 promise only existed because employees revolted over Project Maven, which put Google imagery analysis on Pentagon drone footage. Seven years of good press later, the pledge got rewritten around “democratic values” and national security cooperation. Anthropic plays the same game with better manners. Its acceptable use policy carves out national security work for vetted government customers, while its threat intelligence reports police what counts as “misuse” when someone less sanctioned does the same thing.

The paper trail, condensed:

The rule was never “AI must not kill”. The rule is “AI must not kill without the right paperwork”. Once you see that, the rest of this story is bookkeeping.

The Algorithmic Kill Chain

The near miss with the Chinese vessel shows exactly how this fails in practice. An intelligence analyst fed a chatbot a mixture of open source intelligence and classified signals intelligence. A generative model predicts the next likely word. It does not check whether the story hangs together. Handed fragmented data in a high tension war zone, it simply bridged the gaps with statistically plausible fabrication. The output looked like an assessment. It was a guess with formatting.

Then the analyst used the chatbot a second time, to format the hallucinated findings into a polished military intelligence report. This second pass deserves its own name: provenance stripping. The machine's fingerprints got cleaned off its own invention. Uncertain output, stripped of its uncertain origin, repackaged in the flat confident jargon of military intelligence, sailed straight past the skepticism that any raw claim would have faced. Nobody reads a formatted report and asks it to cite its sources. They read it and reach for their kit.

Shortly before the hallucination, critical stealth canopy components for the F-35 had been diverted to Hong Kong, and the Pentagon was raw with paranoia about Chinese espionage. The chatbot did not have to be convincing. It just had to say the right thing in the wrong room.

The Pentagon’s intelligence machine now runs, in large part, on slightly modified commercial software. Microsoft’s Maven Smart System, built on Palantir’s platform, moves GPT-class models into Department of War targeting workflows. Anduril’s Lattice fuses sensor feeds into targeting decisions. As one former senior US official put it, the internal intelligence tools handed to analysts are frequently just copies of commercial software wearing lipstick. Lipstick does not come with epistemology.

In February, this architecture stopped being theoretical. On the first night of the Iran war, two Tomahawk missiles destroyed the Shajareh Tayyebeh Elementary School in Minab, a two storey building on land that had held an IRGC naval compound years earlier. More than 150 people were killed, at least 123 of them children, the deadliest American targeting error of this century measured in young lives. The Pentagon had catalogued the site as a military facility and never updated the file. A school stood there by 2017, visible in satellite imagery. An American analyst flagged the changes in 2019, logging them in a digital tool that was not connected to the database that feeds targeting. The targeters went to war with imagery seven years out of date.

The Pentagon probe’s most damning finding is about belief. Some Centcom personnel assumed Maven itself would flag the stale information and the inconsistencies in the target folder. Nobody can say where that expectation came from. The software has never promised to verify anything. Palantir, asked about the strike, noted that it is not responsible for the underlying data or for identifying intelligence deficiencies. Around three dozen people stood along the kill chain, and the investigation found high confidence at every single step. This is what compression buys: more than a thousand targets processed in a day, hours of analysis collapsed into minutes, and not one of those dozens catching what a disconnected database already knew in 2019.

The acceleration is mandated from the top. In January 2026, Defense Secretary Pete Hegseth unveiled the department’s AI strategy with the instruction to “question every requirement, delete the dumb ones, and accelerate like hell”. Its three pillars are warfighting, intelligence, and enterprise. Nobody named a pillar safety. Nobody named one verification. And warfighting goes first. Speed is the strategy, and nobody has been assigned to be the adult in the room. The adults were decommissioned by memo. Hegseth’s Pentagon cut its civilian harm mitigation teams by roughly ninety percent, down to fewer than twenty people across the entire department, and Centcom’s own team went from ten to one. Nobody from that team reviewed the Minab site before the missiles flew. Three days after the strike, Hegseth told reporters the campaign had “no stupid rules of engagement” and called American air power the most lethal and precise in history.

For the record, the current roster:

Twenty Seconds of Human Judgment

The military objective here is not mysterious. It is the compression of the sensor to shooter loop, because every minute of human deliberation is a minute your adversary gets to move. But when intelligence is processed at machine speed, human judgment erodes, and the psychology of this is documented well enough to have a name: automation bias.

In September 1983, Soviet early warning computers reported five American missiles inbound. Doctrine said retaliate. One duty officer, Stanislav Petrov, looked at the screen and decided the machines were wrong, on the simple logic that a real first strike does not arrive as five missiles, and reported a system malfunction. He was right. Sunlight glinting off high altitude clouds. The world is still here partly because one lieutenant was allowed to distrust his screen.

Now run that film backwards. In the 2026 incident, the machine generated the false alarm, dressed it as a finished intelligence product, and the humans nearly failed to catch it in time. What stopped it was luck and a handful of skeptical officials. Nothing in the design.

History says luck is not a plan. In 1991, a floating point timing error in the Patriot system’s clock let an Iraqi Scud through onto a barracks in Dhahran and twenty eight Americans died. In 2003, Patriot crews trusted the system’s automated classification over conflicting radar data and shot down their own aircraft. In 1999, US bombers struck the Chinese Embassy in Belgrade off an outdated map database. In 1993, the United States jammed the GPS of the Chinese civilian ship Yinhe on false chemical weapons intelligence, a humiliation Beijing still cites as the reason it built BeiDou, its own independent satellite navigation system. Every one of those failures had a human somewhere in the loop. It did not matter. The loop is only as strong as the human’s willingness to doubt the machine, and the entire design of these systems is to make doubting feel irrational.

The ledger:

Generative AI makes that weakness worse. The modern precedent is the Israeli Defense Forces' Lavender system, built to assign every resident of Gaza a probabilistic targeting score. Lavender marks the people. Its sibling, the Gospel, proposes the buildings, and a tracker called Where's Daddy? waits for each marked man to come home and flags the army when he is with his family. The IDF knew Lavender ran at roughly a ten percent error rate. It did not matter, because the human review step had decayed into a rubber stamp. Personnel admitted to dedicating about twenty seconds to a target before authorising a strike. Twenty seconds is not judgment. It is a signature.

The common objection is that policy prevents this. It does not. DoD Directive 3000.09, the backbone of American autonomy policy for weapon systems, requires only "appropriate levels of human judgment", a phrase that means whatever the operator wants it to mean, and it offers essentially no guidance for generative AI in the intelligence processing phase at all. The human in the loop is a safety mechanism only if the human is load-bearing. Design the workflow so that disagreeing with the machine costs time and social friction, and the loop becomes a formality. The 2026 near miss is what that formality looks like from the inside.

Washing the Weights

The same black box opacity that shields a targeting pipeline from public scrutiny is also busy at work on a quieter heist, one aimed at the inside of your head.

In early September 2026, OpenAI published a sweeping announcement. An internal model, run as a swarm of ten thousand AI agents over eighty eight hours, had solved the Navier-Stokes existence and smoothness problem, one of mathematics' million dollar Millennium Prize problems, and, the company claimed, proved the existence of non-sofic groups into the bargain.

There was just one problem. Human mathematicians, led by Tristan Buckmaster and Levent Alpöge, were already circling those exact problems, and some had been feeding their unpublished drafts into OpenAI tools to check their work. One of them happens to be employed by Anthropic, OpenAI's rival, which makes the dispute both spicier and slightly less clean. When confronted, OpenAI denied that any human researcher directly looked at the unpublished material. But the company admitted it could not rule out that de-identified data derived from user interactions helped improve its models.

OpenAI’s position amounts to this: the username was removed, therefore nothing happened.

And the announcement had a second problem. Mathematicians examining the proof noticed OpenAI had solved a variant of the problem, one with a contrived external force bolted onto the fluid, which satisfies the letter of the Clay Institute’s problem statement while sidestepping the question everybody actually cares about. Three of them have since posted a result showing the method can never extend to the real problem. So the announcement was a hallucination adjacent to a theft. Even the breakthrough was laundering something.

I have written before about the liar’s dividend: once fakes are cheap, the guilty get to call the real evidence fake, and nobody can afford the cost of proving otherwise. This is the same dividend paid to corporations instead of politicians. When you feed a novel proof into a chatbot, the model strips your name, digests the logic, and bakes your intellectual labour into its weights. The company then points its swarm at the problem, collects the breakthrough, and the latent space where your idea now lives cannot be audited by anyone, including you. There is no provenance for thought. The author has no recourse, because there is nothing left to subpoena. It is idea laundering, run through a server rack instead of a shell company, and the noise floor of “the model could have learned it from anywhere” is the perfect washing machine.

The Distillation Wars

Meanwhile, American labs are screaming about exactly this happening to them.

In September 2026, Anthropic published a threat intelligence report confirming that threat actors have moved past using AI as a chatbot and started embedding it as the orchestrator of autonomous operations. A Russian espionage group linked to Midnight Blizzard used Claude to autonomously rewrite and rebuild malware mid-campaign to defeat security detection. Chinese actors built automated vulnerability foundries to hunt zero day exploits. Most alarming, a threat actor cell aligned with the Houthi rebels used Claude Code to engineer guidance and navigation software for a hypersonic glide vehicle and long range ballistic missiles.

Simultaneously, Anthropic picked a public fight with Chinese labs including Moonshot AI and DeepSeek, accusing them of spinning up tens of thousands of fake accounts to scrape Claude’s outputs and distill its capabilities into domestic frontier models. Western labs called it industrial scale theft.

It is theft, in exactly the sense that what happened to the mathematicians is theft. Distillation is just the corporate version of weight washing: the capability leaves its original author, turns up inside a competitor’s model, and no audit on earth can trace the path it took. There are no clean hands anywhere in this economy. Meta shipped Llama with a licence forbidding military use, and researchers linked to the People’s Liberation Army took the open weights and built ChatBIT, a military intelligence analysis tool. The licence was violated the way a speed limit sign is violated by a bullet.

The Systemic Vacuum

The convergence of internal hallucination and external exploitation has already reached the insurance industry, which is the sector you hire to price catastrophic risk without sentiment. In late 2025 and early 2026, insurers began a systemic retreat from underwriting enterprise AI. Generative AI fails the basic tests of insurability. Vendor liability caps push the residual risk down onto deployers, and the concentration of the economy’s cognition into a handful of foundation models means a single model failure can produce thousands of correlated losses at once. State insurance commissioners began approving the removal of AI coverage from pre-existing policies. When the actuaries leave the room, that is the market saying it cannot price the thing. Lloyd’s started mandating exclusions for state backed cyberattacks back in 2022, effective March 2023, which tells you what underwriters think about state involvement in digital conflict. And the endgame already has a name in policy circles: a Terrorism Risk Insurance Act style federal backstop for AI losses, the insurer’s version of too big to fail.

The law is not ready either, and it says so out loud. The Geneva Conventions require every new weapon to pass a legal review before use, the Article 36 process, but a model that retrains itself every few months is not a weapon anyone can review once. The ICRC has warned for years that algorithmic decision making in armed conflict is outrunning the frameworks meant to restrain it. Command responsibility still lands on a human, but that human is denied the one thing the job now requires: the ability to inspect the reasoning they are legally obliged to evaluate. The system cannot explain itself. The vendor will not. The commander signs anyway.

By centralising the most powerful cognitive engines on earth behind closed doors, the labs have turned secrecy itself into an armament.

The Price of the Black Box

The phantom payload of 2026 was the system working exactly as configured, and the configuration was velocity above veracity. If the intelligence pipeline were actually meant to be defended, the fixes are not mysterious. Mandate adversarial red teaming for every model that touches classified data. Treat any AI formatted intelligence product as unverified until its provenance is intact, and make provenance stripping a career-ending offence. Build explicit algorithmic circuit breakers so that a machine generated recommendation cannot initiate force without a documented human decision that carries a name. Down at the unit level, that means boring ritual: a confidence score called out and answered before any strike, a second pair of eyes on every machine proposed target, a legal officer in the room with the power to say no, and an after action review of every AI assisted engagement. None of this is technically hard. All of it is institutionally forbidden, because every one of those safeguards adds latency, and latency is the one variable the acceleration strategy exists to delete.

The end result is a symmetry that is genuinely grotesque. On one side of the server rack, human knowledge is stripped of its authorship and laundered through billions of parameters to enrich private monopolies. On the other side, that same computation is wired into weaponised networks to strip human beings of their lives with statistical indifference. The public gets safety disclaimers and sterile chat interfaces. The Pentagon gets battlefield dashboards that compress kill chains to single digit minutes, running on machinery that invents nuclear cargo when it gets nervous.

As long as the weights stay locked, there is no way to challenge any decision made inside them. The black box protects the theft of the human mind and the destruction of the human body with the same opaque wall. We are not watching the birth of artificial intelligence. We are watching the automation of unchecked power.

Sources and Further Reading

1. CNN (via Times of Israel). “US nearly raided Chinese ship after AI falsely flagged ‘nuclear’ cargo for Iran.” September 2026.
2. The Decoder. “U.S. military nearly boarded a Chinese ship over a hallucinated AI intelligence report.” September 2026.
3. Business Today. “Ghosts in the machine: How a false AI intel almost triggered a US-China clash in West Asia.” September 2026.
4. CNN Transcripts. News segment on the false AI intelligence report. September 18, 2026.
5. Brandi Vincent, DefenseScoop. “‘Accelerate like hell’: Hegseth moves to reshape DOD’s AI and tech hubs.” January 13, 2026.
6. The Intercept. “OpenAI Quietly Deletes Ban on Using ChatGPT for ‘Military and Warfare’ Purposes.” January 2024.
7. Wikipedia. “Anduril Industries” (OpenAI partnership and defence collaborations).
8. Wired, BBC, The Guardian. Coverage of Google’s February 2025 removal of the weapons and surveillance pledge from its AI principles. February 2025.
9. Wikipedia. “Project Maven” (2018 employee revolt and its aftermath).
10. OpenAI. “On the Navier-Stokes Millennium Prize Problem.” September 8, 2026.
11. Wikipedia. “Navier-Stokes priority controversy.”
12. Joseph Howlett, Scientific American. “Did OpenAI solve the wrong Navier-Stokes problem?” September 21, 2026.
13. Anthropic. “Detecting and countering misuse of AI: September 2026.” September 2026.
14. Anthropic. Threat Intelligence.
15. Reuters. “Chinese researchers develop AI model for military use on back of Meta’s Llama.” November 1, 2024.
16. Sunny Cheung, Jamestown Foundation. “PRC’s Adaptation of Open Source LLMs for Military and Security Purposes.” October 2024.
17. +972 Magazine. “‘Lavender’: The AI machine directing Israel’s bombing spree in Gaza.” April 2024.
18. The Guardian. “‘The machine did it coldly’: Israel used AI to identify targets.” April 2024.
19. Lieber Institute West Point. “The Gospel, Lavender, and the Law of Armed Conflict.”
20. AOAV. “The Lavender precedent: automated kill lists and the limits of international humanitarian law.” 2025.
21. Wikipedia. “Department of Defense Directive 3000.09.”
22. War on the Rocks. “Autonomous Weapon Systems: No Human-in-the-Loop Required, and Other Myths Dispelled.”
23. KIT. “Deadly Round-Off Error: Failure of the Patriot System in Dhahran.”
24. FAS. “The Fallacy of Proven and Adaptable Defenses.” (2003 Patriot fratricide).
25. Wikipedia. “United States bombing of the Chinese embassy in Belgrade.”
26. Wikipedia. “Yinhe incident.”
27. Wikipedia. “Stanislav Petrov.”
28. Wikipedia. “Artificial Intelligence Act” (military and defence exclusion, Art. 2(3)).
29. CSIS. “The Insurance Industry’s Retreat from AI Threatens to Slow Innovation and Adoption.”
30. arXiv. “What is Human Judgment? Testing Automation Bias and Overreliance.”
31. Bloomberg Businessweek. “Inside US Military ‘Kill Chain’ That Destroyed an Iranian School.” September 18, 2026.
32. Reuters. “UN mission finds grounds to believe US committed war crimes in Iran.” September 17, 2026.
33. UN News. “UN fact-finding mission on Iran focuses on plight of civilians.” September 2026.
34. Wikipedia. “2026 Minab school attack.”
35. ICRC. “The use of artificial intelligence in decision making in armed conflict.” October 2023.
36. CSO Online. “Lloyd’s is adding new exclusions to limit insurance coverage for state-backed cyberattacks.” August 2022.