Another bleak warning, another round of Cassandra calls about humanity’s immaculate capacity for self-destruction. This month, the UN Secretary-General António Guterres and the head of the International Committee of the Red Cross and Red Crescent (ICRC), Mirjana Spoljaric Egger, issued a joint appeal stating that, “We are now dangerously close to crossing a moral red line: the autonomous targeting of humans by machines.” Both insist that two types of autonomous weapons – unpredictable autonomous weapons and anti-personnel weapons – be prohibited.
In Geneva, the chief of disarmament affairs at UN Geneva, Mélanie Régimbal, held up the fort by reiterating the position taken by the UN that systems selecting or engaging targets and taking human lives in the absence of control and judgment “are politically unacceptable, morally repugnant and should be prohibited under international law.”
Laurent Gisel, head of the arms and the conduct of hostilities division at the ICRC, also spoke from Geneva, describing conflict as “not a story of precision, distinction and restraints” but one “of civilians killed, families displaced, hospitals, schools and homes destroyed at ever greater speed and scale.” In the thick fog of conflict, many variables could come into play. “A combatant can surrender or be wounded and therefore be protected from attack; civilians can enter the area of operations of the autonomous weapons systems; the weapons may encounter a situation that would not have been anticipated.” The dangers of unpredictability in such autonomous systems was hugely problematic. “If it’s not possible to predict it, then how is it possible to limit it?”
These are not remarkable warnings. The UN and ICRC had issued a joint appeal on the subject in 2023, calling upon States to negotiate a legally binding document by 2026. At the level of civil society, the Stop Killer Robots coalition was formed in October 2012 and publicly launched in 2013. The campaign, according to the group, focuses on fostering a world where “we respect each other’s inherent dignity”; take responsibility for choices regarding technology and how it alters the relationships between humans; develop technology to promote peace, justice, human rights, equality and respect for law; and where people work as activists “to build a society and overcome inequalities and systems of oppression.” This is fairly airy stuff, but central is the goal of developing and fortifying “social norms that reject autonomous killing by machine [sic] in warfare, policing, border control and other circumstances”. Control, responsibility, accountability are fundamental features.
That campaign has not made much meaningful progress of late. The faith placed by the military establishment in murderous machines remains an unshakeable one, with the idea of human control over decision-making increasingly thinned in favour of the uncritical human consumer of data. That faith has effectively displaced the scrupulous, incredulous decision maker when it comes to acquiring and hitting targets. Within the first few weeks of the Gaza War, beginning on October 7, 2023, the Israeli Defense Forces (IDF) was making generous use of Lavender, an AI decision-support system (DSS) to generate 37,000 individual targets on a crude rating scale of significance depending on affiliations with armed groups. Other systems used in battle included The Gospel (Habsora), which generates target lists, and Where’s Daddy?, a platform tracking the locations of individuals ahead of the strike. The resulting slaughter to scale tallied. Far from suggesting a seamless integration of human and machine, the IDF’s conduct suggests a misplaced optimism and laziness in deferring to the merits of a killing system.
The disagreement between the artificial intelligence firm Anthropic and the US Department of Defense over the dangerous limits of autonomous systems in war is another case in point. The falling out between the two players in the thanatotic economy was serious enough to see a concerted attempt by the Pentagon to blacklist the firm as a “supply-chain risk”. (This was found on August 27 by the US District Court of the Northern District of California to be an “unlawful retaliation” in violation of the First Amendment and a denial of due process as protected by the Fifth Amendment.) That falling out, however, is not necessarily a boon for those favouring limits on killer robots or fully autonomous military platforms.
In a February 2026 statement, Dario Amodei, the company’s co-founder, conceded that the Pentagon, “not private companies”, made military decisions. “We have never raised objections to particular military operations nor attempted to limit our technology in an ad hoc manner.” Amodei, however, considers himself a politically sentient being, which begs the question as to why he is at the vanguard of denuding humanity’s role in making decisions, notably through encouraging Large Language Models (LLMs) that remove initiative, industry and inventiveness. AI, he claims, can “undermine” democracy, a goal he seems to be pursuing himself. Of personal concern was the use of the company’s technology for mass domestic surveillance, which presented “serious, novel risks to our fundamental liberties” and the use of fully autonomous weapons “that take humans out of the loop entirely and automate selecting and engaging targets”. But even here, his quibbles are brittle, given an admission that fully autonomous systems “may prove critical for our national defense.” The concern, it would seem, lay in the reliability of the systems available and whether they would harm American personnel. “We will not knowingly provide a product that puts America’s warfighters and civilians at risk.”
Two days after Amodei’s statement, the pre-emptive, joint illegal attack on Iran by the US and Israel commenced, seeing yet another gluttonous use of AI DSS. The US military generated targets at speed by turning to Palantir’s Maven Smart System, which integrates Anthropic’s Claude AI to assess surveillance data, generate target lists, and prioritise them. “Many of the targets hit in Iran,” writes Noor Hammad of the International Institute for Strategic Studies (IISS), “have been civilian, including a school and healthcare and residential facilities, illustrating the risks of rapid target generation.” Think tankers can always be relied upon to offer the understatement and euphemism in equal measure.
Gisel has told reporters that there is, as things stand, no “confirmed evidence that autonomous weapons have been used to directly target human beings in the battlefield.” This is a distinction with little difference, given the deferential attitudes shown by the technology fetishists in the IDF, the Pentagon and other militaries. If the system says so, it must be right. There is also a risk that these calls for restraint and a legal system of control will simply fatten the detritus of worry without effect. Ahead of the seventh Review Conference of the Convention on Certain Conventional Weapons (CCW) taking place in November, the human species is unlikely to take a pathway that does not first imperil it before seeking, irretrievably, to bottle the escaped genie. From the creation of atomic weapons to the threats of climate change, the worst will always come before any touted panacea. And that panacea, when it appears, is likely to be a mere palliative.
Dr. Binoy Kampmark was a Commonwealth Scholar at Selwyn College, Cambridge. He currently lectures at RMIT University. Email: bkampmark@gmail.com
