Military and killer robots

Some supporters of autonomous weapons have argued that they will be more accurate or precise than humans, and therefore lead to less collateral damage. For example, Ron Arkin has written, “Robots probably will possess a range of sensors better equipped for battlefield observations than humans have, cutting through the fog of war.” Supporters argue that they will bring increased speed and efficiency to the battlefield, that they would be able to operate in environments with insecure communications, and that they could save lives by decreasing the need for human soldiers and acting as a deterrent.

But similar arguments were made for other indiscriminate weapons in the past, like landmines, cluster munitions, and nuclear weapons. Those weapons claimed hundreds of thousands of victims before being banned by international treaties. By reacting with their environment in unexpected ways, autonomous weapons would increase risks to soldiers and civilians alike. Improved precision can be achieved without removing meaningful human control from the use of force. The potential advantages of autonomous weapons are far outweighed by the serious challenges they pose to international law and security.

Another danger emanating from the deployment of autonomous weapons is dependence on wireless communications. Wireless communications are susceptible to intentional disruption such as hacking, ‘jamming’ and ‘spoofing’, which could make systems inoperable or corrupt their programming. In 2012 researchers used a ‘fake’ GPS communications signal to redirect the path of an unmanned air system, successfully spoofing the system and demonstrating concerns with the security of unmanned and autonomous weapons. In a world where cybersecurity and cyberwar raise growing concerns, more sophisticated hacking could enable complete takeover of the operation of autonomous systems, including potential release of weapons.

Autonomous weapons, which would select and engage targets on the basis of sensor data, would give militaries the ability to target people based on race, ethnicity, gender, style of dress, height, age, pattern of behaviour, or any other available data that could make up a targetable group. Technology is not perfect, it’s not neutral, and killer robots would be vulnerable to technical failures. There are also concerns that killer robots would be much cheaper and easier to produce than other weapons of mass destruction. Thousands of scientists have warned that with no costly or hard-to-obtain raw materials it is possible that autonomous weapons could be mass produced. If development is left unregulated, there is risk of these systems being acquired and deployed by non-state actors or individuals along with states.

There are also ethical, moral, technical, legal, and security problems with autonomous weapons. Machines lack inherently human characteristics like compassion and understanding of human rights and dignity, which are necessary to make complex ethical choices and apply the laws of war. In case of a mistake or an unlawful act, autonomous weapons present an accountability gap, which would make it difficult to ensure justice, especially for victims. The nature of war will drastically change as sending machines in place of troops lowers the threshold for conflict.  Autonomous weapons could also be used in other circumstances, such as in border control and policing.

In a military engagement where lethal force is directed or applied, there is a clear chain of command and accountability. Because militaries function as hierarchical organisations, this command structure is top-down, from the commander who orders the use of force to the person who ‘pulls the trigger’. With autonomous weapons, command and control are threatened, and responsibility and accountability are not so clear-cut.

If an autonomous weapon can select and engage its own targets the chain of command is disrupted. In these systems, upon activation there is a period of time where the weapon system can apply force to a target without additional human approval. Even if some parameters are predetermined, due to the dynamic nature of conflict the machine would autonomously engage targets without direct command. This means the human operator does not determine specifically where, when or against what force is applied.

Issues of explainability, predictability, and replicability mean that targeting and engagement by autonomous weapons also threaten military control. There will be little or no clarity on why or how a killer robot made a specific decision. If those decisions result in errors – like friendly fire or excessive collateral damage – it will be difficult to determine whether it was a result of the machines functioning or adversarial tampering. When command and control are disrupted in this way, the accountability gap also widens. Who is responsible for unlawful acts by autonomous weapons? The people who set the variables in its utility function? The people who programmed it in the first place? The military commander? Who will be held accountable?

Some have argued that control can be maintained through appropriate oversight, or ability to intervene or cancel an attack. However, there are serious concerns over whether human operators would be able to maintain the necessary situational understanding to have meaningful control. The amount of data a human commander would have to review would outstrip human ability to analyse it. The inability to interpret the huge metadata sets will further distance humans from understanding what goes on the battlefield. The speed and reaction of machines compared to humans will increase the pace of war, and the result will be a loss of meaningful human control over the use of force.

One of the main principles of International Humanitarian Law (IHL) is distinction – the requirement to distinguish between combatants and civilians. But in the recent decades, conflicts have increasingly been non-international armed conflicts fought between state forces and non-state actors like guerillas or insurgents. Enemy combatants in this type of warfare are rarely seen wearing standard military uniforms, making it harder to distinguish them from civilians. In this type of warfare, it is often the goal of non-state actors to blend in or appear as civilians in order to gain tactical advantage. Given the difficulties human soldiers face in determining who is a legitimate target, it is easy to see even greater risk posed by the use of autonomous weapons.

A machine would determine if the target was a combatant based purely on programming, likely developed in a sterile laboratory years before the decision to kill was made. Abrogating life and death decisions to a machine is morally, ethically and legally flawed.

John MacBride, LCol (Retd), in open letter on military personnel’s call for a ban on autonomous weapons.

Diplomatic talks concerning autonomy in weapons systems are entering a critical stage, though talks at the United Nations in the Convention on Conventional Weapons have made little progress since 2014. A handful of military powers are stubbornly resisting proposals to launch negotiations on a legally binding instrument addressing autonomy in weapons systems. Meanwhile, military investments in artificial intelligence and emerging technologies continue unabated. If left unchecked, this could result in the further dehumanisation of warfare, and diminished public trust in the many promising and beneficial civilian applications of emerging technologies.

Read more

Stop Killer Robots is not seeking to ban weapons that operate under meaningful human control. We are not opposed to artificial intelligence (AI) or robotics broadly, or even to the use of AI or robotics by the military. We are not proposing a ban on systems without weaponry designed to save lives, such as autonomous explosive ordnance disposal systems, which may operate with or without human control. But we believe there is a line which should never be crossed: life and death decision making should not be delegated to machines.

The development or deployment of autonomous weapons that target people and cannot or don’t operate under meaningful human control will lower the threshold for entry into armed conflict, and any such system that is deployed is vulnerable to hacking, or malfunction, increasing risk to friendly troops and civilians alike. It is our understanding that no military commander would want to cede control on the battlefield to an autonomous weapon.

Technologies are designed and created by people. We have a responsibility to establish boundaries between what is acceptable and what is unacceptable. We have the capacity to do this, to ensure that artificial intelligence and other emerging technologies contribute to the protection of humanity – not its destruction.

At the height of the Cold War, a Russian officer saved the world. On 26 September 1983, Lieutenant Officer Stanislav Petrov decided not to accept that the computer signals alerting him to an imminent attack from US’s nuclear warheads were accurate. What would have happened if he would have hit the ‘I-believe button’ to rubberstamp the system’s recommendation? Ensuring that weapons systems operate under meaningful human control means that life and death decisions are not delegated to machines. The importance of human decision making in a military context is as important now as it was during the Cold War