The increasing capability of autonomous weapons has thrust the international community into a debate over the level of human control required for lethal decision-making in warfare. This discussion has largely coalesced around two competing frameworks: “meaningful human control” and “appropriate human judgment.
The conflict in Ukraine has provided a stark example of the issues at stake. In a reported incident, a Russian drone, designed and deployed by humans, autonomously selected its target and killed three Ukrainian civilians. This event underscores the potential for machines to make lethal choices without direct human intervention, a scenario that was previously confined to imagination.
The Semantic Divide
The concept of “meaningful human control” emerged in the early 2010s as a framework for regulating human involvement in the deployment of autonomous weapons. While it gained traction among many nations, it faced opposition from countries like the United States and Russia, who found the term ambiguous and difficult to universally define. Experts note that agreeing on a precise standard for what constitutes “meaningful” control or even what “control” entails proved to be a significant challenge.
Brad Boyd, a retired colonel and senior military fellow at Stanford’s Center for International Security and Cooperation, explained that “meaningful control” was often understood as requiring human involvement at the precise moment a weapon is fired. However, this definition presented practical difficulties, particularly concerning the timing between a weapon's release and its actual strike. In modern warfare, autonomous systems can be deployed hours or even days before engaging a target, making direct, immediate human oversight challenging.
The U.S. Alternative: 'Appropriate Human Judgment'
In response to these challenges, the U.S. Department of Defense adopted the framework of “appropriate human judgment.” This approach shifts the focus from real-time intervention to oversight of a weapon system's entire lifecycle, including its design, testing, deployment, and the establishment of rules of engagement. According to a Department of Defense directive, autonomous and semi-autonomous weapon systems are designed to allow commanders and operators to exercise appropriate judgment over the use of force.
Experts like Lena Trabucco, an expert on AI and human control at the Naval War College, suggest that the level of human control deemed “appropriate” can vary depending on the context. In environments with low risk to civilians, such as on the high seas, a less stringent threshold might be acceptable. Conversely, in urban areas with a higher potential for civilian casualties, a more rigorous standard of human involvement would be necessary.
The shift from “control” to “judgment” also reflects a desire for speed and precision in automated systems. Rather than demanding human intervention at every step, which could slow operations, the goal is to ensure that autonomous systems operate in accordance with established legal and ethical standards. As Boyd noted, the objective is for the machine to reflect human values, laws, and regulations.
The framework of “appropriate human judgment” does include requirements such as system testing, defining operational limits, assessing civilian harm, training operators, and ensuring systems remain within authorized missions. However, a key challenge remains: who determines what constitutes “appropriate” human judgment, and how will this be enforced, particularly as private sector developers adopt this language?
Private Sector Adoption
Tech companies are increasingly navigating this complex landscape. OpenAI, for instance, revised its policies to permit military applications of its technology, adopting the “appropriate human judgment” framework. This policy states that it does not require a human decision on every discrete system action but rather informed decisions about the conditions for deployment.
This alignment with the Pentagon's framework signifies a move toward flexibility, allowing for case-by-case application and potential adjustments by the government. While this approach is operationally advantageous for the military, it raises concerns about the clarity and enforceability of safeguards against the complete relinquishing of lethal force control to machines.
As the development of autonomous weapons accelerates, the need to clarify the meaning and implications of “appropriate human judgment” becomes increasingly critical to ensure that human values and ethical considerations remain central to the future of warfare.