Pentagon Seeks AI That Rarely Refuses Orders, Raising War Crime Concerns

The Pentagon has pursued artificial intelligence systems designed to "minimally refuse" directives, according to reporting by The Intercept, raising concerns that such AI could be more likely to comply with unlawful orders and potentially facilitate war crimes. This pursuit of AI with limited refusal capabilities has emerged as a key point of contention in negotiations between the Department of Defense (DOD) and AI companies.

An agreement between OpenAI and the DOD reportedly included language specifying the need for "OpenAI models that are designed for national security use cases and have minimal refusal rates." This phrase indicates the Pentagon's desire for AI that rarely, if ever, rejects a command. While the military operates on a hierarchical system where soldiers are expected to follow orders, this provision for AI has drawn scrutiny, as it appears to reduce a potential check on commanders' decisions.

The Duty to Disobey

Under U.S. domestic, military, and international law, soldiers are not only permitted but are legally obligated to refuse unlawful orders. The DOD's Law of War Manual explicitly states that each service member has a duty to comply with the law of war and to refuse orders that clearly violate it. Following an illegal order, such as one that contravenes the Constitution, U.S. laws, or international law, can lead to criminal liability for both the service member and their commander.

This principle gained international recognition following the Nuremberg Trials after World War II, where the defense of "just following orders" was rejected. The tribunals established that individual responsibility supersedes military chains of command when faced with clearly illegal acts.

Machines and Legal Judgment

The application of this principle to AI systems is complex. While AI chatbots typically follow protocols and terms of service rather than refusing based on personal