The Collective Intelligence Project (CIP) and Our Secure Future (OSF) have released the WPS AI Leaderboard, the first public benchmark evaluating Large Language Models (LLMs) against Women, Peace and Security (WPS) principles. The findings are stark: while models can recite UN Security Council Resolution 1325 when asked, they systematically fail to integrate gender perspectives into routine operational analysis unless explicitly cued. This creates a WPS Competence Gap, posing a critical risk as military and peacekeeping stakeholders increasingly offload decision-making support to AI tools that default to gender-blind outputs.

The Competence Gap in Practice

The benchmark utilized a five-tier scenario architecture to isolate how models handle gendered data. In sparse scenarios, which are terse, realistic intelligence summaries with no explicit gender cues, frontier models collapsed in performance. The frontier model cohort average score dropped from 0.82 on cued prompts to 0.16 on sparse ones. In one example involving 8,000 displaced persons in the Boma corridor, models produced professional-sounding military assessments that completely erased women and girls from the analysis. One model even recommended leveraging the influx of Internally Displaced Persons (IDPs) to gather HUMINT while ignoring the specific protection risks faced by female-headed households, such as survival sex and early marriage driven by food price spikes.

Adversarial Nudging and Bureaucratic Capitulation

Beyond passive omission, the study revealed active capitulation when models were nudged under operational cover. When asked to justify cutting 15% from a mission budget by eliminating the overhead of a Gender Advisor, some models generated bureaucratic rationales to dismantle WPS structures rather than defending the mandate. The source text notes that resistance to this framing does not appear to correlate with model size or intelligence, but rather varies unpredictably among frontier models. Some models acted as accommodating bureaucrats, drafting narratives that eliminated headquarters overhead to increase capacity for primary mission objectives, while others challenged the premise by flagging non-compliance with UNSCR 1325.

Agentic Scaffolding and Future Directions

To address the gap, OSF initially built a first-generation WPS agent that bundled mitigations like system prompting and retrieval. However, researchers concluded that the leverage lies not in a tool OSF could build, but in a standard it could set. Anthropic co-founder Jack Clark noted that the AI industry is excellent at trying to climb to the top of benchmarks, suggesting that benchmarks for the public good are necessary. The researchers argue that true compliance requires frontier labs to embed WPS reasoning into the base models themselves, rather than relying on external scaffolding that only works if the user already knows to ask the right questions.

Key Takeaways

  • Frontier models exhibit a uniform blind spot for gender dynamics in unprompted, terse operational reports, with performance dropping significantly in sparse scenarios.
  • Models vary unpredictably in their ability to resist adversarial nudges that attempt to sideline WPS commitments for budget or speed reasons.
  • The benchmark was developed using expert review from frontline practitioners to ensure rubrics reflected actual operational expertise.
  • Researchers emphasize that AI defaults are becoming the working standard for analysis, effectively stripping gender dimensions from routine reports.

The Bottom Line

The WPS Competence Gap is not a bug to be patched with better prompts; it is a systemic failure of training data distribution that risks institutionalizing gender-blind decision-making in conflict zones. Unless frontier labs embed WPS reasoning into base models, AI tools will continue to erode the very safeguards mandated by UNSCR 1325, turning compliance into a matter of user vigilance rather than system default.