AI Benchmarks Under The Spotlight As Washington Turns Them Into Security Assets

TL;DR

Prime Big Deal Days · Oct 6–7Offer from Amazon

Get smart everyday buys delivered free — and shop member deals

• Fast, free delivery on millions of items
• Access to Prime Big Deal Days deals on October 6–7
• Prime Video, Amazon Music and more included

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

The US government has introduced a classified benchmarking system for advanced AI models, designating certain models as ‘covered frontier models.’ This move centralizes oversight and creates a voluntary pre-release review framework. The approach emphasizes security but raises transparency and fairness questions.

On June 2, President Trump signed Executive Order 14409, creating a classified benchmarking process for advanced AI models and designating certain models as covered frontier models. This move signifies a major shift in US AI oversight, centralizing control within the NSA and Treasury, and making security assessments more secretive.

The order mandates that by August 1, 2026, federal agencies—including the NSA, Treasury, and CISA—must establish a classified cyber-capability benchmark for AI models, with the NSA Director making designation decisions. It also introduces a voluntary framework allowing developers to give the government access to models before release, for up to 30 days, with evaluations shared as appropriate. Additionally, the order creates an AI cybersecurity clearinghouse under Treasury to share vulnerability intelligence and allocates funds and personnel to improve AI vulnerability detection and cybersecurity.

While participation in the pre-release review is technically optional, analysts note that trusted partner status—earned through participation—may become a key factor in federal procurement, effectively creating a de facto standard. The process is built around classified benchmarks, which are not accessible or contestable by developers, raising concerns about transparency and oversight.

Implications of Classified AI Benchmarking for US AI Development

This development marks a significant change in US AI policy, moving from a hands-off approach to one emphasizing security and oversight. The classification of benchmarks and the central role of NSA and Treasury could influence industry practices, favoring vendors who participate in government evaluations. It also raises questions about transparency, fairness, and the potential for opaque standards to shape market access and innovation.

For AI developers, especially those competing globally, the order could impact deployment timelines, access to federal contracts, and the development of new models. The shift towards classified assessments contrasts with European models, which favor public, contestable benchmarks, highlighting a divergence in AI governance philosophies.

US AI Oversight Evolution and International Comparisons

Previously, US AI regulation was characterized by a relatively relaxed stance, with the government emphasizing voluntary collaboration. The 2026 executive order represents a pivot, with the NSA and Treasury taking more active roles in setting security standards. This follows earlier actions, such as requiring companies like Anthropic to suspend models with advanced cyber capabilities, signaling that capability assessments already influence market access. The contrast with the EU’s public systemic-risk thresholds—such as 10²⁵ FLOPs—illustrates differing approaches: transparency versus secrecy.

The move reflects concerns over AI security, dual-use capabilities, and national competitiveness, amid ongoing debates about how best to regulate fast-evolving AI technologies.

“Participation in the pre-release framework could become a de facto requirement for federal contracts, blurring voluntary and mandatory distinctions.”

— Industry observer

Unresolved Questions About Transparency and Market Impact

It remains unclear how the classified benchmarks will be developed, whether they will be challenged or reviewed publicly, and how they will influence market competition over time. The extent to which participation in the voluntary framework will become mandatory for federal contracts is also uncertain, as legal and political debates unfold.

Next Steps in US AI Security Oversight and Industry Response

Developers and industry stakeholders will need to decide whether to participate in the pre-release framework ahead of the August 1 deadline. The government is expected to finalize the classification criteria and designation process soon, potentially leading to increased reliance on trusted partner status for federal procurement. Congressional and industry discussions are likely to intensify regarding transparency, fairness, and the scope of future regulations.

Key Questions

What is a ‘covered frontier model’?

A ‘covered frontier model’ is an AI system designated by the NSA as having significant cyber capabilities, subject to classified benchmarking and oversight under the new executive order.

Will participation in the pre-release review be mandatory?

Participation is currently voluntary, but industry analysts suggest that trusted partner status—obtained through participation—may become a de facto requirement for federal contracts.

Why are the benchmarks classified?

The benchmarks are classified to prevent adversaries from exploiting detailed capability thresholds, thereby protecting national security interests.

How does this US approach compare to Europe’s AI regulation?

The EU’s AI Act uses public, contestable thresholds based on compute and risk, whereas the US is adopting a secretive, classified benchmark system emphasizing security and control.

What are the potential risks of classified benchmarks?

Opaque benchmarks may lead to unreviewable standards, potential biases, and market distortions if developers cannot challenge or verify the criteria.

Source: ThorstenMeyerAI.com