arXiv: Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

A new academic study published on arXiv, titled Benchmark Scores Are Pipeline-Dependent, reveals that reported performance scores for cybersecurity large language models are not reliable indicators of real-world capability. The authors audited multiple popular benchmarks and found that minor changes in the evaluation pipeline, such as prompt formatting, answer extraction methods, or scoring thresholds, can produce significantly different results for the same model. This means that a model that appears to excel at tasks like vulnerability detection or phishing analysis in one test setup may perform poorly in another, undermining the validity of current benchmark comparisons.

This finding directly affects any organization that uses or procures AI-powered cybersecurity tools, including managed security service providers, financial institutions, critical infrastructure operators, and enterprise security teams. It also impacts vendors who market their models based on benchmark claims, as well as regulators and auditors who rely on such metrics to assess the safety and efficacy of AI systems under frameworks like the EU AI Act. Any compliance decision that depends on quantitative model performance is now subject to uncertainty.

Compliance teams should immediately treat all existing benchmark scores as provisional and require vendors to disclose their full evaluation pipeline, including code, prompts, and scoring logic. They should also demand independent, pipeline-agnostic testing for high-risk use cases and document any performance claims in their AI risk registers. Moving forward, procurement and internal validation processes must include reproducibility checks, and any model deployment should be accompanied by a clear audit trail of how its performance was measured. This study is a call to shift from trusting headline numbers to verifying the entire evaluation methodology.