SecurityScorecard Launches the World’s First TPRM Benchmark to Test Whether AI Can Defend the Supply Chain and Withstand an Attack

Can AI manage third-party risk without becoming part of the risk? TPRM Benchmark tests both sides of AI in supply chain security.

NEW YORK--(BUSINESS WIRE)--SecurityScorecard today announced the launch of the world’s first AI TPRM Benchmark to evaluate how different AI models perform on real-world Third-Party Risk Management and cyber tasks as well as withstand adversarial manipulation and supply chain attacks. Built on one of the world’s largest third-party risk datasets, the benchmark gives organizations a standardized framework to objectively assess AI capability and resilience before deploying models across critical third-party risk workflows.

As organizations introduce AI into security and third-party risk workflows, they need a way to determine not only whether models can perform the work, but whether they can be trusted under adversarial conditions. Traditional AI benchmarks are not typically designed to reflect the operational realities of TPRM, where models may need to interpret security telemetry, prioritize exposures and evaluate vendor-provided information that could be incomplete, misleading, or manipulated.

The TPRM Benchmark is grounded in real-world third-party risk scenarios leveraging SecurityScorecard’s proprietary supply chain risk analysis and expertise. The result is a practical test of how AI performs against the kinds of decisions, tradeoffs, and adversarial inputs it may encounter in enterprise TPRM environments.

For the first time, the TPRM Benchmark brings two critical dimensions of AI evaluation together:

1. AI Supply Chain Security Capability: How leading AI models perform on real-world TPRM and supply chain security tasks, including exposure triage, breach correlation, vendor risk concentration analysis, and vulnerability prioritization.
2. AI Model Resilience: How well these same models defend themselves from common supply chain and adversarial attacks, such as prompt injection, jailbreaks, and deceptive vendor inputs designed to manipulate their conclusions.

SecurityScorecard’s call to action invites a select number of AI labs, developers, security vendors, and researchers to help define the tests and methods that make up the benchmark. To facilitate participation, SecurityScorecard has open sourced the framework, enabling participants to test models and agents against it and submit results to share in a public leaderboard.

“Putting an AI agent into your vendor-risk program without testing it is like giving a new employee privileged access on day one and hoping for the best,” said Dr. Aleksandr Yampolskiy, CEO and Co-Founder of SecurityScorecard. “The TPRM Benchmark answers two questions every security leader should be asking: can this model do the work, and can it be turned against me? If you can’t answer both, you don’t have an AI advantage. You have a new attack surface.”

A Two-Part Approach to Evaluating AI for Third-Party Risk

Pillar 1: What is the best AI model for TPRM, Cyber and Supply Chain Security Tasks?

The TPRM Benchmark tests models across numerous TPRM disciplines inspired by real-world supply chain risks, including exposure triage, vendor concentration, breach-exposure correlation, remediation planning, fourth-party technology risk and portfolio drift. Each task is generated from a highly realistic snapshot of SecurityScorecard’s vendor-risk telemetry and validated by domain experts before any model is scored.

Three design choices distinguish the TPRM Benchmark from prior AI benchmarks:

• Grounded in measured data. Tasks are built from SecurityScorecard telemetry rather than hypothetical scenarios, giving each evaluation a clear connection to real-world third-party risk.
• Designed to reduce benchmark contamination. The TPRM Benchmark rotates task inputs, protects answer keys and uses safeguards to make it harder for models to simply memorize benchmark content.
• Built for statistically rigorous evaluation. Results include confidence intervals and are measured against a baseline so performance reflects meaningful separation rather than one-off gains.

In testing across a wide variety of model configurations, including closed-frontier, open-weight, and international models, performance varied significantly by discipline. The TPRM Benchmark assigns each model an autonomy readiness level from L0 to L4 based on repeated-trial performance, helping organizations better understand how to best combine human-machine teams for critical cyber tasks. Answering questions, such as: when should an AI model provide increased agency and where is human oversight still required?

Pillar 2: How resilient are models to cyber and supply chain security attacks?

AI agents used in third-party risk workflows may ingest content that vendors or attackers can influence, including questionnaire responses, self-attestations, documents, and inbound communications. The TPRM Benchmark tests how models respond to deception, spin, and other deliberate manipulation of supply chain security data.

Test attacks on the model itself. The TPRM Benchmark model security suite tests models against prompt injection, jailbreak, and other sophisticated attack techniques and grades their resistance from A to F.

• Test attacks embedded in the work. Selected tasks introduce deceptive vendor self-attestations that contradict underlying security telemetry, measuring how much those inputs degrade model accuracy.

“Our initial benchmark runs show open-weight models closing the gap with leading frontier models on supply chain security tasks, but several of those same models remain highly susceptible to adversarial attack,” said Dr. Mike Mylrea, CISO and Chief AI Officer at SecurityScorecard. “As AI increasingly takes on critical security functions, organizations need to know not just which models perform best at each task, but which ones hold up when someone is actively trying to manipulate them.”

The distinction between capability and trustworthiness is also what makes an independent, rigorously designed benchmark so valuable, according to Dr. Yelena Yesha, the Knight Foundation Endowed Chair of Data Science and AI at the Frost Institute for Data Science and Computing (IDSC).

“There are many AI models on the market that deliver the results they promise, but very few adhere to the cybersecurity standards needed for real industry adoption, and that gap is exactly why we need an evaluation platform like this one. I think of a good benchmark like a race track: it shows you which models have real problems under the hood, and it reveals the true limits of what’s possible, which is what makes this work academically meaningful, about scientific discovery, not just about picking the best commercial model off the shelf,” said Dr. Yesha, the Director for the Machine Learning and AI program at IDSC. “It also matters enormously for organizations trying to use AI in third-party risk workflows specifically. It’s not enough to know a model can complete the task. Done right, a benchmark like this is how you get the confidence to actually deploy AI in these workflows, knowing it does the job well and that its security can be trusted.”

Academic experts say the benchmark’s rigor is what sets it apart. Most organizations deploying AI into supply chain workflows have no way to know whether their chosen model can withstand adversarial attack. They optimize for accuracy, cost, or speed, but rarely for resilience. That’s precisely the problem the TPRM Benchmark was built to solve.

“TPRM Bench stands out by testing AI against real supply chain scenarios grounded in empirical security data,” said Yevgeniy Dodis, Professor of Computer Science at New York University and Fellow of the IACR (International Association for Cryptologic Research). “Its rigorous, reproducible framework helps bridge the gap between benchmark performance and real-world security as AI adoption in critical infrastructure accelerates.”

Industry leaders see the same need for shared, transparent measurement.

“You can't secure an attack surface you haven't measured, and TPRM Bench gives security teams a shared, transparent way to evaluate models on capability and attack resistance together,” said Manoj Nair, Chief Technology & Innovation Officer at Snyk. “In our own Snyk research across 3,000 enterprise environments, 74.5% of those with a detected model, reference at least one that scores ‘high’ on an attacker goal under adversarial testing.”

For security leaders, the benchmark also helps decide how much trust to place in AI.

“Preempting the next cyber attack means proving your defenses before they're put to the test,” said Peter Lee, CEO of SimSpace. “Trust in AI must be earned through continuous testing in realistic, production-like conditions. SecurityScorecard's TPRM Benchmark helps build that trust by measuring whether AI can perform critical tasks and resist manipulation. For security leaders, validating AI is essential for deciding how much autonomy to give agents and for pinpointing when humans need to remain in control.”

With AI now used by both attackers and defenders, objective measurement matters even more.

“AI is now part of both attacker and defender tooling, so unexpected model behavior can quickly become an incident for one side or an opportunity for the other. At GreyNoise, we've seen attackers' AI use leave identifiable tells, and we've seen defenders' models make mistakes analyzing packets and TTPs,” said Corey Bodzin, Chief Product Officer of GreyNoise. “Objective, transparent measurement like SecurityScorecard's TPRM AI Benchmark puts evidence of model performance on the table, so defenders can adopt AI with a clear view of its strengths and limits.”

Open Access and Industry Participation

The TPRM Benchmark is open to a select number of AI labs, model providers, security vendors, enterprise security teams, and researchers. Participants can test models and agents against the public benchmark, compare results across capability and attack resistance, and submit results to the open arena.

Get started at tprmbenchmark.com to read the technical report and register for the gym.

Availability

The TPRM Benchmark public tier, arena access, model security suite, and documentation are available today at tprmbenchmark.com/tprm-bench/#evaluate.

Official expert-verified scores are measured on SecurityScorecard's live internal tier; public-tier results are directly comparable for research and self-assessment and are labeled as synthetic-calibrated. Enterprises can request a certified evaluation of a candidate model or agent at [e-mail].

About SecurityScorecard

SecurityScorecard is the global leader in threat-informed third-party risk management (TPRM), securing the world's supply chains. The company delivers a threat-informed approach to TPRM that enables organizations to drive out risk at the source. The platform uses continuous visibility and AI-accelerated workflows to help organizations reduce third-party risk before incidents occur and respond with confidence when they do.

Trusted by 3,000 organizations, including over 70% of the Fortune 100, and recognized as a trusted resource by the U.S. Cybersecurity and Infrastructure Security Agency (CISA). Backed by Evolution Equity Partners, Silver Lake Partners, Sequoia Capital, Google Ventures, and Riverwood Capital, SecurityScorecard delivers end-to-end supply chain cybersecurity that safeguards business continuity.

Protect the supply chain behind your business. Learn more at securityscorecard.com.