Aikido Arena Review: What the Cybersecurity AI Benchmark Actually Measures

Aikido Arena is a useful, unusually legible benchmark for one narrow question: how well does a model-investigator reason about a known vulnerability after an analysis harness has already pointed it to the relevant code? It is not, by itself, a measure of whole-repository discovery, an autonomous penetration test, or a reliable forecast of production security outcomes.

That distinction is the key to reading the Aikido Arena leaderboard. Its 32-case evaluation puts repeated discovery, report quality, and inference spending in the same view. The practical question for a security team is how much of that result transfers to its own workload.

Sources checked November 11, 2026. This methodology and published-data review is by Penligent, an AI pentesting vendor. We did not rerun Aikido’s benchmark. Model names, scores, and prices below are a dated snapshot, not a live ranking or Penligent performance claims.

What Aikido Arena tests

Aikido’s methodology describes GitHub advisory cases, pre-fix commits, a common production harness, short model-agnostic prompts, and three runs. Crucially, investigators receive the vulnerable snippet’s location. This is source-code analysis with deliberately guided attention.

This is a thoughtful design for comparing the investigator stage under a shared harness. Holding the surrounding scanner steady helps isolate differences between models. Directing attention to the vulnerable snippet also avoids spending the benchmark budget on repository navigation. Aikido deserves credit for publishing the setup and for showing the trade-off between single-run recall, pooled recall, precision, and cost rather than presenting one opaque “cyber score.”

How to read pass@1, pass@3, precision, and cost

Pass@1 on the current leaderboard is the average recall across three individual runs. It is the closest of the displayed recall metrics to what one run may deliver; it is not the first run’s score. Pass@3 is the union: a case counts if at least one of the three runs finds it. It answers how broad a pooled three-run set can become, not how often one run succeeds. The difference between these columns makes model variance visible rather than hiding it.

Precision is the share of reported findings accepted after triage, pooled across the runs. It matters because a high-recall system that sends a large volume of rejected issues to engineers may be expensive to operate. Cost is the benchmark’s estimated inference spend at listed model prices. It is useful for relative comparison under this workload; it is not a quote for a production scanner, and does not include every deployment cost such as engineering, orchestration, review, or follow-up validation.

In the snapshot reviewed, GPT-6 Astra reported 75.0% pass@1 recall, 90.6% pass@3 recall, 96.0% precision, and $3,922 for three passes across the 32 cases. DeepSeek V4.1 Flash 1017 reported 70.8%, 87.5%, 86.4%, and $67.32 respectively. Aikido lists $135.23 and $2.40 per finding for those systems. These are vendor-published benchmark figures, not Penligent test results.

The headline ranking can obscure the decision. On this snapshot, the difference between the two models’ pooled recall is one case out of 32, while their listed three-pass costs differ by more than fifty-fold. That is a useful cost-versus-coverage signal for this particular dataset. It does not show that the cheaper model is a drop-in replacement in a team’s own codebase, nor that the more expensive model is unnecessary where missed findings carry a higher cost.

What the price comparison leaves out

A team’s budget decision needs two denominators: unique accepted issues and the work required to act on them. For an internal pilot, we would calculate the following proposed operational metric:

cost per actionable issue =
  (inference + execution + human review + reproduction costs)
  / unique issues accepted with sufficient remediation evidence

This is our evaluation proposal, not an Arena metric. Count a duplicated issue once. Keep an accepted finding separate from a successfully reproduced issue if the workflow requires reproduction. If no issue satisfies the acceptance rule, report the denominator as zero and the metric as undefined; do not turn a cheap run with no usable output into an efficiency win.

Consider a fictional pilot with equal scope: system A costs $100 in execution and needs four review hours; system B costs $300 and needs one review hour. At an illustrative $100 per hour, both cost $500 before reproduction. The example does not estimate any model’s actual review burden. It demonstrates why inference price alone cannot establish the less expensive production choice.

The published Astra and DeepSeek figures also do not reveal whether their found-case sets are nested. Counts of 29 and 28 do not prove the first model found every case the second found plus one. Without the overlap matrix, we cannot price a complementary second model or claim that the additional spend buys a particular missed vulnerability. Request case-level results before designing a model cascade.

Pooling is useful, but it is not repeatability

A three-run union rewards complementary discoveries. A production owner may also care whether a finding appears consistently: a control that surfaces a defect in one run out of three has a different operating profile from one that finds it every time. Report both. Aikido’s August research article discusses run-to-run variation in more detail; its historical campaign should not be substituted for the current board.

Here is a deliberately fictional four-case example. Run one finds A and B; run two finds B and C; run three finds A and D. Each run recalls 50% of the cases, and their union recalls 100%. Yet none of the four cases was found in every run. All three statements are true. A buyer who sees only the union loses the information needed to decide whether one scheduled run is sufficient.

For a pilot, retain a case-by-run matrix with each cell marked found, missed, or incomplete. Compute the average individual recall, union coverage, and intersection coverage from that same matrix. Record timeouts and tool failures explicitly rather than silently excluding them. Incomplete execution is operationally meaningful even when it is not a model-reasoning error.

Do not extrapolate a three-run score into a promise that repeated calls will eventually find everything. Difficult cases can stay difficult across runs, and resource limits can constrain every attempt. Additional runs should earn their place through observed additional accepted cases and cost, with a defined stopping rule.

The most important limitation is also a design choice

Aikido’s investigator receives a known vulnerable location. That makes the test more controlled, and gives the benchmark a legitimate purpose: comparing code reasoning after triage has selected a candidate region. But it removes a major part of real security work. In a whole-repository audit, an agent must decide where to look, map entry points and data flows, manage search budget, and avoid stopping at a plausible but invalid theory. Arena’s result should therefore be read as investigator-stage performance inside Aikido’s harness, not as end-to-end vulnerability discovery by a bare model.

There are further limits to keep in view:

• Thirty-two cases make small score gaps fragile. One case equals 3.125 percentage points of recall. A one-case lead is worth noting, not treating as a universal capability gap.
• Three runs are still a small sample. Pass@3 exposes useful variation, but the union is not a confidence interval and does not tell a buyer how often a single future run will succeed.
• The harness is part of the tested system. Aikido holds it fixed to compare model investigators, which is a strength for that question. Teams using different tools, prompts, retrieval, permissions, context windows, or triage rules should expect different results.
• The public method is not a turnkey independent replication package. The pages explain the workflow, but the materials reviewed do not provide a complete per-case manifest, exact run artifacts, and all adjudication records needed for an outside team to reproduce the full leaderboard end to end.
• Rediscovering known, pre-fix vulnerabilities is not the same task as testing a live application. The Arena scores code-level findings; it does not establish exploitability against a deployed target, operational impact, or the quality of a remediation.

Do not confuse Aikido Arena with CyberArena

The names are easy to mix up in search. Aikido Arena is Aikido Security’s model leaderboard described above. CyberArena is a separate research benchmark: the paper by Bangya Liu and colleagues, revised October 19, 2026, describes symmetric attack-and-defense contests. Two agents simultaneously attack and defend changing services in isolated environments. The paper reports 12 agents, 40 challenges, and 466 matches. These are different tasks, rather than competing scores on the same exam.

Those scores answer different questions. Aikido’s shared-harness design is useful when the goal is to compare investigator models on a bounded known-vulnerability task. CyberArena is more relevant when the question concerns interaction between offensive and defensive agents, persistent patches, service availability, and an opponent that adapts. Neither leaderboard should be treated as a substitute for the other.

A practical way to use the leaderboard

Use Aikido Arena to form a shortlist, not to make a procurement decision by itself. Before adopting a model, run a small internal evaluation on code and issue types that match your stack. Keep the harness, repository revision, permissions, budget, and triage policy constant when comparing models. Report both single-run results and pooled results; separately count accepted findings, duplicates, reviewer time, and cost. Include cases where the correct outcome is “no vulnerability,” so precision is measured against realistic negative examples too.

This separates a model’s reasoning from the surrounding workflow. For example, take a synthetic two-user application whose requirement is that each user can read only their own record. An investigator may correctly identify a missing ownership check when shown the handler. The next stage asks whether it can find that handler on its own. Deployment validation then requires evidence using the two approved test identities, and remediation verification must preserve legitimate access. These are proposed acceptance stages, not a claim that any named model passed them.

Freeze the answer key before testing and have the evaluator adjudicate against it. Record the exact model snapshot, reasoning setting, harness revision, tool list, code revision, retry policy, resource budget, and acceptance rule. Otherwise, a model upgrade, extra hint, or changed triage standard can masquerade as a capability improvement.

For false positives, specify the denominator. Arena’s triager agreement describes accepted versus rejected reports; it is not a conventional false-positive rate over all negative code locations. Those quantities answer different questions. A conservative system may produce few rejected reports while also missing useful findings, so read precision alongside coverage and retain the underlying counts.

Then test the workflow you actually intend to deploy. If the use case is code review, include repository navigation and confirm that the agent can identify the relevant path without being shown the vulnerable line. If the use case is application testing, use authorized targets and record reproducible evidence, scope, and cleanup. A model leaderboard can inform those pilots, but only the pilot can show whether the complete workflow fits your engineering process.

Verdict: Aikido Arena is one of the more useful public cyber-model leaderboards because it makes repeatability, false positives, and cost visible alongside recall. Its strongest contribution is a disciplined comparison inside a consistent harness. Its main caveat is scope: the known-location task measures a downstream reasoning stage, not end-to-end autonomous security testing. Read it for what it tests, compare it with benchmarks that cover other stages, and validate the full workflow on your own authorized environment.

For a broader framework to compare AI pentesting workflows by scope, evidence, and cost, see Penligent’s AI pentesting tools evaluation.

Methodology: AI-assisted synthesis of Aikido’s public Arena methodology, leaderboard, and research post, checked November 11, 2026, plus the public CyberGym and CyberArena descriptions. No independent model run, Aikido product test, or security test of a live application was performed.