TL;DR:
We measured 1,610 production runs of Apex, Cantina’s AI bug hunter. At a calibrated budget it recovered 100% of the criticals and highs from human penetration tests.
More compute, more findings, and we can show you the curve
When you give Apex more compute, you find more bugs. Across 1,610 production runs on real codebases, the same shape shows up every time, and it’s sub-linear. Doubling your compute adds about 40% more findings, not 100%.
Each dot is one run. Both axes are rescaled so 1× equals a median-budget run, which puts runs on very different codebases on a comparable plot. The dark line is the cohort median; the shaded band is the middle 50% of runs. The dashed line is a fit (findings ∝ compute^0.54). That’s where the “40% per doubling” comes from.
“Validated” means a finding survived Apex’s automated validator stage, same definition used in every chart below.
A curve through a cloud of dots could just be an average hiding a lot of noise. It isn’t. When we fit the same power law to each run separately (1,033 runs with enough trajectory to fit), the exponent clusters tightly around the cohort value. The system doesn’t scale efficiently on average; it scales the same way almost every run.
It goes further than a shared exponent. For the runs that go long enough to bend toward their ceiling, if you rescale each one by its own two fitted constants, the compute it takes to get halfway, and the findings it tops out at, every trajectory falls onto a single master curve. Different codebases, different budgets, one shape.
What this means for buyers
Apex’s return-on-compute is predictable: the same sub-linear law holds run after run, codebase after codebase. Doubling compute adds about 40% to your finding count, not 100%; the next dollar still buys findings, but each one buys fewer than the last. Because the curve is universal, your budget maps to expected coverage in advance. You’re planning against measured behaviour, not a hope.
A ceiling exists, and we can tell when an agentic penetration test hits it
Some agentic penetration tests saturate: past a point, more compute stops yielding new findings. Among long agentic penetration tests, about one in four (24%) shows a clear ceiling, but the more striking number is the other side: 74% were still scaling at the budget they were given, with only a 2% sliver too ambiguous to call. Most agentic penetration tests, in other words, still had headroom when they stopped.
The bend is visible directly. Each line is one agentic penetration test, with both axes normalised by that test’s own totals: every curve runs (0%, 0%) → (100%, 100%), so the only thing that varies is the shape. Saturating agentic penetration tests collect most findings early and flatten; still-scaling tests track closer to the linear diagonal. We classify the shape with a standard model-selection test (Hill curve vs. power law); the caption gives the exact threshold.
This is the same saturation that produced the master curve in §1. The Hill fit that flags a ceiling here is exactly the fit that hands each run its two constants, half-saturation compute and finding ceiling, and once you rescale by those, these saturating trajectories are the points that collapse onto the universal curve. Saturation isn’t a separate phenomenon from the scaling law; it is the scaling law, seen at the budgets where it starts to flatten.
The histogram below shows how the cohort splits on this score. The mass sits overwhelmingly to the right (still-scaling, 74%) with a clear-ceiling tail on the left (24%); the indistinguishable middle between −4 and +4 is only 2%, so this is close to a clean two-way split, not a three-way one.
What this means for buyers
Most agentic penetration tests (74%) were still scaling at the budget they were given, meaning more compute would likely have bought more findings. Only about a quarter had clearly saturated, where additional compute is mostly wasted spend. The useful part is that the two cases are distinguishable from the trajectory, and the agentic penetration test acts on it directly: it auto-detects saturation as it runs and stops once it has reached 95% of the estimated finding ceiling. You don’t over-pay on a codebase that has flattened, and you don’t stop short on one that’s still climbing. One honest caveat: a flat trajectory can mean Apex hit a wall or that the codebase genuinely had few bugs left; telling those apart needs the absolute counts, not just the shape.
Severity composition remains roughly flat throughout an agentic penetration test
If you split each agentic penetration test’s progress into 10 equal slices and look at what proportion of findings in each slice are critical, high, medium, low, and informational, the bands barely move. Critical hovers around 5–11% per decile, high around 23–35%, medium around 26–34%. No single decile is preferentially loaded with high-severity findings; any slice of an agentic penetration test is roughly a representative sample of its severity mix.
The one mild trend: high-severity findings are slightly over-represented early (34% in the first decile, 23% in the last) and informational is slightly over-represented late. That’s the only directional shift visible in the data; everything else is decile-to-decile noise.
What this means for buyers
You can’t game an agentic penetration test by stopping early. Severity composition is roughly flat across deciles: a half-budget run returns about half the findings of a full-budget run, distributed across severities in nearly the same proportions. There’s a mild drift (high-severity findings concentrate slightly early, informational slightly late) but it’s small relative to the per-severity counts you lose by cutting compute. Critical and medium counts both scale with budget at essentially the same per-decile rate as total findings.
Reproducibility, broken down by severity
If you rerun the same agentic penetration test on the same codebase, the results will differ, and how much they differ depends on severity and budget. We measured it two ways: a stress test (44 runs on one codebase, deliberately varied across budgets and configurations) that bounds the worst case, and a controlled replicate (3 strictly identical reruns) that shows what reproducibility looks like once that variation is removed.
The solid lines are the two launch-relevant severity classes, critical and high, across the varied-config family. The y-axis is the coefficient of variation in cumulative synth-valid findings at a given compute level, standard deviation divided by mean. Lower is more reproducible. Both drop roughly an order of magnitude as compute grows, from 200–400% CV at small budgets down into the 30–55% range by mid-budget.
The dotted lines are the controlled version of the same test: 3 strictly identical runs, same codebase, same configuration, no deliberate variation. They sit dramatically lower, and both converge to ~5–6% CV by 0.8× of their budget: critical 5%, high 6%, against ~137% and ~95% for the varied-config family at the same relative budget. Strip out config drift and the hunt is near-deterministic. Hover any point to read its CV.
What this means for buyers
Variability shrinks roughly an order of magnitude as compute grows, and a strict same-config re-run is far more reproducible than the varied-config envelope suggests. On 3 identical re-runs, the hunt converged to ~5–6% CV at both critical and high by ~0.8× budget, near-deterministic.
We calibrated the agentic penetration test to match a human penetration test on every codebase
§1 showed Apex’s return-on-compute is predictable and universal. That’s the foundation for a product decision: we engineered the agentic penetration test to climb that curve efficiently, then calibrated how much compute it spends so it always reaches the point where it has matched an independent human penetration test’s critical and high findings, including engagements that cost $500k+ and ran for months. Across the 8 codebases where we have a human penetration test to check against, it did: every critical, every high.
An agentic penetration test is the configuration we run when coverage is the goal: higher per-task budget, deeper exploration, stricter validation. We picked the 8 rows below by availability: every codebase where both an Apex agentic penetration test and an independent human penetration test exist, with no further filtering. Each cell is bugs Apex recovered / human-confirmed bugs the penetration tester posted (any non-withdrawn status). Green = 100%; n/a = no human bugs of that severity on that codebase.
Pool stats across 8 engagements
Per-engagement coverage by severity across the 8 engagements where Apex ran an agentic penetration test against an independent human penetration test on the same codebase. Each cell is bugs Apex recovered / human-confirmed bugs the penetration tester posted (any non-withdrawn status). Green cells are 100% recovery; n/a means the human penetration test produced zero bugs of that severity on that codebase, so coverage is undefined. The bottom row is pool coverage: every human-confirmed bug counted once across all 8 engagements.
Pool stats across all 8: 9/9 critical, 32/32 high, 45/54 medium (83%), combined 86/95 = 91%. On 5 of the 8 codebases, medium coverage is below 100%; the misses are real and concentrated at that severity.
And this table understates what Apex did. It only credits Apex for matching the human penetration tester’s findings. It gives no credit for what Apex found beyond them. On multiple of these engagements, the agentic penetration test surfaced critical and high-severity bugs that the original human penetration test missed entirely, still live in the codebase, which we reported to the teams to fix. The 100% you see here is matched-to-human; the actual finding set was larger.
The calibration mechanism is the saturation stop from §2: the run climbs its own curve and halts once it reaches 95% of its estimated finding ceiling, far enough to clear the human-match point, without burning compute on a codebase that has already flattened. The figures below place each engagement on its own scaling curve from §1. For each codebase, the bars labeled “Compute to match 100% crit + high” and “Where the run reached” show the two measured points. The diamond labeled “Human penetration test cost” shows where that codebase’s actual human penetration test budget would land on the same axis, if it were spent on AI compute at our blended provider rate. Everything is rescaled by the cohort-median run compute (1× = a median production run).
What this means for buyers
The agentic penetration test is calibrated to reach a human penetration test’s critical and high findings, and on every codebase here it did, at a fraction of the compute that the human penetration test budget would buy. Because the scaling curve from §1 is universal, that calibration travels: the same compute target lands at the match point codebase after codebase, not just on average. Two honest qualifiers: “match” is measured against the human’s confirmed set (and as noted above, Apex frequently found more, including live critical and high-severity findings the human penetration testers missed); and this is measured on 8 codebases with human ground truth, a strong, consistent signal rather than a population guarantee.
Run an agentic penetration test on your code
Point Apex at your repository and get critical and high coverage fast. Buying for a portfolio or running a firm? Talk to us about bundle pricing.
