A closed security ticket does not prove the exploit is gone. Fix Review tests the submitted code against the exploit path, safety properties, and supported behavior established during the original investigation.
Submitted fix pull requests failed in two ways: the original vulnerability remained reachable, or the patch introduced another security problem. The figures carry the exact counts and denominators.
- reviews run
- 1,258
- findings checked
- 644
- repositories
- 51
- organizations
- 10
Code target for each review
460 Linked pull requestExact PR checkout
798 Current branchRepository state at review time
Scope and definitions
The published dataset contains disclosure-safe aggregates only. It excludes names and identifiers for tenants, workspaces, repositories, pull requests, findings, users, and commits.
A review is one initiated attempt against a linked pull request or the repository’s current branch. A substantive review passes the no-change preflight and ends with Fix confirmed or Fix Incorrect. The pass chart includes only findings that eventually reached confirmation.
These results are historical; the workflow figure documents current behavior.
Select the code under review
Target identity is part of the verdict. Pull-request mode checks the exact linked checkout. Branch mode reads the current default branch and is intended for a merged fix. Combining those targets would obscure which code the result covers.
Start from a finding, a filtered findings list, a signed pull-request comment, or a scheduled recheck. The workflow guide documents permissions and target-selection rules.
If the finding’s original location has not changed, the run stops before analysis. This preflight returned Fix Pending for 154 attempts within 2.2 seconds; those runs were skipped, not fast confirmations.
What failed inside submitted patches
The observed failures were semantic, not syntactic. A patch could compile, pass its local tests, and still leave the security property false.
97
issue-category entries across 62 fix pull requests
-
01 · Partial coverage
The reported input is blocked, but an equivalent encoding or sibling caller remains reachable.
Required proof
Probe alternate representations and every caller that shares the vulnerable operation.
-
02 · Late validation
The right check runs after the mutation, commit, or external side effect.
Required proof
Establish the invariant before the first irreversible operation.
-
03 · Supported-flow regression
The exploit disappears because a legitimate retry, recovery, or success path also disappears.
Required proof
Exercise the named supported flows as well as negative cases.
-
04 · New attack surface
The change adds attacker-controlled state, fallback behavior, or a new trust transition.
Required proof
Trace the changed dataflow beyond the path in the original report.
The key question is not whether a guard appears in the diff. It is whether the required property holds across equivalent inputs, sibling callers, side-effect ordering, supported flows, and the new dataflow created by the patch.
Evidence-gated verification
The diff is the target, not the source of truth. The reviewer receives the raw finding, validation records, and synthesized remediation. That packet defines the exploit path, attacker preconditions, safety requirements, supported flows, prohibited patch shapes, negative cases, adjacent state transitions, deployment assumptions, and supported code anchors. Unknowns remain explicit.
-
01 · Setup
Check out the selected PR or branch and load the evidence packet.
-
02 · Scope
Map the original exploit and every required safety property into the checkout.
-
03 · Hunt
Exercise negative cases, adjacent paths, ordering, supported flows, and deployment assumptions.
-
04 · Validate
Reject unsupported candidates and preserve missing evidence as uncertainty.
-
05 · Synthesize
Return Fix confirmed, Fix Incorrect, or insufficient context with evidence.
Fix confirmed requires evidence for every first-class safety requirement. Contradictory evidence produces Fix Incorrect; missing evidence produces insufficient context.
For the 1,003 substantive runs, median time to verdict was 37 minutes. The middle half took 27–51 minutes.
Iteration cost
A new commit invalidates the prior verdict.
Output you can act on
Fix Incorrect separates Original finding still needs attention from Additional issue found. After cross-category deduplication, 93 issue writeups remain: every one has analysis, 70 include a specific file.ext:line, and 66 include a supported next fix. Stored analysis is capped at 6,000 characters.
An unmerged pull request can earn Fix confirmed, but it also shows Pending Merge and the finding stays in Todo. After merge, the status changes to Merged and the finding moves to Fixed. A webhook records the merge; a ten-minute sweep covers missed delivery.
Limits
The verdict answers one narrow question: did the reviewed code close this finding? A patch that moves a trust boundary or changes authentication still warrants broader review. Undocumented business rules cannot be inferred reliably.
Amend the pull request and the prior result is stale. Run the review again against the new commit.