Gating an agent’s eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. severity_gate.py is an offline, stdlib-only tool that returns SHIP, REVIEW, or BLOCK. In this post’s fixtures, flipping one severity_class field to critical turns SHIP into BLOCK while the pass-rate holds at 92.5%.

AI disclosure: I wrote severity_gate.py with an AI assistant and ran it myself, offline, before publishing. Every output block below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes (0 / 1 / 2), hashed the STDOUT of each scenario twice to confirm it is byte-for-byte deterministic, and edited every line. The 512 / 31 / 6 figures and the “rounding error” line belong to Ethan Walker’s Dev.to post, not to me; I link the primary sources and keep their numbers in their own paragraphs, away from my fixture counts (40 / 3 / 2, 92.5%).

In short:

• Whether a run is safe to ship is a property of which failures it holds, not of their average. A run can sit at a healthy pass-rate and still carry the one failure that should have stopped the deploy on its own.
• The tool reads a finished eval run (JSON) plus a small policy. Any FAIL in a blocking severity class (default: critical) is a BLOCK, whatever the aggregate says. A below-threshold aggregate with no critical fail is a REVIEW. Everything clean is a SHIP.
• The demo: same 40 cases, same 37/40 = 92.5% pass-rate, same three failing cases. Change one field, severity_class from low to critical on two of them, and the verdict flips SHIP to BLOCK. The pass-rate does not move. The diff is two lines.
• It re-weights failures you already labeled. It does not detect PII, does not judge correctness, does not find new failures. Garbage labels in, garbage gate out.
• Standard library only (json, sys). Offline, keyless, read-only, zero network, deterministic STDOUT. Exit 0 / 1 / 2 for a CI gate. The tool and every fixture are in this post.

A flat pass-rate measures the run. It was never the gate.

Here is the ritual I keep seeing at the end of an agent’s CI pipeline. The eval suite runs, a number comes out, someone reads it, and if it clears the bar the change ships. The number is a mean: passes over total. It is a fine thing to track. It tells you, roughly, whether the run got better or worse than last week.

It is a bad thing to gate on, and the reason is simple arithmetic. A mean gives every failure the same weight. So a run where forty tests failed on trailing whitespace and a run where one test leaked a customer’s email into a log both cost the same amount of pass-rate. If your suite is big enough, the second run rounds off to nothing. The one failure that should stop the deploy is invisible at the resolution of the average.

That is the whole gap this tool sits in. The pass-rate is tracking: a post-hoc measurement of what happened. The decision about what ships is control, and control has to see the distribution the mean flattens. severity_gate.py reads the finished eval results and makes the ship decision on the failures’ severity, not their count.

The rule the mean can’t express

The tool keeps the pass-rate on screen, for reference, and then ignores it for the parts of the decision the mean can’t reach. Three verdicts, one policy:

• BLOCK when at least one failing case is in a blocking severity class. Default policy: critical. This fires no matter what the aggregate is. One critical fail out of ten thousand green cases is still a BLOCK.
• REVIEW when nothing blocking failed, but either a case in a review class failed (default: high), or the flat pass-rate dropped below ship_threshold. A human looks. It is not an auto-ship and it is not a hard stop.
• SHIP only when nothing blocking or review-class failed and the pass-rate meets the threshold.

The severity labels are yours. The tool does not invent them and does not guess them; it reads the severity_class field you put on each case. That is the load-bearing limit, and I come back to it at the end. What the tool contributes is that once you have committed to a label, the ship decision follows it deterministically instead of dissolving into an average.

Run it in sixty seconds

No keys. No network. No install beyond Python. Save the file, point it at a results JSON, run one command. Here is the whole thing, one file, standard library only:

#!/usr/bin/env python3
"""
severity_gate.py -- an offline deploy gate that reads a FINISHED eval run and
decides SHIP / REVIEW / BLOCK from the per-severity distribution of the failures,
not from a single flat pass-rate.

It reads a JSON file of eval RESULTS (it never runs the agent or the eval) plus
an optional policy file. For every case it reads a verdict (pass/fail) and the
severity_class the USER assigned (critical / high / medium / low). It reports the
flat pass-rate for reference, then makes the ship decision on a rule the mean
cannot express:

  BLOCK   -- at least one FAIL whose severity_class is in the policy's block_on
             set (default: critical). Fires regardless of the aggregate.
  REVIEW  -- no blocking fail, but either a FAIL in review_on (default: high),
             or the flat pass-rate is below ship_threshold. A human looks; it is
             not an auto-ship and not a hard block.
  SHIP    -- no blocking fail, no review fail, and pass-rate >= ship_threshold.

The point the tool exists to make: flip one field -- the severity_class on a
handful of already-failing cases -- and the verdict moves SHIP -> BLOCK while the
pass-rate does not move at all. The aggregate MEASURES the run; it was never the
control over what ships.

Offline. Keyless. Read-only. Zero network. Standard library only (json, sys).
No subprocess, no exec, no eval, no import of the analyzed results, no model, no
DB. The fixtures are read with json.load; they are DATA, never executed. Output
is byte-for-byte deterministic across runs.

It does NOT detect PII, does NOT judge whether a verdict is correct, and does NOT
find new failures. It re-weights failures YOU already labeled. Garbage labels in,
garbage gate out. It decouples the ship decision from the scalar mean; it does
not discover anything.

Exit codes (usable as a CI gate):
  0  SHIP
  1  REVIEW or BLOCK  (the verdict text says which, and why)
  2  bad input (missing / unreadable / unparseable file, missing field, unknown
     verdict or severity_class) -- fail-closed

Usage:
  python3 severity_gate.py <results.json> [policy.json]
"""

import json
import sys

DEFAULT_POLICY = {
    "ship_threshold": 0.90,
    "severity_order": ["low", "medium", "high", "critical"],
    "block_on": ["critical"],
    "review_on": ["high"],
}

VALID_VERDICTS = ("pass", "fail")


def _bad(msg):
    print("ERROR: " + msg)
    raise SystemExit(2)


def _load_json(path, what):
    try:
        with open(path, "r") as fh:
            return json.load(fh)
    except OSError as exc:
        _bad("cannot read %s %s: %s" % (what, path, exc))
    except ValueError as exc:
        _bad("cannot parse %s %s: %s" % (what, path, exc))


def load_policy(path):
    if path is None:
        return dict(DEFAULT_POLICY)
    data = _load_json(path, "policy")
    if not isinstance(data, dict):
        _bad("policy %s is not a JSON object" % path)
    pol = dict(DEFAULT_POLICY)
    pol.update(data)
    order = pol["severity_order"]
    if not isinstance(order, list) or not order:
        _bad("policy severity_order must be a non-empty list")
    for key in ("block_on", "review_on"):
        for cls in pol[key]:
            if cls not in order:
                _bad("policy %s lists '%s', not in severity_order %s"
                     % (key, cls, order))
    try:
        pol["ship_threshold"] = float(pol["ship_threshold"])
    except (TypeError, ValueError):
        _bad("policy ship_threshold must be a number")
    return pol


def load_cases(path, order):
    data = _load_json(path, "results")
    cases = data.get("cases") if isinstance(data, dict) else data
    if not isinstance(cases, list) or not cases:
        _bad("results %s must hold a non-empty list of cases" % path)
    default_sev = order[0]
    out = []
    for i, c in enumerate(cases):
        if not isinstance(c, dict):
            _bad("case %d is not an object" % i)
        verdict = c.get("verdict")
        if verdict not in VALID_VERDICTS:
            _bad("case %s has verdict %r, expected 'pass' or 'fail'"
                 % (c.get("case_id", i), verdict))
        sev = c.get("severity_class", default_sev)
        if sev not in order:
            _bad("case %s has severity_class %r, not in severity_order %s"
                 % (c.get("case_id", i), sev, order))
        out.append({
            "case_id": str(c.get("case_id", "case-%d" % i)),
            "verdict": verdict,
            "severity_class": sev,
            "category": str(c.get("category", "")),
        })
    return out


def decide(cases, pol):
    block_on = set(pol["block_on"])
    review_on = set(pol["review_on"])
    total = len(cases)
    passed = sum(1 for c in cases if c["verdict"] == "pass")
    rate = passed / total

    block_fails = [c for c in cases
                   if c["verdict"] == "fail" and c["severity_class"] in block_on]
    review_fails = [c for c in cases
                    if c["verdict"] == "fail" and c["severity_class"] in review_on]

    if block_fails:
        verdict, code = "BLOCK", 1
        reason = ("%d failing case(s) in a blocking severity class (%s)"
                  % (len(block_fails), ", ".join(sorted(block_on))))
    elif review_fails:
        verdict, code = "REVIEW", 1
        reason = ("%d failing case(s) in a review severity class (%s)"
                  % (len(review_fails), ", ".join(sorted(review_on))))
    elif rate < pol["ship_threshold"]:
        verdict, code = "REVIEW", 1
        reason = ("flat pass-rate %.1f%% is below ship_threshold %.1f%%"
                  % (rate * 100, pol["ship_threshold"] * 100))
    else:
        verdict, code = "SHIP", 0
        reason = ("no blocking or review-class failure, pass-rate %.1f%% "
                  ">= ship_threshold %.1f%%" % (rate * 100,
                                                pol["ship_threshold"] * 100))

    return {"total": total, "passed": passed, "failed": total - passed,
            "rate": rate, "verdict": verdict, "reason": reason, "code": code}


def render(cases, pol, d):
    order = pol["severity_order"]
    rank = {c: i for i, c in enumerate(order)}
    block_on, review_on = set(pol["block_on"]), set(pol["review_on"])
    out = ["SEVERITY-GATE REPORT"]
    out.append("cases: %d   pass: %d   fail: %d"
               % (d["total"], d["passed"], d["failed"]))
    out.append("flat pass-rate: %.1f%% (%d/%d)"
               % (d["rate"] * 100, d["passed"], d["total"]))
    out.append("ship_threshold: %.1f%%   block_on: [%s]   review_on: [%s]"
               % (pol["ship_threshold"] * 100,
                  ", ".join(pol["block_on"]), ", ".join(pol["review_on"])))
    out.append("per-severity (fail / total):")
    for cls in reversed(order):  # worst first
        tot = sum(1 for c in cases if c["severity_class"] == cls)
        fl = sum(1 for c in cases
                 if c["severity_class"] == cls and c["verdict"] == "fail")
        mark = "  <- blocking" if cls in block_on else (
            "  <- review" if cls in review_on else "")
        out.append("  %-8s %d / %d%s" % (cls, fl, tot, mark))
    fails = [c for c in cases if c["verdict"] == "fail"]
    fails.sort(key=lambda c: (-rank[c["severity_class"]], c["case_id"]))
    if fails:
        out.append("failing cases (worst severity first):")
        for c in fails:
            cat = (" [%s]" % c["category"]) if c["category"] else ""
            out.append("  - %-8s %s%s" % (c["severity_class"], c["case_id"], cat))
    out.append("decision: %s -- %s" % (d["verdict"], d["reason"]))
    return "\n".join(out)


def main(argv):
    if len(argv) not in (2, 3):
        print("usage: severity_gate.py <results.json> [policy.json]")
        raise SystemExit(2)
    pol = load_policy(argv[2] if len(argv) == 3 else None)
    cases = load_cases(argv[1], pol["severity_order"])
    d = decide(cases, pol)
    print(render(cases, pol, d))
    raise SystemExit(d["code"])


if __name__ == "__main__":
    main(sys.argv)

The inputs are eval results, one JSON object per run: a cases list where each case has a verdict and a severity_class. To keep the runs below reproducible byte for byte, I generate the sample files with a tiny builder that only writes data. The gate reads that data; it never runs it.

#!/usr/bin/env python3
# make_fixtures.py -- writes the sample eval-result files this post runs on.
# It only emits DATA. severity_gate.py reads it with json.load, never executes it.
import json
import os

os.makedirs("fixtures", exist_ok=True)


def case(i, verdict="pass", sev="low", cat="general"):
    return {"case_id": "c%03d" % i, "verdict": verdict,
            "severity_class": sev, "category": cat}


def write(name, cases):
    with open("fixtures/%s" % name, "w") as fh:
        json.dump({"cases": cases}, fh, indent=2, sort_keys=True)
        fh.write("\n")


# 40 cases, 3 fail -> flat pass-rate 37/40 = 92.5%. Two fails are a PII cluster
# left untriaged (severity low); one is an unrelated format miss (medium).
base = [case(i) for i in range(1, 41)]
base[9] = case(10, "fail", "medium", "output_format")
base[19] = case(20, "fail", "low", "pii_leak")
base[29] = case(30, "fail", "low", "pii_leak")
write("ship_run.json", base)

# Same 40 cases, same 37/40 = 92.5%. ONLY the two PII fields change: low->critical.
killer = [dict(c) for c in base]
killer[19]["severity_class"] = "critical"
killer[29]["severity_class"] = "critical"
write("block_run.json", killer)

# Aggregate below the bar, nothing critical/high: 34/40 = 85.0%.
review = [case(i) for i in range(1, 41)]
for i in (5, 12, 18, 23, 31, 37):
    review[i - 1] = case(i, "fail", "medium", "verbosity")
write("review_run.json", review)

# A critical-severity case that PASSED must not block. 20 cases, one low fail.
edge = [case(i) for i in range(1, 21)]
edge[0] = case(1, "pass", "critical", "pii_leak")
edge[14] = case(15, "fail", "low", "typo")
write("edge_run.json", edge)

# Unknown severity label -> fail-closed (exit 2).
bad = [case(1), case(2, "fail", "catastrophic", "pii_leak")]
write("bad_run.json", bad)

The baseline: a green run that isn’t safe to ship

Start with ship_run.json: 40 cases, 37 pass, 3 fail. The pass-rate is 37/40 = 92.5%, comfortably over the 90% bar. Two of the three failures are a pii_leak cluster that nobody has triaged yet, so they carry the default low severity. The third is a formatting miss, medium. My fixture, my run:

$ python3 severity_gate.py fixtures/ship_run.json
SEVERITY-GATE REPORT
cases: 40   pass: 37   fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0%   block_on: [critical]   review_on: [high]
per-severity (fail / total):
  critical 0 / 0  <- blocking
  high     0 / 0  <- review
  medium   1 / 1
  low      2 / 39
failing cases (worst severity first):
  - medium   c010 [output_format]
  - low      c020 [pii_leak]
  - low      c030 [pii_leak]
decision: SHIP -- no blocking or review-class failure, pass-rate 92.5% >= ship_threshold 90.0%

Exit 0. SHIP. This is the run that ships today under a flat pass-rate: the number is green, nothing is labeled critical, out the door it goes. The two pii_leak cases are sitting right there in the report, but at severity low they are just two more failures the mean absorbed.

One field flips ship to block

This is the demo the post exists for. block_run.json is ship_run.json with exactly one thing changed: the two pii_leak cases, the ones already failing, are relabeled from low to critical. Same 40 cases. Same 37 passes. Same 92.5%. The diff:

$ diff fixtures/ship_run.json fixtures/block_run.json
120c120
<       "severity_class": "low",
---
>       "severity_class": "critical",
180c180
<       "severity_class": "low",
---
>       "severity_class": "critical",

Two lines. Nothing about the pass-rate changed, because nothing about the pass/fail counts changed. Now run the gate on it:

$ python3 severity_gate.py fixtures/block_run.json
SEVERITY-GATE REPORT
cases: 40   pass: 37   fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0%   block_on: [critical]   review_on: [high]
per-severity (fail / total):
  critical 2 / 2  <- blocking
  high     0 / 0  <- review
  medium   1 / 1
  low      0 / 37
failing cases (worst severity first):
  - critical c020 [pii_leak]
  - critical c030 [pii_leak]
  - medium   c010 [output_format]
decision: BLOCK -- 2 failing case(s) in a blocking severity class (critical)

Exit 1. BLOCK. The pass-rate line is identical to the SHIP run, character for character: flat pass-rate: 92.5% (37/40). The deploy verdict inverted anyway. This is the thing to sit with. If the mean were a valid ship criterion, a change that leaves the mean untouched could not change the decision. It changed the decision. So the mean was not the criterion; it was a number we let stand in for one.

And it would take one, not two. Relabel a single pii_leak case to critical and the gate blocks, because the policy tolerates zero critical failures. The cluster size is not the point. The point is that a severity edit with no effect on the aggregate has total effect on the verdict.

The falsifiability test: when nothing is critical, the aggregate still rules

If the tool blocked on everything, or ignored the pass-rate entirely, it would be a different kind of useless. So here is the counter-case that would break the thesis if it came out wrong. review_run.json drops the aggregate below the bar, 34/40 = 85.0%, but every failure is benign: six medium cases, nothing critical, nothing high.

$ python3 severity_gate.py fixtures/review_run.json
SEVERITY-GATE REPORT
cases: 40   pass: 34   fail: 6
flat pass-rate: 85.0% (34/40)
ship_threshold: 90.0%   block_on: [critical]   review_on: [high]
per-severity (fail / total):
  critical 0 / 0  <- blocking
  high     0 / 0  <- review
  medium   6 / 6
  low      0 / 34
failing cases (worst severity first):
  - medium   c005 [verbosity]
  - medium   c012 [verbosity]
  - medium   c018 [verbosity]
  - medium   c023 [verbosity]
  - medium   c031 [verbosity]
  - medium   c037 [verbosity]
decision: REVIEW -- flat pass-rate 85.0% is below ship_threshold 90.0%

Exit 1, but REVIEW, not BLOCK, and the reason names the aggregate, not a severity class. This is the third state doing real work. The gate did not throw the pass-rate away; when nothing dangerous failed, the aggregate is exactly what it falls back on. REVIEW is where a low mean lands. BLOCK is reserved for the failure class you said must never ship. Put this run next to the killer and the tool’s shape is clear: it answers to the distribution of severity when severity is present, and to the mean when it is not.

A critical case that passed is not a block

One boundary a careful reader will poke at. Does BLOCK fire on the presence of a critical case, or on a critical failure? It has to be the failure, or you could never ship a suite that tests a critical path at all. edge_run.json has a critical-severity case that passed (your PII redaction test ran and came back clean) and one unrelated low failure.

$ python3 severity_gate.py fixtures/edge_run.json
SEVERITY-GATE REPORT
cases: 20   pass: 19   fail: 1
flat pass-rate: 95.0% (19/20)
ship_threshold: 90.0%   block_on: [critical]   review_on: [high]
per-severity (fail / total):
  critical 0 / 1  <- blocking
  high     0 / 0  <- review
  medium   0 / 0
  low      1 / 19
failing cases (worst severity first):
  - low      c015 [typo]
decision: SHIP -- no blocking or review-class failure, pass-rate 95.0% >= ship_threshold 90.0%

Exit 0. SHIP. Note the per-severity line: critical 0 / 1 — one critical case exists, zero critical cases failed. A passing test on a critical path is exactly the signal you want; blocking on it would punish you for having good coverage. The gate blocks on critical fail, not on critical present.

Who decides what blocks is policy, not the tool

The default policy blocks on critical and reviews on high. That default is mine, and you should not keep it without thinking. The mapping from a failure category to a severity class, and the choice of which classes stop a deploy, is a decision your team owns, not something a linter should hand you. So the policy is a file you pass in. Here is fixtures/strict_policy.json — four lines that also block on high and send medium to review:

{
  "ship_threshold": 0.90,
  "block_on": ["critical", "high"],
  "review_on": ["medium"]
}

Point the same ship_run.json at it:

$ python3 severity_gate.py fixtures/ship_run.json fixtures/strict_policy.json
SEVERITY-GATE REPORT
cases: 40   pass: 37   fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0%   block_on: [critical, high]   review_on: [medium]
per-severity (fail / total):
  critical 0 / 0  <- blocking
  high     0 / 0  <- blocking
  medium   1 / 1  <- review
  low      2 / 39
failing cases (worst severity first):
  - medium   c010 [output_format]
  - low      c020 [pii_leak]
  - low      c030 [pii_leak]
decision: REVIEW -- 1 failing case(s) in a review severity class (medium)

Exit 1. Same run, same 92.5%, but under a policy that treats a medium format miss as review-worthy, the verdict is REVIEW. The tool did not decide that a format miss matters. You did, in the policy file. The gate just applied it the same way every time.

Ethan’s 94% run, stated as his

I did not come up with this from nothing. On July 5, Ethan Walker published a Dev.to post whose title is the whole argument in one line: a 94% pass rate hid a PII leak in 6 test cases. His numbers, from his run, stated as his: 512 test cases, 31 failures, and 6 of those were a PII leak. His line, which I checked against the post before quoting: “Six out of 512 is a rounding error against a flat pass-rate metric.” He also names the tools by hand — DeepEval, Promptfoo, LangSmith — and writes that they “all give you this by default; none of them force severity weighting on you.”

Those figures are his and none of the counts in my runs above are. I built fixtures of a different size on purpose so the two never blur: my numbers are 40 / 3 / 2 at 92.5%, computed by my tool on my synthetic files; his are 512 / 31 / 6 at 94%, from his own eval suite. The tool in this post is one narrow, runnable answer to the thing his title describes: the point where a small critical cluster hides inside a green aggregate.

The general form of it was put well the same week by another writer, posting as aiexplore369zoho, under the heading the mean is lying to you. Their framing: “A benchmark score is a mean.” Reliability, they argue, is a tail statistic, not a central one; the mean can stay flat while the failure rate on a critical slice doubles. That is the concept. Naming the slice that must not fail and gating on it directly, instead of on the average that buries it, is one way to act on it.

Why not just add a smarter judge?

A fair objection: skip the labels, put a stronger model in the loop as an inspector, and let it catch the critical cases. Someone tested exactly that. Posting as zxpmail, they ran three models as agent quality inspectors and reported the opposite of a free lunch: the stronger the model, the more valid work it rejected. Their strongest model reached 0% false positives on garbage but falsely rejected 3 of 4 perfectly valid outputs. Those are their measurements, not mine.

The takeaway I draw for a gate is narrow. A probabilistic inspector trades one error for another and gives you a different answer on reruns. A gate over labels you already committed to is deterministic: same input, same verdict, every run. That determinism is not a nice-to-have here; it is the property the SHA hashes at the end are there to prove.

Where this sits next to the rest

This is a spoke on the pre-execution gate for AI agents cluster, and its object is the deploy decision over a finished eval run. The neighbors ask adjacent questions, and the differences matter:

• The green-checkmark auditor is the inverse case, and worth stating plainly so the two do not get merged. There, the green carries no signal at all: mirror tests that restate the code, with nothing that can ever go red. Here, the signal is present — the failing cases are real and correctly failing — and the mean drowns it. One post is about a checkmark that means nothing; this one is about failures that mean something and get averaged into silence.
• The eval-contamination probe asks whether the score can even be trusted, whether the harness graded the real work or a fabricated artifact. That is a question about whether the number is valid. This post assumes the number is valid and asks whether a valid mean is a valid ship criterion. Different link in the chain.
• Reconciling a scorecard from evidence is the case where the claimed metric might be false — the log does not back it up. Here the 92.5% is true. I am not disputing the number; I am disputing that a true average is what should decide the deploy.
• Your agent returns 200 and lies checks one runtime result against its evidence. That is a single call. This is the aggregate over a whole eval run, the pre-deploy decision that sits above all those single calls.

What this is NOT

I would rather undersell this than have you deploy it as something it isn’t.

• Severity-based gating is not my invention, and I am not claiming it is. CVSS scores security findings by severity; DREAD ranked them; every bug tracker has had a Critical / Major / Minor field for decades; risk-based testing has weighted test outcomes for a very long time. None of that is new. What is narrow here is the target: the agent-eval deploy gate, where a screenshot of a flat pass-rate is still the shipping ritual, packaged as a portable stdlib gate over the JSON your eval tool already emits. I am applying an old principle to a place that mostly still ignores it, not discovering the principle.
• It acts only on the labels you assign. It does not detect PII. It does not judge whether a fail is really a fail or a pass is really a pass. It finds no new failures. Mislabel a critical failure as low and it ships; that is the GIGO limit, and it is real. The tool re-weights failures you already found and classified. It decouples the ship decision from the scalar mean. It does not discover anything.
• It reads results; it does not run the agent or the eval. It is an offline post-processor for a finished run, meant for the step between “eval done” and “merge/deploy.” To generate the verdicts and severities in the first place you still need your eval framework.
• It does not replace DeepEval, Promptfoo, or LangSmith. It is a decision layer that consumes their output. Their default is a flat pass-rate; this is the thin gate you bolt on top of the JSON they produce, so the ship decision reads severity instead of the average. That is the whole seam it fills, and nothing more.
• The taxonomy and the category-to-severity mapping are policy, not truth. critical / high / medium / low and the choice of what blocks are a file you own. The tool ships a default so it runs out of the box; the default is a starting point to argue with, not a standard.
• The counts here are fixture units. The 40 / 3 / 2 and the 92.5% describe the synthetic files in this post. Run it on your own eval output to get numbers that mean something about your agent.

One design choice I am still not certain about: REVIEW and BLOCK share exit 1, and only the verdict text tells them apart. I did that so a CI job has a clean “0 means auto-ship, non-zero means a human decides” contract, with exit 2 reserved for “I could not even read the input.” If your pipeline needs to branch differently on REVIEW versus BLOCK, you would want three non-zero codes, and I could see arguing it either way. I went with the two-state contract because it matches the fail-closed shape of the other gates in this series, but I would not fight hard for it.

Bad input fails closed

A gate that crashes into a green is worse than no gate. Point it at a run with a severity label it does not recognize, and it refuses to decide rather than guess:

$ python3 severity_gate.py fixtures/bad_run.json
ERROR: case c002 has severity_class 'catastrophic', not in severity_order ['low', 'medium', 'high', 'critical']
$ echo $?
2

No path argument, a missing file, an unparseable JSON, a case missing verdict, an unknown severity value: all exit 2, distinct from the exit 1 a REVIEW or BLOCK returns, so your CI can tell “the gate says hold” apart from “the gate could not run.” I ran each scenario twice and hashed the full STDOUT both times, on Python 3.13.5, offline: ship_run is cca7a591..., block_run is 346ee92c..., review_run is 6a27e17f..., edge_run is 99f4e8dc..., and ship_run under the strict policy is 35f9f6bb.... Identical across both runs, every time.

The question I actually want answered

Here is the one I do not have a good number for. For teams shipping an agent behind an eval suite: how many of your failing cases are labeled with a severity at all? Not “do you have an eval,” which most people say yes to, but whether the pii_leak and the trailing-whitespace failure carry different weights when the deploy decision gets made, or whether they both just dent the same pass-rate. My guess is that most suites are still gating on the flat number and the severity field is empty, but that is a guess, and I would like to know if I am wrong.

If this was useful, follow along here for the next runnable gate in the series, and tell me in the comments where a green aggregate has hidden a failure that should have blocked your deploy. I read every one.