Conformance self-test probe reference

The mockforge bench --conformance-self-test driver runs a positive case plus one probe per category against every operation in the spec, and records the result in conformance-self-test.json (plus a human- readable summary in conformance-report.html).

This page is the canonical reference for every probe label the driver can emit. Use it to interpret the negative_caught / negative_missed buckets in the JSON, and the “Missed negatives” drill-down in the HTML report.

Label scheme

<category>:<subcategory>[:<scope>]

The category (the part before the first :) is what the report buckets on. So request-body:type-mismatch:user.email rolls up under request-body, alongside request-body:empty, request-body:required-removed:id, and friends.

Per-location violation entries (#896)

Server-side violation buffer entries are emitted one per location, not one per request. A single POST that fails a query enum (kind) AND a required body field (email) produces:

{ "category": "query",         "reason": "kind: Validation error ...",  "occurrences": 1 }
{ "category": "request-body",  "reason": "email: ... is required",      "occurrences": 1 }

• category mirrors the validator detail’s "path" prefix: query / headers / cookies / parameters (path params) / request-body.
• reason starts with the offending parameter or field name, so grepping query.\$.xgafv-style details works directly against the export.
• With dedup enabled (MOCKFORGE_CONFORMANCE_BUFFER_UNIQUE=true, default), same (method, path, category, reason) repeats collapse into one entry with a bumped occurrences.
• Requests whose validator output carries no structured per-location details[] fall back to the legacy one-entry-per-request shape with the priority classifier (classify_validation_reason).

Positive case (one per operation)

A missed positive (4xx/5xx) means the server rejected a request the spec says is valid. Common cause: missing auth header, missing --base-path, or a spec-derived sample that doesn’t satisfy a non-trivial constraint the synthesiser didn’t notice.

Request-body negatives

Fire whenever the operation declares a request body content type.

Bounded at SCHEMA_MUTATION_CAP = 12 mutations per operation (top 20 properties, top 5 required) so a 100-property body on a 22 000-op spec doesn’t produce a runaway test matrix.

Content-type negatives (rounds 27 / 28 / 32)

Fire on operations that declare application/json as an accepted request-body content type. Each variant covers one way a real client can get the content-type vs the body wrong.

Round 28 wired these into the shared ServerConformanceViolation buffer so the server-side report also records when a server accepts the mismatched content-type, not just the client-side bench.

Variant-b: embedded non-JSON content

Companion probes (round 27) inject an XML / YAML / multipart / urlencoded snippet into the first string field of a spec-shape JSON envelope. The Content-Type header stays application/json and the body parses as valid JSON; only the string-valued payload changes.

Round 35 made these tolerate 4xx server responses, because a server with a pattern / format validator on the target field will (correctly) reject the embedded snippet. That’s why the expected range is 2xx-4xx: only a 5xx is a finding (server crashed parsing the embedded payload).

Round 34 skips these probes entirely when the positive sample has no string leaf to mutate (e.g. a {enabled: boolean} body), so the bench doesn’t emit a probe it knows can’t construct a valid envelope.

Parameter negatives

Security negatives (only when the spec declares a security scheme)

Auth-stripping is case-insensitive on Authorization and walks the operation’s own declared API-key locations, so the probe’s credential is the only thing the server sees.

OWASP injection negatives

Fire only when the operation has an injectable target. Target is chosen once per operation in priority order:

1. First query parameter’s value
2. First string field of the positive JSON body
3. Skip — no false signal for GET /healthz

A 5xx on any OWASP probe is a hard finding (server crashed on the payload). A 2xx is a soft finding (input passed through unfiltered; may or may not be a real vulnerability depending on whether the payload reaches a sink).

How “passed” is decided

Currently response-code-only. Each probe records the expected status range it was designed for (CaseCapture.expected_status_range) and the driver compares the actual HTTP status against that range. The driver uses a tristate ExpectedOutcome enum (round 35):

Probe-by-probe assignment:

• Positive case → Success → 2xx-3xx.
• All request-body / parameter / security / OWASP negatives → ClientError → 4xx.
• content-type-mismatch:* negatives → ClientError → expected 415, evaluated as 4xx.
• embedded-content:* (variant-b) negatives → NotServerError → 2xx-4xx.

A response-body shape validator (in addition to the status-code check) is queued (round 21.3).

5xx is always a finding

The bench never deliberately elicits a 5xx. None of the probe families above expect or tolerate a 5xx. A server emitting 5xx on any probe is a server-side bug to fix, not a noisy test result. That’s true for the NotServerError variants too: the 2xx-4xx tolerance still treats 5xx as a fail.

What --inject-response-violations changes

This flag is sometimes confused with status-class chaos. It is narrowly scoped to response bodies:

• Modifies the response body for synthesized 2xx responses by dropping the first declared required field from the response schema, so the body becomes spec-non-conforming while still parsing as JSON.
• Does not change the HTTP status. A 2xx response stays 2xx; a 4xx response stays 4xx.
• Use case: feed a downstream proxy or conformance harness a known bad-shape body so you can confirm it catches the mismatch.

If you want to exercise downstream handling of varied 4xx / 5xx statuses, use the chaos surface instead:

• mockforge serve --chaos — global random failure injection with configurable per-status weight.
• The route-chaos API under /__mockforge/api/route-chaos: per-route status overrides.

Those are documented in the book’s chaos chapter; they’re intentionally separate knobs from request-body conformance.

Interpreting bucket totals

Negatives [security]: 739 caught / 1812 missed  ⚠

A negative is a deliberately-bad request that should draw a 4xx from the target (the service you are benching), not from MockForge. The two buckets are about the target’s behaviour:

• caught = the target rejected it with a 4xx (its validation held).
• missed = the target accepted it (2xx-3xx) or crashed (5xx) when the probe expected a rejection.

A category that’s all-missed is often the spec telling you that whole validator class isn’t enforced. A category that’s all-caught with no missed is the server doing its job. The interesting reads are partial-misses (some routes enforce, others don’t) — those are usually middleware ordering bugs.

A high “missed” is not automatically a bug

Many probes are spec-valid by construction, so the contract itself permits them and a correct target is right to accept them (counted as “missed” only because the probe hoped for a rejection). Examples:

• a string field accepts any string, so an owasp:sqli payload like ' OR '1'='1 satisfies the schema — only an app-layer WAF, not the contract, would reject it;
• an optional query parameter can be dropped (parameters:missing-query);
• an unconstrained {id} path segment accepts any value (parameters:bad-path-param).

These land in conformance-request-violations.json as typed parameter_negative_probe (or equivalent) rows with a note explaining why they are not a schema breach and that rejecting them is the target’s responsibility. So when parameters or owasp shows 0 caught / N missed, read it as “the contract permits these, and the target chose to accept them” — check the per-probe rows to see which are genuinely unenforced validations versus spec-permitted inputs. The same summary is printed inline under each Negatives [...] line so you don’t have to come back to this page.

Definite issues

Because “missed” needs interpretation, the self-test also prints a Definite issues (N) section and writes a conformance-definite-issues.json sidecar containing ONLY the unambiguous problems, so you can triage many APIs by reading one short list:

• valid_request_rejected — the target refused a spec-valid (positive) request. It broke a legitimate call; this is always a real problem.
• server_error — a probe (positive or negative) drew a 5xx. Even a deliberately-bad request should get a clean 4xx, never a crash.

Spec-valid “missed” negatives are deliberately excluded from this view: those are the ones the caught/missed rollup exists to help you judge. A “caught” 4xx is never listed here — a caught negative means the target did its job (rejected a bad request), so it is not a violation even if that exact error status isn’t enumerated in the spec (use --validate-response-schemas if you want to check error bodies against the spec separately).

HTML drill-down cap

The “Missed negatives” table in conformance-report.html caps at 200 rows by default to keep the HTML file viewable on huge specs. Override with --report-missed-cap N (0 = no cap, show everything). The JSON report always carries the full set regardless of the cap.

CLI summary

mockforge bench \
  --conformance --conformance-self-test \
  --spec your-api.yaml \
  --target https://your-api.example.com/ \
  --base-path /api \
  --conformance-header "Authorization: Bearer real-token" \
  --report-missed-cap 0

Output artefacts in bench-results/:

• conformance-self-test.json — full machine-readable report
• conformance-report.html — human-readable summary (this page’s labels)
• conformance-spec-audit.json — pure spec audit (when --spec is set)

See also the conformance overview for the high-level mode docs.