Semantic Drift Notifications

Experimental. This feature ships in MockForge but is early: the /api/v1/semantic-drift/* endpoints exist and the CLI is mockforge governance semantic analyze; there is no CLI for listing or resolving incidents, and the contract_drift config shown below is not read yet. Behavior, commands, and configuration may change between releases. Report issues at https://github.com/SaaSy-Solutions/mockforge/issues.

Pillars: [Contracts]

Semantic Drift Notifications detect when the meaning of an API changes, not just its structure. This is where AI Contract Diff goes from “nice” to “indispensable.”

Overview

Structural diffs catch obvious breaking changes. Semantic drift detection catches subtle changes that break consumers even when the structure appears compatible:

Description changes that alter meaning
Enum narrowing (values removed)
Soft-breaking changes hidden behind oneOf/anyOf
Nullable → non-nullable changes
Error codes removed

How It Works

Layer 1: Structural Diff

Traditional contract diffing compares:

Field types
Required vs optional
Schema structure
HTTP methods

Layer 2: Semantic Analysis

Semantic drift detection adds:

LLM-Powered Analysis: Understands meaning, not just structure
Rule-Based Detection: Fast detection of common patterns
Soft-Breaking Scoring: Quantifies likelihood of breaking consumers
Confidence Scoring: Indicates how certain the detection is

Detection Types

Description Changes

Detects when field/endpoint descriptions change meaning:

Before:

{
  "status": {
    "type": "string",
    "description": "Order status: pending, processing, shipped"
  }
}

After:

{
  "status": {
    "type": "string",
    "description": "Order status: pending, processing, shipped, cancelled"
  }
}

Detection: New value added, but description change may indicate behavior change.

Enum Narrowing

Detects when enum values are removed:

Before:

{
  "status": {
    "type": "string",
    "enum": ["pending", "processing", "shipped", "cancelled"]
  }
}

After:

{
  "status": {
    "type": "string",
    "enum": ["pending", "processing", "shipped"]
  }
}

Detection: “cancelled” value removed—breaking change for consumers using it.

Nullable Changes

Detects nullable → non-nullable changes hidden behind oneOf:

Before:

{
  "email": {
    "oneOf": [
      {"type": "string"},
      {"type": "null"}
    ]
  }
}

After:

{
  "email": {
    "type": "string"
  }
}

Detection: Field is no longer nullable—breaking for consumers expecting null.

Error Code Removals

Detects when error codes are removed:

Before:

{
  "responses": {
    "400": {...},
    "404": {...},
    "409": {...}
  }
}

After:

{
  "responses": {
    "400": {...},
    "404": {...}
  }
}

Detection: 409 error code removed—breaking for consumers handling conflicts.

Soft-Breaking Changes

Detects changes that may break consumers but aren’t structurally breaking:

Format changes: Email format validation tightened
Constraint changes: Min/max values changed
Pattern changes: Regex pattern modified
Example changes: Examples suggest different behavior

Usage

CLI Commands

# Analyze semantic drift for an endpoint between two contract versions
mockforge governance semantic analyze --before old.yaml --after new.yaml --endpoint /api/users/{id} --method GET

# Write the result to a file
mockforge governance semantic analyze --before old.yaml --after new.yaml --endpoint /api/users/{id} --method GET --output drift.json

Listing incidents is only available over the API; there is no CLI command for listing or resolving incidents.

API Usage

# Analyze semantic drift
POST /api/v1/semantic-drift/analyze
{
  "before_spec": "...",
  "after_spec": "...",
  "endpoint": "/api/users/{id}",
  "method": "GET"
}

# Get incidents
GET /api/v1/semantic-drift/incidents?workspace_id=workspace-123

# Get incident details
GET /api/v1/semantic-drift/incidents/{id}

Semantic Drift Results

Example Result

{
  "semantic_confidence": 0.85,
  "soft_breaking_score": 0.7,
  "change_type": "enum_narrowing",
  "semantic_mismatches": [
    {
      "type": "SemanticEnumNarrowing",
      "severity": "high",
      "location": "/api/orders/{id}",
      "field": "status",
      "description": "Enum value 'cancelled' removed",
      "before": ["pending", "processing", "shipped", "cancelled"],
      "after": ["pending", "processing", "shipped"],
      "confidence": 0.9
    }
  ],
  "llm_analysis": {
    "reasoning": "Removing 'cancelled' status breaks consumers that rely on this value for order cancellation flows.",
    "impact": "High - affects order cancellation workflows",
    "recommendation": "Deprecate 'cancelled' first, then remove in next major version"
  }
}

Understanding Scores

semantic_confidence: 0.0-1.0, how certain the semantic analysis is
soft_breaking_score: 0.0-1.0, likelihood this breaks consumers
change_type: Type of semantic change detected
confidence: Individual mismatch confidence

Configuration

Enable Semantic Drift Detection

# mockforge.yaml
contract_drift:
  semantic_drift:
    enabled: true
    confidence_threshold: 0.65  # Minimum confidence to report
    use_llm_analysis: true  # Enable LLM-powered analysis
    rule_based_detection: true  # Enable rule-based detection

LLM Configuration

contract_drift:
  semantic_drift:
    enabled: true
    llm:
      provider: openai  # or anthropic, local
      model: gpt-4
      temperature: 0.3  # Lower = more deterministic

Integration with Drift Budgets

Semantic drift incidents integrate with drift budgets:

contract_drift:
  drift_budget:
    max_breaking_changes: 2
    semantic_drift:
      enabled: true
      # Semantic drift incidents count toward budget

Webhooks

Semantic drift detection can trigger webhooks:

webhooks:
  - url: https://slack.com/hooks/...
    events:
      - semantic_drift.detected
      - semantic_drift.high_confidence
      - semantic_drift.soft_breaking