Caption Regression Testing

A caption normalizer is refactored, a rounding helper is “simplified”, a library is bumped a minor version — and three thousand cues shift by a frame that no reviewer will ever notice until a broadcaster ingest rejects the package. That is the failure caption regression testing exists to catch: a code change that silently alters validated output. The gap is not correctness in the abstract; the artifacts already passed CI/CD gating for caption builds once. The gap is stability under change — proof that the output a tool produced today is byte-for-byte (or frame-for-frame within a declared tolerance) the same output it produced before the diff landed. Golden-master testing supplies that proof by freezing a known-good serialization of the normalized cue model and asserting every future run reproduces it, so a regression fails a pull request instead of a delivery.

The contract is deliberately asymmetric. Text content is compared exactly — a single changed glyph is a regression, full stop. Timing is compared with a frame-quantized tolerance — ±2 frames under FCC 47 CFR § 79.1 synchronicity, because two serializations that round 0417.084 µs differently must not fail a test that has nothing to do with drift. Everything else — cue count, ordering, region assignment — is exact. This page builds that comparison engine, wires it into a test suite, and hardens it against the four things that make golden files flaky: nondeterministic ordering, floating timestamps, locale, and unmanaged golden drift.

Problem Framing: What a Regression Actually Is

A functional QC gate answers “is this artifact compliant?” A regression test answers a different, orthogonal question: “did my code change what this artifact is?” The two fail on disjoint inputs. A cue can be perfectly compliant — inside every reading-rate and sync threshold — and still be a regression, because last week the same converter emitted a two-line cue and this week it emits one line. Nothing is wrong; something changed, and change to validated output is the risk that silent regressions represent. So the unit under test is not the caption file on disk; it is the deterministic function raw bytes → normalized cue model → canonical serialization. Golden-master testing pins the output of that function.

The thresholds that make the comparison stable rather than flaky are numeric and few. Timing carries a ±2-frame tolerance (66.7 ms at 29.97 fps, 80 ms at 25 fps) so that two mathematically equivalent timecodes never fail a test about drift. Text carries a 0% tolerance — exact string equality after whitespace normalization — because caption text is the payload and any mutation is a defect. Cue count, cue order, and region assignment are exact. The comparison runs in two tiers: a fast SHA-256 hash of the canonical bytes for the overwhelmingly common “nothing changed” path, and a slower structural diff that names the offending cue when the hash misses. That two-tier design keeps a suite of thousands of goldens fast while still producing a legible failure. It reuses the same normalized cue model consumed by the SRT, SCC & WebVTT parsing workflows and gated by CI/CD gating for caption builds, so a golden is a snapshot of exactly the structure the rest of the pipeline trusts.

Pipeline Stage & Prerequisites

Regression tests run in the test suite, not in the delivery pipeline. They execute on every pull request that touches caption tooling — parsers, normalizers, serializers, or their pinned dependencies — and they gate the merge, upstream of the build gate that later validates real deliveries. The golden files live in the repository beside the fixtures that generate them, versioned so a deliberate output change and its reviewed golden update land in the same commit. Nothing here reads a live PTS track or a delivery container; the inputs are small, committed fixtures chosen to exercise each normalization branch.

Pin the caption libraries exactly. A regression suite whose own dependencies float will report phantom failures the moment a transitive bump changes microsecond rounding — the very drift the tolerance exists to absorb, but only up to ±2 frames. The synthetic-track and drift-injection helpers used to build timing fixtures are covered separately in pytest fixtures for drift thresholds; the frame-accurate onset math those fixtures depend on is the same integer-frame convention used by automated sync drift detection.

Step-by-Step Implementation

Step 1 — Normalize into a canonical cue model

Every regression test starts by collapsing format-specific quirks into one canonical, hashable structure. The cue is frozen and ordered so it compares by value, timestamps are snapped to the frame grid, and trailing whitespace is stripped so an editor’s invisible edit cannot masquerade as a caption change.

from dataclasses import dataclass

FPS = 29.97  # NTSC; pass the real rate per fixture

@dataclass(frozen=True, order=True)
class NormalizedCue:
    start_ms: int
    end_ms: int
    region: str
    text: str

def quantize_ms(ms: float, fps: float = FPS) -> int:
    # SMPTE ST 12-1 — snap onset/offset to the frame grid so
    # mathematically-equal times serialize to identical integers
    frame = round(ms / 1000 * fps)
    return round(frame / fps * 1000)

def normalize(raw_cues, fps: float = FPS) -> list[NormalizedCue]:
    cues = []
    for c in raw_cues:
        # collapse line-level whitespace; caption text is compared exactly
        text = "\n".join(line.rstrip() for line in c.text.splitlines()).strip("\n")
        cues.append(NormalizedCue(
            start_ms=quantize_ms(c.start_ms, fps),
            end_ms=quantize_ms(c.end_ms, fps),
            region=(c.region or "default"),
            text=text,
        ))
    # deterministic ordering — never trust the source file's cue order
    return sorted(cues)

Step 2 — Serialize canonically for byte-stable comparison

A golden is only useful if the same model always serializes to the same bytes on every machine. sort_keys=True fixes key order, explicit separators remove platform-dependent whitespace, and ensure_ascii=False keeps accented caption text stable rather than re-encoding it per interpreter.

import json
from typing import Sequence

def canonical_serialize(cues: Sequence[NormalizedCue]) -> str:
    payload = [
        {"start_ms": c.start_ms, "end_ms": c.end_ms,
         "region": c.region, "text": c.text}
        for c in cues
    ]
    # sort_keys + fixed separators => identical bytes across runs and OSes
    return json.dumps(payload, ensure_ascii=False, sort_keys=True,
                      separators=(",", ":")) + "\n"

Step 3 — Hash first, then structural-diff with tolerance

The comparison runs in two tiers. Identical canonical bytes hash identically and pass in constant time — the common case. Only on a hash miss does the slower structural diff run, applying the exact/tolerant split: text and region must match exactly, timing may differ by up to the frame tolerance.

import hashlib

def digest(serialized: str) -> str:
    return hashlib.sha256(serialized.encode("utf-8")).hexdigest()

def structural_diff(current, golden, timing_tol_ms: float = 66.7) -> list[str]:
    diffs: list[str] = []
    if len(current) != len(golden):
        diffs.append(f"cue count {len(current)} != golden {len(golden)}")
    for i, (a, b) in enumerate(zip(current, golden)):
        # FCC 47 CFR § 79.1 — timing tolerant to ±2 frames (66.7 ms @ 29.97)
        if abs(a.start_ms - b.start_ms) > timing_tol_ms:
            diffs.append(f"cue {i} start drift {a.start_ms - b.start_ms:+d} ms")
        if abs(a.end_ms - b.end_ms) > timing_tol_ms:
            diffs.append(f"cue {i} end drift {a.end_ms - b.end_ms:+d} ms")
        if a.text != b.text:            # text is exact — zero tolerance
            diffs.append(f"cue {i} text {b.text!r} -> {a.text!r}")
        if a.region != b.region:        # placement is exact
            diffs.append(f"cue {i} region {b.region!r} -> {a.region!r}")
    return diffs

Step 4 — Wire it into pytest with a controlled update path

The assertion helper hashes for the fast path, falls back to the structural diff, and honours an UPDATE_GOLDENS=1 environment switch so a deliberate output change is regenerated on purpose and reviewed as a diff — never edited by hand.

import os
from pathlib import Path

def load_golden(path: Path) -> list[NormalizedCue]:
    data = json.loads(path.read_text(encoding="utf-8"))
    return [NormalizedCue(**row) for row in data]

def assert_matches_golden(raw_cues, golden_path: Path,
                          fps: float = FPS, timing_tol_ms: float = 66.7) -> None:
    current = normalize(raw_cues, fps)
    serialized = canonical_serialize(current)
    if os.environ.get("UPDATE_GOLDENS") == "1":
        golden_path.write_text(serialized, encoding="utf-8")  # regenerate on demand
        return
    golden = load_golden(golden_path)
    if digest(serialized) == digest(canonical_serialize(golden)):
        return                                               # fast path: byte-identical
    diffs = structural_diff(current, golden, timing_tol_ms)
    assert not diffs, "caption regression:\n" + "\n".join(diffs)

The full approval-testing workflow — line-ending normalization, BOM handling, and the exact *.golden file convention — is expanded in the companion golden-file testing for caption output, which the helper above is the reference implementation for.

Threshold Reference Table

Tune the comparison against this table, not against prose. Timing tolerances are the ±2-frame window expressed per framerate; everything textual is exact.

Verification & Test Pattern

The regression engine is itself code, so it needs tests that prove its two invariants: serialization is order-independent, and the tolerance split behaves — timing within a frame passes, a changed glyph fails.

import pytest
from caption_regression import (
    normalize, canonical_serialize, structural_diff, NormalizedCue,
)

class Raw:
    def __init__(self, start_ms, end_ms, text, region="default"):
        self.start_ms, self.end_ms = start_ms, end_ms
        self.text, self.region = text, region

def test_serialization_is_order_independent():
    cues = [Raw(1000, 3000, "Line one"), Raw(3200, 5000, "Line two")]
    a = canonical_serialize(normalize(cues))
    b = canonical_serialize(normalize(list(reversed(cues))))
    assert a == b  # source ordering must never change the golden

def test_sub_frame_timing_shift_is_not_a_regression():
    golden = normalize([Raw(1000, 3000, "Hello")])
    shifted = normalize([Raw(1020, 3020, "Hello")])  # +20 ms < one frame
    assert structural_diff(shifted, golden, timing_tol_ms=66.7) == []

def test_text_mutation_is_a_regression():
    golden = normalize([Raw(1000, 3000, "Hello")])
    changed = normalize([Raw(1000, 3000, "H3llo")])
    assert structural_diff(changed, golden) != []

Run this suite before any golden is trusted; a comparison engine that has never rejected a known bad input is untested infrastructure, exactly as with the build gate’s fixtures.

Troubleshooting / Failure Modes

Nondeterministic cue ordering : Root cause: a converter emits cues in dict-iteration or thread-completion order, so the same input serializes two different ways and the golden flaps. Fix: sort the cue list by (start_ms, end_ms, region, text) before serialization — the sorted() call in normalize — so order is a property of content, never of the run.

Floating-point timestamps : Root cause: storing times as float seconds lets IEEE 754 rounding differ across interpreters and platforms, producing sub-millisecond diffs that fail an exact compare. Fix: quantize to integer frames on the grid (quantize_ms) and store integer milliseconds; equal timecodes then serialize to equal integers.

Locale-dependent serialization : Root cause: number or string formatting that honours LC_NUMERIC (decimal commas) or non-UTF-8 default encodings makes goldens machine-specific. Fix: serialize with explicit encoding="utf-8" and ensure_ascii=False, keep all values as integers and str, and never route caption text through locale-aware formatters.

Unmanaged golden drift : Root cause: engineers regenerate goldens reflexively whenever a test fails, so the snapshots track the bug instead of catching it. Fix: gate regeneration behind an explicit UPDATE_GOLDENS=1, require the golden diff in the same reviewed pull request as the code change, and treat an unexplained golden churn as a review blocker.

Whitespace and line-ending noise : Root cause: a CRLF fixture on Windows or a trailing space from an editor changes the bytes without changing the caption. Fix: normalize line endings and rstrip each line during normalize, so only semantically meaningful text differences surface.

Dependency micro-version skew : Root cause: a floating pycaption bump changes internal rounding and every timing field shifts a millisecond, flooding the structural diff. Fix: pin caption libraries exactly in the lockfile; let the ±2-frame tolerance absorb legitimate sub-frame variance, and treat a broad timing shift as the version-bump signal it is.

Operational Notes

Golden files are small, text, and diffable — commit them, review them, and never .gitignore them. Group them beside the fixtures that generate them so a reviewer sees the input, the code change, and the output delta in one place. At scale, parametrize one test function across a directory of fixtures rather than writing a function per asset; a single parametrized test over a hundred *.golden files reports each failure by fixture id and keeps the suite maintainable. Keep the hash fast-path first so the common “nothing changed” run stays in constant time even across thousands of snapshots.

Regression testing sits alongside, not inside, delivery. A merge that changes output legitimately flows on to CI/CD gating for caption builds for compliance validation and then to scheduled QC report generation for the audit rollup; the regression suite’s only job is to guarantee that whatever ships is the output someone reviewed. Where timing itself is the thing under test — a normalizer that must not introduce drift across a framerate matrix — pair these goldens with the drift-aware detection in automated sync drift detection so a frame of introduced skew fails a test rather than an ingest.

Related

• Golden-file testing for caption output — the approval-test helper with UPDATE_GOLDENS and frame tolerance in full.
• CI/CD gating for caption builds — the compliance gate that runs downstream of a passing regression suite.
• Scheduled QC report generation — where output verdicts roll up into the audit record.
• Automated sync drift detection — the frame-accurate timing check that complements exact-text regression.
• Pytest fixtures for drift thresholds — reusable synthetic-track fixtures for the timing goldens here.

Part of: Automated QC Validation & Reporting — the deterministic caption QC and reporting reference.