Golden-File Testing for Caption Output

The failure this page solves is an approval test that fails on noise instead of on regressions: a converter emits a WebVTT or SRT file that is semantically identical to the reviewed golden, but the test rejects it because a UTF-8 BOM crept onto line one, the runner wrote CRLF endings, or a timestamp rounded from 00:00:01.001 to 00:00:01.000 — a one-millisecond, sub-frame difference well inside the FCC 47 CFR § 79.1 ±2-frame sync tolerance. A naive assert actual == golden treats all three as failures. A correct golden-file harness canonicalizes away the byte noise, compares literal caption text exactly, and compares each embedded timestamp within a frame tolerance — so it fails only when the converter’s output actually changed.

"""golden.py — approval testing for caption converter output with frame tolerance."""
import os
import re
from pathlib import Path

# 1 frame in ms per broadcast rate; ±2 frames is the FCC 47 CFR § 79.1 window
FRAME_MS = {23.976: 1000 / 23.976, 25: 40.0, 29.97: 1000 / 29.97}
TS = re.compile(r"(\d{2}):(\d{2}):(\d{2})[.,](\d{3})")   # WebVTT '.' and SRT ',' both match

def _to_ms(m: "re.Match") -> int:
    h, mm, s, ms = (int(m.group(i)) for i in range(1, 5))
    return ((h * 60 + mm) * 60 + s) * 1000 + ms

def canonicalize(text: str) -> str:
    text = text.lstrip("")                          # strip a UTF-8 BOM
    text = text.replace("\r\n", "\n").replace("\r", "\n") # normalize CRLF / CR to LF
    return text.rstrip("\n") + "\n"                       # exactly one trailing newline

def _tokenize(line: str) -> list:
    # split a line into timestamp tokens (compared with tolerance) and text (exact)
    out, last = [], 0
    for m in TS.finditer(line):
        if m.start() > last:
            out.append(("text", line[last:m.start()]))
        out.append(("time", _to_ms(m)))
        last = m.end()
    out.append(("text", line[last:]))
    return out

def _line_matches(actual: str, golden: str, tol_ms: float) -> bool:
    a, g = _tokenize(actual), _tokenize(golden)
    if len(a) != len(g):
        return False
    for (ak, av), (gk, gv) in zip(a, g):
        if ak != gk:
            return False
        if ak == "time" and abs(av - gv) > tol_ms:        # timing: tolerant
            return False
        if ak == "text" and av != gv:                     # text: exact, zero tolerance
            return False
    return True

def approve(actual: str, golden_path: Path, fps: float = 29.97, frames: int = 2) -> None:
    actual = canonicalize(actual)
    if os.environ.get("UPDATE_GOLDENS") == "1":
        golden_path.write_text(actual, encoding="utf-8")  # controlled regeneration
        return
    golden = canonicalize(golden_path.read_text(encoding="utf-8-sig"))
    tol_ms = frames * FRAME_MS[fps]
    a_lines, g_lines = actual.splitlines(), golden.splitlines()
    if len(a_lines) != len(g_lines):
        raise AssertionError(f"line count {len(a_lines)} != golden {len(g_lines)}")
    bad = [f"L{i + 1}: {g!r} -> {a!r}"
           for i, (a, g) in enumerate(zip(a_lines, g_lines))
           if not _line_matches(a, g, tol_ms)]
    if bad:
        raise AssertionError("caption output regression:\n" + "\n".join(bad))

A pytest case is then a two-line call — render the converter’s output and approve it against a committed *.golden file:

from pathlib import Path
from golden import approve
from my_converter import convert            # your SCC/SRT -> WebVTT converter

def test_program_001_webvtt_output():
    out = convert(Path("fixtures/program_001.scc").read_text(encoding="utf-8"))
    approve(out, Path("goldens/program_001.vtt.golden"), fps=29.97, frames=2)

Code walkthrough

canonicalize runs first on both sides so the comparison never sees byte noise. It strips a leading  BOM, rewrites CRLF and lone-CR line endings to LF, and pins exactly one trailing newline. Reading the golden with encoding="utf-8-sig" is a second BOM guard — a golden authored by a Windows editor loses its signature on load. After canonicalization, any surviving difference is a real content difference.

_tokenize is what separates this harness from a plain string diff. It walks each line and splits it into an ordered list of ("time", ms) and ("text", literal) tokens using a single regex that matches both the WebVTT HH:MM:SS.mmm and SRT HH:MM:SS,mmm grammars. A cue line like 00:00:01.000 --> 00:00:03.500 becomes a time token, the literal --> text token, and a second time token — so the arrow and any positioning text stay exact while the two timestamps become tolerant numeric comparisons.

_line_matches enforces the asymmetric contract. Token kinds and counts must line up; timestamps pass when they are within tol_ms; literal text must be identical. The tolerance is computed once in approve as frames * FRAME_MS[fps] — two frames at 29.97 fps is 66.7 ms, at 25 fps 80 ms, at 23.976 fps 83.4 ms — so the same helper serves NTSC, PAL, and film-rate deliveries by passing the real fps.

approve is the entry point and carries the update workflow. With UPDATE_GOLDENS=1 set it overwrites the golden with the canonicalized output and returns — the only sanctioned way to change a golden, so regeneration is a deliberate, reviewable act rather than a hand edit. Without it, a line-count mismatch fails immediately (the cheap structural check), then each line is compared token-wise and every mismatch is reported with its line number and the golden-to-actual delta, so a failing test names exactly what moved.

The line-oriented design is a deliberate trade. A structural comparison of a parsed cue model — the approach in the parent caption regression testing helper — is robust to reordering but discards the exact byte layout a downstream tool consumes; this harness instead pins the emitted file line for line, catching a stray blank line or a changed cue-settings string that a model-level diff would normalize away. The cost is that it assumes stable cue ordering, so it belongs on converters whose output order is already deterministic. Run it as a first line of defence on the serialized artifact, and keep the model-level goldens for anything whose internal ordering is not guaranteed.

Threshold reference table

Edge cases & known gotchas

• Line-ending drift: a golden committed on Linux (LF) and regenerated on Windows (CRLF) differs on every line until canonicalize normalizes both — never skip it, and set * text eol=lf in .gitattributes so the stored golden stays LF.
• BOM on reload: author or hand-edit a golden in an editor that adds a UTF-8 BOM and a naive read_text("utf-8") keeps the ; read with utf-8-sig (as above) so the signature is consumed.
• Cue ordering: this line-oriented harness assumes the converter emits cues in a stable order. If yours does not, sort cues before serialization — the model-level caption regression testing helper sorts the normalized cue list for exactly this reason.
• Tolerance vs exact confusion: only the four-group timestamp pattern is tolerant; a numeric ID line or a cue-settings coordinate is plain text and compared exactly. Do not widen the regex to match arbitrary digits or you will silently tolerate changed cue numbers.
• Frame rate mismatch: passing fps=29.97 to approve a 25 fps PAL golden applies a 66.7 ms window where 80 ms is correct, so a legitimate one-frame PAL shift can fail. Thread the real source rate through per fixture.

Integration hook

This helper is the concrete, output-level implementation of the approach framed in caption regression testing: where the parent compares a canonical JSON snapshot of the normalized cue model, this compares the converter’s serialized text — the bytes a downstream mux actually consumes — with the same exact-text, tolerant-timing split. Use the model-level goldens to lock the internal structure and these output-level goldens to lock the emitted file, then feed the timing fixtures from pytest fixtures for drift thresholds so the frame tolerance here is exercised at each rate boundary.

Related

• Caption regression testing — the parent method: golden-master snapshots of the normalized cue model.
• Detecting sync drift in automated QC pipelines — the frame-accurate PTS check behind the timing tolerance here.
• Pytest fixtures for drift thresholds — synthetic tracks that exercise the per-rate frame tolerance.

Part of: Automated QC Validation & Reporting — the deterministic caption QC and reporting reference.