Enforcing Character Rate Limits in QC

Character-rate validation is the gate that stops captions from outrunning the viewer. When text density climbs above what a CEA-608 decoder can flush — or above what a human can read before the cue clears — the result is dropped glyphs, on-screen stutter, forced scrolling, and a documented accessibility failure under FCC 47 CFR § 79.1 (synchronicity and completeness). This step sits inside Automated QC Validation & Reporting as a deterministic checkpoint: it converts a parsed caption track into an instantaneous characters-per-second (CPS) signal, asserts that signal against jurisdictional ceilings, and emits a pass/fail verdict with the exact offending cue indices.

Problem framing

The constraint is physical before it is regulatory. CEA-608 carries two bytes of caption data per frame, so at 29.97 fps the channel sustains roughly 29.97 characters per second of airtime — a hard ceiling no authoring tool can exceed without the decoder buffer overrunning. Production validators do not run at that ceiling; they enforce a sustained 20 CPS with short bursts permitted to 30 CPS for dense dialogue or emergency-alert overlays, and the more conservative EBU-TT-D / BBC reading-rate guidance caps sustained density at 17 CPS (≈ 160–180 WPM) so viewers with cognitive or visual impairments can keep pace.

Three things make this hard to get right:

• Tokenization correctness. A naive len(text) over-counts: control codes, mid-cue line breaks, and inter-word spaces are not glyphs a decoder paints, so they must be excluded from the CPS numerator the same way a CEA-608 decoder accounts for buffer occupancy.
• Window choice. Per-cue chars / duration hides bursts that straddle a cue boundary. A rolling window over the whole track catches the dense moment that a per-cue average smooths away.
• Non-destructive remediation. A rate fix must never silently truncate dialogue — trading a completeness violation for a rate one re-breaks § 79.1. Enforcement flags; it never deletes words.

A spike in character density is also a reliable proxy for upstream timing damage, which is why this check is a prerequisite for automated sync drift detection: rapid cue turnover inflates timestamp jitter and produces false drift warnings unless density is validated first.

Pipeline stage & prerequisites

Rate enforcement runs in the post-parse, pre-render stage. By this point the upstream parser has already produced a normalized cue model — the same canonical model described in SRT, SCC & WebVTT parsing workflows — with every timestamp converted to an integer millisecond epoch. Running the check before SRT timestamp normalization is a mistake: drop-frame notation and TTML tick offsets will distort the duration denominator and throw the CPS math off.

Required tooling:

For SCC sources the cue model comes from the state machine described in parsing SCC with Python libraries; for HLS/DASH sources it comes from WebVTT cue extraction & validation. The rate checker itself is format-agnostic — it consumes the normalized events and nothing else.

Step-by-step implementation

1. Normalize cues into a uniform event model

Collapse every parser’s output into one CaptionEvent shape so the rate logic never branches on format. Strip the BOM during decode, not after.

from dataclasses import dataclass
import pysrt

@dataclass
class CaptionEvent:
    start_ms: int
    end_ms: int
    text: str

def load_srt_events(path: str) -> list[CaptionEvent]:
    """Parse an SRT file into millisecond-epoch caption events."""
    # cue.start.ordinal is integer milliseconds — avoids float rounding
    return [
        CaptionEvent(c.start.ordinal, c.end.ordinal, c.text)
        for c in pysrt.open(path, encoding="utf-8-sig")  # 'utf-8-sig' strips a UTF-8 BOM
    ]

2. Count only the glyphs the decoder paints

Visible-character accounting mirrors CEA-608 buffer occupancy: spaces, newlines, and control sequences do not consume a display cell in the CPS sense.

import re

_CONTROL = re.compile(r"[\x00-\x1f]")  # strip control codes (CEA-608 PAC/midrow artifacts)

def visible_chars(text: str) -> int:
    """Count glyphs a decoder renders: drop control codes, newlines, and spaces."""
    flattened = _CONTROL.sub("", text).replace("\n", " ")
    return len(flattened.replace(" ", ""))  # spaces excluded from CPS numerator

3. Compute an instantaneous rate with a sliding window

A rolling deque over time-sorted events gives an exact instantaneous CPS without the O(n²) cost of re-scanning the track per cue. Expired events fall off the left of the window as the parser advances.

from collections import deque

def rolling_cps(
    events: list[CaptionEvent],
    window_ms: int = 1000,
    sustained_cps: float = 20.0,   # CEA-608 operational sustained ceiling
    burst_cps: float = 30.0,       # short-burst allowance (EAS / dense dialogue)
) -> list[tuple[int, float, str]]:
    """Return (timestamp_ms, rate_cps, status) for each cue onset.

    status is PASS (<= sustained), BURST (sustained < r <= burst), or FAIL (> burst).
    """
    ordered = sorted(events, key=lambda e: e.start_ms)
    window: deque[CaptionEvent] = deque()
    results: list[tuple[int, float, str]] = []

    for event in ordered:
        window.append(event)
        now = event.start_ms
        # Purge events whose onset is older than the window
        while window and (now - window[0].start_ms) >= window_ms:
            window.popleft()

        chars = sum(visible_chars(e.text) for e in window)
        rate = chars / (window_ms / 1000.0)

        if rate <= sustained_cps:
            status = "PASS"
        elif rate <= burst_cps:
            status = "BURST"   # tolerated transient — log, do not fail
        else:
            status = "FAIL"    # FCC 47 CFR § 79.1 — exceeds decoder buffer budget
        results.append((now, round(rate, 2), status))

    return results

4. Reduce the signal to a gating verdict

CI/CD needs a single boolean. Collapse the per-onset series into a deterministic exit contract and surface the offending indices so an editor — or an automated re-timer — can act without re-reading the file.

def evaluate(events: list[CaptionEvent], window_ms: int = 1000) -> dict:
    """Collapse the rolling series into an audit-ready verdict."""
    series = rolling_cps(events, window_ms=window_ms)
    failures = [(t, r) for t, r, s in series if s == "FAIL"]
    return {
        "max_cps": max((r for _, r, _ in series), default=0.0),
        "failure_count": len(failures),
        "first_failure_ms": failures[0][0] if failures else None,
        # EBU-TT-D / BBC sustained reading-rate ceiling for cross-check
        "verdict": "fail" if failures else "pass",
    }

Deeper handling — multi-language payloads, overlapping-cue resolution, and frame-accurate timecode conversion — is covered in how to enforce FCC character rate limits programmatically.

Threshold reference table

The 17 CPS accessibility figure and the 20 CPS operational figure are not in conflict: 17 CPS is the reader-comfort target enforced for compliance against EBU-TT-D, while 20/30 CPS is the decoder-survival envelope. A strict pipeline asserts the lower of the two for the target jurisdiction. Jurisdiction-specific ceilings are enumerated in the FCC Part 79 compliance checklist and the Ofcom Code on subtitling standards.

Verification & test pattern

Validate the algorithm against hand-built fixtures whose expected CPS you can compute by hand. The assertions below pin both a clean pass and a deliberate over-budget burst.

def test_rolling_cps_flags_burst():
    events = [
        CaptionEvent(0, 900, "hello there"),        # 10 visible chars in ~1s -> 10 CPS
        CaptionEvent(1000, 1900, "this is a very dense caption line that overruns"),
    ]
    series = rolling_cps(events, window_ms=1000)

    # First onset: 10 visible chars over a 1s window -> well under the 20 CPS ceiling
    assert series[0][1] <= 20.0 and series[0][2] == "PASS"

    # Dense second cue must breach the 30 CPS burst ceiling
    assert series[1][2] == "FAIL"

def test_visible_chars_excludes_whitespace_and_controls():
    # newline + spaces + a control code must not inflate the count
    assert visible_chars("ab\ncd\x01 ef") == 6

def test_clean_track_passes():
    events = [CaptionEvent(0, 2000, "a calm short line")]
    assert evaluate(events)["verdict"] == "pass"

Run these as part of the same suite that gates the build; the pattern slots directly into the pytest fixtures described in CI/CD gating for caption builds.

Troubleshooting / failure modes

Whitespace-inflated density. Artificial carriage returns or padding raise per-line cell occupancy without changing raw CPS, yet still trigger decoder truncation or forced scrolling. Fix: assert characters-per-line (≤ 32 for CEA-608) independently of CPS — both checks are required.

Per-cue averaging hides bursts. A long cue with a dense opening clause passes a per-cue average while overrunning the decoder in its first 400 ms. Fix: always use the sliding window; never divide chars by full cue duration.

Overlapping cue windows. Two cues whose display windows intersect double-count characters and make the active window ambiguous. Fix: assert cue[n].end <= cue[n+1].start and clamp to a non-negative gap before rate evaluation.

Encoder starvation (rate floor breach). Sustained density below ~5 CPS over a long span signals dropped cues or malformed timestamps, not compliant captions. Fix: alarm on the floor as well as the ceiling.

BOM-corrupted first cue. A UTF-8 BOM left in the file breaks the first cue’s text and skews its character count. Fix: decode with utf-8-sig (step 1) or detect with charset_normalizer before parsing.

Multibyte glyph miscounting. Counting bytes instead of code points over-charges accented or CJK captions. Fix: operate on str (code points), never bytes, and normalize to NFC first.

Operational notes

Memory must scale with window size, not asset duration. The deque holds only the events inside the current window, so peak memory is bounded regardless of whether the asset is a 30-second promo or a four-hour feature — provided the caller streams cues rather than materializing the whole sorted list for very large tracks. For batch QC across thousands of assets, drive the checker with async batch caption processing and cap worker-pool concurrency so I/O on shared caption storage does not saturate; each job is stateless and consumes exactly one caption track. Wrap rolling_cps in a generator when integrating into a streaming pipeline so violations yield in real time instead of buffering a feature-length asset. Finally, serialize every verdict — including the per-onset CPS series — into the telemetry store that feeds scheduled QC report generation, so character-rate failures are queryable by clause alongside every other QC dimension during an audit.

Frequently Asked Questions

Should spaces count toward CPS? No. A CEA-608 decoder’s buffer pressure tracks painted glyphs, so spaces, newlines, and control codes are excluded from the CPS numerator. They are counted separately for characters-per-line checks.

Why a sliding window instead of per-cue CPS? Per-cue division averages away bursts that straddle cue boundaries. A rolling window over the whole track reports the true instantaneous peak — the value that actually overruns the decoder.

Is the limit 17, 20, or 30 CPS? All three describe different ceilings: 17 CPS is the EBU-TT-D reader-comfort target, 20 CPS is the operational sustained limit, and 30 CPS is the short-burst decoder-survival envelope. Enforce the lowest one your target jurisdiction requires.

Can the checker auto-fix violations? It can flag dense cues for splitting or window extension, but it must never truncate dialogue — that trades a rate violation for an FCC § 79.1 completeness violation. Remediation is logged and re-validated, never applied blind.

Related

• Automated sync drift detection — frame-accurate PTS alignment that depends on clean density signals.
• Scheduled QC report generation — windowed aggregation and audit storage for rate verdicts.
• CI/CD gating for caption builds — the stateless gate and exit contract this check plugs into.
• How to enforce FCC character rate limits programmatically — multi-language payloads and overlapping-cue resolution.
• SRT timestamp normalization — the millisecond-epoch normalization this check requires upstream.

Part of: Automated QC Validation & Reporting — the broadcast caption QC reference.