Caption Encoding & Charset Detection

Caption files arrive in whatever encoding the tool that last touched them happened to emit, and a wrong guess about those bytes corrupts the very first cue or throws mid-parse. A single ingest queue routinely mixes clean UTF-8, UTF-8 with a byte-order mark, UTF-16 from a Windows export, and Windows-1252 or Latin-1 from a legacy authoring suite — and none of them announce themselves in the filename. Hand any of those bytes to a parser that assumes UTF-8 and one of two things happens: a UnicodeDecodeError halts the job at the first non-ASCII byte, or, worse, the bytes decode “successfully” into mojibake that passes parsing and only surfaces as garbled text at playout. The fix is to make encoding detection a deterministic, auditable stage that runs before any timestamp or cue is read, converts every asset to canonical UTF-8, and refuses to guess when the evidence is weak. This is the front door to every workflow in SRT, SCC & WebVTT parsing workflows: nothing downstream is trustworthy until the bytes are known.

The contract this page enforces is narrow and testable. Every file resolves to exactly one source encoding with an attached confidence score; a byte-order mark is treated as authoritative and short-circuits statistics entirely; a statistical guess below a fixed confidence floor is not decoded but quarantined for review; and the emitted artifact is always canonical UTF-8 with no BOM and LF line endings, so every stage after this one can assume clean bytes. Detection is deterministic — the same bytes always yield the same verdict — which is what makes a green ingest reproducible evidence rather than a lucky roll.

Problem Framing: Detection Is a Precondition, Not a Repair

Encoding handling fails when it is treated as error recovery — a try/except wrapped around a parser that “fixes” bad bytes after the parse has already started. By then the damage is done: the parser has consumed a leading byte-order mark as a glyph, split a multi-byte character across a chunk boundary, or decoded a Windows-1252 en-dash as a Latin-1 control code. Detection must be a discrete stage that runs on the raw bytes before a single cue is tokenized, and it must produce a decision, not a decoded string. That decision is either “decode as encoding X” or “do not decode — quarantine.”

Three numeric properties make the decision enforceable. First, a byte-order mark is authoritative: if the first two-to-four bytes match a known BOM signature, the encoding is settled at confidence 1.0 and no statistics run. Second, statistical detection carries a confidence score on a fixed 0.0–1.0 scale, and there is a hard floor — 0.90 in the reference below — under which the file is not decoded. Third, the output is canonical: exactly one byte-order mark is stripped, line endings collapse to LF, and the file is re-emitted as UTF-8 with no BOM, so downstream stages have zero encoding variance to reason about. A file that decodes at 0.62 confidence is not “probably fine” — it is a coin-flip that will corrupt a proper noun, and the pipeline treats it as a hard stop rather than a silent pass.

The stakes are the same accuracy obligation that governs the rest of the caption stack. A dropped or substituted accented character pushes an English-language track under the 99% character-accuracy floor that FCC 47 CFR § 79.1 sets, so an encoding mistake is a compliance defect, not a cosmetic one. Getting the bytes right here is what lets parsing SCC with Python libraries and SRT timestamp normalization assume a clean UTF-8 input and never re-implement charset logic of their own.

Pipeline Stage & Prerequisites

Detection is the first stage after ingest and strictly before tokenization. It reads bytes from object storage or stdin, never trusting a filename extension or an HTTP charset header, both of which are routinely wrong on archival assets. Its only output to the rest of the pipeline is a normalized UTF-8 byte stream plus a small verdict record — source encoding, confidence, whether a BOM was stripped — that flows into the audit manifest. When it runs at archive scale it sits inside the worker pool described in async batch caption processing, one detection per asset, with quarantined files diverted to a review queue rather than failing the whole batch.

Required tooling:

Pin these from pre-built wheels in the job lockfile. charset_normalizer is preferred over the older chardet because it is actively maintained, ships without a C extension, and returns a normalized chaos metric that maps cleanly to a confidence floor; chardet is retained only as an optional cross-check when two independent detectors disagreeing is itself a signal to quarantine.

Step-by-Step Implementation

Step 1 — Sniff for a byte-order mark first

A byte-order mark is a deterministic signal that needs no statistics, so it is tested first. The one trap is ordering: the UTF-32 LE mark (FF FE 00 00) begins with the same two bytes as the UTF-16 LE mark (FF FE), so the four-byte signatures must be tested before the two-byte ones or a UTF-32 file is misread as UTF-16.

from typing import Optional

# BOM signatures, longest first. UTF-32 LE (FF FE 00 00) shares its first two
# bytes with UTF-16 LE (FF FE), so 4-byte marks MUST be tested before 2-byte
# marks (Unicode 15.0 §23.8) or a UTF-32 file is misdetected as UTF-16.
BOM_TABLE = (
    (b"\x00\x00\xfe\xff", "utf-32-be", 4),
    (b"\xff\xfe\x00\x00", "utf-32-le", 4),
    (b"\xef\xbb\xbf",     "utf-8-sig", 3),
    (b"\xff\xfe",         "utf-16-le", 2),
    (b"\xfe\xff",         "utf-16-be", 2),
)


def sniff_bom(raw: bytes) -> Optional[tuple[str, int]]:
    """Return (encoding, bom_len) when a byte-order mark is present, else None.
    A BOM is authoritative — it settles the encoding without any statistics."""
    for sig, encoding, length in BOM_TABLE:
        if raw.startswith(sig):
            return encoding, length
    return None

Step 2 — Score the encoding statistically when there is no BOM

Most SRT and WebVTT files carry no BOM, so the fallback is statistical. charset_normalizer.from_bytes(raw).best() returns the single most-likely CharsetMatch — or None when nothing decodes — and its chaos attribute is a normalized 0.0–1.0 measure of how “noisy” the best decode looks. A confidence of 1 - chaos gives a monotonic score where 1.0 is a clean decode and values near 0 mean the detector is essentially guessing.

from charset_normalizer import from_bytes


def detect_charset(raw: bytes) -> tuple[str, float]:
    """Statistical detection for BOM-less bytes.

    Returns (encoding, confidence) where confidence = 1 - chaos, clamped to
    [0.0, 1.0]. charset_normalizer is the maintained successor to chardet and
    scores candidates by byte-distribution chaos rather than a fixed table.
    """
    best = from_bytes(raw).best()
    if best is None:
        return "unknown", 0.0
    confidence = max(0.0, min(1.0, 1.0 - float(best.chaos)))
    return best.encoding, round(confidence, 3)

Step 3 — Gate on confidence and decide normalize vs quarantine

The verdict combines both signals. A BOM short-circuits to confidence 1.0; otherwise the statistical confidence is compared to the floor. Below the floor the file is not decoded — decoding a low-confidence guess is exactly how mojibake enters the pipeline — so the action is quarantine.

from dataclasses import dataclass

CONFIDENCE_FLOOR = 0.90   # below this, the guess is untrustworthy — quarantine


@dataclass
class EncodingVerdict:
    source_encoding: str
    confidence: float
    had_bom: bool
    action: str            # "normalize" | "quarantine"


def resolve_encoding(raw: bytes) -> EncodingVerdict:
    bom = sniff_bom(raw)
    if bom is not None:
        encoding, _ = bom
        # A BOM is authoritative: confidence is 1.0 by definition.
        return EncodingVerdict(encoding, 1.0, had_bom=True, action="normalize")

    encoding, confidence = detect_charset(raw)
    action = "normalize" if confidence >= CONFIDENCE_FLOOR else "quarantine"
    return EncodingVerdict(encoding, confidence, had_bom=False, action=action)

Step 4 — Normalize to canonical UTF-8 with a single BOM strip

Only files with a normalize verdict are decoded. Decoding via the utf-8-sig, utf-16, and utf-32 codecs consumes one byte-order mark automatically, but a file mangled by a buggy converter can carry a doubled BOM, so a single explicit lstrip("") removes any residual U+FEFF without a fragile double-strip. Line endings collapse to LF, and the re-encode uses plain utf-8 (never utf-8-sig) so no BOM is written back.

def normalize_to_utf8(raw: bytes) -> tuple[bytes, EncodingVerdict]:
    """Decode with the resolved encoding, strip any leading BOM, and re-emit
    canonical UTF-8 (no BOM, LF line endings). Raises on a quarantined file."""
    verdict = resolve_encoding(raw)
    if verdict.action == "quarantine":
        raise ValueError(
            f"charset confidence {verdict.confidence:.2f} < {CONFIDENCE_FLOOR:.2f}: "
            f"guessed {verdict.source_encoding!r} — quarantined for review"
        )

    # errors='strict' guarantees a partial/mojibake decode can never be returned.
    text = raw.decode(verdict.source_encoding, errors="strict")
    # Remove any residual U+FEFF (handles a doubled BOM the codec left behind).
    text = text.lstrip("")
    # Collapse CRLF/CR to LF so chunked decoders never split a codepoint on \r\n.
    text = text.replace("\r\n", "\n").replace("\r", "\n")
    return text.encode("utf-8"), verdict   # canonical UTF-8, no BOM

For legacy SRT specifically — the single most common problem source — the Windows-1252 versus UTF-8 disambiguation deserves its own byte-level heuristic, covered in detecting Windows-1252 in legacy SRT; that page refines the detect_charset step for the exact C1-range punctuation that trips up statistical detectors on short files.

Threshold Reference Table

Every constant the detector asserts, with its source. Tune the config against this table, not against prose.

The one policy that is non-negotiable is the replacement column: errors='ignore' and errors='replace' are never used, because both silently delete or substitute characters and drop the file below the FCC accuracy floor. If a codec cannot decode the bytes strictly, the answer is a different codec or the quarantine queue — never a lossy decode.

Verification & Test Pattern

The only honest test of a detector is a round trip: take text of known content, encode it into each target encoding (with and without a BOM), and assert that detection resolves the right source encoding and that normalization reproduces the original characters exactly. Pin a low-confidence fixture too, and assert it quarantines rather than decodes.

import pytest
from caption_encoding import (
    resolve_encoding, normalize_to_utf8, sniff_bom, CONFIDENCE_FLOOR,
)

SAMPLE = "1\n00:00:01,000 --> 00:00:03,500\n“Café” — déjà vu\n"

@pytest.mark.parametrize("encoding,bom", [
    ("utf-8", b""),
    ("utf-8-sig", b""),          # codec adds its own BOM
    ("utf-16", b""),             # codec adds its own BOM
    ("cp1252", b""),
])
def test_round_trip_recovers_text(encoding, bom):
    raw = bom + SAMPLE.encode(encoding)
    out, verdict = normalize_to_utf8(raw)
    assert verdict.action == "normalize"
    # Normalized output must decode back to the exact original characters.
    assert out.decode("utf-8").replace("\r\n", "\n") == SAMPLE

def test_bom_is_authoritative():
    raw = b"\xff\xfe" + "hi".encode("utf-16-le")
    assert sniff_bom(raw) == ("utf-16-le", 2)
    assert resolve_encoding(raw).confidence == 1.0

def test_low_confidence_is_quarantined(monkeypatch):
    import caption_encoding as ce
    monkeypatch.setattr(ce, "detect_charset", lambda raw: ("iso-8859-5", 0.40))
    with pytest.raises(ValueError):
        normalize_to_utf8(b"\x80\x81\x82 ambiguous bytes")

Run this suite as its own CI step before the detector is ever pointed at production assets. The round-trip fixtures double as regression protection: a library upgrade that changes chaos scoring will move a fixture across the floor, and the test catches it before an ingest silently starts quarantining clean files.

Troubleshooting / Failure Modes

Windows-1252 smart quotes decode as UTF-8 “successfully” : Root cause: a curly quote (0x93/0x94) or en-dash (0x96) is a lone byte in the 0x80–0x9F range that is illegal as standalone UTF-8, so a strict UTF-8 decode raises — but a lenient one substitutes U+FFFD. Fix: never decode leniently; try cp1252 before iso-8859-1 so the byte maps to the intended glyph, and see detecting Windows-1252 in legacy SRT.

Double BOM leaves a stray glyph at the head of the file : Root cause: a file round-tripped through a buggy converter carries EF BB BF EF BB BF; the codec consumes only the first mark and the second decodes to a visible U+FEFF in front of the first cue. Fix: lstrip("") after decode removes every residual mark, not just one.

UTF-16 file with no BOM detected as Latin-1 : Root cause: BOM-less UTF-16 has a NUL byte in every other position; a byte-oriented detector can score it as a single-byte encoding full of control codes. Fix: add a pre-check for a high ratio of 0x00 bytes at even or odd offsets and force the UTF-16 candidate; if the ratio is ambiguous, quarantine rather than guess.

Mojibake that already passed a previous stage : Root cause: an upstream tool decoded CP1252 as Latin-1 and re-encoded to UTF-8, baking â€" where an em-dash belonged; the bytes are now valid UTF-8 and no detector can recover the original. Fix: scan post-normalization for the classic Ã/â€ mojibake bigrams and reject rather than trust the “clean” UTF-8.

Pure-ASCII file scored at low confidence and quarantined : Root cause: a file with no bytes above 0x7F is genuinely ambiguous — every ASCII-superset encoding decodes it identically — and some detectors report that ambiguity as low confidence. Fix: treat a decode that is valid ASCII as UTF-8 at high confidence regardless of the detector’s chaos score, since the byte-identical result is safe.

Confidence floor tuned too low lets mojibake through : Root cause: relaxing the floor to clear a backlog of quarantined files trades review cost for silent corruption. Fix: keep the floor at 0.90, and instead of lowering it, add the Windows-1252 heuristic so genuinely-recoverable legacy files clear the floor on real evidence.

Operational Notes

At single-file scale detection is a millisecond concern; at archive scale it is a throughput and triage concern. Read bytes from object storage or stdin rather than copying onto local disk, and cap the number of bytes the statistical detector inspects — the first 64 KB of a file is more than enough to fix an encoding, and feeding it a feature-length transcript wastes CPU without improving the verdict. Run one detection per asset inside the bounded worker pool from async batch caption processing so a batch of thousands cannot collectively exhaust the host.

Treat the quarantine queue as a first-class output, not an error dump. Every quarantined file carries its guessed encoding and confidence, so an operator reviews a ranked list — highest-confidence-but-still-sub-floor first — rather than an undifferentiated pile. Stream the verdict records (encoding, confidence, BOM flag) into the manifest that scheduled QC report generation aggregates, so a rising rate of Windows-1252 fallbacks from one vendor becomes a visible trend and a source-side conversation rather than a recurring silent fix. Once the bytes are canonical UTF-8, the parse stage inherits a single invariant — clean, BOM-free, LF-terminated text — and never needs to reason about encoding again.

Related

• Parsing SCC with Python libraries — the CEA-608 tokenizer that consumes the canonical UTF-8 this stage produces.
• Detecting Windows-1252 in legacy SRT — the byte-level C1-range heuristic behind the CP1252-vs-UTF-8 decision.
• SRT timestamp normalization — the sibling cleanup stage that frame-quantizes timing once text is decoded.
• Async batch caption processing — the bounded worker pool that runs one detection per asset at archive scale.
• Scheduled QC report generation — where per-file encoding verdicts roll up into an auditable trend.

Part of: SRT, SCC & WebVTT Parsing Workflows — the broadcast caption parsing reference.