Detecting Windows-1252 in Legacy SRT

SRT files exported by legacy Windows caption tools are almost always Windows-1252, and the specific bytes that give them away sit in the 0x80–0x9F C1 range: curly single quotes 0x91/0x92, curly double quotes 0x93/0x94, and the en-dash and em-dash at 0x96/0x97. Those exact bytes are the failure mode: as lone bytes they are illegal in UTF-8 (they can only ever appear as continuation bytes following a lead byte), so a strict UTF-8 decode raises UnicodeDecodeError at the first smart quote, while a naive detector or a lenient decode turns a subtitle’s dialogue quotes into U+FFFD replacement characters or mojibake. A whole-file statistical detector often can not resolve this on short subtitle files — a two-line SRT with one em-dash gives it almost nothing to work with. What does resolve it deterministically is a byte-level heuristic that asks a precise question: are there code points in the C1 range that are valid Windows-1252 glyphs but cannot be part of any valid UTF-8 multibyte sequence? If yes, and strict UTF-8 fails, the file is Windows-1252. This is the CP1252-vs-UTF-8 refinement of the byte sniffing in caption encoding and charset detection.

import codecs

# Windows-1252 assigns printable glyphs to 27 code points in the C1 range
# (0x80-0x9F) that ISO-8859-1 leaves as control codes. As LONE bytes these are
# illegal in UTF-8 (WHATWG Encoding §5) — they can only appear as continuation
# bytes after a lead byte 0xC0-0xF4 — so their unaccompanied presence is a
# smoking gun for Windows-1252.
CP1252_SIGNATURE_BYTES = frozenset({
    0x80, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C,
    0x8E, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9A, 0x9B,
    0x9C, 0x9E, 0x9F,
})
# The punctuation that dominates legacy subtitle text: curly quotes and dashes.
CP1252_COMMON = frozenset({0x91, 0x92, 0x93, 0x94, 0x96, 0x97})  # ‘ ’ “ ” – —


def _utf8_decodes_strict(raw: bytes) -> bool:
    try:
        raw.decode("utf-8", errors="strict")
        return True
    except UnicodeDecodeError:
        return False


def choose_srt_encoding(raw: bytes) -> dict:
    """Decide between utf-8 and windows-1252 for a legacy SRT payload.

    Returns the encoding to decode with, a 0.0-1.0 confidence, and the byte
    evidence behind the call — never a decoded (and possibly lossy) string.
    """
    if raw.startswith(codecs.BOM_UTF8):                     # EF BB BF
        return {"encoding": "utf-8-sig", "confidence": 1.0, "reason": "UTF-8 BOM present"}

    high_bytes = [b for b in raw if b >= 0x80]
    c1_hits = sum(1 for b in raw if b in CP1252_SIGNATURE_BYTES)
    common_hits = sum(1 for b in raw if b in CP1252_COMMON)
    utf8_ok = _utf8_decodes_strict(raw)

    # Decisive case: a C1 signature byte that broke strict UTF-8 cannot be
    # anything but Windows-1252 punctuation. Weight confidence toward the
    # common quote/dash code points that legacy SRT actually emits.
    if c1_hits and not utf8_ok:
        confidence = min(1.0, 0.80 + 0.20 * common_hits / c1_hits)
        return {"encoding": "windows-1252", "confidence": round(confidence, 3),
                "reason": f"{c1_hits} lone C1 byte(s), strict UTF-8 failed"}

    # Clean UTF-8 with real multibyte structure (lead bytes >= 0xC0): trust it.
    if utf8_ok and any(b >= 0xC0 for b in raw):
        return {"encoding": "utf-8", "confidence": 0.99, "reason": "valid UTF-8 multibyte sequences"}

    # Valid UTF-8 but pure ASCII (no high bytes): the two encodings are
    # byte-identical here, so UTF-8 is safe though the file is ambiguous.
    if utf8_ok and not high_bytes:
        return {"encoding": "utf-8", "confidence": 0.50, "reason": "pure ASCII — encodings identical"}

    # High bytes present but no valid UTF-8 multibyte structure and no C1
    # signature: Latin-1-style upper range. CP1252 is the safe superset.
    return {"encoding": "windows-1252", "confidence": 0.60,
            "reason": "high bytes without UTF-8 structure"}


def decode_legacy_srt(raw: bytes) -> tuple[str, dict]:
    verdict = choose_srt_encoding(raw)
    text = raw.decode(verdict["encoding"], errors="strict")  # strict: never lossy
    return text.lstrip(""), verdict

Code walkthrough

CP1252_SIGNATURE_BYTES is the heart of the heuristic. It lists the 27 code points that Windows-1252 fills in the C1 range where ISO-8859-1 has only control codes. The property that makes them a signature is not that they are printable in CP1252 — it is that as lone bytes they are structurally illegal in UTF-8. A byte like 0x92 is only ever valid in UTF-8 as a continuation byte immediately after a lead byte in the 0xC0–0xF4 range; standing alone at the start of a codepoint it fails a strict decode. So the presence of these bytes plus a failed strict UTF-8 decode is not a guess, it is a proof.

choose_srt_encoding runs the decision as a short ladder. A UTF-8 BOM ends it immediately. Otherwise it gathers three cheap byte counts — all high bytes, the C1 signature hits, and the subset of common quote/dash code points — and one boolean, whether the whole buffer decodes as strict UTF-8. The decisive branch fires when there are C1 signature bytes and strict UTF-8 failed: that combination is impossible for genuine UTF-8, so the verdict is Windows-1252. Confidence is floored at 0.80 and lifted toward 1.0 in proportion to how many of the offending bytes are the common punctuation legacy SRT actually contains, so a file full of smart quotes scores higher than one with a single stray byte.

The remaining branches handle the non-decisive cases. Valid UTF-8 carrying real multibyte sequences (any lead byte >= 0xC0) is trusted at 0.99 — those sequences do not occur by accident in CP1252 text. Valid UTF-8 that is pure ASCII is genuinely ambiguous, because every ASCII-superset encoding produces identical bytes; UTF-8 is chosen but flagged at 0.50 so the caller knows the call was a tie. The final fallback covers high bytes with neither UTF-8 structure nor a C1 signature — the upper Latin-1 range like 0xE9 (é) — where Windows-1252 is the safe choice because it is a strict superset of Latin-1’s printable range. decode_legacy_srt then decodes strictly and strips any residual BOM, so a caller gets clean text plus the evidence record for the audit trail.

Threshold reference table

Edge cases & known gotchas

• Pure ASCII is unresolvable, not UTF-8. A file with no byte above 0x7F decodes identically under UTF-8, CP1252, and Latin-1. The heuristic returns UTF-8 at 0.50 confidence, but treat that as “no evidence either way” — do not let a downstream stage record it as a confirmed UTF-8 detection.
• Latin-1 vs Windows-1252 collide outside the C1 range. For bytes 0xA0–0xFF the two encodings are nearly identical, so a file whose only high bytes are 0xE9 (é) or 0xF1 (ñ) can not be told apart by this test. CP1252 is the correct default because it is a strict superset; the distinction only matters if a source genuinely uses the 0x80–0x9F control codes, which caption text never does.
• A UTF-8 BOM overrides the byte scan entirely. The BOM check runs first for a reason: a file can carry EF BB BF and still contain bytes that look CP1252-ish in isolation. Trust the BOM and decode utf-8-sig — the byte counts are only consulted when there is no mark.
• Mixed-encoding files defeat a single verdict. An SRT concatenated from a UTF-8 segment and a CP1252 segment will fail strict UTF-8 and score as CP1252, corrupting the UTF-8 portion on decode. When c1_hits is high but a chunk of the file also holds valid multibyte sequences, quarantine rather than decode — one encoding can not describe the whole file.
• Double-encoded mojibake looks like clean UTF-8. If an upstream tool already read CP1252 as Latin-1 and re-encoded to UTF-8, the smart quotes are now valid multibyte bytes spelling ’. This heuristic will (correctly) call the bytes UTF-8; catching that damage needs a separate mojibake-bigram scan, not an encoding sniff.

Integration hook

This function is the CP1252-vs-UTF-8 specialization of the detect_charset step in caption encoding and charset detection. Where the parent stage runs a general statistical detector and gates on a confidence floor, this heuristic gives a deterministic answer for the one case that detectors handle worst — short legacy SRT with a handful of C1 punctuation bytes — and returns the same shape of verdict record (encoding plus confidence) so it slots straight into the parent’s normalize-or-quarantine gate before the bytes ever reach a parser. The clean UTF-8 it produces is what lets SRT timestamp normalization frame-quantize timing without tripping over a raw 0x96 byte.

Related

• Caption encoding and charset detection — the parent stage: BOM sniffing, confidence gating, and quarantine policy.
• Fixing UTF-8 encoding errors in SCC files — the same CP1252 fallback applied to Scenarist SCC rather than SRT.
• SRT timestamp normalization — the sibling cleanup stage that runs once the SRT text is decoded.

Part of: SRT, SCC & WebVTT Parsing Workflows — the broadcast caption parsing reference.