detect_filetype()/partition() raise UnicodeDecodeError on non-UTF-8 text files that start with "{" or "["

[2sumtech]

Describe the bug

A plain-text file that is not UTF-8 encoded (e.g. cp1252/latin-1) and whose first non-whitespace character is { or [ — a log file with [timestamp] lines, an INI-style [section] config — crashes file-type detection with UnicodeDecodeError instead of being classified as TXT. The crash surfaces from partition(), partition(file=...), and detect_filetype() directly.

To Reproduce

import io
from unstructured.partition.auto import partition

payload = "[2026-08-19] München report\n".encode("cp1252")
partition(file=io.BytesIO(payload))

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 14: invalid start byte

Same with a .txt file path. Requires libmagic installed (the standard setup, including the official docker image) so the file routes through the text differentiator; on environments where libmagic reports JSON files as text/plain (noted in a code comment in filetype.py), actual JSON files in legacy encodings crash the same way.

Expected behavior

The file should be classified (as TXT here) and partitioned; a content-sniffing predicate should never raise on arbitrary text input.

Root cause

_TextFileDifferentiator._is_json (unstructured/file_utils/filetype.py) runs json.load(file) on the raw byte stream and catches only json.JSONDecodeError. json.load auto-detects only UTF-8/16/32 (the JSON interchange encodings, RFC 8259), so any other charset raises UnicodeDecodeError, which escapes. The 4096-char text_head gate decodes with fallback charset detection, so it happily passes non-UTF-8 text through to the raw-bytes parse. This is a regression of #705 (fixed by #707 in 0.7.3): the old _is_text_file_a_json worked on the decoded text head.

Environment

unstructured 0.26.3 (current main, 104b585), Python 3.11, libmagic via python-magic.

I have a fix ready and will open a PR.