Describe the bug
Delimiter sniffing in partition_csv() decodes its sample line-by-line (unstructured/partition/csv.py, _CsvPartitioningContext.delimiter). readlines() splits the raw bytes on 0x0A, which in UTF-16 lands in the middle of a code unit, and only the first fragment carries the BOM. For UTF-16-LE — what str.encode("utf-16") and Excel's "Unicode Text" export produce — the first fragment is odd-length and decoding raises UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x0a ... truncated data, even though the caller passed the correct encoding and pd.read_csv handles the same bytes fine.
UTF-16-BE doesn't raise, but silently decodes every line after the first as mojibake, so the sniffer sees corrupted data. (The existing stanley-cups-utf-16.csv example doc is BE, which is why tests never hit this.)
To Reproduce
import io
from unstructured.partition.csv import partition_csv
data = "name,age\nAlice,30\nBob,25\n".encode("utf-16")
partition_csv(file=io.BytesIO(data), encoding="utf-16")
# UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x0a in position 18: truncated data
Fails identically with filename=. pd.read_csv(io.BytesIO(data), encoding="utf-16") parses the same bytes correctly.
Expected behavior
The file partitions successfully when the correct encoding is supplied (and delimiter sniffing sees correctly decoded text for both UTF-16 variants).
Environment
unstructured 0.26.3, Python 3.11, reproduced on current main.
I have a fix ready and will open a PR.
Describe the bug
Delimiter sniffing in
partition_csv()decodes its sample line-by-line (unstructured/partition/csv.py,_CsvPartitioningContext.delimiter).readlines()splits the raw bytes on0x0A, which in UTF-16 lands in the middle of a code unit, and only the first fragment carries the BOM. For UTF-16-LE — whatstr.encode("utf-16")and Excel's "Unicode Text" export produce — the first fragment is odd-length and decoding raisesUnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x0a ... truncated data, even though the caller passed the correctencodingandpd.read_csvhandles the same bytes fine.UTF-16-BE doesn't raise, but silently decodes every line after the first as mojibake, so the sniffer sees corrupted data. (The existing
stanley-cups-utf-16.csvexample doc is BE, which is why tests never hit this.)To Reproduce
Fails identically with
filename=.pd.read_csv(io.BytesIO(data), encoding="utf-16")parses the same bytes correctly.Expected behavior
The file partitions successfully when the correct encoding is supplied (and delimiter sniffing sees correctly decoded text for both UTF-16 variants).
Environment
unstructured 0.26.3, Python 3.11, reproduced on current
main.I have a fix ready and will open a PR.