Python CSV Header Has a BOM: Decode Before DictReader Splits Columns

By ON Lab · Sources checked November 8, 2026

If Python’s CSV reader shows an unexpected character in the first column name, check the file’s opening bytes before editing the header. For a file known to be UTF-8, encoding="utf-8-sig" removes an opening UTF-8 byte order mark during decoding. This also matters when the first header is quoted: leaving the mark in place can prevent the quote from being treated as the start of a quoted field.

First establish which boundary failed

A dictionary lookup such as row["id"] can fail because the header is different, because the delimiter is wrong, or because the file has no header at all. A BOM is only one possibility. Work on a copy and inspect both the bytes and the parsed field names. Do not print real customer records into a public issue.

from pathlib import Path
import csv
path = Path("input.csv")
print(path.read_bytes()[:3].hex())
with path.open(encoding="utf-8", newline="") as stream:
    reader = csv.DictReader(stream)
    print(repr(reader.fieldnames))

The UTF-8 signature is efbbbf. If the opening bytes differ, this test has not established a UTF-8 BOM. A UTF-16 file needs its actual encoding; changing the separator will not repair that mismatch. If the header contains an ordinary leading space, inspect the producing system rather than assuming every unexpected character is a signature.

Why the first quoted header matters

We created a local fixture with an opening UTF-8 signature followed by this content. The identifier is text, and the first column name contains a comma.

"last,name",id
"Lee, Min",0012

With plain UTF-8 decoding, our Python 3.9.6 run produced three header fields: the first began with U+FEFF followed by "last, the second was name", and the third was id. The data had only two fields, so the reader assigned None to the final key. This was a column-boundary problem as well as an odd first key. Renaming the first dictionary key after parsing would not reconstruct those misaligned columns.

Decode before CSV parsing

import csv
from pathlib import Path
with Path("input.csv").open(encoding="utf-8-sig", newline="") as stream:
    reader = csv.DictReader(stream)
    expected = ["last,name", "id"]
    if reader.fieldnames != expected:
        raise ValueError(f"Unexpected header: {reader.fieldnames!r}")
    for row in reader:
        print(row["last,name"], row["id"])

On the same fixture, the fields became ["last,name", "id"]; the row mapped to {"last,name": "Lee, Min", "id": "0012"}. The leading zeros remained because this example never converts the identifier to a number. This changes the read operation, not the original file. The codec also accepts UTF-8 text without that signature, so a separate BOM-removal rewrite is unnecessary for these two UTF-8 cases.

If the corrected reader still disagrees

A global replace("\ufeff", "") changes text throughout the file and can hide an upstream concatenation problem. Stripping quotes or splitting each line on commas discards the structure the CSV parser is meant to preserve. Keep the smallest failing input and check its complete header after decoding.

Choose the smallest useful alternative

The standard-library reader and codec require no third-party parser. The historical CPython report supplies a minimal signature example; our quoted-header fixture adds a check for structural misalignment and a specific expected schema. This is not an encoding detector. For opening a CSV in Excel, see the separate original-bytes import check; that interface has a different decoding boundary.

Method: current Python 3.13 documentation, a historical CPython report, and an original Python 3.9.6 fixture on macOS. Both codecs were run against identical synthetic bytes; no user data or spreadsheet application was used. The result does not establish that your file is UTF-8.

Sources

• docs.python.org: source guidance
• docs.python.org: source guidance