bug: Parsing of zebrad.toml on Windows fails due to Byte Order Mark (BOM) added to the beginning of the file

[Anynomouss]

What happened?

I expected to zebra to be able to load the config file it created itself on Windows:
When generating a zebrad.toml and loading it:

.\zebrad.exe generate > zebrad.toml 
C:\Users\$USER$\AppData\Local\zebrad.toml

The generate file is not parsed properly because of BOM markers added at the beginning of the file resulting into an error

error: Zebra could not load the provided configuration file and/or environment variables.This might mean you are using a deprecated format of the file, or are attempting to
configure deprecated or unknown fields via environment variables.You can generate a valid config by running "zebrad generate", and diff it against yours to examine any format inconsistencies.
error: zebrad fatal error: I/O operation failed: Configuration error: TOML parse error at line 1, column 3
|
1 | ��# Default configuration for zebrad.

When opening the file in a hex editor it shows FF FE BOM markers are added by Windows since it by default generates text files with Byte Order Mark (BOM) for UTF-16 Little Endian. It would be better to scan for these bytes and skip or remove them. The zebrad.toml config would be loaded normally on Windows. Opening the file with Notepad++, going to encoding and saving as UTF-8 (without BOM) manually fixes the issues. An easy fix would be to ignore errors from the first line, or ignore errors in lines that do not contain '=' since those are comments and not relevant for loading config parameters.

Zebra Version

4.4.1

Which operating systems does the issue happen on?

• Linux
• macOS
• Windows
• Other OS

[natalieesk]

Comment Generated by Claude Code

Triage note before implementation — the "strip a BOM" framing doesn't resolve this report, and there's a design decision to make first:

• A UTF-8 BOM (EF BB BF) already loads correctly on main: the config/toml parser tolerates it (verified empirically by loading a UTF-8-BOM config through the current loader).
• This report's hex dump shows FF FE — a UTF-16 LE BOM, i.e. the whole file is UTF-16-encoded (the old Windows PowerShell 5.1 > redirection default). That isn't a leading-prefix strip: read_to_string can't decode it at all, so a fix would need to detect the BOM and decode UTF-16 (LE/BE) → UTF-8 before parsing.
• Modern PowerShell 7 writes UTF-8 (no BOM), so this mainly affects Windows PowerShell 5.1 users.

Open question: should Zebra accept UTF-16-encoded config files? If yes, the fix is byte-level BOM detection + UTF-16 decoding (with UTF-16 LE/BE + UTF-8-BOM test vectors). Moving to S-needs-design for a team call.

[conradoplg]

I don't think we should accept UTF-16-encoded files