When the input encoding is set with yaml_parser_set_encoding(), yaml_parser_determine_encoding() (which strips a leading BOM during auto-detection) is skipped, so the BOM reaches the scanner and is handled in yaml_parser_scan_to_next_token():
/* Allow the BOM mark to start a line. */
if (parser->mark.column == 0 && IS_BOM(parser->buffer))
SKIP(parser);SKIP() increments mark.column, so the first line's tokens start at column 1 instead of 0. A root-level block mapping then terminates at the first newline, and parsing fails with "did not find expected ".
Reproduction: parse "\xEF\xBB\xBFa: b\nc: d\n" with the encoding set to YAML_UTF8_ENCODING. Only the a: b pair is emitted before the error. The same input parses fully with YAML_ANY_ENCODING.
Per YAML 1.2 §5.2 the BOM is not content, so this skip should not advance mark.column.
Found via ruby/psych#331 (Ruby's Psych always sets the encoding explicitly, so BOM-prefixed documents were silently truncated).
When the input encoding is set with
yaml_parser_set_encoding(),yaml_parser_determine_encoding()(which strips a leading BOM during auto-detection) is skipped, so the BOM reaches the scanner and is handled inyaml_parser_scan_to_next_token():SKIP()incrementsmark.column, so the first line's tokens start at column 1 instead of 0. A root-level block mapping then terminates at the first newline, and parsing fails with "did not find expected ".Reproduction: parse
"\xEF\xBB\xBFa: b\nc: d\n"with the encoding set toYAML_UTF8_ENCODING. Only thea: bpair is emitted before the error. The same input parses fully withYAML_ANY_ENCODING.Per YAML 1.2 §5.2 the BOM is not content, so this skip should not advance
mark.column.Found via ruby/psych#331 (Ruby's Psych always sets the encoding explicitly, so BOM-prefixed documents were silently truncated).