Search before asking
Description
A column-pruned Hive TEXTFILE query reads only the columns listed in _col_idxs, but the field splitter still scans every separator of each line — including trailing fields that are never used. For wide tables with many unused trailing columns this is wasted work proportional to the unused tail length of every row.
Proposal
On the query path, after the reader resolves _col_idxs, compute max(_col_idxs) + 1 and pass it to the splitter as a split limit. The single-char, non-escape path of HiveTextFieldSplitter then stops as soon as it has emitted that many fields. Each emitted field is still bounded by its own separator, so the produced prefix is byte-identical to a full split, and the reader only needs those prefix columns — query results are unchanged.
The escape-configured path, the multi-char (KMP) path, and EncloseCsvTextFieldSplitter ignore the limit and split fully, so escaped separators and enclose quoting are unaffected. The load path is unaffected.
Use case
Faster scans of wide Hive TEXTFILE tables when queries select only a prefix of the columns.
Related issues
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
A column-pruned Hive
TEXTFILEquery reads only the columns listed in_col_idxs, but the field splitter still scans every separator of each line — including trailing fields that are never used. For wide tables with many unused trailing columns this is wasted work proportional to the unused tail length of every row.Proposal
On the query path, after the reader resolves
_col_idxs, computemax(_col_idxs) + 1and pass it to the splitter as a split limit. The single-char, non-escape path ofHiveTextFieldSplitterthen stops as soon as it has emitted that many fields. Each emitted field is still bounded by its own separator, so the produced prefix is byte-identical to a full split, and the reader only needs those prefix columns — query results are unchanged.The escape-configured path, the multi-char (KMP) path, and
EncloseCsvTextFieldSplitterignore the limit and split fully, so escaped separators and enclose quoting are unaffected. The load path is unaffected.Use case
Faster scans of wide Hive
TEXTFILEtables when queries select only a prefix of the columns.Related issues
No response
Are you willing to submit PR?
Code of Conduct