You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, the number of shuffle mappers is equal to the number of input files, with each mapper reading a single input file. However:
For very large files, we may want multiple mappers reading segments of a file in parallel for the sake of memory utilization and/or maximizing read throughput.
For many very small files, we may want multiple files to be read by a single mapper for the sake of reducing task overhead and/or possibly maximizing read throughput.
(1) has already been brought up by a user as a concrete need. We should therefore support this by allowing num_mappers != num_files, and letting num_mappers be set by the user with a best-effort default as a fallback.
Implementation Notes
(2) (i.e. num_files > num_mappers) won't involve much more than the typical np.array_split() pattern, e.g.
(1) (i.e. num_files < num_mappers), however, will require us to read a file starting at a certain offset; given that the Pandas read_parquet() API doesn't expose that functionality, the most likely course of action would be using the pyarrow Parquet reading facilities directly. Here is an example of such usage, although we will most likely want to use a ParquetDataset with split_row_groups=True and rely on the row group granularity for the MVP.
Right now, the number of shuffle mappers is equal to the number of input files, with each mapper reading a single input file. However:
(1) has already been brought up by a user as a concrete need. We should therefore support this by allowing
num_mappers != num_files, and lettingnum_mappersbe set by the user with a best-effort default as a fallback.Implementation Notes
(2) (i.e.
num_files > num_mappers) won't involve much more than the typicalnp.array_split()pattern, e.g.(1) (i.e.
num_files < num_mappers), however, will require us to read a file starting at a certain offset; given that the Pandasread_parquet()API doesn't expose that functionality, the most likely course of action would be using the pyarrow Parquet reading facilities directly. Here is an example of such usage, although we will most likely want to use aParquetDatasetwithsplit_row_groups=Trueand rely on the row group granularity for the MVP.