Given that the Python GIL prevents us from running the PandasEngine (from #24) across multiple cores, we should have the ability to partition the input data into a number of separate processes and execute those in parallel. Probably some sort of transform_reduce would work. We'll need to make sure we handle separate interpreters correctly. My thought is that we keep each PandasEngine single threaded and simply spawn a bunch of them in the PythonTiler itself.
Given that the Python GIL prevents us from running the PandasEngine (from #24) across multiple cores, we should have the ability to partition the input data into a number of separate processes and execute those in parallel. Probably some sort of
transform_reducewould work. We'll need to make sure we handle separate interpreters correctly. My thought is that we keep each PandasEngine single threaded and simply spawn a bunch of them in thePythonTileritself.