PXF fdw supports parallel data loading.

[MisterRaindrop]

Parallel support for FDW is still an early-stage commit. It requires modifications to the Cloudberry kernel to support parallelism, as well as refactoring the FDW code in PXF.

Currently, I have a demo that can run through simple cases.

However, I have a question:

since we now rely on the stable branch of Cloudberry, can I still submit code to the stable branch REL_2_STABLE?

[MisterRaindrop]

How are the versions of PXF and the cloudberry kernel planned?

[ostinru]

How are the versions of PXF and the cloudberry kernel planned?

For Greenplum+PXF these component versions they were loosely connected. Probably, only major versions should match (e.g. Greenplum 6.x and PXF 6.x). In production we have clusters with latests Greenplum 6 and PXF 6.2.3 (there were a bug fixed in lates 6.10.x).

So, I think we can try to keep single version of PXF for all Cloudberry versions (unless we need changes in Cloudberry core).

[my-ship-it]

Parallel support for FDW is still an early-stage commit. It requires modifications to the Cloudberry kernel to support parallelism, as well as refactoring the FDW code in PXF.

Currently, I have a demo that can run through simple cases.

However, I have a question:

since we now rely on the stable branch of Cloudberry, can I still submit code to the stable branch REL_2_STABLE?

I think there is no problem. Generally, plugins are based on the stable branch of the kernel, and the stable branch ensures the compatibility of ABI interfaces.

[MisterRaindrop]

Parallel support for FDW is still an early-stage commit. It requires modifications to the Cloudberry kernel to support parallelism, as well as refactoring the FDW code in PXF.

Currently, I have a demo that can run through simple cases.

However, I have a question:

since we now rely on the stable branch of Cloudberry, can I still submit code to the stable branch REL_2_STABLE?

I think there is no problem. Generally, plugins are based on the stable branch of the kernel, and the stable branch ensures the compatibility of ABI interfaces.

The issue is that parallel testing requires reliance on the modified Cloudberry kernel code I worked on, otherwise, effective testing cannot be conducted.

[ostinru]

Parallel support for FDW is still an early-stage commit.

Which part of pipeline Cloudberry => pxf_fdw => pxf => S3/HDFS are you going to query concurrently? Do you want to send more than one HTTP requests per one postgreSQL process?

[MisterRaindrop]

Parallel support for FDW is still an early-stage commit.

Which part of pipeline Cloudberry => pxf_fdw => pxf => S3/HDFS are you going to query concurrently? Do you want to send more than one HTTP requests per one postgreSQL process?

The parallel capability of FDW in Cloudberry (PostgreSQL parallelization) is utilized. When parallelism is enabled, multiple processes are launched to perform queries in parallel. This involves kernel support for parallelism in Cloudberry, modifications to the FDW and server in Cloudberry-PXF, and I have already successfully run simple test cases.