Support open-table formats

[sandugood]

Any plans on supporting open-table formats? Specifically asking about Iceberg.

[gabotechs]

This project ships the fundamentals to support Iceberg amongst other open-table formats, but there has been no discussions about default support for it.

My team is also interested in Iceberg support in datafusion-distributed, and are actively working/contributing towards it.

It would make sense to host somewhere an out-of-the-box integration with iceberg-rust and datafusion-distributed, and this project is open to hosting that under a flag.

cc @toutane, do you think having a default iceberg integration offered by datafusion-distributed would make sense?

[toutane]

Hi @sandugood, hi @gabotechs,

Yes, I think having a default integration between iceberg-rust and datafusion-distributed makes complete sense.

The question for me, however, is whether this should be based on the existing DataFusion integration, iceberg-datafusion (the crate under the iceberg-rust project), or not. It provides a basic TableProvider and a leaf node built on top of the primitives from the main iceberg-rust crate.

[gabotechs]

Yeah, I think basing this on iceberg-rust would be great.

The only thing that is still unclear to me is where it should be hosted: if we host it here, we introduce a dependency on iceberg-rust and upgrades will delay up until iceberg-rust gets upgraded, which I don't love.

[toutane]

I don't think iceberg-rust would agree to host it either. There's actually an open PR on iceberg-rust to add a Ballista integration, but the maintainers aren't on board with that direction - apache/iceberg-rust#2613 (comment).

The third option would be to have a dedicated repo under datafusion-contrib. What do you think?

[gabotechs]

Yeah, that can be an option indeed