Spark CBO considers columnar operator by generic interface

[jinchengchenghh]

In Apache Spark, adding cost calculation for columnar format operators helps the optimizer make better decisions.

Columnar execution can be faster due to better CPU efficiency and compression, but it also introduces extra costs, especially when converting between row and column formats. If these costs are not considered, the optimizer might choose inefficient plans.

By including columnar operator costs, Spark can:

• Properly compare row-based and columnar execution
• Avoid unnecessary data format conversions
• Improve overall query performance and stability

For example, the smj and shuffled hash join depends on the actual native engine implementation, and filter after scan may still keeps the encode caused the broadcast table size estimation incorrect, sort algorithm costs are different.

This is implemented in Spark as a generic interface, so other native engines can extend it and report costs based on their own implementations.

In short, it ensures Spark uses columnar execution only when it is truly beneficial.