From 57f1b76449983ea9d0a726001e5e2e6abc7459ec Mon Sep 17 00:00:00 2001
From: Yuming Wang <yumwang@ebay.com>
Date: Sun, 19 Jul 2026 22:44:45 +0800
Subject: [PATCH] [GLUTEN-12564][CORE] Exclude DefaultAllocationManagerFactory
 from gluten-package shade bundle
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The shaded gluten-package.jar bundles DefaultAllocationManagerFactory.class
(both the CORE path in Arrow <= 15.x and the unsafe/netty subpaths in
Arrow >= 16.x) because the shade relocation excludes
org.apache.arrow.memory.** from relocation but still keeps the class
files in the bundle. The class is redundant — it always exists in the
upstream arrow-memory-unsafe / arrow-memory-netty jar that is on the
runtime classpath.

Starting with Arrow 16.0.0, CheckAllocator.check() inspects the URL of
DefaultAllocationManagerFactory.class and throws
"IllegalStateException: Unknown allocation manager type to infer"
when the URL does not contain "memory-core", "memory-unsafe", or
"memory-netty". When the class is loaded from inside
gluten-package.jar the URL matches none of these patterns (in
particular for Arrow <= 15.x where the class lives at the CORE path
org/apache/arrow/memory/DefaultAllocationManagerFactory.class).

This is currently silent on upstream CI because:
  - Spark 3.3/3.4/3.5 community uses Arrow 12.0.1 whose CheckAllocator
    predates the URL check.
  - Spark 4.0/4.1 uses Arrow 18.1.0; the spark-4.0 profile overrides
    arrow.version to 18.1.0 so gluten-package bundles the unsafe/-
    subpath class whose URL contains "/org/apache/arrow/memory/unsafe/"
    and matches the URL pattern.

But the bug manifests whenever a Spark distribution ships Arrow >=
16.0.0 while Gluten is built with arrow.version <= 15.x (e.g. a
downstream Spark 3.5 fork that upgrades Arrow from 12.0.1 to 18.3.0
while Gluten's spark-3.5 profile still uses arrow.version=15.0.0).

Fix: exclude the three known DefaultAllocationManagerFactory class
locations from the shaded bundle so the classloader always finds the
class in the upstream arrow-memory-*.jar, whose URL contains a
recognized substring. The class is functionally identical (same FACTORY
field providing an AllocationManager.Factory); the only behavioral
change is that Gluten stops forcing the unsafe allocator and lets
Arrow's own default selection apply (which is netty per
DefaultAllocationManagerOption.java:80).

Closes #12564.
---
 package/pom.xml | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/package/pom.xml b/package/pom.xml
index 55318f85cf..513a3352bd 100644
--- a/package/pom.xml
+++ b/package/pom.xml
@@ -163,6 +163,25 @@
                     <exclude>META-INF/NOTICE.txt</exclude>
                     <exclude>LICENSE.txt</exclude>
                     <exclude>NOTICE.txt</exclude>
+                    <!--
+                      Exclude DefaultAllocationManagerFactory from the shade bundle.
+                      Arrow 16.0.0+'s CheckAllocator inspects the URL of this class
+                      and throws "Unknown allocation manager type to infer" if the
+                      URL does not contain "memory-core", "memory-unsafe", or
+                      "memory-netty". When the class is bundled inside
+                      gluten-package.jar the URL does not match any of these
+                      patterns (in particular for Arrow <= 15.x where the class
+                      lives at org/apache/arrow/memory/DefaultAllocationManager
+                      Factory.class), breaking allocator initialization whenever
+                      the runtime Arrow version is >= 16.0.0 (e.g. Spark
+                      distributions that ship a newer Arrow than Gluten's
+                      build-time arrow.version). Defer to the upstream
+                      arrow-memory-unsafe / arrow-memory-netty jar which provides
+                      a recognizable URL.
+                    -->
+                    <exclude>org/apache/arrow/memory/DefaultAllocationManagerFactory.class</exclude>
+                    <exclude>org/apache/arrow/memory/unsafe/DefaultAllocationManagerFactory.class</exclude>
+                    <exclude>org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.class</exclude>
                   </excludes>
                 </filter>
                 <filter>