feat(DBI): Phase 8 (RootClass) + Phase 9/9b (switch to upstream DBI.pm + PurePerl, DBD::JDBC base driver)

dev/modules/dbi_test_parity.md

@@ -471,7 +471,7 @@ Triage these once Phase 1 & 2 are done and we have clean output.
 
 ## Progress Tracking
 
-### Current Status: Phases 1–6 landed on `fix/dbi-test-parity` (PR #546). Callbacks, HandleSetErr, and errstr accumulation landed in Phase 6.
+### Current Status: Phases 1–7 landed on `fix/dbi-test-parity` (PR #546, merged). Phase 8 (RootClass), Phase 9 (upstream DBI.pm switch), and Phase 9b (JDBC restoration) landed on `feature/dbi-phase8-and-arch-switch`.
 
 ### Completed
 
@@ -663,6 +663,142 @@ Triage these once Phase 1 & 2 are done and we have clean output.
       by removing the `!exists` guard in the connect-attr
       re-application path)
 
+- [x] **2026-04-23 — Phase 8: RootClass subclassing.**
+  - `DBI->connect` detects `RootClass` (explicit attr or invocant
+    class ≠ DBI), eagerly `require`s it, and reblesses the outer
+    dbh/drh/sth into `${RootClass}::db` / `::dr` / `::st`. Failed
+    `require` dies unconditionally so `eval { connect(...) }` leaves
+    `$@` set for inspection (real DBI's behaviour).
+  - RootClass is stashed on the inner dbh so prepared sths inherit
+    it via `_new_sth`.
+  - `_new_sth` now inherits the error-handling attributes
+    (`RaiseError`, `PrintError`, `PrintWarn`, `RaiseWarn`,
+    `HandleError`, `HandleSetErr`, `ShowErrorStatement`, `Warn`)
+    from the parent dbh — without this, `set_err` on an sth
+    couldn't fire `RaiseError` because it looks them up on the
+    inner hash.
+  - `DBI::_::OuterHandle::_dispatch_packages` detects the
+    dr/db/st suffix via `isa()` for subclass-reblessed handles.
+  - `DBD::_::db::clone` propagates `RootClass` (plus `CompatMode`,
+    `RaiseError`, `PrintError`) to the cloned handle.
+  - **Per-test deltas:**
+    - `t/30subclass.t`: 19/43 → **43/43**
+    - `t/06attrs.t`: 142/166 → 145/166
+
+### Phase 9 (in progress): architectural switch to upstream DBI.pm
+
+**Status: Phase 9 + 9b landed (2026-04-23).**
+
+A spike confirmed that **upstream DBI.pm 1.647 + DBI::PurePerl load
+and run under PerlOnJava with only a 3-line shim**. This led to a
+two-commit architectural switch:
+
+#### Phase 9 — replace our DBI.pm + _Handles.pm with upstream
+
+- `src/main/perl/lib/DBI.pm` — upstream DBI 1.647 DBI.pm unchanged
+  except for a 4-line PerlOnJava patch: force
+  `$ENV{DBI_PUREPERL} = 2` before the XSLoader-vs-PurePerl decision
+  block so DBI::PurePerl is always used and XSLoader::load is never
+  attempted (PerlOnJava has no XS).
+- `src/main/perl/lib/DBI/PurePerl.pm` — upstream DBI::PurePerl 1.47
+  unchanged.
+- Deleted `src/main/perl/lib/DBI/_Handles.pm` (~1500 lines) and
+  `src/main/perl/lib/DBI/_Utils.pm` — PurePerl provides the same
+  functionality.
+- Bug prerequisite: PerlOnJava now walks `@Pkg::ISA` on qualified
+  method calls (`$obj->Pkg::method()`). DBI.pm:1345 relies on
+  `$drh->DBD::_::dr::STORE($k, $v)` routing via
+  `@DBD::_::dr::ISA = qw(DBD::_::common)` to
+  `DBD::_::common::STORE`. Fixed in RuntimeCode.java.
+
+#### Phase 9b — restore JDBC path via DBD::JDBC base driver
+
+Phase 9 disabled JDBC-backed DBDs because our Java-registered
+methods were under `package DBI` and got shadowed by upstream's
+`sub connect` / `sub prepare` / etc. This re-plumbs them as a
+proper upstream-style DBD:
+
+- New `PerlModuleBase.registerMethodInPackage(pkg, perlName,
+  javaName)` helper for arbitrary-package registration.
+- `DBI.initialize()` now registers Java methods under
+  `DBD::JDBC::{dr,db,st}::*` instead of under `DBI::`. `connect`
+  lives on `::dr`, `prepare`/`do`/`disconnect`/transactions/`*info`
+  live on `::db`, `execute`/`fetchrow_*`/`rows`/`bind_*` on `::st`.
+- All `bless` targets in `DBI.java` retargeted from `DBI::db` /
+  `DBI::st` / `DBI::dr` to the `DBD::JDBC` equivalents.
+- `DBI.initialize()` is now called from `GlobalContext` at
+  startup (no longer via XSLoader, which doesn't fire with
+  `DBI_PUREPERL=2`).
+- New `src/main/perl/lib/DBD/JDBC.pm` base driver: provides
+  `driver()` factory, `DBD::JDBC::dr/db/st` classes that inherit
+  from `DBD::_::{dr,db,st}` and wire the Java methods in.
+  `DBD::JDBC::st` aliases `fetch` and `fetchrow` to
+  `fetchrow_arrayref` so Perl's MRO doesn't fall through to
+  DBD::_::st's recursive defaults.
+- `src/main/perl/lib/DBD/SQLite.pm` rewritten to inherit from
+  `DBD::JDBC`. Its `::dr::connect` translates DSN via
+  `_dsn_to_jdbc` before delegating to `DBD::JDBC::dr::connect`.
+- `src/main/perl/lib/DBD/Mem.pm` deleted — the bundled upstream
+  DBI ships a real pure-Perl DBD::Mem (built on SQL::Statement)
+  which our shim was shadowing. Removing the shim lets the real
+  upstream driver run under PerlOnJava, which matters for
+  `t/54_dbd_mem.t`.
+
+#### Per-test deltas after Phase 9 + 9b
+
+| Test | Phase 8 (pre-switch) | Phase 9+9b |
+|---|---|---|
+| 01basics.t | 100/130 (halts) | **130/130** |
+| 03handle.t | 94/137 | 134/137 |
+| 06attrs.t | 145/166 | 164/166 |
+| 08keeperr.t | 84/91 | 88/91 |
+| 12quote.t | 5/10 | 10/10 |
+| 14utf8.t | 10/16 | 15/16 |
+| 15array.t | 16/55 | **50/55** |
+| 17handle_error.t | 84/84 | 84/84 |
+| 20meta.t | 3/8 | 8/8 |
+| 30subclass.t | 43/43 | 43/43 |
+| 31methcache.t | 24/49 | **49/49** |
+| 09trace.t | 99/99 | **99/99** (kept after defensive SCOPE_EXIT_CLEANUP fix) |
+| 40/41/42/43 (profile) | 13/84 | SKIP (legit PurePerl skip) |
+| 70callbacks.t | 65/81 | SKIP (legit PurePerl skip) |
+
+8 files go from partial-fail to full-pass; 4 more go from badly
+broken to ≥95%. Profile/Callbacks/Kids/swap_inner_handle are
+legitimately SKIPped by PurePerl — these are the XS-only features
+upcoming Phase 10 will reimplement in Java.
+
+#### Known issues for follow-up
+
+1. **t/09trace.t** previously regressed 99→1 due to a PerlOnJava
+   interpreter bug. **Fixed in this branch** with a defensive
+   guard in `BytecodeInterpreter.SCOPE_EXIT_CLEANUP` that tolerates
+   non-scalar values in a my-scalar slot (root cause is a compiler
+   bug leaving a `RuntimeList` in a scalar register after
+   `my $x = { ternary-returning-list }`; the guard is the
+   minimal-risk fix, the proper emitter fix is tracked separately).
+   Now 99/99.
+2. **Full `jcpan -t DBI` baseline not yet re-run.** Per-test numbers
+   extrapolate to ~5800–6300 passing subtests (from the 4940/6570
+   Phase 7 baseline), but a full run would confirm.
+
+### Phase 10 (planned): reimplement XS-only features in Java
+
+Upstream DBI::PurePerl explicitly skips some XS features with
+warnings like `"$h->{Profile} attribute not supported for DBI::PurePerl"`.
+These are the roadmap for the next round of Java work:
+
+- **Profile dispatch hook** — single biggest block (91 tests in
+  t/40..43_prof_*.t). Upstream XS wraps every dispatched method in
+  a timing frame that bumps `$h->{Profile}{Data}{$path...}`. We'd
+  hook `DBI::dispatch` (via method wrapping in the Java shim) to
+  do the same.
+- **Callbacks** — 65-test block (t/70callbacks.t). Fire
+  `$h->{Callbacks}{$method}` (or `*`) before/around dispatch.
+- **Kids/ActiveKids/CachedKids** auto-bookkeeping on parent handles.
+- **swap_inner_handle**, **take_imp_data** round-trip.
+- **XS-level trace formatter** (per-handle trace fh + PerlIO layers).
+
 ### Next Steps
 
 Remaining high-signal individual-test failures (running

src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java

@@ -172,9 +172,36 @@ public static RuntimeList execute(InterpretedCode code, RuntimeArray args, int c
                             }
 
                             case Opcodes.SCOPE_EXIT_CLEANUP -> {
-                                // Scope-exit cleanup for a my-scalar register
+                                // Scope-exit cleanup for a my-scalar register.
+                                //
+                                // Root cause for the defensive `instanceof` check
+                                // below: a my-scalar declared inside a
+                                // short-circuiting expression
+                                //   if (COND_A and COND_B and defined((my $x = ...)->{k})) {...}
+                                // may never run its MY_SCALAR initialisation if
+                                // COND_A or COND_B short-circuits. The compiler
+                                // has already allocated a register for `$x`, but
+                                // that register may be holding a temp value left
+                                // over from an earlier statement (e.g. a
+                                // CREATE_LIST result from an unrelated block
+                                // whose register was later recycled). When the
+                                // enclosing scope exits, SCOPE_EXIT_CLEANUP runs
+                                // on `$x`'s register and finds a non-scalar.
+                                //
+                                // This is safe to ignore because the user never
+                                // observes `$x` in that short-circuit path (their
+                                // code is inside the same block and also skipped).
+                                // `scopeExitCleanup` only has work to do on real
+                                // RuntimeScalars (IO-owner fd recycling,
+                                // refCount decrement for blessed refs with
+                                // DESTROY, and captureCount tracking for
+                                // closures); a non-scalar slot simply has no
+                                // cleanup obligation.
                                 int reg = bytecode[pc++];
-                                RuntimeScalar.scopeExitCleanup((RuntimeScalar) registers[reg]);
+                                RuntimeBase slot = registers[reg];
+                                if (slot instanceof RuntimeScalar rs) {
+                                    RuntimeScalar.scopeExitCleanup(rs);
+                                }
                                 registers[reg] = null;
                             }
 
@@ -317,6 +344,9 @@ public static RuntimeList execute(InterpretedCode code, RuntimeArray args, int c
                                 int dest = bytecode[pc++];
                                 int src = bytecode[pc++];
                                 RuntimeBase srcVal = registers[src];
+                                if (dest == 51 && srcVal instanceof RuntimeList) {
+                                    new RuntimeException("TRACE ALIAS dest=51 src=" + src + " putting list in reg 51, srcVal=" + srcVal).printStackTrace();
+                                }
                                 registers[dest] = isImmutableProxy(srcVal) ? ensureMutableScalar(srcVal) : srcVal;
                             }
 

src/main/java/org/perlonjava/core/Configuration.java

@@ -33,7 +33,7 @@ public final class Configuration {
      * Automatically populated by Gradle/Maven during build.
      * DO NOT EDIT MANUALLY - this value is replaced at build time.
      */
-    public static final String gitCommitId = "5acad7563";
+    public static final String gitCommitId = "1cdf0926f";
 
     /**
      * Git commit date of the build (ISO format: YYYY-MM-DD).
@@ -48,7 +48,7 @@ public final class Configuration {
      * Parsed by App::perlbrew and other tools via: perl -V | grep "Compiled at"
      * DO NOT EDIT MANUALLY - this value is replaced at build time.
      */
-    public static final String buildTimestamp = "Apr 23 2026 09:55:16";
+    public static final String buildTimestamp = "Apr 23 2026 13:55:28";
 
     // Prevent instantiation
     private Configuration() {

src/main/java/org/perlonjava/runtime/perlmodule/DBI.java

@@ -32,35 +32,48 @@ public DBI() {
     /**
      * Initializes and registers all DBI methods.
      * This method must be called before using any DBI functionality.
+     *
+     * With the switch to upstream DBI.pm + DBI::PurePerl, methods are now
+     * registered under DBD::JDBC::{dr,db,st} sub-packages so upstream's
+     * dispatch (which looks up $h->{ImplementorClass}::method) routes here
+     * for JDBC-backed dbhs. DBD::SQLite / DBD::Mem etc. inherit from these.
      */
     public static void initialize() {
         // Create new DBI instance
         DBI dbi = new DBI();
         try {
-            // Register all supported DBI methods
-            dbi.registerMethod("connect", null);
-            dbi.registerMethod("prepare", null);
-            dbi.registerMethod("execute", null);
-            dbi.registerMethod("fetchrow_arrayref", null);
-            dbi.registerMethod("fetchrow_hashref", null);
-            dbi.registerMethod("rows", null);
-            dbi.registerMethod("disconnect", null);
-            dbi.registerMethod("last_insert_id", null);
-            dbi.registerMethod("begin_work", null);
-            dbi.registerMethod("commit", null);
-            dbi.registerMethod("rollback", null);
-            dbi.registerMethod("bind_param", null);
-            dbi.registerMethod("bind_param_inout", null);
-            dbi.registerMethod("bind_col", null);
-            dbi.registerMethod("table_info", null);
-            dbi.registerMethod("column_info", null);
-            dbi.registerMethod("primary_key_info", null);
-            dbi.registerMethod("foreign_key_info", null);
-            dbi.registerMethod("type_info", null);
-            dbi.registerMethod("ping", null);
-            dbi.registerMethod("available_drivers", null);
-            dbi.registerMethod("data_sources", null);
-            dbi.registerMethod("get_info", null);
+            // dr-level: connect creates a dbh. available_drivers / data_sources
+            // are class-level but also registered here for backwards compat.
+            dbi.registerMethodInPackage("DBD::JDBC::dr", "connect", "connect");
+            dbi.registerMethodInPackage("DBD::JDBC::dr", "data_sources", "data_sources");
+
+            // db-level: SQL prep / execute / transaction / info methods.
+            dbi.registerMethodInPackage("DBD::JDBC::db", "prepare", "prepare");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "disconnect", "disconnect");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "last_insert_id", "last_insert_id");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "begin_work", "begin_work");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "commit", "commit");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "rollback", "rollback");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "ping", "ping");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "table_info", "table_info");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "column_info", "column_info");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "primary_key_info", "primary_key_info");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "foreign_key_info", "foreign_key_info");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "type_info", "type_info");
+            dbi.registerMethodInPackage("DBD::JDBC::db", "get_info", "get_info");
+
+            // st-level: execute / fetch / bind / row-count methods.
+            dbi.registerMethodInPackage("DBD::JDBC::st", "execute", "execute");
+            dbi.registerMethodInPackage("DBD::JDBC::st", "fetchrow_arrayref", "fetchrow_arrayref");
+            dbi.registerMethodInPackage("DBD::JDBC::st", "fetchrow_hashref", "fetchrow_hashref");
+            dbi.registerMethodInPackage("DBD::JDBC::st", "rows", "rows");
+            dbi.registerMethodInPackage("DBD::JDBC::st", "bind_param", "bind_param");
+            dbi.registerMethodInPackage("DBD::JDBC::st", "bind_param_inout", "bind_param_inout");
+            dbi.registerMethodInPackage("DBD::JDBC::st", "bind_col", "bind_col");
+
+            // Legacy: available_drivers and data_sources as DBI-class methods.
+            // Upstream DBI.pm defines available_drivers itself; register only
+            // what it doesn't already provide.
         } catch (NoSuchMethodException e) {
             System.err.println("Warning: Missing DBI method: " + e.getMessage());
         }
@@ -155,7 +168,7 @@ public static RuntimeList connect(RuntimeArray args, int ctx) {
             dbh.put("Name", new RuntimeScalar(jdbcUrl));
 
             // Create blessed reference for Perl compatibility
-            RuntimeScalar dbhRef = ReferenceOperators.bless(dbh.createReference(), new RuntimeScalar("DBI::db"));
+            RuntimeScalar dbhRef = ReferenceOperators.bless(dbh.createReference(), new RuntimeScalar("DBD::JDBC::db"));
             return dbhRef.getList();
         }, dbh, "connect('" + jdbcUrl + "','" + dbh.get("Username") + "',...) failed");
     }
@@ -236,7 +249,7 @@ public static RuntimeList prepare(RuntimeArray args, int ctx) {
             sth.put("NUM_OF_PARAMS", new RuntimeScalar(numParams));
 
             // Create blessed reference for statement handle
-            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
 
             dbh.get("sth").set(sthRef);
 
@@ -831,7 +844,7 @@ public static RuntimeList table_info(RuntimeArray args, int ctx) {
 
             // Create statement handle for results
             RuntimeHash sth = createMetadataResultSet(dbh, rs);
-            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
             return sthRef.getList();
         }, dbh, "table_info");
     }
@@ -864,7 +877,7 @@ public static RuntimeList column_info(RuntimeArray args, int ctx) {
             ResultSet rs = metaData.getColumns(catalog, schema, table, column);
 
             RuntimeHash sth = createMetadataResultSet(dbh, rs);
-            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
             return sthRef.getList();
         }, dbh, "column_info");
     }
@@ -952,7 +965,7 @@ private static RuntimeList columnInfoViaPragma(RuntimeHash dbh, Connection conn,
         result.put("has_resultset", scalarTrue);
         sth.put("execute_result", result.createReference());
 
-        RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+        RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
         return sthRef.getList();
     }
 
@@ -974,7 +987,7 @@ public static RuntimeList primary_key_info(RuntimeArray args, int ctx) {
             ResultSet rs = metaData.getPrimaryKeys(catalog, schema, table);
 
             RuntimeHash sth = createMetadataResultSet(dbh, rs);
-            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
             return sthRef.getList();
         }, dbh, "primary_key_info");
     }
@@ -1001,7 +1014,7 @@ public static RuntimeList foreign_key_info(RuntimeArray args, int ctx) {
                     fkCatalog, fkSchema, fkTable);
 
             RuntimeHash sth = createMetadataResultSet(dbh, rs);
-            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
             return sthRef.getList();
         }, dbh, "foreign_key_info");
     }
@@ -1015,7 +1028,7 @@ public static RuntimeList type_info(RuntimeArray args, int ctx) {
             ResultSet rs = metaData.getTypeInfo();
 
             RuntimeHash sth = createMetadataResultSet(dbh, rs);
-            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBI::st"));
+            RuntimeScalar sthRef = ReferenceOperators.bless(sth.createReference(), new RuntimeScalar("DBD::JDBC::st"));
             return sthRef.getList();
         }, dbh, "type_info");
     }