fix: Type::Tiny Phase 6 — tie infrastructure, lexical sub parsing, splice context

dev/modules/type_tiny.md

@@ -8,10 +8,10 @@ and many CPAN modules. This document tracks the work needed to make
 
 ## Current Status
 
-**Branch:** `feature/type-tiny-support`
+**Branch:** `feature/type-tiny-phase6`
 **Module version:** Type::Tiny 2.010001 (375 test programs)
-**Pass rate:** 99.5% (2476/2488 individual tests in tests that ran, 6 files with real failures)
-**Phase:** 5e complete (2026-04-09)
+**Pass rate:** 99.7% (3029/3038 individual tests, 5 files with real failures)
+**Phase:** 6 complete (2026-04-10)
 
 ### Baseline Results
 
@@ -296,7 +296,7 @@ Type::Tie, _HalfOp overloading, etc.) as time permits.
 
 ## Progress Tracking
 
-### Current Status: Phase 5 completed — 99.0% pass rate (2879/2907)
+### Current Status: Phase 6 completed — 99.7% pass rate (3029/3038)
 
 ### Results History
 
@@ -306,6 +306,7 @@ Type::Tie, _HalfOp overloading, etc.) as time permits.
 | Phase 4 | 186 | 57 | — | — | — |
 | Phase 5a | 318 | 13 | 2812 | 2869 | 98.0% |
 | Phase 5b | 331 | 10 | 2879 | 2907 | 99.0% |
+| Phase 6 | 347 | 5 | 3029 | 3038 | 99.7% |
 
 ### Completed Phases
 - [x] Phase 1: `looks_like_number` string parsing (2026-04-09)
@@ -454,55 +455,137 @@ Type::Tie, _HalfOp overloading, etc.) as time permits.
     - `Type-Params/multisig-gotonext.t` (1/6 → 8/8) — eval local restore fix
     - `00-begin.t` (0/0 → 1/1) — eval local restore fix
 
-### Remaining Failing Test Files (after Phase 5e)
+- [x] Phase 6: Tie infrastructure, lexical sub parsing, splice context (2026-04-10)
+  - **Splice list context in interpreter:** Function calls in splice replacement positions
+    now evaluated in LIST context (not SCALAR), matching the JVM backend behavior. The
+    interpreter's `executeSplice` was calling functions in SCALAR context, causing only one
+    value to be inserted instead of the full list.
+  - **Lexical sub + statement modifier:** Lexical subs (`my sub name`) now recognized as
+    function calls before statement modifier keywords (`if`, `unless`, `while`, `until`,
+    `for`, `foreach`, `when`). Previously, `quuux if 1` treated `quuux` as a bareword.
+  - **`tie(my(@arr), ...)` prototype parsing:** Backslash prototype `\[$@%*]` now correctly
+    handles parenthesized `my` declarations. `my(@bar)` produces
+    `OperatorNode("my", ListNode(OperatorNode("@")))` — the extra ListNode wrapper is
+    now unwrapped via `unwrapMyListDeclaration()` helper.
+  - **`return @tied_array` in eval STRING:** `materializeSpecialVarsInResult` was iterating
+    `arr.elements` directly (the empty ArrayList backing TieArray), bypassing FETCHSIZE/FETCH.
+    Now dispatches through `getList()` for tied arrays and hashes.
+  - Files: `CompileOperator.java`, `SubroutineParser.java`, `PrototypeArgs.java`, `RuntimeCode.java`
+  - Tests fixed:
+    - `Eval-TypeTiny/lexical-subs.t` (11/12 → 12/12) — lexical sub + statement modifier
+    - `Eval-TypeTiny/aliases-tie.t` (6/11 → 10/11) — tie prototype + return fix (remaining 1 is DESTROY)
+    - `Type-Tie/basic.t` (1/2 → 3/3) — tie on arrays now supported
+    - `Type-Tie/01basic.t` (15/17 → 17/17) — splice list context
+    - `Types-Standard/tied.t` (0/0 → 27/27) — tie on arrays now supported
+    - `Type-Library/exportables.t` (0/0 → 11/11) — resolved by eval require fix
+    - `Type-Library/exportables-duplicated.t` (0/1 → 1/1) — eval require fix
+    - `Type-Tiny-Enum/basic.t` (17/17 → 25/25) — unlocked more tests
+    - `Moo/basic.t` (4/5 → 5/5), `Moo/coercion.t` (9/19 → 19/19),
+      `Moo/exceptions.t` (13/15 → 15/15), `Moo/inflation.t` (9/11 → 11/11)
+    - `Moo/coercion-inlining-avoidance.t` (0/0 → 14/14), `v2-multi.t` (1/1 → 5/5)
+
+### Remaining Failing Test Files (after Phase 6)
+
+**Tests with actual subtest failures (5 files, 9 individual failures):**
 
-**Tests with 0 subtests (9 files, missing features/deps):**
+| Test | Result | Root Cause |
+|------|--------|-----------|
+| `Error-TypeTiny-Assertion/basic.t` | 28/29 | B::Deparse output differs (known limitation) |
+| `Eval-TypeTiny/basic.t` | 11/12 | DESTROY not implemented (JVM GC) |
+| `Eval-TypeTiny/aliases-tie.t` | 10/11 | DESTROY not implemented (JVM GC) |
+| `Type-Tie/06clone.t` | 3/6 | Clone::PP doesn't preserve tie magic |
+| `Type-Tie/06storable.t` | 3/6 | Storable::dclone doesn't preserve tie magic |
+
+**Tests with 0 subtests / skipped (23 `!` in runner, mostly CWD or missing deps):**
 
 | Test | Issue |
 |------|-------|
 | `Eval-TypeTiny/aliases-native.t` | `\$var = \$other` ref aliasing not supported |
-| `Eval-TypeTiny/aliases-tie.t` | TIESCALAR not found (class loading issue) |
-| `Type-Library/exportables.t` | `+Rainbow` sub not found (exporter edge case) |
 | `Type-Registry/lexical.t` | `builtin::export_lexically` not implemented |
 | `Type-Tiny-Enum/exporter_lexical.t` | `builtin::export_lexically` not implemented |
 | `Types-Standard/strmatch-allow-callbacks.t` | `(?{...})` code blocks in regex |
 | `Types-Standard/strmatch-avoid-callbacks.t` | `(?{...})` code blocks in regex |
-| `Types-Standard/tied.t` | Unsupported variable type for `tie()` |
-| `gh1.t` | Dies early |
-
-**Tests with actual subtest failures (6 files, 12 failures):**
-
-| Test | Result | Root Cause |
-|------|--------|-----------|
-| `Error-TypeTiny-Assertion/basic.t` | 28/29 | B::Deparse output differs |
-| `Eval-TypeTiny/lexical-subs.t` | 11/12 | Lexical sub without parens returns bareword |
-| `Type-Tie/01basic.t` | 15/17 | Tied array edge cases |
-| `Type-Tie/06clone.t` | 3/6 | Clone::PP doesn't preserve tie magic |
-| `Type-Tie/06storable.t` | 3/6 | Storable::dclone doesn't preserve tie magic |
-| `Type-Tie/basic.t` | 1/2 | Unsupported tie on arrays |
-
-**Tests with runner flakiness (varies per run, ~5 Moo tests):**
-
-| Test | Best Result | Issue |
-|------|-------------|-------|
-| `Type-Library/exportables-duplicated.t` | 0/1 | `caller()` corruption after eval require |
-| `Moo/basic.t` | 4/5 | Moo isa coercion |
-| `Moo/coercion.t` | 9/19 | Moo coercion inlining |
-| `Moo/exceptions.t` | 13/15 | Exception `->value` metadata |
-| `Moo/inflation.t` | 9/11 | Moo → Moose inflation |
-| `gh14.t` | 0/1 | Deep coercion edge case |
+| `gh1.t` | Missing `Math::BigFloat` dependency |
+| Various Type-Library/*, Type-Tiny-*/basic.t | Test runner CWD issue — pass when run from Type-Tiny dir |
+
+- [x] Phase 6b: Fix sprintf warnings, `local` restoration on `last`, spurious sprintf warning (2026-04-10)
+  - **sprintf/printf warnings fired unconditionally:** All sprintf/printf warnings
+    ("Invalid conversion", "Missing argument", "Redundant argument") used plain
+    `WarnDie.warn()` which always emits warnings. Changed to `WarnDie.warnWithCategory()`
+    with the `"printf"` category, matching Perl 5 behavior where these warnings only
+    fire under `use warnings` or `use warnings "printf"`.
+  - **`local` variable restoration on `last` exit (3 fixes):**
+    - JVM backend (EmitStatement.java): Added `Local.localSetup/localTeardown` wrapping
+      For3Node (while/for loops, bare blocks) so `last` exits that bypass the body block's
+      own cleanup still restore `local` variables.
+    - JVM backend (EmitControlFlow.java): Non-local `last`/`next`/`redo` now routes
+      through `returnLabel` instead of direct `ARETURN`, ensuring the subroutine's
+      `popToLocalLevel()` cleanup runs when `last LABEL` crosses subroutine boundaries
+      (e.g., test.pl's `sub skip { local $^W=0; last SKIP }`).
+    - Bytecode interpreter (BytecodeCompiler.java): Added `GET_LOCAL_LEVEL/POP_LOCAL_LEVEL`
+      wrapping For3Node for both bare blocks and while/for loops, matching the JVM backend.
+  - **Spurious sprintf "isn't numeric" warning:** `SprintfOperator.java` was calling
+    `getDouble()` on arbitrary string arguments when checking for Inf/NaN on invalid format
+    specifiers. Now only checks DOUBLE type values and known Inf/NaN string literals,
+    avoiding the spurious warning.
+  - Files: `SprintfOperator.java`, `WarnDie.java`, `EmitStatement.java`,
+    `EmitControlFlow.java`, `BytecodeCompiler.java`
+  - Test impact: `op/sprintf2.t` recovers 1 test (1651 → 1652), restoring baseline.
+
+### Remaining Issues from `./jcpan --jobs 8 -t Type::Tiny`
+
+| Issue | Impact | Details |
+|-------|--------|---------|
+| `builtin::export_lexically` | 2 tests | PerlOnJava reports `$]=5.042` so `Exporter::Tiny` takes the native lexical sub path, but `builtin::export_lexically` is not implemented. Affects `Type-Registry/lexical.t`, `Type-Tiny-Enum/exporter_lexical.t`. |
+| `sprintf "%{"` warning | Cosmetic | Fixed in Phase 6b — warning now properly gated by `use warnings "printf"`. Not a test failure; `Types::Standard::Tied` has `use warnings` so the warning is correct but was previously also firing in no-warnings contexts. |
+| `Math::BigFloat` missing | 1 test | Core Perl module not bundled with PerlOnJava. Only `t/40-bugs/gh1.t` requires it. Would need porting `Math::BigInt` + `Math::BigFloat` (large effort). |
+
+### Phase 7: Clone/Storable tie preservation (completed 2026-04-10)
+
+**Goal:** Fix `Type-Tie/06clone.t` (3/6 → 6/6) and `Type-Tie/06storable.t` (3/6 → 6/6).
+
+Both tests create tied variables via `Type::Tie`, clone them, and verify the clone
+still enforces type constraints. Tests 2/4/6 failed because the clone lost tie magic.
+
+**7a. Replace custom Clone::PP with CPAN Clone::PP 1.08:**
+- Replaced our custom 77-line `Clone/PP.pm` with CPAN Clone::PP 1.08.
+- CPAN version handles ties, `clone_self` / `clone_init` hooks, depth limiting, and
+  circular reference detection.
+- However, Clone::PP's tie handling is too simplistic for Type::Tie (it calls
+  `tie %$copy, ref $tied` without constructor arguments), so we also needed 7c.
+
+**7b. Fix Storable::dclone to handle tied variables:**
+- `Storable.java` `deepClone()` now detects `TieHash`, `TieArray`, `TieScalar` backing.
+- For tied hashes/arrays: deep-clones the handler object, creates a new Tie* wrapper,
+  and copies data through the tied interface (FETCH/STORE).
+- For tied scalars: adds `TIED_SCALAR` case to clone the handler and re-tie.
+- Fixed STORABLE_freeze/thaw hook to create the correct reference type (ARRAY vs HASH)
+  for the thaw object — Type::Tie::BASE is array-based, not hash-based.
+- Files: `Storable.java`
+
+**7c. Create Java-based Clone module:**
+- Created `Clone.java` as a proper Java XS implementation of `Clone::clone`.
+- Handles tied hashes, tied arrays, tied scalars, blessed objects, circular references,
+  and depth limiting — equivalent to the XS Clone module.
+- `Clone.pm` loads it via XSLoader (falls back to Clone::PP if unavailable).
+- Files: `Clone.java`
+
+**Bundled tests added:**
+- `src/test/resources/module/Clone-PP/t/` — 7 test files from CPAN Clone::PP 1.08
+- Type-Tie tests are run via `./jcpan -t Type::Tiny` (not bundled; Type::Tie is part of Type-Tiny CPAN dist)
 
 ### Next Steps
-1. Address Tie-related failures (4 files, requires deeper tie infrastructure work)
-2. Investigate `caller()` corruption after `eval require` (exportables-duplicated.t)
-3. Investigate Moo coercion failures (5 test files)
-4. Consider B::Deparse output compatibility (1 test)
+1. Consider implementing scope-exit hooks for DESTROY (2 test files)
+2. Consider B::Deparse output compatibility (1 test)
+3. Fix test runner CWD handling for tests that reference `./lib`, `./t/lib`
+4. Consider bundling `Math::BigFloat` / `Math::BigInt` (low priority, 1 test)
+5. Consider implementing `builtin::export_lexically` (low priority, 2 tests)
 
 ### Open Questions
-- `ArrayRef[Int] | HashRef` triggers `Can't call method "isa" on unblessed reference`
-  at Type/Tiny/Union.pm line 60 — separate runtime issue, not parser-related
-- Test runner shows variable error counts (29-66 `!` errors) due to parallel JVM startup
-  contention — actual failures are consistent across runs
+- The 23 `!` errors in the test runner are mostly CWD-related: tests use `./lib` and `./t/lib`
+  which require running from the Type-Tiny distribution directory
+- All 5 Moo tests pass when run from the correct CWD
+- `builtin::export_lexically` would require lexical scoping machinery — complex to implement properly
 
 ---
 

src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java

@@ -5326,6 +5326,17 @@ public void visit(For3Node node) {
                 emitInt(0);
             }
 
+            // Save local variable level so that `last` exits restore `local` variables.
+            // The body's BlockNode has its own GET_LOCAL_LEVEL/POP_LOCAL_LEVEL, but `last`
+            // bypasses the body's POP_LOCAL_LEVEL. This outer pair catches that case.
+            // On normal exit, the body already restored locals, so this is a no-op.
+            int blockLocalLevelReg = -1;
+            if (FindDeclarationVisitor.containsLocalOrDefer(node)) {
+                blockLocalLevelReg = allocateRegister();
+                emit(Opcodes.GET_LOCAL_LEVEL);
+                emitReg(blockLocalLevelReg);
+            }
+
             // Push loop info so that redo/next/last inside bare blocks work
             // (Perl 5 allows redo/next/last in bare blocks)
             // Unlabeled bare blocks are targets for unlabeled redo/next/last;
@@ -5372,8 +5383,14 @@ public void visit(For3Node node) {
                 patchJump(exitPcPlaceholder, exitPc);
             }
 
-            // Patch last (break) PCs to jump past the block
+            // Patch last (break) PCs to jump to local cleanup (or past the block if no locals).
+            // POP_LOCAL_LEVEL must be at endPc so `last` runs it.
+            // On normal exit this is a no-op since the body's POP_LOCAL_LEVEL already ran.
             int endPc = bytecode.size();
+            if (blockLocalLevelReg >= 0) {
+                emit(Opcodes.POP_LOCAL_LEVEL);
+                emitReg(blockLocalLevelReg);
+            }
             for (int pc2 : loopInfo.breakPcs) {
                 patchJump(pc2, endPc);
             }
@@ -5394,6 +5411,17 @@ public void visit(For3Node node) {
             node.initialization.accept(this);
         }
 
+        // Save local variable level so that `last` exits restore `local` variables.
+        // The body's BlockNode has its own GET_LOCAL_LEVEL/POP_LOCAL_LEVEL, but `last`
+        // bypasses the body's POP_LOCAL_LEVEL. This outer pair catches that case.
+        // On normal exit, the body already restored locals, so POP_LOCAL_LEVEL is a no-op.
+        int for3LocalLevelReg = -1;
+        if (FindDeclarationVisitor.containsLocalOrDefer(node)) {
+            for3LocalLevelReg = allocateRegister();
+            emit(Opcodes.GET_LOCAL_LEVEL);
+            emitReg(for3LocalLevelReg);
+        }
+
         // Step 2: Push loop info onto stack for last/next/redo
         int loopStartPc = bytecode.size();
         // do-while is NOT a true loop (can't use last/next/redo); while/for are true loops
@@ -5497,8 +5525,14 @@ public void visit(For3Node node) {
             emitInt(loopStartPc);
         }
 
-        // Step 10: Loop end - patch the forward jump (last jumps here)
+        // Step 10: Loop end - restore local variables and patch jumps.
+        // POP_LOCAL_LEVEL must be at loopEndPc so `last` runs it.
+        // On normal exit (condition false) this is a no-op since the body already cleaned up.
         int loopEndPc = bytecode.size();
+        if (for3LocalLevelReg >= 0) {
+            emit(Opcodes.POP_LOCAL_LEVEL);
+            emitReg(for3LocalLevelReg);
+        }
         if (loopEndJumpPc != -1) {
             patchJump(loopEndJumpPc, loopEndPc);
         }

src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java

@@ -1174,12 +1174,17 @@ private static void visitSplice(BytecodeCompiler bc, OperatorNode node) {
         if (!arrayOp.operator.equals("@")) bc.throwCompilerException("splice requires array variable: splice @array, ...");
         int arrayReg = resolveArrayOperand(bc, new OperatorNode("splice", arrayOp, node.tokenIndex), "splice");
         List<Integer> argRegs = new ArrayList<>();
+        // Compile splice arguments in LIST context so replacement values
+        // (after offset and length) are properly flattened.
+        int savedCtx = bc.currentCallContext;
+        bc.currentCallContext = RuntimeContextType.LIST;
         for (int i = 1; i < list.elements.size(); i++) { list.elements.get(i).accept(bc); argRegs.add(bc.lastResultReg); }
+        bc.currentCallContext = savedCtx;
         int argsListReg = bc.allocateRegister();
         bc.emit(Opcodes.CREATE_LIST); bc.emitReg(argsListReg); bc.emit(argRegs.size());
         for (int argReg : argRegs) bc.emitReg(argReg);
         int rd = bc.allocateOutputRegister();
-        bc.emit(Opcodes.SPLICE); bc.emitReg(rd); bc.emitReg(arrayReg); bc.emitReg(argsListReg); bc.emit(bc.currentCallContext);
+        bc.emit(Opcodes.SPLICE); bc.emitReg(rd); bc.emitReg(arrayReg); bc.emitReg(argsListReg); bc.emit(savedCtx);
         bc.lastResultReg = rd;
     }
 

src/main/java/org/perlonjava/backend/bytecode/OpcodeHandlerExtended.java

@@ -299,18 +299,18 @@ public static int executeStringConcatAssign(int[] bytecode, int pc, RuntimeBase[
         }
         RuntimeScalar target = (RuntimeScalar) registers[rd];
         // Remember if target was BYTE_STRING before concatenation.
-        // In PerlOnJava, "upgrading" from BYTE_STRING to STRING doesn't change bytes
-        // (unlike Perl where bytes > 127 get re-encoded), so we preserve BYTE_STRING
-        // in .= to prevent false UTF-8 flag contamination of binary buffers.
+        // Only preserve BYTE_STRING when the concat result itself is BYTE_STRING
+        // (both operands were non-UTF-8). When concat produces STRING (at least
+        // one operand was UTF-8), preserve the UTF-8 flag per Perl semantics.
         boolean wasByteString = (target.type == RuntimeScalarType.BYTE_STRING);
         RuntimeScalar result = StringOperators.stringConcat(
                 target,
                 (RuntimeScalar) registers[rs]
         );
         target.set(result);
-        // Preserve BYTE_STRING type when the target was byte string and the result
-        // still fits in Latin-1 (all chars <= 255)
-        if (wasByteString && target.type == RuntimeScalarType.STRING) {
+        // Preserve BYTE_STRING type only when both the target was byte string AND
+        // the concat result was also byte string (meaning the RHS was also non-UTF-8)
+        if (wasByteString && result.type == RuntimeScalarType.BYTE_STRING && target.type == RuntimeScalarType.STRING) {
             String s = target.toString();
             boolean fits = true;
             for (int i = 0; i < s.length(); i++) {

src/main/java/org/perlonjava/backend/jvm/EmitControlFlow.java

@@ -126,8 +126,15 @@ static void handleNextOperator(EmitterContext ctx, OperatorNode node) {
                     "(Lorg/perlonjava/runtime/runtimetypes/ControlFlowType;Ljava/lang/String;Ljava/lang/String;I)V",
                     false);
 
-            // Return the tagged list (will be detected at subroutine return boundary)
-            ctx.mv.visitInsn(Opcodes.ARETURN);
+            // Return the tagged list via returnLabel so that local variable teardown
+            // (popToLocalLevel) runs before the method exits. A direct ARETURN would
+            // bypass the cleanup at returnLabel, leaving `local` variables un-restored.
+            if (ctx.javaClassInfo.returnLabel != null) {
+                ctx.mv.visitVarInsn(Opcodes.ASTORE, ctx.javaClassInfo.returnValueSlot);
+                ctx.mv.visitJumpInsn(Opcodes.GOTO, ctx.javaClassInfo.returnLabel);
+            } else {
+                ctx.mv.visitInsn(Opcodes.ARETURN);
+            }
             return;
         }