Fix utf8::valid() for CPAN::Meta parsing

dev/design/JCPAN_DATETIME_FIXES.md

@@ -0,0 +1,311 @@
+# jcpan DateTime Fix Plan
+
+## Overview
+
+This document tracks all errors and warnings that occur when running `jcpan install DateTime` with a clean cache, and the plan to fix them.
+
+## Error Categories
+
+### 1. CRITICAL: DateTime::Locale Installation Fails
+
+**Root Cause**: `File::stat.pm` is missing
+
+```
+Can't locate File/stat.pm in @INC at /Users/fglock/.perlonjava/lib/File/ShareDir/Install.pm line 11
+```
+
+**Impact**: DateTime::Locale cannot be configured, which means DateTime tests fail with:
+```
+Can't locate DateTime/Locale.pm in @INC
+```
+
+**Solution**: Implement `File::stat.pm` - a stub or full implementation
+
+**Priority**: HIGH
+
+---
+
+### 2. IPC::Open3 Read-Only Modification Error
+
+**Error**:
+```
+open3: Modification of a read-only value attempted
+         at IPCOpen3.java line 162
+```
+
+**Impact**: Many module tests fail in `t/00-compile.t` type tests that use open3 to test module compilation
+
+**Solution**: Fix IPCOpen3.java line 162 to handle read-only values
+
+**Priority**: MEDIUM
+
+---
+
+### 3. Missing Core Modules
+
+| Module | Used By | Priority |
+|--------|---------|----------|
+| File::stat | File::ShareDir::Install | HIGH |
+| IO::Select | Various | MEDIUM |
+| PerlIO::encoding | Encode tests | LOW |
+| encoding.pm | Encode | LOW |
+
+---
+
+### 4. Encode Module Issues
+
+**Errors**:
+```
+Can't locate object method "decode" via package "ISO-8859-1"
+Can't locate object method "encode" via package "UTF-8"
+Can't locate object method "encodings" via package "Encode"
+Undefined subroutine &Encode::define_encoding called
+```
+
+**Root Cause**: Encode is an XS module; PerlOnJava has a Java implementation but some methods are missing
+
+**Solution**: Add missing methods to Encode.java:
+- `encodings()`
+- Ensure `decode`/`encode` work with encoding names as package names
+
+**Priority**: MEDIUM
+
+---
+
+### 5. Version Format Errors
+
+**Error**:
+```
+Invalid version format (version required) at Version.java line 334
+```
+
+**Solution**: Handle edge cases in Version.java
+
+**Priority**: LOW
+
+---
+
+### 6. CPAN::Meta::Requirements Warning
+
+**Error**:
+```
+Use of uninitialized value in numeric gt (>) at jar:PERL5LIB/CPAN/Meta/Requirements.pm line 215
+```
+
+**Solution**: Check for undef before numeric comparison
+
+**Priority**: LOW
+
+---
+
+### 7. Test::Builder Overload Issue
+
+**Error**:
+```
+Undefined subroutine &*version::("" called at jar:PERL5LIB/Test/Builder.pm line 771
+```
+
+**Solution**: Fix overload handling for version objects
+
+**Priority**: LOW
+
+---
+
+### 8. Exporter require_version Missing [FIXED]
+
+**Error**:
+```
+Can't locate object method "require_version" via package "Testing"
+```
+
+**Solution**: Added `require_version` method to Java `Exporter.java` which delegates to `UNIVERSAL::VERSION`
+
+**Files changed**:
+- `src/main/java/org/perlonjava/runtime/perlmodule/Exporter.java`
+
+**Priority**: MEDIUM - **FIXED** (commit 70ce06938)
+
+---
+
+### 9. Too Many Arguments for like()
+
+**Error**:
+```
+Too many arguments for main::like at t/conflicts.t line 109
+```
+
+**Root Cause**: Test::More's `like()` has different prototype handling
+
+**Priority**: LOW (cosmetic test failure)
+
+---
+
+### 10. Carp.pm Bareword Error with CORE::GLOBAL::require [FIXED]
+
+**Error**:
+```
+Bareword "Exporter" not allowed while "strict subs" in use at jar:PERL5LIB/Carp.pm line 224
+```
+
+**Root Cause**: When `CORE::GLOBAL::require` is overridden and a module uses `require Bareword;` under strict subs, the bareword was parsed as an expression instead of using require's special bareword-to-filename conversion.
+
+**Solution**: Added special handling in `ParsePrimary.java` for `require` when `CORE::GLOBAL::require` is overridden:
+1. Parse the argument using standard require handling (converts bareword to filename string)
+2. Build a subroutine call node with `&CORE::GLOBAL::require(...)` form
+
+**Files changed**:
+- `src/main/java/org/perlonjava/frontend/parser/ParsePrimary.java`
+
+**Priority**: MEDIUM - **FIXED** (commit 70ce06938)
+
+---
+
+### 11. JVM VerifyError: Inconsistent Stackmap Frames [FIXED]
+
+**Error**:
+```
+java.lang.VerifyError: Inconsistent stackmap frames at branch target 518
+Exception Details:
+  Location:
+    org/perlonjava/anon195.apply(...) @507: goto
+  Reason:
+    Current frame's stack size doesn't match stackmap.
+```
+
+**Triggered by**: perl5's `File/stat.pm` (imported via sync.pl)
+
+**Minimal Reproducer**:
+```perl
+no strict "refs";
+my $result = defined eval { &{"Fcntl::S_IFX"} };
+```
+
+The issue requires BOTH of these:
+1. `no strict "refs"` in scope (enables symbolic subroutine calls)
+2. `defined eval { &{"symbolic_ref"} }` (eval-wrapped symbolic sub call with defined check)
+
+**Works** (any element missing):
+- `eval { &{"..."} }` without `defined` - OK
+- `defined eval { die "x" }` - OK (no symbolic ref call)
+- `defined eval { Func() }` - OK (direct call, not symbolic)
+- With `use strict "refs"` - OK (throws error at compile time)
+
+**Root Cause Analysis**: The bytecode generator in `EmitVariable.java` creates inconsistent stack states when handling control flow markers from subroutine calls inside eval blocks.
+
+When `&{"symbolic_ref"}` is called:
+1. `RuntimeCode.apply()` may return a `RuntimeControlFlowList` marker (LAST/NEXT/REDO)
+2. The code checks `isNonLocalGoto()` and branches to handle the marker
+3. For unlabeled LAST/NEXT, the original code jumped directly to loop labels with an **empty stack**
+4. But the `applyNoControlFlow` merge point expects a **RuntimeList on the stack** (from DUP or cfSlot load)
+5. JVM frame computation fails because paths arriving at merge point have different stack states
+
+**Specific bytecode issue** (from disassembly before fix):
+```
+L9   checkUnlabeled
+     ILOAD 29; ICONST_0; IF_ICMPEQ L11  // LAST?
+     ILOAD 29; ICONST_1; IF_ICMPEQ L12  // NEXT?
+     ILOAD 29; ICONST_2; IF_ICMPEQ L13  // REDO?
+     GOTO L8
+L11  GOTO L7     // LAST: empty stack, jumps to L7
+L12  GOTO L7     // NEXT: empty stack, jumps to L7  
+L13  GOTO L6     // REDO: OK, goes to redo label
+L8   ALOAD 27    // Load cfSlot, falls through to L7
+L7   FRAME [...] [RuntimeList]  // Expects value on stack!
+     ASTORE 25
+```
+
+**Fix** (commit 280d03af2): Push the control flow marker (from cfSlot) onto the stack before jumping to the merge point:
+```java
+mv.visitLabel(isLast);
+mv.visitVarInsn(Opcodes.ALOAD, cfSlot);  // Push marker
+mv.visitJumpInsn(Opcodes.GOTO, applyNoControlFlow);
+
+mv.visitLabel(isNext);
+mv.visitVarInsn(Opcodes.ALOAD, cfSlot);  // Push marker
+mv.visitJumpInsn(Opcodes.GOTO, applyNoControlFlow);
+```
+
+**Files changed**:
+- `src/main/java/org/perlonjava/backend/jvm/EmitVariable.java` lines 710-721
+
+**Impact**: File::stat.pm and other modules using `eval { &{...} }` pattern now load correctly.
+
+**Priority**: HIGH (blocks File::stat and potentially other complex modules) - **FIXED**
+
+---
+
+## Implementation Plan
+
+### Phase 1: Critical (enables DateTime to install)
+
+1. **Fix JVM VerifyError for complex control flow** (Issue #11)
+   - Debug why File::stat.pm causes stackmap frame inconsistency
+   - Create minimal reproduction case
+   - Fix bytecode emitter
+   - Files: `EmitterMethodCreator.java`, `EmitterVisitor.java`, `EmitControlFlow.java`
+
+2. **Implement File::stat.pm stub** (if JVM fix takes too long)
+   - Create `src/main/perl/lib/File/stat.pm`
+   - Implement basic stat() wrapper returning object with standard fields
+   - File: `src/main/perl/lib/File/stat.pm`
+
+### Phase 2: High Priority (reduces test failures)
+
+2. **Fix IPC::Open3 read-only error**
+   - File: `src/main/java/org/perlonjava/runtime/perlmodule/IPCOpen3.java`
+   - Line 162: clone value before modification
+
+3. **Add Encode::encodings() method**
+   - File: `src/main/java/org/perlonjava/runtime/perlmodule/Encode.java`
+
+4. **Implement require_version in UNIVERSAL**
+   - File: `src/main/java/org/perlonjava/runtime/perlmodule/Universal.java`
+
+### Phase 3: Medium Priority
+
+5. **Fix Version.java edge cases**
+6. **Fix CPAN::Meta::Requirements undef check**
+7. **Implement IO::Select stub**
+
+### Phase 4: Low Priority (polish)
+
+8. **Fix Test::Builder overload handling**
+9. **Fix Carp.pm bareword issue**
+10. **Fix like() prototype handling**
+
+---
+
+## Progress Tracking
+
+### Completed
+- [x] Phase 16: utf8::valid() fix for CPAN::Meta parsing (2026-03-20)
+- [x] ExtUtils::MakeMaker MYMETA.yml meta-spec v2 format (2026-03-20)
+- [x] Added File::stat.pm via import system (2026-03-20) - but triggers JVM bug
+- [x] JVM VerifyError fix for eval block control flow (2026-03-20, commit 280d03af2)
+- [x] Add missing S_* mode constants to Fcntl.pm (2026-03-20, commit 94974ba79)
+  - File::stat.pm now loads successfully
+- [x] Exporter::require_version() implementation (2026-03-20, commit 70ce06938)
+- [x] CORE::GLOBAL::require bareword handling fix (2026-03-20, commit 70ce06938)
+- [x] Exporter version check in import (2026-03-21, commit a2e0aa131)
+  - First argument starting with digit is now treated as version check
+  - Fixes: "Symbol 0.03 not allowed for export in package File::ShareDir::Install"
+- [x] MakeMaker $(INST_LIB) variable expansion (2026-03-21, commit f42f9125c)
+  - Fixes modules with explicit PM hash using Make-style variables
+
+### In Progress
+- None
+
+### All fixes complete!
+- `jcpan install DateTime` works successfully
+- DateTime module loads and functions correctly:
+  ```
+  ./jperl -e 'use DateTime; print DateTime->now->strftime("%Y-%m-%d")'
+  2026-03-21
+  ```
+
+---
+
+## Related Documents
+
+- `dev/design/cpan_client.md` - Main CPAN client documentation
+- `dev/design/xsloader.md` - XSLoader implementation

dev/design/cpan_client.md

@@ -355,6 +355,58 @@ All major DateTime issues have been fixed. The 7 remaining test failures are:
 
 ---
 
+## Phase 16: utf8::valid() Fix for CPAN::Meta Parsing (2026-03-20)
+
+### Problem Statement
+
+When installing DateTime with empty caches, CPAN::Meta::YAML parsing would fail with:
+```
+Read an invalid UTF-8 string (maybe mixed UTF-8 and 8-bit character set).
+Did you decode with lax ":utf8" instead of strict ":encoding(UTF-8)"?
+```
+
+This error prevented proper parsing of META.yml/MYMETA.yml files, which meant test dependencies like Test::Without::Module and CPAN::Meta::Check were not being properly detected.
+
+### Root Cause
+
+CPAN::Meta::YAML validates strings before parsing:
+```perl
+if ( utf8::is_utf8($string) && ! utf8::valid($string) ) {
+    die "Read an invalid UTF-8 string...";
+}
+```
+
+The `utf8::valid()` function in PerlOnJava was using `CharsetDetector` which was fundamentally wrong:
+- It converted the string to bytes using the default charset
+- Then tried to detect if those bytes were UTF-8
+- This always failed for properly decoded Unicode strings
+
+### Solution
+
+Rewrote `utf8::valid()` in `Utf8.java` to correctly check string validity:
+- **For character strings (UTF-8 flag on)**: Validates that surrogate pairs are properly formed
+- **For byte strings (UTF-8 flag off)**: Attempts to decode bytes as UTF-8
+
+### Files Changed
+
+- `src/main/java/org/perlonjava/runtime/perlmodule/Utf8.java` - Fixed `valid()` method
+
+### Test Results
+
+The fix allows CPAN::Meta::YAML to properly parse MYMETA.yml files, enabling CPAN.pm to detect and install test dependencies.
+
+---
+
+## Known Remaining CPAN Issues
+
+| Issue | Status | Impact |
+|-------|--------|--------|
+| File::stat.pm missing | Not implemented | DateTime::Locale installation fails |
+| IPC::Open3 read-only error | Bug in IPCOpen3.java | Some module tests fail |
+| Test::Harness UTF-8 error | Pre-existing | Some test output parsing fails |
+
+---
+
 ## Related Documents
 
 - `dev/design/xsloader.md` - XSLoader/Java integration

dev/import-perl5/config.yaml

@@ -172,6 +172,10 @@ imports:
   - source: perl5/lib/File/Compare.pm
     target: src/main/perl/lib/File/Compare.pm
 
+  # File::stat - by-name interface to stat() (required by IO::Dir)
+  - source: perl5/lib/File/stat.pm
+    target: src/main/perl/lib/File/stat.pm
+
   # From core library
   - source: perl5/lib/Tie/Array.pm
     target: src/main/perl/lib/Tie/Array.pm
@@ -620,6 +624,11 @@ imports:
   - source: perl5/cpan/Term-ANSIColor/lib/Term/ANSIColor.pm
     target: src/main/perl/lib/Term/ANSIColor.pm
 
+  # Class::Struct - Declare struct-like datatypes as Perl classes
+  # Required by File::stat.pm
+  - source: perl5/lib/Class/Struct.pm
+    target: src/main/perl/lib/Class/Struct.pm
+
   # Add more imports below as needed
   # Example with minimal fields:
   # - source: perl5/lib/SomeModule.pm