Merge pull request #327 from fglock/feature/module-build-support

Module::Build support

Author: fglock

dev/design/cpan_client.md

@@ -435,20 +435,24 @@ When a built-in function like `shift`, `pop`, `caller`, etc. is followed by `->`
    - Needs stub similar to ExtUtils::MakeMaker
    - Blocks: modules that only provide Build.PL
 
-2. **Core module detection** - Medium priority
-   - CPAN.pm doesn't recognize built-in modules (strict, warnings, Exporter, etc.)
-   - Option A: Add version stubs to built-in modules
-   - Option B: Configure CPAN.pm to skip core modules
-   - Option C: Add core module versions to a metadata file
+2. ~~**Core module detection**~~ - ✅ Resolved
+   - CPAN::DistnameInfo now installable via jcpan
+   - Warning about it no longer appears
 
 3. **Test running improvements** - Low priority
    - `make test` uses fork which isn't supported in PerlOnJava
    - Current workaround: `notest("install", "Module")`
    - Long-term: Consider IPC::Open3 for test harness
 
-4. **YAML.pm improvements** - Low priority
-   - Warning: "YAML version '0.01' is too low"
-   - Current stub is minimal; better YAML parsing would help with META.yml
+4. ~~**YAML.pm improvements**~~ - ✅ FIXED
+   - Updated YAML.pm version to 1.31 (matches CPAN version)
+   - "YAML version '0.01' is too low" warning no longer appears
+   - Our YAML.pm wraps YAML::PP which provides full functionality
+
+- [x] **Phase 9a: YAML version update** (2026-03-17)
+  - Updated YAML.pm $VERSION from 0.01 to 1.31
+  - Silences "YAML version too low" warning in CPAN.pm
+  - CPAN.pm requires >= 0.60; our YAML::PP-based implementation is fully capable
 
 ### Open Questions
 - How important is Safe compartmentalization for users?

dev/design/fork_open_emulation.md

@@ -0,0 +1,215 @@
+# Fork-Open Emulation for PerlOnJava
+
+## Problem Statement
+
+Perl's `open FH, "-|"` (2-arg piped open) uses fork() to create a child process:
+
+```perl
+my $pid = open *FH, "-|";
+if ($pid) {
+    # Parent: read from FH (child's stdout)
+    my $output = <FH>;
+    close FH;
+} else {
+    # Child: exec the command
+    exec @cmd;
+}
+```
+
+The JVM cannot support `fork()` - there's no way to split the JVM into two identical processes.
+However, the 3-arg form `open FH, "-|", @cmd` works fine because it just spawns a process and
+pipes its output - no fork needed.
+
+## Solution: Runtime Fork-Open Emulation
+
+Instead of complex AST transformations, we detect the fork-open pattern at runtime and
+emulate it by deferring the pipe creation until `exec` is called.
+
+### How It Works
+
+1. **When `open FH, "-|"` is called without a command:**
+   - Don't fail immediately
+   - Store a "pending fork-open" state with the filehandle reference
+   - Return 0 (child PID) to make the code take the "child" branch
+
+2. **When `exec @cmd` is called:**
+   - Check for pending fork-open state
+   - If pending: create the pipe using 3-arg semantics with @cmd
+   - Return to the "parent" code path (after the if/else) with pipe ready
+   - The "parent" branch code will then read from FH normally
+
+3. **Reset the pending state on:**
+   - Any successful `open` call (new filehandle operation)
+   - Any `close` call
+   - End of the current statement/block (safety)
+
+### State Machine
+
+```
+                    open FH, "-|"
+    [NORMAL] ─────────────────────────> [PENDING_FORK_OPEN]
+       │                                        │
+       │ open/close                             │ exec @cmd
+       │ (reset)                                │
+       ▼                                        ▼
+    [NORMAL] <────────────────────────── [PIPE_READY]
+                   continue execution      (return to parent path)
+```
+
+### Execution Flow Example
+
+```perl
+# Original code:
+my $pid = open *FH, "-|";      # Returns 0, sets PENDING state
+if ($pid) {                     # False, skip parent branch
+    return <FH>;
+} else {
+    exec "ls", "-la";           # Detects PENDING, creates pipe, 
+                                # throws special "return to parent" signal
+}
+# After exec's special return, $pid is now truthy, FH is ready
+# Code continues after the if/else with the pipe working
+```
+
+### Implementation Details
+
+#### 1. Pending State Storage (thread-local)
+
+```java
+// In a new class or IOOperator
+public class ForkOpenState {
+    private static final ThreadLocal<PendingForkOpen> pendingState = new ThreadLocal<>();
+    
+    public static class PendingForkOpen {
+        public RuntimeScalar fileHandle;
+        public int tokenIndex;  // For error messages
+    }
+    
+    public static void setPending(RuntimeScalar fh, int tokenIndex) { ... }
+    public static PendingForkOpen getPending() { ... }
+    public static void clear() { ... }
+    public static boolean hasPending() { ... }
+}
+```
+
+#### 2. Modified `open` (IOOperator.java)
+
+```java
+// In openPipe or open method:
+if (mode.equals("-|") && commandList.isEmpty()) {
+    // Fork-open mode without command
+    ForkOpenState.setPending(fileHandle, tokenIndex);
+    return new RuntimeScalar(0);  // Return 0 = "child" branch
+}
+```
+
+#### 3. Modified `exec` (SystemOperator.java)
+
+```java
+public static RuntimeScalar exec(RuntimeList args, ...) {
+    if (ForkOpenState.hasPending()) {
+        PendingForkOpen pending = ForkOpenState.getPending();
+        ForkOpenState.clear();
+        
+        // Create the pipe using 3-arg semantics
+        RuntimeList openArgs = new RuntimeList();
+        openArgs.add(pending.fileHandle);
+        openArgs.add(new RuntimeScalar("-|"));
+        openArgs.addAll(args);  // The command from exec
+        
+        RuntimeIO fh = RuntimeIO.openPipe(openArgs);
+        // ... set up the filehandle ...
+        
+        // Throw special exception to return to "parent" path
+        throw new ForkOpenCompleteException(processId);
+    }
+    
+    // Normal exec behavior
+    ...
+}
+```
+
+#### 4. Exception Handling
+
+```java
+public class ForkOpenCompleteException extends RuntimeException {
+    public final int pid;
+    public ForkOpenCompleteException(int pid) { this.pid = pid; }
+}
+```
+
+The calling code needs to catch this and return the PID to make the "parent" branch execute.
+
+#### 5. Reset Points
+
+Add `ForkOpenState.clear()` calls to:
+- `IOOperator.open()` - at the start, before any operation
+- `IOOperator.close()` - when closing any filehandle
+- Potentially in error handlers
+
+### Supported Patterns
+
+This approach handles various fork-open idioms:
+
+```perl
+# Classic if/else pattern
+my $pid = open FH, "-|";
+if ($pid) { ... } else { exec @cmd }
+
+# unless pattern  
+my $pid = open FH, "-|";
+unless ($pid) { exec @cmd }
+...parent code...
+
+# or-exec pattern (common idiom)
+open FH, "-|" or exec @cmd;
+
+# Defined-or pattern
+my $pid = open FH, "-|";
+exec @cmd unless defined $pid;
+```
+
+### Limitations
+
+1. **Code between open and exec**: If there's significant code between `open` and `exec`
+   in the "child" branch, it will execute. This matches Perl behavior where the child
+   does run that code before exec.
+
+2. **Multiple fork-opens**: Only one pending fork-open at a time per thread. Nested
+   fork-opens would need stack-based state (future enhancement if needed).
+
+3. **Non-exec child code**: If the child branch does something other than exec (like
+   `exit` or complex processing), it won't work. This is a limitation of not having
+   real fork.
+
+### Testing
+
+```perl
+# Test 1: Basic fork-open pattern
+my $pid = open my $fh, "-|";
+if ($pid) {
+    my $line = <$fh>;
+    print "Got: $line";
+    close $fh;
+} else {
+    exec "echo", "hello";
+}
+
+# Test 2: or-exec pattern
+open my $fh, "-|" or exec "echo", "hello";
+my $line = <$fh>;
+print "Got: $line";
+close $fh;
+```
+
+### Related Files
+
+- `src/main/java/org/perlonjava/runtime/operators/IOOperator.java` - open implementation
+- `src/main/java/org/perlonjava/runtime/operators/SystemOperator.java` - exec implementation
+- `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeIO.java` - pipe handling
+
+### References
+
+- Perl open documentation: https://perldoc.perl.org/functions/open
+- Module::Build `_backticks` method uses this pattern
+- IPC::Open2/Open3 also use fork-open patterns

dev/design/moo_support.md

@@ -325,35 +325,19 @@ All tests meet or exceed the baseline (20260312T075000):
 
 ## Known Issues (Remaining Moo Test Failures)
 
+All remaining test failures are expected and require Java features that are not available:
+
 ### Issue: DEMOLISH Not Being Called (Expected - Not Supported)
-**Tests affected**: t/demolish-basics.t (3 failures)
+**Tests affected**: demolish-*.t (6 failures)
 **Symptom**: Object destructors (DEMOLISH methods) are not called when objects go out of scope
 **Root cause**: DESTROY/fork/threads are not supported in PerlOnJava (they compile but throw at runtime)
 **Status**: Expected failure - these features are out of scope for PerlOnJava
 
-### Issue: SUPER::new Not Working in Extended Classes - FIXED (Phase 13)
-**Tests affected**: t/extends-non-moo.t
-**Symptom**: `Undefined subroutine &Package::SUPER::new called`
-**Root cause**: Only `SUPER::method` was supported, not `Package::SUPER::method`
-**Status**: ✅ FIXED - RuntimeCode.java now handles `::SUPER::` pattern
-
-### Issue: Regex Escaping in Error Messages (quotemeta) - FIXED (Phase 12)
-**Tests affected**: t/accessor-coerce.t, t/accessor-isa.t (many failures)
-**Symptom**: `plus\_three` vs `plus_three`, `less\_than\_three` vs `less_than_three`
-**Root cause**: quotemeta was escaping `_` (underscore) which Perl doesn't escape
-**Status**: ✅ FIXED - StringOperators.java now treats `_` as alphanumeric
-
-### Issue: Role Application Error Messages
-**Tests affected**: t/compose-roles.t (4 failures)  
-**Symptom**: Missing error messages when required attributes are not provided
-**Root cause**: Error throwing in role composition may not propagate correctly
-**Status**: Needs investigation
-
-### Issue: Spurious "Odd number of elements in anonymous hash" Warnings
-**Tests affected**: Various tests when run via TAP::Harness
-**Symptom**: Warnings appear in TAP::Harness but not when running tests directly
-**Root cause**: Unknown - standard Perl does NOT emit these warnings
-**Status**: Needs investigation - add stack trace to RuntimeHash.java to identify source
+### Issue: Weak References Not Supported (Expected - Java GC Limitation)
+**Tests affected**: accessor-weaken*.t (20 failures), no-moo.t (5 failures)
+**Symptom**: Weak references don't work as expected in Java's garbage collector
+**Root cause**: Java's GC is fundamentally different from Perl's reference counting
+**Status**: Expected failure - would require extensive changes to RuntimeScalar
 
 ## Remaining jcpan Improvements
 
@@ -686,33 +670,25 @@ Moo tests run via `jcpan -t Moo`. Recent fixes (Phases 12-13) should improve pas
     - `parseStackTraceElement()` returns the `#line`-adjusted filename for caller() reporting
   - **Result**: Tests 15, 18 now PASS; tests 19-26 now run (were previously skipped due to parse errors)
 
+- [x] Phase 38: croak-locations.t tests 27-28 now passing (2026-03-17)
+  - Tests 27-28 were listed as failing but now pass without additional changes
+  - The fixes from Phase 29 (correct line numbers) and Phase 37 (#line directive) resolved these
+  - Test 27: Delegated method croak now correctly reports call site
+  - Test 28: Role default isa now correctly reports application location
+  - **Result**: croak-locations.t 29/29 tests passing (100%)
+
 ### Current Status
 
-**Test Results (after Phase 37):**
-- **Moo**: 64/71 test programs passing (90%), 806/839 subtests passing (96%)
+**Test Results (after Phase 38 - croak-locations.t fully passing):**
+- **Moo**: 65/71 test programs passing (91.5%), 808/839 subtests passing (96.3%)
 - **Mo**: 28/28 test programs passing (100%), 144/144 subtests passing (100%)
 
-**Remaining Failures (categorized):**
-1. **accessor-weaken tests** (20 failures) - Expected, weak references not supported in Java GC
-2. **croak-locations.t** (2 failures) - Tests 27, 28: delegated method croak and role default isa
-3. **demolish tests** (6 failures) - Expected, DESTROY not supported
-4. **no-moo.t** (5 failures) - Namespace cleanup requires weak references
-
-**croak-locations.t test 27 analysis**:
-- Test: `Method::Generate::Accessor::_generate_delegation - user croak`
-- Expected: `LocationTestFile line 22` (where delegated method is called)
-- Got: `(eval N) line 50` (internal constructor code)
-- Issue: Carp is blaming the generated constructor instead of the user's call site
-- This is a deeper Carp stack-walking issue with Sub::Quote-generated code
-
-**croak-locations.t test 28 analysis**:
-- Test: `Moo::Role::create_class_with_roles - default fails isa`
-- Expected: `LocationTestFile line 21` (where `apply_roles_to_object` is called)
-- Got: `LocationTestFile line 18` (where the object was created)
-- Issue: Carp is blaming object creation instead of role application
-- Related to how default values and isa checks interact with stack walking
-
-**Expected failures** (not fixable without fundamental changes):
+**Remaining Failures (all expected - require Java features not available):**
+1. **accessor-weaken*.t** (20 failures) - Weak references not supported in Java GC
+2. **demolish-*.t** (6 failures) - DESTROY not supported
+3. **no-moo.t** (5 failures) - Namespace cleanup requires weak references
+
+**All remaining failures require fundamental Java GC limitations:**
 - Weak references: accessor-weaken tests (20), no-moo.t cleanup (5)
 - DESTROY/GC: demolish tests (6)
 
@@ -799,36 +775,32 @@ that didn't communicate with the compiler's strict checking.
 #### Phase 36: croak-locations.t Tests 15, 18 (Completed)
 **Status**: Completed 2026-03-17 (merged into Phase 37 above)
 
-Tests 15 and 18 are now fixed. The remaining tests 27-28 involve:
-- Test 27: Delegated method croak - Carp blames generated code instead of call site
-- Test 28: Role default isa - Carp blames object creation instead of role application
-
-These require deeper investigation into how Carp walks the stack for Sub::Quote-generated code.
+Tests 15 and 18 are now fixed. Tests 27-28 were also fixed by Phase 29 and 37 (see Phase 38).
 
 ---
 
-**Revised Priority Order** (considering deferred implementations):
+**Revised Priority Order** (all high-impact items completed):
 
 | Priority | Phase | Impact | Status | Effort |
 |----------|-------|--------|--------|--------|
 | 1 | ~~B::Deparse (33)~~ | ~~1 test~~ | **Completed** | ~~Medium~~ |
 | 2 | ~~Mo strict.t (35)~~ | ~~1 test~~ | **Completed** | ~~Low~~ |
 | 3 | ~~Interpreter caller() (34)~~ | ~~Parity~~ | **Completed** | ~~Medium~~ |
 | 4 | ~~croak-locations.t 15,18 (36/37)~~ | ~~2 tests~~ | **Completed** | ~~Medium~~ |
-| 5 | **croak-locations.t 27,28** | 2 tests | Complex | High |
+| 5 | ~~croak-locations.t 27,28~~ | ~~2 tests~~ | **Completed** | ~~High~~ |
 | 6 | DESTROY (31) | 6 tests | **Deferred** | High |
 | 7 | Weak References (32) | 25 tests | **Deferred** | High |
 
-**Actionable items** (can be investigated):
-1. **croak-locations.t 27-28**: Complex Carp stack walking for Sub::Quote-generated code
-
-**Deferred** (need design maturation):
-- Phase 31 (DESTROY): Requires scope-based tracking, complex GC interaction
+**All actionable items completed!** Remaining failures (31 subtests) require:
+- Phase 31 (DESTROY): Scope-based tracking, complex GC interaction
 - Phase 32 (Weak refs): Memory impact concern, need alternative to adding field
 
-**Achievable test improvement without deferred features**:
-- Current: 64/71 Moo tests (90%), 806/839 subtests (96%), 28/28 Mo tests (100%)
-- The bulk of remaining failures (31 tests) require DESTROY or weak refs
+**Final achievable state reached**:
+- Moo: 65/71 test programs (91.5%), 808/839 subtests (96.3%)
+- Mo: 28/28 test programs (100%), 144/144 subtests (100%)
+
+The 31 remaining failing subtests all require DESTROY or weak reference support,
+which are fundamentally limited by Java's GC model.
 
 ### PR Information
 - **Branch**: `feature/moo-support` (PR #319 - merged)

dev/design/todo.md

@@ -1,5 +1,8 @@
 # TODO
 
+## Warnings to Implement
+- `"my" variable $x masks earlier declaration in same scope` warning
+
 ## More Difficult, and Low Impact
 - `goto()` to jump to a label in the call stack
 - Thread