CPAN Phase 6: Enable CPAN.pm for pure Perl module installation

.cognition/skills/debug-windows-ci/SKILL.md

@@ -0,0 +1,179 @@
+# Debug PerlOnJava Windows CI Failures
+
+## Overview
+
+This skill helps debug test failures that occur specifically in the Windows CI/CD environment but pass locally on macOS/Linux.
+
+## When to Use
+
+- Tests pass locally on macOS/Linux but fail on Windows CI
+- Windows-specific path handling issues
+- Shell command differences between platforms
+- File I/O issues on Windows
+
+## CI/CD Structure
+
+### GitHub Actions Workflow
+
+The CI runs on `windows-latest` using:
+- Java 21 (Temurin)
+- Gradle for build
+- Maven for tests (`make ci` runs `mvn clean test`)
+
+### Viewing CI Logs
+
+```bash
+# List recent CI runs
+gh run list --branch <branch-name> --limit 5
+
+# View failed test logs
+gh run view <run-id> --log-failed
+
+# Filter for specific errors
+gh run view <run-id> --log-failed 2>&1 | grep -E "FAILURE|error|not ok"
+
+# Get test count summary
+gh run view <run-id> --log-failed 2>&1 | grep "Tests run:"
+```
+
+## Common Windows CI Issues
+
+### 1. Cwd/getcwd Issues
+
+**Symptom**: "Cannot chdir back to : 2" or "Undefined subroutine &Cwd::cwd called"
+
+**Root Cause**: The Perl `Cwd.pm` uses shell backticks (`` `cd` ``) on Windows which doesn't work in PerlOnJava.
+
+**Solution**: PerlOnJava provides `Internals::getcwd` which uses Java's `System.getProperty("user.dir")`. The Cwd.pm has been modified to use this when available.
+
+**Key Files**:
+- `src/main/perl/lib/Cwd.pm` - Perl module with platform-specific fallbacks
+- `src/main/java/org/perlonjava/runtime/perlmodule/Internals.java` - Java implementation of getcwd
+
+### 2. Temp File Creation Issues
+
+**Symptom**: "Cannot open/create <filename>: open failed"
+
+**Root Cause**: 
+- Windows uses different path separators (`\` vs `/`)
+- Temp directory permissions may differ
+- File locking behavior differs on Windows
+
+**Debugging**:
+```bash
+# Check temp path in error message
+gh run view <run-id> --log-failed 2>&1 | grep "open failed"
+```
+
+### 3. $^O Detection
+
+PerlOnJava sets `$^O` based on the Java `os.name` property:
+- Windows: `MSWin32`
+- macOS: `darwin`
+- Linux: `linux`
+
+**Key File**: `src/main/java/org/perlonjava/runtime/runtimetypes/SystemUtils.java`
+
+### 4. Shell Command Differences
+
+Windows CI may fail when Perl code uses:
+- Backticks with Unix commands
+- `system()` calls assuming Unix shell
+- Path separators in shell commands
+
+## Debugging Workflow
+
+### Step 1: Identify the Failing Test
+
+```bash
+# Get list of failing tests
+gh run view <run-id> --log-failed 2>&1 | grep "testUnitTests.*FAILURE"
+```
+
+### Step 2: Map Test Number to File
+
+```bash
+# List tests in order (tests are numbered alphabetically)
+ls -1 src/test/resources/unit/*.t | sort | nl | grep "<number>"
+```
+
+### Step 3: Analyze the Error
+
+```bash
+# Get full context around error
+gh run view <run-id> --log-failed 2>&1 | grep -A10 "unit\\<test>.t"
+```
+
+### Step 4: Check if Pre-existing
+
+```bash
+# Compare with master branch CI
+gh run list --branch master --limit 3
+gh run view <master-run-id> --log-failed
+```
+
+## Platform-Specific Code Patterns
+
+### Checking for Windows in Perl
+
+```perl
+if ($^O eq 'MSWin32') {
+    # Windows-specific code
+}
+```
+
+### Checking for Windows in Java
+
+```java
+if (SystemUtils.osIsWindows()) {
+    // Windows-specific code
+}
+```
+
+### Safe Cross-Platform getcwd
+
+```perl
+# In Cwd.pm, use Internals::getcwd if available
+if (eval { Internals::getcwd(); 1 }) {
+    *getcwd = \&Internals::getcwd;
+}
+```
+
+## Test File Locations
+
+- Unit tests: `src/test/resources/unit/*.t`
+- Perl5 test suite: `perl5_t/t/`
+- Java tests: `src/test/java/org/perlonjava/`
+
+## Related Files
+
+- `.github/workflows/gradle.yml` - CI workflow definition
+- `Makefile` - Build targets including `ci`
+- `src/main/java/org/perlonjava/runtime/perlmodule/Cwd.java` - Java Cwd stub
+- `src/main/perl/lib/Cwd.pm` - Perl Cwd implementation
+
+## Troubleshooting Checklist
+
+1. [ ] Is the failure Windows-specific? (Check if macOS/Linux CI passes)
+2. [ ] Is it a new regression or pre-existing? (Compare with master)
+3. [ ] Does it involve file paths or shell commands?
+4. [ ] Does it use Cwd or directory operations?
+5. [ ] Is `$^O` being checked correctly?
+6. [ ] Are there any `defined &Subroutine` checks that might behave differently?
+
+## Adding Debug Output
+
+To debug CI issues, you can temporarily add print statements to Perl modules:
+
+```perl
+# Add to Cwd.pm to debug
+warn "DEBUG: \$^O = $^O";
+warn "DEBUG: Internals::getcwd available: " . (eval { Internals::getcwd(); 1 } ? "yes" : "no");
+```
+
+Then check CI logs:
+```bash
+gh run view <run-id> --log-failed 2>&1 | grep "DEBUG:"
+```
+
+Remember to remove debug output before final commit.

dev/design/cpan_client.md

@@ -224,7 +224,7 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
 
 ## Progress Tracking
 
-### Current Status: Phase 5 complete
+### Current Status: Phase 6 mostly complete - CPAN.pm functional for pure Perl modules
 
 ### Completed
 - [x] Analyze CPAN.pm dependencies (2024-03-13)
@@ -316,11 +316,86 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
 - `src/main/perl/lib/ExtUtils/MakeMaker/Config.pm` - Config wrapper
 - `src/main/java/org/perlonjava/runtime/runtimetypes/GlobalContext.java` - Added ~/.perlonjava/lib to @INC
 
-### Next Steps
-1. Consider a minimal CPAN download helper (pure Perl, no build step)
-2. Expand user documentation with more examples
-3. Add more commonly-needed pure Perl modules
-4. Test with real CPAN modules (pure Perl ones)
+- [x] **Phase 6: CPAN.pm Support** (2024-03-14, in progress)
+  - **Safe.pm stub created**: Minimal implementation for CPAN.pm metadata evaluation
+    - `reval()` uses `eval` with `no strict 'vars'` (CPAN metadata is trusted)
+    - Supports CHECKSUMS file evaluation ($cksum hash)
+  - **CPAN.pm imported**: Added to config.yaml with CPAN/*, CPAN::Meta::*, Parse::CPAN::Meta
+  - **Parser fixes for CPAN.pm compatibility**:
+    - File test operators with function call operands (`-f func()`)
+    - Block argument parsing for undefined functions (`func { } @args`)
+    - File test with qualified package names (`-f CPAN::find_perl`)
+  - **Regex fix**: Character class ranges like `[A-Z-0-9]` now parse correctly
+  - **try/catch feature gating**: `try` and `catch` only keywords when `use feature 'try'` enabled
+    - Allows Try::Tiny to work correctly without feature flag
+  - **CPAN.pm now loads and can install pure Perl modules**:
+    ```bash
+    ./jperl -MCPAN -e 'CPAN::Shell->install("Try::Tiny")'
+    # Downloads, validates checksums, installs to ~/.perlonjava/lib/
+    ```
+
+### Files Changed (Phase 6)
+- `src/main/perl/lib/Safe.pm` - New stub for CPAN.pm metadata evaluation
+- `dev/import-perl5/config.yaml` - Added CPAN.pm, CPAN/*, CPAN::Meta::*, Parse::CPAN::Meta
+- `src/main/java/org/perlonjava/backend/jvm/EmitOperatorFileTest.java` - Fixed file test with function calls
+- `src/main/java/org/perlonjava/frontend/parser/SubroutineParser.java` - Block args for undefined functions
+- `src/main/java/org/perlonjava/frontend/parser/ParsePrimary.java` - File test with qualified names, try/catch gating
+- `src/main/java/org/perlonjava/runtime/regex/ExtendedCharClass.java` - Character class range fix
+- `src/main/java/org/perlonjava/runtime/regex/RegexPreprocessorHelper.java` - Character class range fix
+- `src/main/perl/lib/ExtUtils/MM.pm` - Platform selection and MM alias, inherits from MM_Unix/MM_Win32
+- `src/main/perl/lib/ExtUtils/MM_Unix.pm` - parse_version and maybe_command for Unix
+- `src/main/perl/lib/ExtUtils/MM_Win32.pm` - Windows-specific maybe_command
+- `src/main/perl/lib/ExtUtils/MakeMaker.pm` - Added stub Makefile generation
+- `src/main/perl/lib/CPAN/Distribution.pm` - Fork fallback using system() when d_fork is false
+
+### Phase 6 Continued (2024-03-14)
+- **MM->parse_version() implemented**: Required by CPAN.pm to check installed module versions
+  - Uses regex extraction for common VERSION patterns (avoids package block scoping issues)
+  - Platform-specific modules: MM_Unix.pm (Unix/macOS), MM_Win32.pm (Windows)
+  - MM alias: `package MM; @ISA = qw(ExtUtils::MM);` for CPAN.pm compatibility
+- **Stub Makefile generation**: MakeMaker now creates minimal Makefile that CPAN.pm recognizes
+  - CPAN.pm reports "Makefile.PL -- OK" and "make -- OK"
+  - make targets: all, test, install, clean (no-ops since files installed directly)
+- **maybe_command() implemented**: Checks if file is executable (Unix: -x, Windows: .exe/.com/.bat/.cmd)
+
+### Known Issues (Phase 6)
+1. **fork() fallback implemented**: CPAN::Distribution patched to use system() when $Config{d_fork} is false
+   - Tests run without fork, losing timeout and signal handling
+   - Works for normal test scenarios
+2. **Dependency resolution**: CPAN.pm tries to install core modules (Exporter, strict, warnings)
+   - These are built-in but CPAN.pm doesn't detect them
+   - May need to stub module versions or configure CPAN.pm to skip core
+3. ~~**YAML size limit**: Large YAML metadata exceeds SnakeYAML's 3MB limit~~ **FIXED**
+   - Increased YAML::PP code point limit to 50MB
+4. **parse_version warnings**: "Error while parsing version" appears but doesn't affect functionality
+   - May be related to alarm/eval interaction in CPAN::Module
+
+### Next Steps (Phase 6 Remaining)
+1. **Core module detection** (Medium priority)
+   - CPAN.pm doesn't recognize built-in modules (strict, warnings, Exporter, etc.)
+   - Need to either provide version stubs or configure CPAN.pm to skip core modules
+   - Option: Add core module versions to a metadata file
+
+2. **Test running** (Medium priority)
+   - `make test` uses fork which isn't supported in PerlOnJava
+   - Current workaround: `notest("install", "Module")`
+   - Long-term: Consider IPC::Open3 for test harness
+
+3. **YAML.pm improvements** (Low priority)
+   - Warning: "YAML version '0.01' is too low"
+   - Current stub is minimal; better YAML parsing would help with META.yml
+
+4. **CPAN::Meta::Requirements warnings** (Low priority)
+   - `"our" variable @ISA redeclared` warnings
+   - Cosmetic issue in imported CPAN module
+
+5. **Module::Build support** (Phase 6b)
+   - Some CPAN modules use Module::Build instead of MakeMaker
+   - Needs stub similar to ExtUtils::MakeMaker
+
+6. **jcpan wrapper script** (Phase 6e)
+   - User-friendly `jcpan install Module` command
+   - Sets up paths and invokes CPAN.pm with notest option
 
 ### Open Questions
 - Should we create a PerlOnJava-specific minimal CPAN download tool?
@@ -331,3 +406,7 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
 - ✅ cpanm feasibility: cpanm requires ExtUtils::MakeMaker which needs `make` - not suitable for PerlOnJava
 - ✅ Archive::Zip: Implemented using java.util.zip
 - ✅ ExtUtils::MakeMaker: Reimplemented for PerlOnJava to skip `make` and install pure Perl modules directly
+- ✅ Safe.pm: Stub implementation using `eval` with `no strict 'vars'` - sufficient for trusted CPAN metadata
+- ✅ Try::Tiny compatibility: `try`/`catch` now feature-gated, module works correctly
+- ✅ parse_version: Implemented using regex extraction to avoid package block scoping issues in compiled modules
+- ✅ Makefile creation: Stub Makefile satisfies CPAN.pm's checks

dev/import-perl5/config.yaml

@@ -451,9 +451,51 @@ imports:
   - source: perl5/lib/Symbol.pm
     target: src/main/perl/lib/Symbol.pm
 
+  # Note: Cwd.pm is NOT imported - we use a customized version that integrates
+  # with Internals::getcwd and Internals::abs_path for cross-platform support.
+  # The upstream version would try to load XSLoader which doesn't exist.
+
   # Note: IPC::Open2 and IPC::Open3 are NOT imported - we use custom
   # implementations with Java ProcessBuilder (see IPCOpen3.java)
 
+  # Phase 6: CPAN.pm and dependencies
+  # CPAN.pm - Main CPAN client
+  - source: perl5/cpan/CPAN/lib/CPAN.pm
+    target: src/main/perl/lib/CPAN.pm
+
+  # CPAN/Distribution.pm - protected because we added fork fallback for PerlOnJava
+  - source: perl5/cpan/CPAN/lib/CPAN/Distribution.pm
+    target: src/main/perl/lib/CPAN/Distribution.pm
+    protected: true
+
+  - source: perl5/cpan/CPAN/lib/CPAN
+    target: src/main/perl/lib/CPAN
+    type: directory
+
+  # CPAN::Meta - Metadata handling for CPAN distributions
+  - source: perl5/cpan/CPAN-Meta/lib/CPAN/Meta.pm
+    target: src/main/perl/lib/CPAN/Meta.pm
+
+  - source: perl5/cpan/CPAN-Meta/lib/CPAN/Meta
+    target: src/main/perl/lib/CPAN/Meta
+    type: directory
+
+  # Parse::CPAN::Meta - Parse META.yml and META.json
+  - source: perl5/cpan/CPAN-Meta/lib/Parse/CPAN/Meta.pm
+    target: src/main/perl/lib/Parse/CPAN/Meta.pm
+
+  # CPAN::Meta::YAML - Read and write CPAN metadata YAML files
+  - source: perl5/cpan/CPAN-Meta-YAML/lib/CPAN/Meta/YAML.pm
+    target: src/main/perl/lib/CPAN/Meta/YAML.pm
+
+  # CPAN::Meta::Requirements - Version requirements handling
+  - source: perl5/cpan/CPAN-Meta-Requirements/lib/CPAN/Meta/Requirements.pm
+    target: src/main/perl/lib/CPAN/Meta/Requirements.pm
+
+  - source: perl5/cpan/CPAN-Meta-Requirements/lib/CPAN/Meta/Requirements
+    target: src/main/perl/lib/CPAN/Meta/Requirements
+    type: directory
+
   # Add more imports below as needed
   # Example with minimal fields:
   # - source: perl5/lib/SomeModule.pm

src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java

@@ -3413,6 +3413,24 @@ void compileVariableDeclaration(OperatorNode node, String op) {
                 lastResultReg = rd;
                 return;
             }
+            // local *{expr} - localize a typeglob with dynamic name
+            if (node.operand instanceof OperatorNode sigilOp3
+                    && sigilOp3.operator.equals("*")
+                    && sigilOp3.operand instanceof BlockNode blockNode) {
+                // Compile the expression inside the block to get the name
+                if (blockNode.elements.size() == 1) {
+                    compileNode(blockNode.elements.getFirst(), -1, RuntimeContextType.SCALAR);
+                } else {
+                    compileNode(blockNode, -1, RuntimeContextType.SCALAR);
+                }
+                int nameReg = lastResultReg;
+                int rd = allocateOutputRegister();
+                emit(Opcodes.LOCAL_GLOB_DYNAMIC);
+                emitReg(rd);
+                emitReg(nameReg);
+                lastResultReg = rd;
+                return;
+            }
             // General fallback for any lvalue expression (matches JVM backend behavior)
             // Handles: local $hash{key}, local $array[index], local $obj->method->{key}, etc.
             if (node.operand instanceof BinaryOperatorNode binOp) {

src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java

@@ -1594,6 +1594,10 @@ public static RuntimeList execute(InterpretedCode code, RuntimeArray args, int c
                                 pc = InlineOpcodeHandler.executeLocalGlob(bytecode, pc, registers, code);
                             }
 
+                            case Opcodes.LOCAL_GLOB_DYNAMIC -> {
+                                pc = InlineOpcodeHandler.executeLocalGlobDynamic(bytecode, pc, registers);
+                            }
+
                             case Opcodes.GET_LOCAL_LEVEL -> {
                                 pc = InlineOpcodeHandler.executeGetLocalLevel(bytecode, pc, registers);
                             }

src/main/java/org/perlonjava/backend/bytecode/Disassemble.java

@@ -1452,6 +1452,12 @@ public static String disassemble(InterpretedCode interpretedCode) {
                     case Opcodes.LOCAL_GLOB:
                         sb.append("LOCAL_GLOB r").append(interpretedCode.bytecode[pc++]).append(" = pushLocalVariable(glob '").append(interpretedCode.stringPool[interpretedCode.bytecode[pc++]]).append("')\n");
                         break;
+                    case Opcodes.LOCAL_GLOB_DYNAMIC: {
+                        int lgdRd = interpretedCode.bytecode[pc++];
+                        int lgdNameReg = interpretedCode.bytecode[pc++];
+                        sb.append("LOCAL_GLOB_DYNAMIC r").append(lgdRd).append(" = pushLocalVariable(glob r").append(lgdNameReg).append(")\n");
+                        break;
+                    }
                     case Opcodes.GET_LOCAL_LEVEL:
                         sb.append("GET_LOCAL_LEVEL r").append(interpretedCode.bytecode[pc++]).append("\n");
                         break;