Add CPAN Phase 1-3: Core modules and IPC::Open2/Open3

dev/design/cpan_client.md

@@ -37,7 +37,7 @@ CPAN.pm has deep dependencies that make it challenging to port. The main blocker
 | **Archive::Tar** | ✅ Done | Medium | Imported via sync.pl |
 | **Archive::Zip** | ❌ Missing | Medium | Zip handling - Java has built-in support |
 | **Net::FTP** | ✅ Done | Medium | Imported via sync.pl |
-| **IPC::Open3** | ❌ Missing | Medium | Process I/O - needs Java ProcessBuilder |
+| **IPC::Open3** | ✅ Done | Medium | Custom implementation using Java ProcessBuilder |
 | **IO::Socket** | ✅ Done | Medium | Imported via sync.pl |
 | **Dumpvalue** | ✅ Done | Low | Imported via sync.pl |
 
@@ -224,7 +224,7 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
 
 ## Progress Tracking
 
-### Current Status: Phase 2 complete
+### Current Status: Phase 3 complete
 
 ### Completed
 - [x] Analyze CPAN.pm dependencies (2024-03-13)
@@ -248,6 +248,16 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
   - Parser fix: `@{${...}}` nested dereference now works in push/unshift
   - SysHostname.java XS module - provides ghname() via InetAddress.getLocalHost()
   - XSLoader caller() support - load() now uses caller() when no argument provided
+- [x] **Phase 3: Process Control** (2024-03-13)
+  - IPC::Open2, IPC::Open3 - custom implementation using Java ProcessBuilder
+  - IPCOpen3.java XS module loaded via XSLoader
+  - ProcessInputHandle.java, ProcessOutputHandle.java for process stream I/O
+  - Works on both Windows (WaitpidOperator) and POSIX (RuntimeIO)
+  - pipe() - fixed autovivification to handle undefined variables (like open())
+  - fcntl() - implemented with jnr-posix native support and fallback stub
+  - ioctl() - implemented with jnr-posix native support and fallback stub
+  - Prototype parsing fix - typeglob arguments now use =~ precedence level
+  - Reference comparison fix - `\$x == \undef` no longer crashes (NPE in getDoubleRef)
 
 ### Files Changed (Phase 2)
 - `dev/import-perl5/config.yaml` - Added IO::Socket, IO::Zlib, Archive::Tar, Net::*, Tie::StdHandle, File::Spec imports
@@ -257,11 +267,26 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
 - `src/main/java/org/perlonjava/runtime/perlmodule/SysHostname.java` - New XS module for Sys::Hostname
 - `src/main/java/org/perlonjava/runtime/perlmodule/XSLoader.java` - Added caller() support for no-argument load()
 
+### Files Changed (Phase 3)
+- `src/main/java/org/perlonjava/runtime/perlmodule/IPCOpen3.java` - XS module for open2/open3
+- `src/main/java/org/perlonjava/runtime/io/ProcessInputHandle.java` - IOHandle for process stdout/stderr
+- `src/main/java/org/perlonjava/runtime/io/ProcessOutputHandle.java` - IOHandle for process stdin
+- `src/main/perl/lib/IPC/Open2.pm`, `src/main/perl/lib/IPC/Open3.pm` - Custom wrappers using XSLoader
+- `src/main/java/org/perlonjava/runtime/operators/IOOperator.java` - pipe() autovivification, fcntl(), ioctl()
+- `src/main/java/org/perlonjava/runtime/operators/OperatorHandler.java` - Added fcntl/ioctl descriptors
+- `src/main/java/org/perlonjava/frontend/parser/PrototypeArgs.java` - Fixed typeglob prototype parsing
+- `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeScalar.java` - Fixed getIntRef()/getDoubleRef() NPE
+- `dev/import-perl5/config.yaml` - Removed IPC::Open2/Open3 imports (custom implementation)
+
 ### Next Steps
-1. Phase 3: Process control (IPC::Open3)
-2. Evaluate cpanm as alternative to CPAN.pm
+1. Phase 4: Evaluate cpanm as alternative to CPAN.pm
+2. Consider Archive::Zip implementation using java.util.zip
+3. Document "how to add a CPAN module" for users
 
 ### Open Questions
 - Is cpanm lighter on dependencies than CPAN.pm?
 - Should we create a PerlOnJava-specific minimal CPAN client?
 - How important is Safe compartmentalization for users?
+
+### Resolved Questions
+- ✅ fork() alternative: IPC::Open2/Open3 now use Java ProcessBuilder

dev/design/functional_interface_implementation.md

@@ -0,0 +1,188 @@
+# PerlSubroutine Functional Interface Implementation Plan
+
+## Overview
+
+This document tracks the implementation of replacing `MethodHandle`-based subroutine invocation with a `PerlSubroutine` functional interface. This fixes MethodHandle conversion errors that occur at runtime.
+
+## Problem Statement
+
+The current implementation uses `MethodHandle` for invoking compiled subroutines:
+```java
+// Current approach - prone to signature mismatch errors
+if (isStatic) {
+    result = (RuntimeList) this.methodHandle.invoke(a, callContext);
+} else {
+    result = (RuntimeList) this.methodHandle.invoke(this.codeObject, a, callContext);
+}
+```
+
+This causes errors like:
+```
+cannot convert MethodHandle(anon200,RuntimeArray,int)RuntimeList to (RuntimeArray,int)RuntimeList
+```
+
+The error occurs when the cached MethodHandle signature doesn't match the invocation pattern (static vs instance).
+
+## Solution
+
+Replace MethodHandle with a functional interface that has a fixed signature:
+
+```java
+@FunctionalInterface
+public interface PerlSubroutine {
+    RuntimeList apply(RuntimeArray args, int callContext) throws Exception;
+}
+```
+
+Benefits:
+1. **Type safety**: Fixed signature eliminates conversion errors
+2. **Performance**: Direct interface calls are faster than MethodHandle.invoke()
+3. **JIT optimization**: Better inlining opportunities
+4. **Simplicity**: No need for separate `codeObject` field - the subroutine IS the object
+
+## Scope
+
+### What Changes
+1. JVM-compiled subroutines (EmitterMethodCreator)
+2. RuntimeCode invocation logic
+3. PerlModuleBase static method registration
+4. Inline method cache (callCached)
+
+### What Doesn't Change
+1. InterpretedCode - already overrides `apply()`, no MethodHandle used
+2. eval STRING - uses either JVM or interpreter path, both covered
+3. API signatures - `apply(RuntimeArray, int)` remains the same
+
+## Implementation Phases
+
+### Phase 1: Create Interface (COMPLETED)
+- [x] Create `PerlSubroutine.java` functional interface
+- File: `src/main/java/org/perlonjava/runtime/runtimetypes/PerlSubroutine.java`
+
+### Phase 2: Update EmitterMethodCreator (COMPLETED)
+- [x] Add `implements PerlSubroutine` to generated classes
+- [x] Change `cw.visit()` to include interface in interfaces array
+- File: `src/main/java/org/perlonjava/backend/jvm/EmitterMethodCreator.java`
+- Line ~437: Changed from `null` to `new String[]{"org/perlonjava/runtime/runtimetypes/PerlSubroutine"}`
+
+### Phase 3: Update RuntimeCode (COMPLETED)
+- [x] Add `public PerlSubroutine subroutine;` field
+- [x] Add constructor `RuntimeCode(PerlSubroutine subroutine, String prototype)`
+- [x] Keep `methodHandle` and `codeObject` for backward compatibility during migration
+- [x] Update `defined()` to check `subroutine != null`
+- [x] Update `copy()` to copy `subroutine` field
+- File: `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java`
+
+### Phase 4: Update makeCodeObject() (COMPLETED)
+- [x] Cast codeObject to PerlSubroutine: `PerlSubroutine subroutine = (PerlSubroutine) codeObject;`
+- [x] Create RuntimeCode with subroutine: `new RuntimeCode(subroutine, prototype)`
+- File: `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java`
+- Method: `makeCodeObject()` (~line 1181)
+
+### Phase 5: Update RuntimeCode.apply() (COMPLETED)
+- [x] Prefer `subroutine.apply()` over `methodHandle.invoke()`
+- [x] Keep methodHandle path as fallback for backward compatibility
+- File: `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java`
+- Methods: `apply()` (~line 2019, ~line 2097)
+
+### Phase 6: Update PerlModuleBase (SKIPPED)
+- Note: Keeping methodHandle approach for PerlModuleBase to preserve caller() stack behavior
+- Early attempts to change this broke export_to_level tests
+- May revisit later if needed
+- File: `src/main/java/org/perlonjava/runtime/perlmodule/PerlModuleBase.java`
+
+### Phase 7: Update Inline Cache (callCached) (COMPLETED)
+- [x] Change cache to check `subroutine != null || methodHandle != null`
+- [x] Prefer `cachedCode.subroutine.apply()` over MethodHandle
+- [x] Fall back to methodHandle when subroutine not available
+- File: `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java`
+- Method: `callCached()` (~line 1237)
+
+### Phase 8: Update SubroutineParser and InterpretedCode (COMPLETED)
+- [x] Update `SubroutineParser` to set `placeholder.subroutine` for deferred compilation
+- [x] Update `subExists` check to include `subroutine != null`
+- [x] Make `InterpretedCode` implement `PerlSubroutine` interface
+- Files: SubroutineParser.java, InterpretedCode.java
+
+### Phase 9: Testing (COMPLETED)
+- [x] Run `./gradlew test` - all tests pass
+- [x] Run comp/require.t - 1743/1747 pass (previously had MethodHandle errors)
+- [x] No MethodHandle conversion errors observed
+- [x] Basic subroutine calls, closures, and method calls work correctly
+
+### Phase 10: Cleanup (PARTIAL - 2024-03-13)
+- [x] Remove redundant `methodHandle` lookups in SubroutineParser deferred compilation
+- [ ] Remove `methodHandle` field from RuntimeCode (blocked: PerlModuleBase still uses it)
+- [ ] Remove `codeObject` field (blocked: still needed for __SUB__ field access)
+- [ ] Remove `methodHandleCache` (low priority - not causing issues)
+- [ ] Remove `isStatic` field (blocked: PerlModuleBase uses it)
+
+Note: Full cleanup is blocked because PerlModuleBase uses methodHandle for static Java
+methods. This is intentional to preserve caller() stack behavior for built-in modules.
+
+## File Change Summary
+
+| File | Changes |
+|------|---------|
+| PerlSubroutine.java | NEW - functional interface |
+| EmitterMethodCreator.java | Add interface to generated classes |
+| InterpretedCode.java | Implements PerlSubroutine interface |
+| RuntimeCode.java | Add subroutine field, update apply(), makeCodeObject(), callCached() |
+| SubroutineParser.java | Set subroutine field, removed redundant methodHandle lookups |
+| RuntimeScalar.java | Constructor disambiguation |
+| GlobalVariable.java | Constructor disambiguation |
+
+## Backward Compatibility
+
+During migration:
+1. Keep both `subroutine` and `methodHandle` fields
+2. Prefer `subroutine` when available, fall back to `methodHandle`
+3. This allows gradual migration and easy rollback
+
+## Risk Assessment
+
+- **Risk**: Medium - touches core subroutine dispatch
+- **Mitigation**: Keep methodHandle as fallback, comprehensive testing
+- **Rollback**: Can revert to methodHandle-only if issues found
+
+## Testing Strategy
+
+1. Unit tests via `./gradlew test`
+2. Integration tests via perl5_t/t test suite
+3. Specific focus on:
+   - comp/require.t (MethodHandle error)
+   - Method calls with closures
+   - Inline cache behavior
+   - eval STRING execution
+
+## Progress Tracking
+
+### Current Status: Implementation Complete (Phase 9 passed)
+
+### Completed Phases (2024-03-13)
+- [x] Phase 1: Create PerlSubroutine interface
+- [x] Phase 2: Update EmitterMethodCreator - generated classes implement PerlSubroutine
+- [x] Phase 3: Update RuntimeCode - added subroutine field, constructor, copy(), defined()
+- [x] Phase 4: Update makeCodeObject() - casts to PerlSubroutine
+- [x] Phase 5: Update RuntimeCode.apply() - prefers subroutine over methodHandle
+- [x] Phase 6: PerlModuleBase - SKIPPED (preserves caller() stack behavior)
+- [x] Phase 7: Update callCached() inline cache
+- [x] Phase 8: Update SubroutineParser and InterpretedCode
+- [x] Phase 9: Testing - all tests pass, no MethodHandle conversion errors
+
+### Files Changed
+- `src/main/java/org/perlonjava/runtime/runtimetypes/PerlSubroutine.java` (NEW)
+- `src/main/java/org/perlonjava/backend/jvm/EmitterMethodCreator.java`
+- `src/main/java/org/perlonjava/backend/bytecode/InterpretedCode.java`
+- `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java`
+- `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeScalar.java` (constructor disambiguation)
+- `src/main/java/org/perlonjava/runtime/runtimetypes/GlobalVariable.java` (constructor disambiguation)
+- `src/main/java/org/perlonjava/frontend/parser/SubroutineParser.java`
+
+### Next Steps (Optional)
+1. Run more extensive tests from perl5_t/t suite
+2. Consider Phase 10 cleanup to remove deprecated methodHandle fields
+
+## Related Documents
+- `dev/design/functional_subroutines.md` - Original design proposal
+- `AGENTS.md` - Project guidelines

dev/import-perl5/config.yaml

@@ -451,6 +451,9 @@ imports:
   - source: perl5/lib/Symbol.pm
     target: src/main/perl/lib/Symbol.pm
 
+  # Note: IPC::Open2 and IPC::Open3 are NOT imported - we use custom
+  # implementations with Java ProcessBuilder (see IPCOpen3.java)
+
   # Add more imports below as needed
   # Example with minimal fields:
   # - source: perl5/lib/SomeModule.pm

docs/about/changelog.md

@@ -9,8 +9,8 @@ Release history of PerlOnJava. See [Roadmap](roadmap.md) for future plans.
 - Add `defer` feature
 - Non-local control flow: `last`/`next`/`redo`/`goto LABEL`
 - Tail call with trampoline for `goto &NAME` and `goto __SUB__`
-- Add modules: `Time::Piece`, `TOML`, `DirHandle`, `Dumpvalue`, `Sys::Hostname`, `IO::Socket`, `IO::Socket::INET`, `IO::Socket::UNIX`, `IO::Zlib`, `Archive::Tar`, `Net::FTP`, `Net::Cmd`.
-- Add operators: `flock`, `syscall`.
+- Add modules: `Time::Piece`, `TOML`, `DirHandle`, `Dumpvalue`, `Sys::Hostname`, `IO::Socket`, `IO::Socket::INET`, `IO::Socket::UNIX`, `IO::Zlib`, `Archive::Tar`, `Net::FTP`, `Net::Cmd`, `IPC::Open2`, `IPC::Open3`.
+- Add operators: `flock`, `syscall`, `fcntl`, `ioctl`.
 - Bugfix: parser now handles `@{${...}}` nested dereference in push/unshift.
 - Bugfix: regex octal escapes `\10`-`\377` now work correctly.
 - Bugfix: operator override in Time::Hires now works.

docs/reference/feature-matrix.md

@@ -523,6 +523,8 @@ my @copy = @{$z};         # ERROR
 - ✅  **`DATA`**: `DATA` file handle is implemented.
 - ✅  **`truncate`**: File truncation
 - ✅  **`flock`**: File locking with LOCK_SH, LOCK_EX, LOCK_UN, LOCK_NB
+- ✅  **`fcntl`**: File control operations (stub + native via jnr-posix)
+- ✅  **`ioctl`**: Device control operations (stub + native via jnr-posix)
 - ✅  **`syscall`**: System calls (SYS_gethostname)
 
 ### Socket Operations
@@ -725,6 +727,8 @@ The `:encoding()` layer supports all encodings provided by Java's `Charset.forNa
 - 🚧  **POSIX** module.
 - 🚧  **Unicode::Normalize** `normalize`, `NFC`, `NFD`, `NFKC`, `NFKD`.
 - ✅  **Archive::Tar** module.
+- ✅  **IPC::Open2** module.
+- ✅  **IPC::Open3** module.
 - ✅  **Net::FTP** module.
 - ✅  **Net::Cmd** module.
 - ❌  **Safe** module.

src/main/java/org/perlonjava/backend/bytecode/BytecodeInterpreter.java

@@ -549,7 +549,7 @@ public static RuntimeList execute(InterpretedCode code, RuntimeArray args, int c
                             // TYPE AND REFERENCE OPERATORS (opcodes 102-105) - Delegated
                             // =================================================================
 
-                            case Opcodes.DEFINED, Opcodes.REF, Opcodes.BLESS, Opcodes.ISA, Opcodes.PROTOTYPE,
+                            case Opcodes.DEFINED, Opcodes.DEFINED_GLOB, Opcodes.REF, Opcodes.BLESS, Opcodes.ISA, Opcodes.PROTOTYPE,
                                  Opcodes.QUOTE_REGEX, Opcodes.QUOTE_REGEX_O -> {
                                 pc = executeTypeOps(opcode, bytecode, pc, registers, code);
                             }
@@ -2014,7 +2014,7 @@ private static int executeComparisons(int opcode, int[] bytecode, int pc,
 
     /**
      * Execute type and reference operations.
-     * Handles: DEFINED, REF, BLESS, ISA, PROTOTYPE, QUOTE_REGEX
+     * Handles: DEFINED, DEFINED_GLOB, REF, BLESS, ISA, PROTOTYPE, QUOTE_REGEX
      */
     private static int executeTypeOps(int opcode, int[] bytecode, int pc,
                                       RuntimeBase[] registers, InterpretedCode code) {
@@ -2027,6 +2027,17 @@ private static int executeTypeOps(int opcode, int[] bytecode, int pc,
                 registers[rd] = defined ? RuntimeScalarCache.scalarTrue : RuntimeScalarCache.scalarFalse;
                 return pc;
             }
+            case Opcodes.DEFINED_GLOB -> {
+                // defined *$var - check if glob is defined without throwing strict refs
+                // Format: DEFINED_GLOB rd scalar_reg pkg_string_idx
+                int rd = bytecode[pc++];
+                int scalarReg = bytecode[pc++];
+                int pkgIdx = bytecode[pc++];
+                String pkg = code.stringPool[pkgIdx];
+                RuntimeScalar scalar = registers[scalarReg].scalar();
+                registers[rd] = GlobalVariable.definedGlob(scalar, pkg);
+                return pc;
+            }
             case Opcodes.REF -> {
                 int rd = bytecode[pc++];
                 int rs = bytecode[pc++];

src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java

@@ -438,6 +438,36 @@ private static void visitDieWarn(BytecodeCompiler bc, OperatorNode node, String
         }
     }
 
+    /**
+     * Handles `defined` operator with special case for `defined *$var`.
+     * Perl allows `defined *$var` even under strict refs without auto-vivifying.
+     */
+    private static void visitDefined(BytecodeCompiler bc, OperatorNode node) {
+        // Check for special case: defined *$var
+        if (node.operand instanceof ListNode listNode && listNode.elements.size() == 1) {
+            Node operand = listNode.elements.getFirst();
+            // Handle defined(+expr) by unwrapping the +
+            if (operand instanceof OperatorNode opNode && opNode.operator.equals("+")) {
+                operand = opNode.operand;
+            }
+            if (operand instanceof OperatorNode opNode && opNode.operator.equals("*")) {
+                // defined *$var - use special handling that doesn't throw strict refs
+                opNode.operand.accept(bc);
+                int scalarReg = bc.lastResultReg;
+                int pkgIdx = bc.addToStringPool(bc.getCurrentPackage());
+                int rd = bc.allocateOutputRegister();
+                bc.emit(Opcodes.DEFINED_GLOB);
+                bc.emitReg(rd);
+                bc.emitReg(scalarReg);
+                bc.emit(pkgIdx);
+                bc.lastResultReg = rd;
+                return;
+            }
+        }
+        // Default case: regular defined
+        emitSimpleUnary(bc, node, Opcodes.DEFINED);
+    }
+
     private static void visitPopShiftOp(BytecodeCompiler bc, OperatorNode node, short opcode) {
         int arrayReg = resolveArrayOperand(bc, node, node.operator);
         int rd = bc.allocateOutputRegister();
@@ -590,7 +620,7 @@ public static void visitOperator(BytecodeCompiler bytecodeCompiler, OperatorNode
                     bytecodeCompiler.isIntegerEnabled() ? Opcodes.INTEGER_BITWISE_NOT : Opcodes.BITWISE_NOT);
             case "binary~" -> emitSimpleUnary(bytecodeCompiler, node, Opcodes.BITWISE_NOT_BINARY);
             case "~." -> emitSimpleUnary(bytecodeCompiler, node, Opcodes.BITWISE_NOT_STRING);
-            case "defined" -> emitSimpleUnary(bytecodeCompiler, node, Opcodes.DEFINED);
+            case "defined" -> visitDefined(bytecodeCompiler, node);
             case "wantarray" -> { int rd = bytecodeCompiler.allocateOutputRegister(); bytecodeCompiler.emit(Opcodes.WANTARRAY); bytecodeCompiler.emitReg(rd); bytecodeCompiler.emitReg(2); bytecodeCompiler.lastResultReg = rd; }
             case "time" -> { int rd = bytecodeCompiler.allocateOutputRegister(); bytecodeCompiler.emit(Opcodes.TIME_OP); bytecodeCompiler.emitReg(rd); bytecodeCompiler.lastResultReg = rd; }
             case "getppid" -> { int rd = bytecodeCompiler.allocateOutputRegister(); bytecodeCompiler.emitWithToken(Opcodes.GETPPID, node.getIndex()); bytecodeCompiler.emitReg(rd); bytecodeCompiler.lastResultReg = rd; }