Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 27 additions & 7 deletions dev/design/cpan_client.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ CPAN.pm has deep dependencies that make it challenging to port. The main blocker
| **ExtUtils::MakeMaker** | ❌ Missing | Very High | Build system - huge module with many dependencies |
| **LWP::UserAgent** | ❌ Missing | Medium | Web client (HTTP::Tiny exists as alternative) |
| **Archive::Tar** | ✅ Done | Medium | Imported via sync.pl |
| **Archive::Zip** | ❌ Missing | Medium | Zip handling - Java has built-in support |
| **Archive::Zip** | ✅ Done | Medium | Java implementation using java.util.zip |
| **Net::FTP** | ✅ Done | Medium | Imported via sync.pl |
| **IPC::Open3** | ✅ Done | Medium | Custom implementation using Java ProcessBuilder |
| **IO::Socket** | ✅ Done | Medium | Imported via sync.pl |
Expand Down Expand Up @@ -224,7 +224,7 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)

## Progress Tracking

### Current Status: Phase 3 complete
### Current Status: Phase 4 complete

### Completed
- [x] Analyze CPAN.pm dependencies (2024-03-13)
Expand Down Expand Up @@ -258,6 +258,20 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
- ioctl() - implemented with jnr-posix native support and fallback stub
- Prototype parsing fix - typeglob arguments now use =~ precedence level
- Reference comparison fix - `\$x == \undef` no longer crashes (NPE in getDoubleRef)
- [x] **Phase 4: CPAN Client Evaluation & Archive::Zip** (2024-03-13)
- **cpanm analysis complete**: cpanm bundles (fatpacks) its dependencies but requires ExtUtils::MakeMaker
- ExtUtils::MakeMaker is the critical blocker - it's the CPAN build system that runs `make`
- Since PerlOnJava doesn't use native compilation, a traditional CPAN client isn't feasible
- **Archive::Zip implemented**: Full Java implementation using java.util.zip
- Read/write zip files
- Add files, strings, and directories
- Extract individual members or entire archive
- Works with system `unzip` command for verification
- Created user documentation: `docs/guides/using-cpan-modules.md`
- How to check module availability
- List of included modules
- How to add pure Perl modules
- Example scripts for downloading CPAN modules

### Files Changed (Phase 2)
- `dev/import-perl5/config.yaml` - Added IO::Socket, IO::Zlib, Archive::Tar, Net::*, Tie::StdHandle, File::Spec imports
Expand All @@ -278,15 +292,21 @@ This is already working for many modules (Pod::*, Test::*, Getopt::Long, etc.)
- `src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeScalar.java` - Fixed getIntRef()/getDoubleRef() NPE
- `dev/import-perl5/config.yaml` - Removed IPC::Open2/Open3 imports (custom implementation)

### Files Changed (Phase 4)
- `src/main/java/org/perlonjava/runtime/perlmodule/ArchiveZip.java` - New Java implementation
- `src/main/perl/lib/Archive/Zip.pm` - Perl wrapper with XSLoader
- `docs/guides/using-cpan-modules.md` - User documentation for adding CPAN modules

### Next Steps
1. Phase 4: Evaluate cpanm as alternative to CPAN.pm
2. Consider Archive::Zip implementation using java.util.zip
3. Document "how to add a CPAN module" for users
1. Consider a minimal CPAN download helper (pure Perl, no build step)
2. Expand user documentation with more examples
3. Add more commonly-needed pure Perl modules

### Open Questions
- Is cpanm lighter on dependencies than CPAN.pm?
- Should we create a PerlOnJava-specific minimal CPAN client?
- Should we create a PerlOnJava-specific minimal CPAN download tool?
- How important is Safe compartmentalization for users?

### Resolved Questions
- ✅ fork() alternative: IPC::Open2/Open3 now use Java ProcessBuilder
- ✅ cpanm feasibility: cpanm requires ExtUtils::MakeMaker which needs `make` - not suitable for PerlOnJava
- ✅ Archive::Zip: Implemented using java.util.zip
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,24 @@ Java equivalents are easier to write and maintain than C/XS. The same API surfac

---

## CPAN Installation Challenges

**ExtUtils::MakeMaker assumes a C toolchain.**

What breaks without C:
- `Makefile.PL` → generates C build rules
- XS modules → need compiler + linker
- Shared libraries → not produced

**What works today:**
- 300+ modules bundled in JAR
- Pure-Perl CPAN modules run unmodified
- Hand-written Java replacements for XS

**Open question:** cpanm integration, automated XS→Java

---

## Module Loading

`require` converts `Module::Name` → `Module/Name.pm`, searches `@INC`, caches in `%INC`.
Expand Down
2 changes: 1 addition & 1 deletion docs/about/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Release history of PerlOnJava. See [Roadmap](roadmap.md) for future plans.
- Add `defer` feature
- Non-local control flow: `last`/`next`/`redo`/`goto LABEL`
- Tail call with trampoline for `goto &NAME` and `goto __SUB__`
- Add modules: `Time::Piece`, `TOML`, `DirHandle`, `Dumpvalue`, `Sys::Hostname`, `IO::Socket`, `IO::Socket::INET`, `IO::Socket::UNIX`, `IO::Zlib`, `Archive::Tar`, `Net::FTP`, `Net::Cmd`, `IPC::Open2`, `IPC::Open3`.
- Add modules: `Time::Piece`, `TOML`, `DirHandle`, `Dumpvalue`, `Sys::Hostname`, `IO::Socket`, `IO::Socket::INET`, `IO::Socket::UNIX`, `IO::Zlib`, `Archive::Tar`, `Archive::Zip`, `Net::FTP`, `Net::Cmd`, `IPC::Open2`, `IPC::Open3`.
- Add operators: `flock`, `syscall`, `fcntl`, `ioctl`.
- Bugfix: parser now handles `@{${...}}` nested dereference in push/unshift.
- Bugfix: regex octal escapes `\10`-`\377` now work correctly.
Expand Down
273 changes: 273 additions & 0 deletions docs/guides/using-cpan-modules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,273 @@
# Using CPAN Modules with PerlOnJava

## Overview

PerlOnJava includes many common CPAN modules and supports adding additional pure Perl modules to your projects.

## Checking Module Availability

To check if a module is available in PerlOnJava:

```bash
./jperl -e 'use Module::Name; print "Available\n"'
```

## Available Modules

PerlOnJava includes:

### Core Modules
- `strict`, `warnings`, `utf8`, `feature`
- `Carp`, `Config`, `Cwd`, `Exporter`
- `File::Spec`, `File::Basename`, `File::Copy`, `File::Find`, `File::Path`, `File::Temp`
- `IO::File`, `IO::Handle`, `FileHandle`
- `Getopt::Long`, `Getopt::Std`

### Data Processing
- `JSON` - JSON encoding/decoding
- `YAML` - YAML parsing
- `TOML` - TOML parsing
- `Text::CSV` - CSV parsing
- `Data::Dumper` - Data structure dumping
- `Storable` - Data serialization

### Cryptography & Encoding
- `Digest::MD5`, `Digest::SHA` - Hash algorithms
- `MIME::Base64`, `MIME::QuotedPrint` - Encoding
- `Encode` - Character encoding

### Network & Web
- `HTTP::Tiny` - HTTP client
- `Socket` - Low-level socket support
- `IO::Socket::INET` - TCP/IP sockets
- `Net::FTP` - FTP client

### Archives
- `Archive::Tar` - Tar file handling
- `Archive::Zip` - Zip file handling
- `Compress::Zlib` - Compression
- `IO::Zlib` - Compressed I/O

### Database
- `DBI` - Database interface (with JDBC backend)

### Testing
- `Test::More`, `Test::Simple`, `Test::Builder`

### Time
- `Time::HiRes` - High-resolution time
- `Time::Piece` - Time manipulation
- `POSIX` - POSIX functions including strftime

## Adding Pure Perl Modules

If you need a CPAN module that's not included, you can often add pure Perl modules directly.

### Method 1: Local lib Directory

Create a `lib` directory in your project and add modules there:

```bash
mkdir -p myproject/lib
cp /path/to/Some/Module.pm myproject/lib/Some/Module.pm
```

Run with:

```bash
./jperl -Imyproject/lib myscript.pl
```

### Method 2: PERL5LIB Environment Variable

```bash
export PERL5LIB=/path/to/your/modules
./jperl myscript.pl
```

### Finding Pure Perl Modules

To check if a CPAN module is pure Perl:

1. Visit https://metacpan.org/pod/Module::Name
2. Look at the source files
3. If there's only `.pm` files (no `.xs` or `.c` files), it's pure Perl

### Example: Adding a Simple Module

```bash
# Download a module from CPAN
curl -O https://cpan.metacpan.org/authors/id/X/XX/XXXX/Module-Name-1.00.tar.gz

# Extract
tar xzf Module-Name-1.00.tar.gz

# Copy the lib directory
cp -r Module-Name-1.00/lib/* myproject/lib/
```

## Modules with XS Components

Some CPAN modules have XS (C/C++) components that won't work directly. For these modules:

1. **Check if PerlOnJava has a Java port** - Many common XS modules have Java implementations
2. **Look for pure Perl alternatives** - e.g., use `JSON` instead of `JSON::XS`
3. **Request a port** - Open an issue at the PerlOnJava repository

### Common XS Modules with Java Alternatives

| XS Module | Java Alternative in PerlOnJava |
|-----------|-------------------------------|
| JSON::XS | JSON (built-in) |
| Compress::Raw::Zlib | Compress::Zlib (built-in) |
| Digest::MD5 (XS part) | Digest::MD5 (Java implementation) |
| DBI (XS part) | DBI (JDBC backend) |
| Time::HiRes (XS part) | Time::HiRes (Java implementation) |

## Working with Archive Files

### Reading a Zip File

```perl
use Archive::Zip qw(:ERROR_CODES);

my $zip = Archive::Zip->new();
my $status = $zip->read('archive.zip');
die "Read failed" unless $status == AZ_OK;

for my $member ($zip->members()) {
print $member->fileName(), "\n";
}
```

### Creating a Zip File

```perl
use Archive::Zip qw(:ERROR_CODES);

my $zip = Archive::Zip->new();
$zip->addFile('document.txt');
$zip->addString("Hello!", 'hello.txt');

my $status = $zip->writeToFileNamed('output.zip');
```

### Working with Tar Files

```perl
use Archive::Tar;

# Read
my $tar = Archive::Tar->new('archive.tar.gz');
my @files = $tar->list_files();
$tar->extract();

# Create
my $tar = Archive::Tar->new();
$tar->add_files('file1.txt', 'file2.txt');
$tar->write('output.tar.gz', COMPRESS_GZIP);
```

## HTTP Requests

```perl
use HTTP::Tiny;

my $http = HTTP::Tiny->new();
my $response = $http->get('https://api.github.com/repos/perl/perl5');

if ($response->{success}) {
print $response->{content};
}
```

## Downloading and Extracting CPAN Modules

Here's a helper script to download and extract a CPAN module:

```perl
#!/usr/bin/env jperl
use strict;
use warnings;
use HTTP::Tiny;
use Archive::Tar;
use File::Temp qw(tempfile);
use File::Path qw(make_path);

my $module = shift or die "Usage: $0 Module::Name\n";
my $dest = shift || 'lib';

# Convert module name to path
(my $path = $module) =~ s/::/-/g;

# Query MetaCPAN for download URL
my $http = HTTP::Tiny->new();
my $resp = $http->get("https://fastapi.metacpan.org/v1/download_url/$module");

if (!$resp->{success}) {
die "Could not find $module on CPAN\n";
}

# Parse JSON response
use JSON;
my $data = decode_json($resp->{content});
my $url = $data->{download_url};

print "Downloading $url\n";
my $tarball = $http->get($url);

if (!$tarball->{success}) {
die "Download failed\n";
}

# Save to temp file
my ($fh, $filename) = tempfile(SUFFIX => '.tar.gz');
print $fh $tarball->{content};
close $fh;

# Extract
my $tar = Archive::Tar->new($filename);
my @files = $tar->list_files();

make_path($dest);
for my $file (@files) {
next unless $file =~ m{/lib/(.+\.pm)$};
my $target = "$dest/$1";
my $dir = $target;
$dir =~ s{/[^/]+$}{};
make_path($dir);

my $content = $tar->get_content($file);
open my $out, '>', $target or die "Cannot write $target: $!";
print $out $content;
close $out;
print "Installed $target\n";
}

unlink $filename;
print "Done!\n";
```

## Troubleshooting

### "Can't locate Module.pm in @INC"

The module is not installed. Check:
1. Is it a pure Perl module? XS modules won't work directly.
2. Is the module in your lib path? Use `-I/path/to/lib`

### "Can't load Java XS module"

This means the module requires an XS implementation that hasn't been ported to Java. Check if there's a pure Perl alternative.

### Module loads but functions don't work

Some modules may load but have unsupported features:
- Check if the module uses XS functions internally
- Some Perl built-in functions may not be fully implemented

## Getting Help

- **PerlOnJava Repository**: https://github.com/fglock/PerlOnJava
- **Issues**: Report missing modules or compatibility problems
- **Feature Matrix**: See `docs/reference/feature-matrix.md` for supported features
1 change: 1 addition & 0 deletions docs/reference/feature-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -727,6 +727,7 @@ The `:encoding()` layer supports all encodings provided by Java's `Charset.forNa
- 🚧 **POSIX** module.
- 🚧 **Unicode::Normalize** `normalize`, `NFC`, `NFD`, `NFKC`, `NFKD`.
- ✅ **Archive::Tar** module.
- ✅ **Archive::Zip** module.
- ✅ **IPC::Open2** module.
- ✅ **IPC::Open3** module.
- ✅ **Net::FTP** module.
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/perlonjava/core/Configuration.java
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ public final class Configuration {
* Automatically populated by Gradle/Maven during build.
* DO NOT EDIT MANUALLY - this value is replaced at build time.
*/
public static final String gitCommitId = "d0091f5b5";
public static final String gitCommitId = "ccaba0dc3";

/**
* Git commit date of the build (ISO format: YYYY-MM-DD).
Expand Down
Loading
Loading