An experimental framework that uses AI (LangChain4j + GPT-4o) to generate, compile, execute, and compare Java collection implementations across multiple frameworks.
This project explores how different Java collection libraries solve the same data processing problems. Given a set of predefined scenarios (e.g., "find high-value customers", "group orders by status"), the system:
- Generates Java code using an LLM for each scenario and framework
- Compiles the code in-memory with validation
- Executes the solution against deterministic test datasets
- Compares results across frameworks for equivalence
| Framework | ID | Description |
|---|---|---|
| Java Streams | streams |
Standard Java Stream API |
| Eclipse Collections | eclipse |
High-performance collections library |
| Google Guava | guava |
Google's core Java libraries |
| Plain Java | plain |
Traditional loops without streams |
- Java 25 with preview features enabled
- Maven 3.8+
- OpenAI API key
-
Clone the repository
-
Create a
.envfile in the project root:OPENAI_API_KEY=your-api-key-here -
Build the project:
mvn clean install
# Run all scenarios with Java Streams on DATASET_A (default)
mvn exec:java -pl generator \
-Dexec.mainClass="com.donaldw.gh.collections.generator.cli.ExperimentCli"
# Run specific scenario with all frameworks + equivalence testing
mvn exec:java -pl generator \
-Dexec.mainClass="com.donaldw.gh.collections.generator.cli.ExperimentCli" \
-Dexec.args="-s SCEN-012 -f all -e"
# Run on all datasets with verbose output
mvn exec:java -pl generator \
-Dexec.mainClass="com.donaldw.gh.collections.generator.cli.ExperimentCli" \
-Dexec.args="-d all -f streams -v"| Option | Description |
|---|---|
-s, --scenario <id> |
Scenario ID to run (repeatable) |
-f, --framework <name> |
Framework: streams, eclipse, guava, plain, or all |
-d, --dataset <name> |
Dataset: DATASET_A, DATASET_B, DATASET_C, or all |
-o, --output <dir> |
Output directory for results (default: output) |
-m, --model <name> |
LLM model to use (default: gpt-4o) |
-e, --equivalence |
Run equivalence tests across frameworks |
-p, --persist |
Persist generated solutions to generated-solutions module |
--clean |
Remove all generated solutions and output files |
-v, --verbose |
Verbose output |
--debug |
Enable LLM tracing (logs all messages, tool calls, and results) |
--no-export |
Don't export metrics to files |
-h, --help |
Show help |
# Run all tests
mvn clean test
# Run only unit tests (skip LLM integration tests)
mvn test -DexcludedGroups=integrationcollections-experiments/
├── domain/ # Domain model (Customer, Order, Product, Money, etc.)
├── datasets/ # Deterministic test datasets (A, B, C)
├── scenarios/ # Scenario definitions in JSON
├── contracts/ # Code generation contracts (ScenarioSolution interface)
├── generator/ # LLM integration, compilation, execution, metrics
│ ├── cli/ # Command-line interface
│ ├── compiler/ # In-memory Java compilation
│ ├── context/ # Domain context for prompts
│ ├── equivalence/ # Cross-framework comparison
│ ├── framework/ # Framework definitions & isolation
│ ├── harness/ # Execution harness with timeout
│ ├── metrics/ # Metrics collection & export
│ ├── normalize/ # Code normalization & comparison
│ ├── persist/ # Solution persistence to disk
│ ├── pipeline/ # Pipeline orchestration
│ └── prompts/ # LLM prompt generation
├── generated-solutions/ # Persisted generated code (reviewable in IDE)
├── PLAN.md # Original implementation plan
├── TASKS.md # Completed task list
└── pom.xml # Parent POM
The framework includes 20 predefined scenarios covering common collection operations:
- SCEN-001: High-Value Customers (filter + sort)
- SCEN-002: Orders by Status (grouping + counting)
- SCEN-003: Top Products by Quantity (aggregation + limit)
- SCEN-004: Revenue by Category (map-reduce)
- SCEN-012: Customer Tier Distribution (enum grouping)
- ... and 15 more
See scenarios/src/main/resources/scenarios/scenarios.json for the full list.
Three deterministic datasets with fixed-seed random generation:
| Dataset | Customers | Products | Characteristics |
|---|---|---|---|
| DATASET_A | 10 | 20 | Small, evenly distributed |
| DATASET_B | 30 | 50 | Medium, skewed distribution |
| DATASET_C | 200 | 500 | Large, realistic patterns |
Results are exported to the output directory:
metrics.json- Detailed metrics for each runmetrics.csv- Same data in CSV formatsummary.json- Aggregated summary statistics
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Scenarios │────▶│ CodeGenerator│────▶│ FrameworkCompiler│
│ (JSON) │ │ (LangChain4j)│ │ (In-Memory) │
└─────────────┘ └──────────────┘ └─────────────────┘
│
▼
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Datasets │────▶│ExecutionHarness│◀───│ CompiledSolution│
│ (A, B, C) │ │ (Timeout) │ │ (Bytecode) │
└─────────────┘ └──────────────┘ └─────────────────┘
│
▼
┌──────────────┐
│EquivalenceRunner│
│ (Compare) │
└──────────────┘
│
▼
┌──────────────┐
│MetricsCollector│
│ (Export) │
└──────────────┘
- Framework Isolation: Prevents cross-framework imports (e.g., no Guava in Eclipse scenarios)
- In-Memory Compilation: Uses
javax.toolsAPI for fast compilation - Timeout Support: Configurable execution timeout (default 30s)
- Result Equivalence: Compares results across frameworks using semantic equality
- Comprehensive Metrics: Timing, code complexity, success rates by framework/scenario
- Code Normalization: Strips comments, sorts imports for comparison
- Solution Persistence: Optionally save generated code to reviewable module
- Self-Correcting Generation: LLM uses tool calling to validate and fix compilation errors
- Java 25 with preview features
- LangChain4j 1.10.0 for LLM integration
- Eclipse Collections 13.0.0 for alternative collections
- Google Guava 33.5.0-jre for alternative collections
- JUnit 6.0.1 for testing
- SLF4J 2.0.17 for logging
- Jackson for JSON parsing
- Spotless 3.1.0 for code formatting
MIT License