Skip to content

WIP: Attribute system improvements + \K regex fix #423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fglock merged 4 commits into from
Apr 2, 2026
Merged

WIP: Attribute system improvements + \K regex fix #423

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
43719d1
Fix \K regex assertion and restore attribute validation
fglock Apr 2, 2026
9ac685c
Add \K regex assertion to changelog and feature matrix
fglock Apr 2, 2026
a83b507
Fix variable attribute dispatch for list declarations
fglock Apr 2, 2026
94618a9
Update attributes.md design doc with Phase 8 progress
fglock Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
232 changes: 45 additions & 187 deletions dev/design/attributes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ Variable attribute dispatch happens in the **emitter/compiler** — when a `my`/

## Progress Tracking

### Current Status: Phase 7 partially complete (isDeclared + Attribute::Handlers + error format)
### Current Status: Phase 8 — \K regex fix + variable attribute list dispatch complete

### Completed Phases

Expand Down Expand Up @@ -201,215 +201,73 @@ Variable attribute dispatch happens in the **emitter/compiler** — when a `my`/
- Fixed variable attribute error format with BEGIN failed suffix
- Files: `RuntimeCode.java`, `RuntimeGlob.java`, `SubroutineParser.java`

### Current Test Results (2026-04-02)

| File | Before | After | Delta |
|------|--------|-------|-------|
| attrs.t | 49/130 → 111/158* | 152/158 | +41 |
| attrproto.t | 3/52 | 51/52 | +48 |
| attrhand.t | 0/0 | 4/4 | +4 |
| uni/attrs.t | 10/34 | 29/34 | +19 |
| **Total** | **62/216** | **236/248** | **+112** |
- [x] Phase 8: \K regex fix + variable attribute list dispatch (2026-04-02)
- Fixed `\K` (keep left) regex assertion in `m//` and `s///`
- `\K` was silently stripped from patterns; now inserts `(?<perlK>)` named capture
- Substitution preserves text before `\K` position, adjusts match variables
- Capture group numbering offsets internal perlK group for user captures
- Restored compile-time/runtime attribute validation (was blocked by \K bug)
- Fixed list variable attribute dispatch: `my ($x,$y) : attr` now properly
propagates attribute annotations from parent node to each child in both
JVM emitter (EmitVariable.java) and interpreter (BytecodeCompiler.java)
- Files: `RegexPreprocessorHelper.java`, `RegexPreprocessor.java`, `RuntimeRegex.java`,
`OperatorParser.java`, `Attributes.java`, `EmitVariable.java`, `BytecodeCompiler.java`

### Current Test Results (2026-04-02, PR #423)

| File | Original | After PR #420 | After PR #423 | Total Delta |
|------|----------|---------------|---------------|-------------|
| attrs.t | 49/130 | 152/158* | 158/159** | +109 |
| attrproto.t | 3/52 | 51/52 | 51/52 | +48 |
| attrhand.t | 0/0 | 4/4 | 4/4 | +4 |
| uni/attrs.t | 10/34 | 29/34 | 35/35 | +25 |
| decl-refs.t | — | 346/408 | 346/408 | — |
| **Total** | **62/216** | **236/248** | **248/254** | **+186** |

\* attrs.t grew from 130 to 158 tests because the test no longer crashes partway through.
\*\* Only failure is TODO test 155 (RT #3605: ternary/attribute parsing ambiguity).

### Remaining Failures Analysis
### Remaining Failures Analysis (updated 2026-04-02)

#### attrproto.t: 4 remaining (48-51)
#### attrproto.t: 1 remaining (48)

**Root cause: `my sub` parser missing attribute loop after prototype**
**Root cause: `my sub` prototype not stored on RuntimeCode in interpreter backend**

| Test | Issue |
|------|-------|
| 48 | `my sub lexsub1(bar) : prototype(baz) {}` — `:prototype(baz)` not parsed |
| 49 | Illegal proto warning not emitted for `(bar)` on lexical sub |
| 50 | Illegal proto warning not emitted for `(baz)` on lexical sub |
| 51 | "Prototype overridden" warning not emitted |

**Fix:** In `StatementResolver.java`, after parsing `(prototype)` for `my sub`, add:
1. Call `emitIllegalProtoWarning()` for the parenthesized prototype
2. A second `while (peek(parser).text.equals(":"))` attribute-parsing loop

**Effort:** Small — straightforward parser fix.

#### attrs.t: 24 remaining

**Group A: `attributes::get` not returning built-in attrs (8 tests: 35-42)**

| Test | Expected | Got | Issue |
|------|----------|-----|-------|
| 35 | `"method Z"` | `"method"` | `FETCH_CODE_ATTRIBUTES` result not merged with built-in attrs |
| 36 | `"lvalue"` | `""` | `_fetch_attrs` not returning `lvalue` for predeclared subs |
| 37 | `"lvalue method"` | `""` | Same — multiple built-in attrs not returned |
| 38 | `"lvalue"` | `""` | `lvalue` on predeclared then defined sub not fetched |
| 39 | `"method"` | `""` | `method` on already-defined sub not fetched |
| 40 | `"method Z"` | `"Z"` | `method` from built-in + `Z` from FETCH not combined |
| 41-42 | `2`, `4` | `1`, `2` | Variable `tie` via `MODIFY_SCALAR_ATTRIBUTES` — `my $x : A0` dispatch missing |
| 48 | `eval 'my sub lexsub1(bar) : prototype(baz) {}; prototype \&lexsub1'` returns empty — `\&lexsub` produces REF instead of CODE in interpreter |

**Root cause:** `_fetch_attrs` in `Attributes.java` doesn't return `lvalue`/`method` from `RuntimeCode.attributes`. Tests 41-42 need variable attribute dispatch from the parser/emitter.
**Root cause:** In the interpreter (eval STRING), `\&lexsub` returns a REF reference (to the scalar holding the code) instead of a CODE reference. This is an interpreter parity issue with how lexical subs are stored and referenced — not an attribute-specific bug. Tests 49-52 now pass (warnings are correct).

**Fix:**
- Fix `_fetch_attrs` to filter and return built-in CODE attrs (`lvalue`, `method`, `const`)
- For 41-42: implement variable attribute dispatch in emitter (Phase 2)

**Group B: Variable attribute dispatch missing (4 tests: 27-28, 41-42)**
#### attrs.t: 1 remaining (155 — TODO)

| Test | Issue |
|------|-------|
| 27 | `my A $x : plugh` — `MODIFY_SCALAR_ATTRIBUTES` not called, no "may clash" warning |
| 28 | Same for multiple attrs |
| 41-42 | `my $x : A0` in loop — tie via MODIFY_SCALAR_ATTRIBUTES not happening |

**Fix:** Implement variable attribute dispatch. When the parser sees `my $x : Foo`, generate `attributes::->import(__PACKAGE__, \$x, "Foo")`. This requires emitter changes.

**Group C: Error detection issues (5 tests: 20, 44-45, 87, uni/23)**

| Test | Expected | Got | Issue |
|------|----------|-----|-------|
| 20 | Error with quoted attr names | Error without quotes | Error message formatting: attrs need double-quoting |
| 44 | `Can't declare scalar dereference in "our"` | `Invalid SCALAR attribute: foo` | Parser doesn't detect `our ${""} : foo` as dereference |
| 45 | `Can't declare scalar dereference in "my"` | `Invalid SCALAR attribute: bar` | Same for `my $$foo : bar` |
| 87 | `Global symbol "$nosuchvar" requires` | `Invalid CODE attribute: foo` | Strict error should be emitted instead of attr error |
| 154 | (TODO test) No separator error | Gets separator error | `$a ? my $var : my $othervar` — `:` parsed as attr separator |

**Fix:** Multiple parser improvements needed. Tests 44-45 need dereference detection. Test 87 needs strict checking before attribute validation. Test 154 is a known TODO.

**Group D: `:const` attribute (2 tests: 140, 145)**

| Test | Expected | Got | Issue |
|------|----------|-----|-------|
| 140 | `Useless use of attribute "const"` warning | No warning | `const` not handled in `_modify_attrs` |
| 145 | `32487` (const closure value) | `undef` | `:Const` -> `const` via MODIFY_CODE_ATTRIBUTES not applied |

**Fix:** Implement `:const` in `Attributes.java._modify_attrs()` — call the anon sub immediately and capture return value.

**Group E: `MODIFY_CODE_ATTRIBUTES` returning custom error (2 tests: 32, uni/16)**

| Test | Expected | Got |
|------|----------|-----|
| 32 | `X at ` (die in handler) | `Invalid CODE attribute: foo` |

**Root cause:** When `MODIFY_CODE_ATTRIBUTES` dies, the die message should propagate. Currently the error is being replaced by the default "Invalid CODE attribute" message.

**Group F: Closure prototype handling (3 tests: 124-126)**

| Test | Expected | Got |
|------|----------|-----|
| 124 | `Closure prototype called` error | Empty `$@` |
| 125 | `Closure prototype called` error | `Not a CODE reference` |
| 126 | `undef` | `"referencing closure prototype"` |

**Root cause:** Closure prototypes (stubs with captured lexicals) should die with "Closure prototype called" when invoked. This is a runtime feature, not an attribute-specific issue.

**Group G: Error message suffix (3 tests: 155-157)**

| Test | Expected | Got |
|------|----------|-----|
| 155 | `...at - line 1.\nBEGIN failed--compilation aborted at - line 1.` | `...at - line 1, near ""` |
| 156 | Same pattern for arrays | Same |
| 157 | Same pattern for hashes | Same |

**Root cause:** `fresh_perl_is` tests run `./jperl` as a subprocess. The error message format is `"at - line 1."` + `"BEGIN failed--compilation aborted"` suffix. PerlOnJava produces `"at - line 1, near \"\""` instead.
| 155 | (TODO test) `$a ? my $var : my $othervar` — `:` parsed as attr separator instead of ternary |

**Fix:** Two issues: (1) error location format, (2) missing "BEGIN failed" propagation.
**Status:** This is a known Perl 5 edge case (RT #3605). The TODO marker means it's expected to fail.

#### uni/attrs.t: 11 remaining
#### uni/attrs.t: 0 remaining — FULLY PASSING (35/35)

These mirror attrs.t failures with Unicode identifiers:
- Tests 8, 11-12, 16-18, 20-21, 23, 30-31 — same root causes as attrs.t groups A-F above
### Next Steps

### Next Steps (Priority Order)
Most attribute tests now pass. The remaining work is lower-priority:

#### Phase 2: `attributes::get` built-in attrs (HIGH — 8 tests)
#### Interpreter parity: `\&lexsub` in eval STRING (1 test: attrproto.t 48)

Fix `_fetch_attrs` in `Attributes.java` to return built-in CODE attributes (`lvalue`, `method`, `const`) from `RuntimeCode.attributes`. This is a small Java change.
In the interpreter, `\&lexsub` creates a REF (to the scalar holding the code) instead of a CODE reference. This affects `prototype \&lexsub` and is a general interpreter parity issue, not attribute-specific.

- **Files:** `Attributes.java`
- **Tests fixed:** attrs.t 35-40, uni/attrs.t equivalent
- **Effort:** Small

#### Phase 3: Variable attribute dispatch (MEDIUM — 6+ tests)

When the parser encounters `my $x : Foo` or `our @arr : Bar`, generate calls to `attributes::->import(__PACKAGE__, \$var, @attrs)`. This requires:

1. In the JVM emitter (`EmitVariable.java` or `EmitOperator.java`): when a variable declaration has `"attributes"` annotation, emit `attributes::->import(PKG, \$var, @attrs)`
2. In the bytecode interpreter: same
3. Timing: compile-time for `our`, run-time for `my`/`state`

- **Files:** `EmitVariable.java`, `CompileAssignment.java` (interpreter)
- **Tests fixed:** attrs.t 27-28, 41-42; uni/attrs.t 11-12, 17-18
- **Effort:** Medium

#### Phase 4: `my sub` attribute parsing (SMALL — 4 tests)

Add attribute parsing after prototype in `StatementResolver.java` `my sub` path:
1. Call `emitIllegalProtoWarning()` for `(proto)` syntax
2. Add second `:` attribute loop after prototype

- **Files:** `StatementResolver.java`
- **Tests fixed:** attrproto.t 48-51
- **Effort:** Small

#### Phase 5: `:const` attribute (SMALL — 2 tests)

Implement `:const` in `Attributes.java._modify_attrs()`:
- When `const` is applied to an already-defined sub, emit "Useless use" warning
- When `const` is applied during sub definition, invoke the sub immediately and replace with constant

- **Files:** `Attributes.java`, possibly `SubroutineParser.java`
- **Tests fixed:** attrs.t 140, 145
- **Effort:** Small-Medium

#### Phase 6: Error message improvements (MEDIUM — 7 tests)

1. Quote attribute names in error messages with `"attr"` format (test 20)
2. Detect `our ${""}` and `my $$foo` as dereferences before attribute processing (tests 44-45)
3. Ensure `MODIFY_CODE_ATTRIBUTES` die propagates correctly (tests 32, uni/16)
4. Fix "BEGIN failed--compilation aborted" error suffix (tests 155-157)
5. Ensure strict errors take priority over attribute errors (test 87)

- **Files:** `Attributes.java`, `SubroutineParser.java`, parser error handling
- **Tests fixed:** attrs.t 20, 32, 44-45, 87, 155-157; uni/attrs.t 8, 16, 20-21, 23
- **Effort:** Medium

#### Phase 7: Closure prototype feature (LOW — 4 tests)

PerlOnJava does not have the "closure prototype" concept that Perl 5 has. In Perl 5, when a named sub is compiled that closes over lexical variables, the initial CV (before cloning) is a "closure prototype" — it has the captured variable slots but they are not yet bound to specific pad instances. This prototype is accessible via `MODIFY_CODE_ATTRIBUTES` (`$_[1]` before the sub is fully instantiated). Calling a closure prototype should die with "Closure prototype called".

**What needs to be implemented:**
1. Detect when a RuntimeCode is a closure prototype (has captured variable slots but the closure hasn't been instantiated/cloned yet)
2. In `RuntimeCode.apply()`, check for the prototype state and die with "Closure prototype called" instead of executing the body
3. The prototype should still be referenceable (test 126: `\&{$proto}` should return a ref to it)

**Test details:**
- Test 124: `eval { $proto->() }` — should die with `/^Closure prototype called/`
- Test 125: `eval { () = &$proto }` — should die with `/^Closure prototype called/`
- Test 126: `\&{$proto}` — should return a reference (referencing closure prototype)

- **Files:** `RuntimeCode.java`, possibly `EmitSubroutine.java`
- **Tests fixed:** attrs.t 124-126; uni/attrs.t 30-31
- **Effort:** Medium — requires implementing a new concept in the runtime

#### Phase 8: Attribute::Handlers (LOW — 4 tests)

The infrastructure (`attributes.pm`, CHECK blocks, MODIFY_CODE_ATTRIBUTES) is now in place. The remaining blockers are likely edge cases in glob manipulation, ref-identity comparison, or `undef &sub` syntax within `Attribute::Handlers.pm` internals.

- **Files:** Possibly runtime fixes
- **Tests fixed:** attrhand.t 1-4
- **Effort:** Unknown — needs investigation
- **Files:** Interpreter's variable reference handling
- **Effort:** Medium — requires changes to how lexical subs are dereferenced in the interpreter

### Estimated Final Results

| Phase | Tests Fixed | Cumulative |
|-------|-----------|------------|
| Current | — | 205/244 (84%) |
| Phase 2 | +8 | 213/244 (87%) |
| Phase 3 | +6 | 219/244 (90%) |
| Phase 4 | +4 | 223/244 (91%) |
| Phase 5 | +2 | 225/244 (92%) |
| Phase 6 | +12 | 237/244 (97%) |
| Phase 7 | +4 | 241/244 (99%) |
| Phase 8 | +4 | 244/244 (100%) |
| Status | Tests Passing | Pass Rate |
|--------|-------------|-----------|
| Current (PR #423) | 248/254 | 97.6% |
| If interpreter \&lexsub fixed | 249/254 | 98.0% |

The only remaining attrs.t failure is TODO test 155 (expected failure).

### Open Questions

Expand Down
1 change: 1 addition & 0 deletions docs/about/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Release history of PerlOnJava. See [Roadmap](roadmap.md) for future plans.
open FH, "-|" or exec @cmd;
- Bugfix: parser now handles `@{${...}}` nested dereference in push/unshift.
- Bugfix: regex octal escapes `\10`-`\377` now work correctly.
- Bugfix: `\K` (keep left) assertion now works in `m//` and `s///`.
- Bugfix: operator override in Time::Hires now works.
- Bugfix: internal temp variables are now pre-initialized.
- Optimization: faster list assignment.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/feature-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ my @copy = @{$z}; # ERROR
- ✅ **Unicode Properties**: Add regex properties supported by Perl but missing in Java regex.
- ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, or `{n,m}+`, which disable backtracking, are not supported.
- ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported.
- ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`.
- ✅ **Preprocessor**: `\Q`, `\L`, `\U`, `\l`, `\u`, `\E` are preprocessed in regex.
- ✅ **Overloading**: `qr` overloading is implemented. See also [overload pragma](#pragmas).

Expand Down
4 changes: 4 additions & 0 deletions src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2722,6 +2722,10 @@ void compileVariableDeclaration(OperatorNode node, String op) {
throwCompilerException("Unsupported variable type in list declaration: " + sigil);
}

// Runtime attribute dispatch for list variable declarations.
// Attributes are stored on the parent my/state node, propagate to each element.
emitVarAttrsIfNeeded(node, reg, sigil);

varRegs.add(reg);
wrapWithRef.add(isDeclaredReference);
}
Expand Down
Loading
Loading