Skip to content

fix: detect __DATA__/__END__ and no Module; terminators in source filters#389

Merged
fglock merged 5 commits into
masterfrom
feature/filter-terminator-detection
Mar 27, 2026
Merged

fix: detect __DATA__/__END__ and no Module; terminators in source filters#389
fglock merged 5 commits into
masterfrom
feature/filter-terminator-detection

Conversation

@fglock
Copy link
Copy Markdown
Owner

@fglock fglock commented Mar 27, 2026

Summary

Improves source filter handling to detect terminator patterns and stop filtering at the appropriate point.

Problem

When applying source filters, Filter::Simple uses terminators like __DATA__, __END__, and no Module; to mark where filtering should stop. Without proper terminator detection:

  • Content after __DATA__/__END__ was being lost or filtered incorrectly
  • Code after no Module; was being filtered when it should pass through unchanged

Solution

After each filter chunk, check if the output ends with a terminator pattern:

  • __DATA__ or __END__ at end of line
  • no ModuleName; at start of line (with optional comment)

When a terminator is detected, append the remaining source unchanged instead of continuing to filter.

Test Results

Filter::Simple tests:

Test Before After
t/data.t ❌ 3/6 ✅ 6/6
t/filter.t ❌ 5/6 ✅ 6/6
t/code_no_comments.t
t/export.t
t/import.t
t/filter_only.t ❌ (pre-existing POD bug)

Note: t/filter_only.t fails due to a separate PerlOnJava bug where content between =end formatname and =cut is not treated as POD.

Test Plan

  • make - all unit tests pass
  • Filter::Simple t/data.t - passes
  • Filter::Simple t/filter.t - passes
  • Log::Log4perl t/049Unhide.t - still passes

Generated with Devin

fglock and others added 2 commits March 27, 2026 17:37
…ters

When applying source filters, detect when the filtered output ends with
a terminator pattern (__DATA__, __END__, or "no Module;") and stop
filtering at that point. The remaining source after the terminator is
appended unchanged.

This is important for Filter::Simple which uses these terminators to
mark where the filter should stop processing. Without this fix:
- Content after __DATA__/__END__ would be lost or filtered incorrectly
- Code after "no Module;" would be filtered when it should pass through

Fixes:
- Filter::Simple t/data.t - DATA section now preserved correctly
- Filter::Simple t/filter.t - "no Module;" terminator now handled

Note: t/filter_only.t still fails due to a separate POD parsing bug
(content between =end and =cut not treated as POD).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Per perlpod documentation, =end formatname only ends a =begin block,
but does not exit the POD section. Content between =end and =cut
should still be treated as POD documentation, not code.

This fixes Filter::Simple filter_only.t which uses =begin/=end blocks.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock fglock force-pushed the feature/filter-terminator-detection branch from fe6a95b to 9153c9b Compare March 27, 2026 16:37
fglock and others added 3 commits March 27, 2026 17:52
…odel

In native Perl, source filters process remaining source incrementally
during parsing, so each filter completes before the next filter module
is loaded. In PerlOnJava, we tokenize upfront then apply filters, so
multiple filter modules may be loaded before filters run.

This caused @Transforms (a package variable in Filter::Simple) to
accumulate transforms from different modules. When the second filter
ran, its $multitransform closure would include the first module
transforms, causing incorrect behavior.

Fix: Make @Transforms lexical in FILTER_ONLY so each call has its own
transform list. Mark Filter::Simple as protected in config.yaml.

Also adds Filter::Simple to bundled modules (with tests).

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock fglock merged commit ba110f9 into master Mar 27, 2026
2 checks passed
@fglock fglock deleted the feature/filter-terminator-detection branch March 27, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant