Skip to content

Conversation

@madeline-scyphers
Copy link
Member

feat: add new DRAM rule parser submodule for traits and distill
New python Lark base rule parser that defines rule grammer
for traits grammer that can be reused for distill and product.
This rule parser is more accurate and less error prone than
the older traits parser completely custom coded. THis fixes
a number of bugs where things were being double counted and not
parsed correctly. This also allows distill, traits, and product
eventually to all use the same rule parsing code.

Rule parser is implemented with polars instead of Pandas for a
few reasons. It is a bit faster, and it allows lazy query planning.
The annotation df is not currently lazy (eager rn), but with plans
to allow it to be lazy. Lazy DataFrames can be more memory efficient
by only loading the data/columns needed, and doing query optimization
to speed up and require less memory for intermediate steps.

New python Lark base rule parser that defines rule grammer
for traits grammer that can be reused for distill and product.
This rule parser is more accurate and less error prone than
the older traits parser completely custom coded. THis fixes
a number of bugs where things were being double counted and not
parsed correctly. This also allows distill, traits, and product
eventually to all use the same rule parsing code.

Rule parser is implemented with polars instead of Pandas for a
few reasons. It is a bit faster, and it allows lazy query planning.
The annotation df is not currently lazy (eager rn), but with plans
to allow it to be lazy. Lazy DataFrames can be more memory efficient
by only loading the data/columns needed, and doing query optimization
to speed up and require less memory for intermediate steps.
@madeline-scyphers madeline-scyphers added enhancement New feature or request package Package structure refactor Refactoring without or minimal functional changes labels Jan 21, 2026
@github-project-automation github-project-automation bot moved this to To Sort in DRAM Jan 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request package Package structure refactor Refactoring without or minimal functional changes

Projects

Status: To Sort

Development

Successfully merging this pull request may close these issues.

2 participants