Feature/dram ag distill with new rule parser for distill, traits, and product #480
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat: add new DRAM rule parser submodule for traits and distill
New python Lark base rule parser that defines rule grammer
for traits grammer that can be reused for distill and product.
This rule parser is more accurate and less error prone than
the older traits parser completely custom coded. THis fixes
a number of bugs where things were being double counted and not
parsed correctly. This also allows distill, traits, and product
eventually to all use the same rule parsing code.
Rule parser is implemented with polars instead of Pandas for a
few reasons. It is a bit faster, and it allows lazy query planning.
The annotation df is not currently lazy (eager rn), but with plans
to allow it to be lazy. Lazy DataFrames can be more memory efficient
by only loading the data/columns needed, and doing query optimization
to speed up and require less memory for intermediate steps.