Template Based Processor

[ianjosephwilson]

• Rewrite Processor To Be Template Based
  • Enumerate interpolation cases to simplify logic and establish framework to hone correctness, specially around text types.
• Move Node Processor
  • Create submodule nodes/
    • Move Node processor into nodes/processor.py
    • Move nodes.py into nodes/nodes.py
  • Sync tests between the node processor and the str processor using parameterized tests.
• Change Component Signature(s)

def FunctionComponent(children: Template, **kwargs) -> Template:
    return t""
def FactoryComponent(children: Template, **kwargs) -> Callable[[], Template]:
    def ComponentCallable() -> Template:
        return t""
    return ComponentCallable

• Change Component Children Processing Order
  • Children is a Template so the actual contents are not processed until after the component is invoked.
• Change Content Interpolation Rules
  • Zero-arg functions no longer have special treatment:
    • to_html(t'<div>{(lambda: "dynamic")}</div>') != '<div>dynamic</div>'
    • A new formatter named callback can be used which then interpolates the return value
      • to_html(t'<div>{(lambda: "dynamic"):callback}</div>') == '<div>dynamic</div>'
  • Boolean values no longer have special treatment:
    • to_html(t'<div>{False}</div>') == "<div>False</div>"
    • None is still ignored: to_html(t'<div>{None}</div>') == "<div></div>"

[AhnafCodes]

Move Node processor into nodes.py

Current state: nodes.py contains only data types. processor.py contains all resolution logic.

Current separation is clean — nodes.py is "output types" and processor.py is "template resolution." Moving Node processing logic into nodes.py would blur this(eventually if not immediate)
Currently processor.py → nodes.py one-way dependency is cleaner.

[AhnafCodes]

Why change API signature, html -> to_html ?

i would not mind html -> H atleast terse
H(t'<div>{count}</div>'), but why?

name collisions? but, then one can always do import html as to_html

[AhnafCodes]

to_html(t'<div>{False}</div>') == "<div>False</div>"

this breaks {condition and template} or {condition and some_func() } which pretty common used in templating.

[ianjosephwilson]

@davepeck @pauleveritt Can you guys take another look at this when you get a chance? It is a very large conceptual change RE: the component signature and the default processor.

I walked back the more ambitious pre-compiled template concept and now both processors are using TNode directly during processing. I have both implementations lined up with each other, tdom/processor.py:to_html and tdom/nodes/processor.py:to_node. They are using shared tests.

The interpolation value options are more strict because I was trying to makes the rules more well defined.

[pauleveritt]

Thanks for all the work @ianjosephwilson I'll see if @davepeck has time this week to go through it together. (Dave: Any afternoon/evening my time is fine.)

[davepeck]

@ianjosephwilson Thank you for this -- I think I have time this week to re-focus energies and read over in more detail. How does this PR relate to the other open and draft PRs we currently have?

(@AhnafCodes -- thank you for hopping in here, and nice to have you on board! I agree that {condition and "foo"} is something we want to support. It's a very common idiom in JSX-land and, when in doubt, I think we should draw our intuitions from there.)

[ianjosephwilson]

@ianjosephwilson Thank you for this -- I think I have time this week to re-focus energies and read over in more detail. How does this PR relate to the other open and draft PRs we currently have?

@davepeck At this point it might be easier to just browse the processor.py and processor_test.py files instead of diffs to get a feel for what changed. I think these are the major things:

---	Iterative Processor?	Precompilation?	TNodes parsed?	Internally uses Node?	Template Component Signature	Reduced Dynamic Intepretation
This PR	Yes	No	Yes	No	Yes	Yes
Transformer POC (closed)	Yes	Yes	Yes	No	Yes	Yes
Main	No	No	Yes	Yes	No	No

• Iterative Processor
  • Uses a stack/queue with a loop to process the templates and writes the html strings out as it goes. Usually this uses less memory, keeps the internals internal and doesn't have issues with the function stack limit (although that would probably be unlikely).
• Precompilation
  • Instead of walking the TNode tree every processing call and dynamically dispatching an interpolation "handler" the handlers were mostly pre-determined and stored in a "precompiled" Template. This "flattened" the processing step into simply walking a Template instead of traversing the TNode tree. You foresaw this as being too complicated and I think the SVG issue was really the straw that broke the camels back, at least for now.
• Internally use Nodes
  • The prior Node processor would create a tree of Nodes and then "stringify" the nodes into a string.
• Template Component Signature
  • Simplified component interface where children and output are instances of Template
• Reduced Dynamic Intepretation
  • Component output is more like a "tag" and less like a "primitive value"
  • Tried to reduce the number of checks per interpolation (for performance, api surface area, debugging, verification, etc.)
    • No special handling for booleans
    • No special handling for callables
    • No special handling for Node

[ianjosephwilson]

@davepeck Would it be easier if I broke some of the smaller less related pieces out of this and we merged those first?

[davepeck]

@davepeck Would it be easier if I broke some of the smaller less related pieces out of this and we merged those first?

I'm sort of working through it in one fell swoop right now. Thanks for all the guide posts here. I do think there are some separable pieces we could consider teasing apart into separate PRs, but I dunno... that may be more work than it's worth.

[ianjosephwilson]

I put some smaller changes in #109 to reduce the size of this PR. I can rebase it if those go in.

[davepeck]

I just merged in the main branch, which now has the SVG work and has several tool updates. ruff and ty, in particular, get big new versions that had several new complaints to address. Think we're back to sanity; sorry for the perturbation here.

[ianjosephwilson]

@davepeck Okay, I didn't have great luck with ty. pyright seemed more mature for handling more complicated edge cases. Maybe most of the thrashing was from a ruff update though right?

I would probably put the cast() calls back in, at least in the short term, I'm not sure why those were stripped out. They were there to help dial in the interpolation behavior/typing during this finalization phase because otherwise the interpolation values have no type.

Do you think you're still thinking about merging this? Or did you have questions about it?

[davepeck]

The cast() calls led to strange complaints from the latest ty but I agree, they probably belong back in there for clarity. Will do.

Stepping back: I've had lots of conversations in the past ~6 weeks with developers about what they actually want. Here are my takeaways:

1. They want html() to return a final rendered string; no point in returning intermediate types like Node
2. They don't care about perf so long as it's roughly the same as Django, which is to say: really slow.
3. They won't use tdom until key tools — formatting, linting, syntax highlighting, and (some day) type checking — "just work" in their editor (which these days means: vscode)
4. They need precise documentation that their favorite coding agent can also use to write valid HTML t-strings and take advantage of features like attribute spreading, etc.
5. They want CSS support; exactly what this looks is TBD but, at minimum, there's probably a css() and some story for <style> tags in HTML

Your PR addresses (1) and says "we can do better" for (2). Which... good!

I'm still sitting on this PR. Directionally, it's where we want to go. But: to merge it is to commit to maintaining it; I'm not ready yet. I'm still exploring variations on this PR's theme.

[ianjosephwilson]

The cast() calls led to strange complaints from the latest ty but I agree, they probably belong back in there for clarity. Will do.

Stepping back: I've had lots of conversations in the past ~6 weeks with developers about what they actually want. Here are my takeaways:

1. They want html() to return a final rendered string; no point in returning intermediate types like Node

Okay, I'm just going to take Node out then. It takes a lot of code and work to keep in here. We could experiment with middleware in some of the ways I suggested before or bring Node back if needed.

2. They don't care about perf so long as it's roughly the same as Django, which is to say: really slow.

Performance is definitely on the back burner now but these changes do remove the recursion because that was traditionally a big memory problem in Python but maybe it is not anymore? I think it is easy to remove and would guarantee some level of stability so I think it is worth it. I would be open to new information though.

3. They won't use tdom until key tools — formatting, linting, syntax highlighting, and (some day) type checking — "just work" in their editor (which these days means: vscode)

I was pushing for more well defined behavior for this reason which is a big part of these changes: #106

4. They need precise documentation that their favorite coding agent can also use to write valid HTML t-strings and take advantage of features like attribute spreading, etc.

The documentation will need to be adjusted but that is blocked by carving out well defined behavior.

5. They want CSS support; exactly what this looks is TBD but, at minimum, there's probably a css() and some story for <style> tags in HTML

This seems like this could be an addition or extension but it shouldn't block current progress. It would be nice to be able to use tdom without further dependencies if possible.

Your PR addresses (1) and says "we can do better" for (2). Which... good!

I'm still sitting on this PR. Directionally, it's where we want to go. But: to merge it is to commit to maintaining it; I'm not ready yet. I'm still exploring variations on this PR's theme.

I'm going to draft this and try to put out a better draft. Let me know if you think meeting again could help sort out some of the issues.

[davepeck]

Okay, I'm just going to take Node out then.

Yeah, that's the right answer, I think.

these changes do remove the recursion

Yeah, and to be clear I'm in favor of removing recursion if there's a low-complexity way to do so. I don't quite feel we've gotten there yet. It's also probably worth calling out that: what the real-world benefits are is... harder to say at the moment. Comes back to having actual users.

[pauleveritt]

Anything I can do to try and keep a tdom-node package alongside? I can read from a rendered template, but perhaps I can learn more about the template internals and do something smarter?

About tracking component location and static paths, I guess I need to give up on this and come up with a workaround, make people do it manually.