added generation of type specific 'MarshalJSON' & 'UnmarshalJSON' implementations by sailingKieler · Pull Request #98 · TypeFox/fastbelt

sailingKieler · 2026-06-08T07:02:49Z

Contributes support for exporting and importing ASTs in Json.
Includes:

a generator component producing json_gen.go files
- they contribute methods turning AstNodes into impls of json.Marshaler & json.Unmarshaler
generic MarshalJSON & UnmarshalJSON implementations for Reference
util/json/json.go demonstrates the procedure of importing an AST in json into a Fastbelt container
a simple round trip AST export and import test including cross references to other documents (circular) by means of the Arithmetics test language,
- focus is on successfully resolving the (cross document) cross-references
a roundtrip AST export and import test (no external cross-refs by now) by means of the Grammar language,
- focus is on completeness of feature processing and cardinalities:
  boolean, string, AstNode, Reference x 1, ?, *, +

Note:

Minor supplements in fixture.go
Fix of a potential nil pointer de-reference in symbol.go

There's one known potential issue:
During the import of an AST in json the Token instances are always initialized with token type Token_ID.
The leads to a compiler issue if there is no ID token type.

We could improve this by an analysis of the Parser rules, but that will never be complete.
Imagine the grammar defines interfaces of types that are never instantiated by parser rules, but the data are always imported vom Json snapshots.
To mitigate this we might offer an opportunity in the grammar to determine a token type for fields of type string or composite. To be discussed.

spoenemann · 2026-06-10T12:32:14Z

+	}
+	// We expect at this point that all references have been attempted to resolve.
+	if !r.resolved.Load() {
+		return nil, errors.New("reference not resolved")


Proposal: generate a partial reference like in Langium:

https://github.com/eclipse-langium/langium/blob/7ad7c76e85134185146e1a99bd352e7527ed1fec/packages/langium/src/serializer/json-serializer.ts#L174-L177

This is just preliminary, I don't know which context instance to rely on for calling Ref(...), yet
(since this function is called by json.Mashall(...), not by ourselves), and whether we should attempt to resolve here, at all.

AFAICS, r.resolved is true if resolution has been attempted, regardless of success or failure.
Hence, this guards stops the serialization, e.g., if the document hasn't been build until DocStateLinked at least.

spoenemann · 2026-06-10T12:33:20Z

+		RefText string `json:"refText"`
+		URI     string `json:"uri,omitempty"`
+		Err     string `json:"error,omitempty"`
+	}{


This struct def should be exported. And I propose aligning this with Langium for cross-framework compatibility:

https://github.com/eclipse-langium/langium/blob/7ad7c76e85134185146e1a99bd352e7527ed1fec/packages/langium/src/serializer/json-serializer.ts#L96-L105

This struct def should be exported.

Why?
It's purely technical and kind of a vehicle.
It might make sense to rely in the two (Un)Marshal methods on the same definition, though.

And I propose aligning this with Langium for cross-framework compatibility:

That's my plain, too.

spoenemann · 2026-06-10T12:43:10Z

+		Err     string `json:"error,omitempty"`
+	}{
+		RefText: r.unit.String(),
+		URI:     uri,


In Langium, we omit the URI if the reference target is in the same document. This can reduce the size of generated JSON by a lot.

Thought: can we include something like a URI converter here as well? Exporting JSONs with absolute file URIs would be pretty useless in many cases.

https://github.com/eclipse-langium/langium/blob/7ad7c76e85134185146e1a99bd352e7527ed1fec/packages/langium/src/serializer/json-serializer.ts#L161-L167

In Langium, we omit the URI if the reference target is in the same document. This can reduce the size of generated JSON by a lot.

That's achieved by the omitempty setting. Of course, the prop value shouldn't be set upfront.

Thought: can we include something like a URI converter here as well? Exporting JSONs with absolute file URIs would be pretty useless in many cases.

Yes, that's my plan.

sailingKieler

Thanks for the hints Miro.
This state is a very early one, it doesn't represent the one I have in mind already.

I also seek for a solution that is compatible with Langium, s.t. we could use it as an interface for data exchange

More changes are coming soon :-)

sailingKieler · 2026-06-10T13:17:38Z

+		Err     string `json:"error,omitempty"`
+	}{
+		RefText: r.unit.String(),
+		URI:     uri,


In Langium, we omit the URI if the reference target is in the same document. This can reduce the size of generated JSON by a lot.

That's achieved by the omitempty setting. Of course, the prop value shouldn't be set upfront.

Thought: can we include something like a URI converter here as well? Exporting JSONs with absolute file URIs would be pretty useless in many cases.

Yes, that's my plan.

sailingKieler · 2026-06-10T13:21:09Z

+		RefText string `json:"refText"`
+		URI     string `json:"uri,omitempty"`
+		Err     string `json:"error,omitempty"`
+	}{


This struct def should be exported.

Why?
It's purely technical and kind of a vehicle.
It might make sense to rely in the two (Un)Marshal methods on the same definition, though.

And I propose aligning this with Langium for cross-framework compatibility:

That's my plain, too.

sailingKieler · 2026-06-10T13:28:15Z

+	}
+	// We expect at this point that all references have been attempted to resolve.
+	if !r.resolved.Load() {
+		return nil, errors.New("reference not resolved")


This is just preliminary, I don't know which context instance to rely on for calling Ref(...), yet
(since this function is called by json.Mashall(...), not by ourselves), and whether we should attempt to resolve here, at all.

AFAICS, r.resolved is true if resolution has been attempted, regardless of success or failure.
Hence, this guards stops the serialization, e.g., if the document hasn't been build until DocStateLinked at least.

Lotes · 2026-07-15T07:32:41Z

Just an idea: We are only generating a JSON here, right?
If someone wants to use the output, he needs to model the meta model, first. Or he hopes everything is like in his imagination.
A lot of languages understand JSON Schema and can generate type information from the schema. Then, deserializing and validating the output should be very simple.

Lotes

Here are some comments. In the reference part I was not so deep.
I hope it helps :-) ...
About the mentioned "embed trick", I am not sure whether there are downsides. Using it within the tests should be fine.

Lotes · 2026-07-15T07:34:17Z

+
+func GenerateJSON(grammar grammar.Grammar, packageName string) string {
+	node := codegen.NewNode()
+	node.AppendLine("// Code generated by typefox.dev/fastbelt/cmd/fastbelt. DO NOT EDIT.")


We could move this line to a constant that is shared over all generators.

Actually, this is AI generated code.
In case we need to update it we wanna prompt a coding agent to update it consistently, don't we? 😉

Lotes · 2026-07-15T08:46:00Z

+	builder := service.MustGet[workspace.Builder](f.Services())
+
+	t.Run("selfContained", func(t *testing.T) {
+		t.Cleanup(cleanup)


In my eyes it looks a bit weird that you clean up before doing something.

defer t.Cleanup(cleanup) would clean up after the test was processed.

From the docs of *common.Cleanup(...):
Cleanup registers a function to be called when the test (or subtest) and all its subtests complete.

Thus, it is supposed to do what we expect, i.e., do clean up once the test is completed.

Lotes · 2026-07-15T08:51:26Z

+	})
+}
+
+const selfContainedJson = `{


We could use embedded files here.
Then the "code space" gets not polluted with big escaped JSON strings.
But I am also fine like it is now. Just wanted to show an opportunity to save > 200 LOC that are actually a different language.

As discussed in the meeting, it won't be super beneficial here but should keep that opportunity in mind for more complex tests.

Lotes · 2026-07-15T08:57:22Z

+		var stringListFields = map[string]string{}
+		for _, field := range fields {
+			if field.Array && (field.GType == TOKEN_TYPE || field.GType == COMPOSITE_TYPE) {
+				varName := strings.ToLower(field.Name[:1]) + field.Name[1:]


Move out repeating snippet like name concatenation to a function.
Or may collect frequently used variable names, instead of computing them in place.
For: varName, jsonTag.
Also because you do not have to change N places if you decide that the names shall change.

Sourced strings.ToLower(field.Name[:1]) + field.Name[1:] out to the composition of the FieldInfo description data in type_generator.go.

Lotes · 2026-07-15T09:08:15Z

+}
+
+func generateDispatchingUnmarshalFunc(node codegen.Node, g grammar.Grammar) {
+	node.AppendLine(`// Unmarshal decodes data into an instance of type T by reading the "$type" field,`)


This function outputs a string that differs only in the grammar name.
I am wondering if we can generate this part in a less verbose way.
Actually it is like a utility function that could almost live in a separated file besides the generated functions. Can we move it out the generated folder? The advantage would be that you get displayed syntax errors for free.

Indeed, it is basically a utility function, and I had it in the framework first.
However, it appeared fairly orphaned there with almost no context, so I added it to the generated code, also because it's called in the generated code.
This way it will always be consistent with the generated code.
It's a matter of taste.

I'm in favor of moving this function to the framework by expecting the syntheticFactories map as an argument.

Lotes · 2026-07-15T09:10:37Z

+}
+
+func generateFunctions(grammar grammar.Grammar, node codegen.Node) {
+	node.AppendLine(("func newToken(tokenType *core.TokenType, view string) *core.Token {"))


This utility function does not differ from other generations. It would make sense to move it outside the generated folder.

It also a utility that is tightly coupled to its callers that we might change depending on the open question regarding the token type, therefore it's non-exported.
In the face of #128 we might even drop it and inline its impl.

Lotes · 2026-07-15T09:11:51Z

+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			grammmar, err := os.ReadFile(tt.path)


Think of using the embed trick from above?

I though about this for a moment, but it doesn't really make sense to me because of:

this is a test, not production to be bundled into a self-contained executable.

I would have to create a string var for each file to embed.

I want the execution to fail with an ordinary test failure instead of some compiler error if any file is absent/any path is wrong

Lotes · 2026-07-15T09:14:12Z

+// assertEqualAst recursively compares two AST subtrees node by node. Diffs are
+// reported via t.Errorf with the node's document path from core.PathOf.
+// Returns true when at least one difference was detected.
+func assertEqualAst(t testing.TB, expected, actual core.AstNode) bool {


It looks cleaner if these helper function would live in a own file.
But I think it is ok FTM, as long as there is only one test file that uses them.

Moved it into a separate file named test/doc_fixture_util.go, as the function seems to be too long & complex to add it to one of the existing files in test.

Lotes · 2026-07-15T09:18:32Z

+// 'err' contains the error msg if the reference was unresolvable before serializing
+// With this function Reference[T] implements json.Unmarshaler.UnmarshalJSON() and the method is called by `json.Unmarshal()`.
+func (r *Reference[T]) UnmarshalJSON(data []byte) error {
+	aux := &struct {


You should make a private type here, because it is repeating.

That private type will be visible in the entire Fastbelt core package, which doesn't make any sense.
Yes we could chose a name the makes misuses obvious but, honestly, I don't see any benefit in that.

…lementations * also added generic 'MarshalJSON' & 'UnmarshalJSON' implementation for 'Reference'

…abling re-use

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite	Current: `c7832b9`	Previous: `d803417`	Ratio
`BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s`	`13.02` MB/s	`5.21` MB/s	`2.50`

This comment was automatically generated by workflow using github-action-benchmark.

spoenemann

Great! I like the overall approach.

spoenemann · 2026-07-21T11:34:22Z

+// This program and the accompanying materials are made available under the
+// terms of the MIT License, which is available in the project root.
+
+package fastbelt


The package name is inconsistent with the containing folder name. In addition, json is not a good package name because it conflicts with the standard library package. As this file currently exports only one function, I suggest moving it to the root of this module.

spoenemann · 2026-07-21T11:56:29Z

+// UnmarshalAndBuildDocument uses the "encoding/json" entry point to unmarshal rootNode based on the given data string and builds the document.
+// Similar to parsing-based document loading the ast build up first including the reference, while the resolution of the references is done during the linking phase of the building process.
+// For properly linking references to other documents, a helper object is attached to the context given to builder providing access to other documents.
+func UnmarshalAndBuildDocument[T core.AstNode](sc *service.Container, document *core.Document, rootNode T, data []byte, ctx context.Context) error {


The ctx parameter should be the first per context package convention.

Wouldn't it be more useful to split the "Unmarshal" and the "Build" part into separate utilities? The former could go to the fastbelt (root) package, the latter to the workspace package.

Actually, my intention is to give a blue print how to do unmarshaling with cross-ref resolution.
We could move it into the docs, too, but then it's not compile checked anymore.

Therefore, I was hesitant to put it into the root package and pollute the namespace with these nice-to-have parts.
I'm fine with putting into util.

spoenemann · 2026-07-21T12:24:01Z

+			// in case a document in the document manager hasn't been build yet or has been reset
+			// and is not included in the current build cycle its exported symbols container may be absent;
+			// map 'nil' to an empty seq in such cases
+			return func(yield func(*SymbolDescription) bool) {}


We have EmptySymbolDescriptions for this. BTW shall we rename that to EmptySymbolSeq to be consistent with the SymbolSeq type?

BTW shall we rename that to EmptySymbolSeq to be consistent with the SymbolSeq type?

Makes sense IMHO.

spoenemann · 2026-07-21T12:29:25Z

+	return nil
+}
+
+type JsonLinkingHelper interface {


I'd consider making all these helper definitions private, including the NewJsonReferenceGetter function below. We can still make them public later if we see legitimate use cases.

In any case, please add some documentation to explain how these things work together.

including the NewJsonReferenceGetter function below.

Makes sense, the uppercase initial is probably a leftover.

sailingKieler · 2026-07-22T15:22:32Z

@spoenemann, @msujew One thing I'm still undecided about:
We could add the interfaces [encoding/]json.Marshaler and [encoding/]json.Unmarshaler to the generated AstNode interfaces, i.e. here:

fastbelt/internal/generator/type_generator.go

Lines 176 to 178 in b789ac2

    
           node.AppendLine("type ", iface.Name(), " interface {") 
        
           node.Indent(func(n codegen.Node) { 
        
           	n.AppendLine("core.AstNode")

Then we can call MarshalJSON() and UnmarshalJSON() directly without casting or using the reflective API.
However, formatting is only provided by json.MarshalIndent(...), and for type dispatching while reading and unmarshaling without a pre-created instance we need to use the (currently generated) generic Unmarshal[...](...) (maybe UnmarshalAstNode?) function.
Therefore I didn't add the interfaces in type_generator.go, yet.
What do you think?

spoenemann reviewed Jun 10, 2026

View reviewed changes

sailingKieler commented Jun 10, 2026

View reviewed changes

sailingKieler force-pushed the cs/json-marshalling branch 2 times, most recently from 23336ec to 10ec511 Compare June 17, 2026 18:23

msujew mentioned this pull request Jun 23, 2026

Fix completion after grammar action #116

Merged

sailingKieler force-pushed the cs/json-marshalling branch 4 times, most recently from f72abbc to cc2a141 Compare July 9, 2026 13:08

sailingKieler marked this pull request as ready for review July 9, 2026 13:30

sailingKieler force-pushed the cs/json-marshalling branch from cc2a141 to 8ca6a6c Compare July 9, 2026 13:45

Lotes reviewed Jul 15, 2026

View reviewed changes

sailingKieler added 15 commits July 20, 2026 12:53

added generation of type specific 'MarshalJSON' & 'UnmarshalJSON' imp…

9100115

…lementations * also added generic 'MarshalJSON' & 'UnmarshalJSON' implementation for 'Reference'

progress on 'json_generator.go'

dad7938

unmarshaling

8034706

improvement in 'util/json.go'

159c628

hot fix in reference.go

8107a78

progress, linking seems to work

63d94b9

refinements

bc88da5

more refinements

470ae50

first json test

d5a1707

fix in symbols handling in 'symbols.go'

69b0365

some supplements to 'fixture.go'

009da0d

json roundtrip test with/for grammar language

fd301e3

refined benchmark to capture relevant measures only

058f223

optimization of slice allocation during unmarshaling

666503a

updates after rebase, moved 'AssertEqualAst' into '/fastbelt/test' en…

2ffe69e

…abling re-use

sailingKieler force-pushed the cs/json-marshalling branch from 631fd58 to 666503a Compare July 20, 2026 11:33

github-actions Bot reviewed Jul 20, 2026

View reviewed changes

sailingKieler force-pushed the cs/json-marshalling branch from 84e47c7 to c1e8c43 Compare July 20, 2026 12:54

some consolidation suggested in review

c7832b9

sailingKieler force-pushed the cs/json-marshalling branch from c1e8c43 to c7832b9 Compare July 20, 2026 12:58

spoenemann reviewed Jul 21, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

sailingKieler commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sailingKieler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lotes commented Jul 15, 2026

Uh oh!

Lotes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sailingKieler Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

Uh oh!

spoenemann left a comment

Choose a reason for hiding this comment

Uh oh!

sailingKieler commented Jun 8, 2026 •

edited

Loading

sailingKieler Jul 20, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

sailingKieler Jul 22, 2026 •

edited

Loading