Stabilize user-defined functions

[casey]

User-defined functions were added in #3247 with the following syntax:

foo(a, b) := a + b

They are currently unstable, since I'm not sure about the syntax. The alternative that I'm considering is:

fn foo(a, b) := a + b

I personally think the syntax without fn looks better, but with fn allows grepping for function definitions with ^fn, however I'm not sure if that's a huge advantage.

Feedback on the syntax is most appreciated, in either direction. I'd like to stabilize user-defined functions soon, since it's only the syntax which is holding them back.

[hrumhurum]

Mathematically speaking, function and data are the same thing. Using that assumption, the syntax without fn is more preferable.

Historically, math prevails even when we have strong personal opinions.

[CaptaiNiveau]

I mean if it's just about grepping, you could just do ^\w+\( and be set, too. I'll play with it tomorrow, but I think I agree with @hrumhurum

[casey]

The other issue that I didn't think of is disambiguation in the expression language, if the expression language ever permitted function definitions.

In the context of expressions f(a,b) is a valid function call, so to disambiguate a function call from a function definition f(a,b) := ... you would have to use an unlimited amount of lookahead, whereas fn f is already disambiguable.

[CaptaiNiveau]

Hm really? Couldn't you do ^\w+\([^\)]\) := to ensure it's a function definition? It wouldn't trigger for nested function calls, either. Not sure what you're referring to with expression language tho - asked llm, something about the parser?

[casey]

I'm not really being precise. In Just, the parser operates in a few contexts. One is at the top level, when it's parsing items like recipes, settings, assignments, and function definitions. Then, in certain places within items, it starts parsing expressions, which I'm imprecisely calling "the expression language". This happens in a few places, for example, on the right side of := in assignments, or in recipe argument defaults.

At the top level, IDENT ( is not ambiguous, because a function call cannot appear in that position. When parsing an expression, IDENT ( is ambiguous, because a function call can appear in that position.

The parser doesn't use regular expressions to recognize what it's about to parse, just because regular expressions are super fragile. It processes the text one token at a time, and uses some amount of lookahead, i.e., looking at tokens past the current token, to disambiguate what's coming next.

An example which would break the regular expression is:

foo() \
  := "bar"

A function definition with a line continuation before the :=. The line continuation is just whitespace, so it doesn't bother the parser, but it does break the regular expression.

Another example is:

foo(a=("a")) := a

In this case, the default argument ("a") contains a closing-paren, which causes the regex not to match, even though it is a function definition.

In general, regular expressions just aren't powerful enough to make parsing decisions. Especially HTML ;)

For the parser to disambiguate a function definition from a function call, without a leading fn, it has to look ahead a potentially unbounded amount of tokens, which is pretty rough.

This already happens when parsing a recipe at the top-level, since recipes can be named after keywords (set, alias, etc, are valid recipe names), so we have to disambiguate with lookahead, which is super gross, and a source of bugs.

Fun fact, one of the very few non-backwards compatible changes I made when just hit 1.0 was changing assignment from using = to using :=, because = can appear in recipe signatures to introduce argument defaults, and having = be used both there and in assignments made disambiguation really awful.

[hrumhurum]

For a manual immediate parser it makes sense to use a keyword for disambiguation. For example, this is the way Pascal language was designed.

Manual language parsing may be fast, but it's rough in terms of development and maintenance. Usually, language developers define a language grammar file, which is then processed by an automated tool producing the actual parser code. When you work on a grammar level, doing a lookahead is a piece of cake - the implementation will be autogenerated and will be reasonably fast.

To fn or not to fn is the question of the project ideology:

1. If you prioritize fast manual parsing without lookaheads, having a fn keyword is a way to go forward
2. Otherwise, fn is not strictly required as it has no purpose neither for the language nor for its users

Since just is a command runner, favoring a syntax without lookahead aligns well with the simplicity and philosophy of traditional Unix tools.

[duncanawoods]

The binary might be task runner but the language is read and written by humans using other tools.

Generally I'd recommend designing syntax with highlighting / LSP support as core concerns. IMHO they are table-stakes for a modern usable language and developing them in tandem can guide you away from some language-design clangers.

Consider:

• a textmate grammar (archaic but widely supported)
• a tree-sitter grammar (quickest path to creating an LSP)

I have no comment on fn but were any proposed syntax tortuous to support then I'd consider it a red-flag for the language rather than "someone else's problem".

---

FYI current IDE support for just is not great : extensions are unmaintained, break on certain syntax or newer AI slop that I would trust less than an emailed executable. Given the risks of malicious extensions, I hope for IDE support from language developers these days.

If it helps, this is the textmate grammar of the most popular vscode extension with 60k users despite being unmaintained for years and highlighting that can fail catastrophically:

https://github.com/skellock/vscode-just/blob/master/syntaxes/just.tmLanguage.json