Adding a new language

[kubukoz]

Hi! I'm trying to add a new language in my cloned version of this repository.

I ran ./gradlew assemble, then made sure I can run the build script for the included Python grammar (./build.py -o out src/test/tree-sitter-python was successful), but when I tried it with my language there was no success.

I made these changes:

// in ai_serenade_treesitter_Languages.cc, added this

#ifdef TS_LANGUAGE_SMITHYQL
extern "C" TSLanguage* tree_sitter_smithyql();
JNIEXPORT jlong JNICALL
Java_ai_serenade_treesitter_Languages_smithyql(JNIEnv* env, jclass self) {
  return (jlong)tree_sitter_smithyql();
}
#endif

// in Languages.java, added this

public static native long smithyql();

and tried to run build.py with the path to my language's config. It's really simple, I followed the tree-sitter docs to define a trivial grammar (source_file: $ => 'hello') and it was working in the tree-sitter CLI.

When I run ./build.py -o out ../smithy-playground/tree-sitter-smithyql -v, I get this:

Output

/Users/kubukoz/projects/java-tree-sitter/./build.py:5: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.ccompiler
rm -f lib/src/*.o libtree-sitter.a libtree-sitter.dylib libtree-sitter.0.dylib libtree-sitter.0.0.dylib
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/get_changed_ranges.o lib/src/get_changed_ranges.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/language.o lib/src/language.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/lexer.o lib/src/lexer.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/node.o lib/src/node.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/parser.o lib/src/parser.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/query.o lib/src/query.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/stack.o lib/src/stack.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/subtree.o lib/src/subtree.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/tree.o lib/src/tree.c
cc -O3 -Wall -Wextra -Werror -std=gnu99 -fPIC -Ilib/src -Ilib/include   -c -o lib/src/tree_cursor.o lib/src/tree_cursor.c
ar rcs libtree-sitter.a lib/src/get_changed_ranges.o lib/src/language.o lib/src/lexer.o lib/src/node.o lib/src/parser.o lib/src/query.o lib/src/stack.o lib/src/subtree.o lib/src/tree.o lib/src/tree_cursor.o
cc  -dynamiclib -Wl,-install_name,/usr/local/lib/libtree-sitter.0.dylib lib/src/get_changed_ranges.o lib/src/language.o lib/src/lexer.o lib/src/node.o lib/src/parser.o lib/src/query.o lib/src/stack.o lib/src/subtree.o lib/src/tree.o lib/src/tree_cursor.o  -o libtree-sitter.0.0.dylib
ln -sf libtree-sitter.0.0.dylib libtree-sitter.dylib
ln -sf libtree-sitter.0.0.dylib libtree-sitter.0.dylib
creating /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users
creating /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz
creating /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects
creating /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects/java-tree-sitter
creating /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects/java-tree-sitter/lib
c++ -O3 -DTS_LANGUAGE_SMITHYQL=1 -I/Users/kubukoz/projects/java-tree-sitter/lib -I/Users/kubukoz/projects/java-tree-sitter/tree-sitter/lib/include -I/nix/store/hpggzh7afi4qs2nm92gsn4drharb858d-zulu11.48.21-ca-jdk-11.0.11/include -I/nix/store/hpggzh7afi4qs2nm92gsn4drharb858d-zulu11.48.21-ca-jdk-11.0.11/include/darwin -c /Users/kubukoz/projects/java-tree-sitter/lib/ai_serenade_treesitter_TreeSitter.cc -o /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects/java-tree-sitter/lib/ai_serenade_treesitter_TreeSitter.o
c++ -O3 -DTS_LANGUAGE_SMITHYQL=1 -I/Users/kubukoz/projects/java-tree-sitter/lib -I/Users/kubukoz/projects/java-tree-sitter/tree-sitter/lib/include -I/nix/store/hpggzh7afi4qs2nm92gsn4drharb858d-zulu11.48.21-ca-jdk-11.0.11/include -I/nix/store/hpggzh7afi4qs2nm92gsn4drharb858d-zulu11.48.21-ca-jdk-11.0.11/include/darwin -c /Users/kubukoz/projects/java-tree-sitter/lib/ai_serenade_treesitter_Languages.cc -o /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects/java-tree-sitter/lib/ai_serenade_treesitter_Languages.o
cc -O3 -std=c99 -DTS_LANGUAGE_SMITHYQL=1 -I../smithy-playground/tree-sitter-smithyql/src -I/Users/kubukoz/projects/java-tree-sitter/tree-sitter/lib/include -I/nix/store/hpggzh7afi4qs2nm92gsn4drharb858d-zulu11.48.21-ca-jdk-11.0.11/include -I/nix/store/hpggzh7afi4qs2nm92gsn4drharb858d-zulu11.48.21-ca-jdk-11.0.11/include/darwin -c ../smithy-playground/tree-sitter-smithyql/src/parser.c -o /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/../smithy-playground/tree-sitter-smithyql/src/parser.o
cc -shared -dynamiclib /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects/java-tree-sitter/lib/ai_serenade_treesitter_TreeSitter.o /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/Users/kubukoz/projects/java-tree-sitter/lib/ai_serenade_treesitter_Languages.o /var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/tmpuc1m8tdqtree_sitter_language/../smithy-playground/tree-sitter-smithyql/src/parser.o -L/Users/kubukoz/projects/java-tree-sitter/tree-sitter -o out.dylib /Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(node.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(parser.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(tree_cursor.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(tree.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(subtree.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(language.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(lexer.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(get_changed_ranges.o)) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: object file (/Users/kubukoz/projects/java-tree-sitter/tree-sitter/libtree-sitter.a(stack.o)) was built for newer macOS version (12.0) than being linked (11.0)
Undefined symbols for architecture arm64:
  "operator delete(void*)", referenced from:
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorNew in ai_serenade_treesitter_TreeSitter.o
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorDelete in ai_serenade_treesitter_TreeSitter.o
  "operator new(unsigned long)", referenced from:
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorNew in ai_serenade_treesitter_TreeSitter.o
  "___gxx_personality_v0", referenced from:
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorNew in ai_serenade_treesitter_TreeSitter.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Traceback (most recent call last):
  File "/nix/store/v4iwl6az3wax9nw0wl42kjpq4vfdd7j4-python3-3.10.5/lib/python3.10/distutils/unixccompiler.py", line 200, in link
    self.spawn(linker + ld_args)
  File "/nix/store/v4iwl6az3wax9nw0wl42kjpq4vfdd7j4-python3-3.10.5/lib/python3.10/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/nix/store/v4iwl6az3wax9nw0wl42kjpq4vfdd7j4-python3-3.10.5/lib/python3.10/distutils/spawn.py", line 91, in spawn
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/cc' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kubukoz/projects/java-tree-sitter/./build.py", line 137, in <module>
    build(args.repositories, args.output, args.arch, args.verbose)
  File "/Users/kubukoz/projects/java-tree-sitter/./build.py", line 109, in build
    compiler.link_shared_object(
  File "/nix/store/v4iwl6az3wax9nw0wl42kjpq4vfdd7j4-python3-3.10.5/lib/python3.10/distutils/ccompiler.py", line 713, in link_shared_object
    self.link(CCompiler.SHARED_OBJECT, objects,
  File "/nix/store/v4iwl6az3wax9nw0wl42kjpq4vfdd7j4-python3-3.10.5/lib/python3.10/distutils/unixccompiler.py", line 202, in link
    raise LinkError(msg)
distutils.errors.LinkError: command '/usr/bin/cc' failed with exit code 1

Note, I'm running on an ARM Mac - the error seems to suggest that has something to do with it. Can you give me any hints?

Many thanks. The grammar can be found here.

[kubukoz]

ah, I see I was missing a tree-sitter submodule. Adding that helped :)

[tmacwill]

glad everything is working now!

[kubukoz]

Just wondering what the recommended next step would be - should I maintain a fork with these changes, or would you take a PR to add this new language?

[tmacwill]

we can take a PR with a new language. thanks!

[kubukoz]

Update: Looks like it was a false positive, I was getting a successful run but it wasn't actually modifying the dylib - I had built this with the Vue syntax in the meantime, and vue was included in the dylib (smithyql was not).

I tried doing the same with the Scala syntax later, but that fails with the same error:

Undefined symbols for architecture arm64:
  "operator delete(void*)", referenced from:
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorNew in ai_serenade_treesitter_TreeSitter.o
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorDelete in ai_serenade_treesitter_TreeSitter.o
  "operator new(unsigned long)", referenced from:
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorNew in ai_serenade_treesitter_TreeSitter.o
  "___gxx_personality_v0", referenced from:
      _Java_ai_serenade_treesitter_TreeSitter_treeCursorNew in ai_serenade_treesitter_TreeSitter.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[kubukoz]

Apparently this is a Mac issue. The errors suggested I was compiling as C instead of C++ (or at least without some libraries - I don't know much about c/cpp so this is just a guess). I forced the if cpp: branch of build.py to be triggered, and it worked!

Wondering if this should be changed upstream - setting cpp = True makes sense for me (an outsider) because the .cc files in this project seem to use c++ features like the delete/new keywords.

[sogaiu]

@kubukoz Thanks for documenting your experience in this issue.

The errors suggested I was compiling as C instead of C++ (or at least without some libraries - I don't know much about c/cpp so this is just a guess). I forced the if cpp: branch of build.py to be triggered, and it worked!

I think I encountered a similar situation and AFAICT it's not limited to macos.

If the tree-sitter grammar in question happens to have an external scanner written in C++ (so there's a src/scanner.cc in the grammar repository directory), the linking step of libjava-tree-sitter in build.py ends up using -lstdc++.

(The canonical tree-sitter-python -- which is used in the gradle tests -- does have scanner.cc so the built libjava-tree-sitter gets linked in that manner.)

Not every grammar has an external scanner though, and even if there is one, it might be in C.

The grammar I tried with doesn't have an external scanner and (I think as a consequence) building with build.py doesn't produce something that's usable. The kind of error that shows up looks like this:

Exception in thread "main" java.lang.UnsatisfiedLinkError: .../java-tree-sitter/libjava-tree-sitter.so: ...tree-sitter-java/libjava-tree-sitter.so: undefined symbol: __gxx_personality_v0
	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
	at java.base/java.lang.Runtime.load0(Runtime.java:755)
	at java.base/java.lang.System.load(System.java:1953)
	at jts.<clinit>(jts.java:8)
Done

When I searched for undefined symbol: __gxx_personality_v0, I ended up at a helpful StackOverflow question, that had information suggesting the use of -lstdc++ when linking.

Changing the linking step of libjava-tree-sitter to use -lstdc++ fixed the issue I was experiencing.

Wondering if this should be changed upstream - setting cpp = True makes sense for me (an outsider) because the .cc files in this project seem to use c++ features like the delete/new keywords.

If by "changed upstream" you mean "change build.py appropriately to account for C++ usage in the linking step", I'm inclined to agree :)

@tmacwill What do you think?

Update: I see in #18 that there's some code to address the C++ situation discussed here.