Remove include and exclude fields from wesl.toml

[stefnotch]

We don't have any use-cases for those fields. At least I'm not aware of any.

And apparently neither Rust nor Typescript seem to truly require such configs.

Cargo doesn't actually use its includes/excludes field for anything important. That field is purely for publishing.
https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields

And I also noticed that Rust Analyzer only has inclusions and exclusions as a very advanced config. One that is rarely ever used, and is implementation specific. https://rust-analyzer.github.io/book/configuration#vfs.extraIncludes
It ignores the cargo includes/excludes fields.

Typescript is also happy to ignore the "includes" and "excludes" fields. Their exact behaviour seems underspecified to me, but from what I can tell, they're only so that the language server/other tools know which set of files to start with. If I import another file from outside the "includes", then Typescript will be happy to do that. And will start suggesting autoimports from that file everywhere in my project.

dummy-ts.zip

Use case: Excluding tests that are a part of the repository

Bundler specific configuration.

The language server should still analyze those test files. And the code formatter should still format the code in them.

e.g. wesl-rs, will follow the Cargo.toml configuration, and thus end up excluding files that are marked as exclude. Implementation detail: This happens because wesl-rs projects will ship .wesl files, and run the build script on the library user's machine.

Use case: Excluding "snapshot" tests

Just don't put those in your src folder. Then neither the language server, nor the bundler will try to use them.

[mighdoll]

I don't think we can include everything that's in the project directory containing wesl.toml. The users project directory might well include shader files that aren't part of their app: app test shaders, documentation samples, reference shaders temporarily downloaded from the internet, archive copies of legacy shaders, node_modules/, etc.

We could require that users only put shaders into a well known directory like src/ or shaders/, and never put tests or doc examples into there. But I think we concluded earlier that users would quickly want more flexibility to arrange things as they'd like. Lygia for example, wasn't willing to adjust their directory structure to accomodate msl.

TypeScript has flexible include/exclude controls and AFAIK most tools respect the tsconfg.json controls. The TS (and wgsl/wesl) language server may be special though - if the user opens a file not mentioned in the config they don't want to just ignore it.

[mighdoll]

On discord we noted the possibility of using root in wesl.toml to specify one directory that contains the shaders. So the directory we'd scan for shader files wouldn't have to be the root of the project or a magic directory name.

The question is whether specifying one directory is sufficient control vs. the more expressive include/exclude controls we have currently.

[mighdoll]

Here's two quick examples where we'd want more control than just one root to specify the shaders:

• Lygia keeps it's shaders in folders at the root of the project ./math/, ./sdf/, etc. They like their structure and have long resisted the idea of changing the structure for the sake of particular ecosystems. We could set root to ./, but that would include all the random shader files mentioned above we don't want.
• It's quite common the js/ts world to keep tests in the same directory as the source code, or in a subdirectory per source folder. src/foo/test/bar.test.ts, src/foo/bar.test.ts. The test shaders should be near the test code. If we set root to src, tools would find a mix of test shaders and app shaders. But most tools will want to know the difference between test and app (e.g. the cli linker shouldn't include a test shader when linking the app, the doc generator shouldn't generate docs for test shaders, etc.)

More generally, the js/ts world has a lot of variety in how people structure their directories..

[stefnotch]

@mighdoll Good examples. I'd argue that both of them are underserved with excludes right now, since most tools do want to work with those files.
(Language server, code formatter, linter, test runner, ...)

As far as I can tell, the issues in those cases are

• the linker should exclude them from the final bundle
• lygia users should not be able to import test files
• the language server should not suggest imports from the tests in non test files

[mighdoll]

test shaders are interesting to discuss... doc gen, cli linker, bundle extensions, packager tools don't want test shaders. But the lang server, cli formatter, cli linter would want test shaders.

(related: we were musing on discord about possibly handling test shaders specially. perhaps with multiple configs like wesl.toml and test-wesl.toml, or adding dev-include to wesl.toml.)

app test shaders, documentation samples, reference shaders temporarily downloaded from the internet, archive copies of legacy shaders, node_modules/, etc.

no tool wants to include all of these..

Also note that some packages will have in src/foo/test/node_modules, so the simpler test/ example will also drag in the issue of wanting to exclude all the uncontrolled stuff that might be in node_modules/.

[stefnotch]

Regarding test shaders, I can think of 4 different strategies. My preferences is the visibility strategy

• visibility: As long as you don't make your tests public from the project root, then the documentation generator and the bundler would do the right thing. Well, the linkers might still bundle them, but the users of a library couldn't import them. Omitting the tests can be done via dead code elimination.
• conditional compilation: That's how Rust handles tests. With #[cfg(test)]. cargo test enables that, everything else doesn't enable that condition.
• adding an extra wesl.toml file in the tests and docs folders. Then those would be separate.
• dev-include: I haven't seen anything comparable in other ecosystems. But I see no reason why this wouldn't work.

node_modules/

Reasonable, we will want something to exclude that folder. No tool wants to look at it.
That's actually a quite interesting case: All Typescript users would expect node_modules to be excluded by default.

[mighdoll]

lots of ideas! but I'm not seeing a way w/o finer grain controls yet:
visibility - does this work for libraries? libraries don't have a root shader..
conditional - seems pretty verbose compared to alternatives, lots of tools will need to understand std condition set.
extra wesl.toml - doens't solve the case with test files in same dir as src, not uncommon in js land.
dev-include - analogy is to devDependencies, but in any case orthogonal to include/exclude. could discuss further in #160.

So i'm not yet seeing a better solution for the test case above, and we haven't discussed the shaders in project root case..

I like the idea of simplification in principle, but not yet seeing how we can handle the variation in typical js projects without some glob selectors. The effort of supporting include/exclude (as say tsconfig.json) and as currently spec'd doesn't seem too high.. or is it hard for some reason I don't yet see?

[stefnotch]

visibility - does this work for libraries? libraries don't have a root shader..

The imports spec guarantees that there's a root shader. import package refers to a module. Even if it's an implicit one.
So this works for libraries.

conditional - seems pretty verbose compared to alternatives, lots of tools will need to understand std condition set.

Yeah, it'll require a lot of @if(test) statements. Rust circumvents this by being able to do @if(test) mod my_tests;. Aka only including the entire file with tests if that condition is enabled.

Maybe we could do the same with @if(test) import package::my_tests;? But that seems iffy. I guess we have the usual problem of it only being possible to apply conditional compilation to items, and not a set of items.

extra wesl.toml - doens't solve the case with test files in same dir as src, not uncommon in js land.

We're free to ignore cases that currently don't exist. We can just say that such a directory structure is unsupported.
Of course if we can do better, we should do better.

dev-include

Yeah, let's keep dev-include as a separate idea.

The effort of supporting include/exclude (as say tsconfig.json) and as currently spec'd doesn't seem too high

One problem is that currently

• wgsl-analyzer will ignore include/exclude. Working on all files, except for node_modules/target, is the better behavior
• same goes for wgslfmt
• wesl-rs doesn't really need the fields, because the Cargo packaging approach is sufficiently different

So we have something in the spec that's currently would only be used by wesl-js when packaging shaders.

However, the behaviours that we want are

* the _linker_ should exclude them from the final bundle

* lygia users should not be able to import test files

* the language server should not suggest imports from the tests in non test files

The first point is Javascript specific. wesl-rs already can use the cargo facilities for excluding files from the final bundle.

And the "should not suggest imports from ..." is not properly covered by an excludes field. Like it doesn't have a clear interpretation. What if I exclude tests and documentation examples? Test stuff should be importable from other test stuff. But it shouldn't suggest imports from documentation stuff.

Visibility would have a much clearer interpretation. Same goes for the nested wesl.toml files.
Heck, they'd even give you proper error messages when you add an incorrect import statement :)

Now that I think of it, @if(test) is a tricky one. The language server could tell when an item is @if(test) code. And then, it would only suggest those auto-imports if you're currently editing another item that's annotated with @if(test)? Should it do that more generally for all @if conditions? 🤔

[mighdoll]

It seems like we should work through the various cases of where shaders might be in the project and which tools want to include which shaders.

wgsl-analyzer will ignore include/exclude. Working on all files, except for node_modules/target, is the better behavior

I'm think this strategy would fail in Lygia. Aside from tests, we'll need to be able to select .wesl files but not the .wgsl files in the same directory. Lygia's wesl.toml currently uses include for this.

Lygia is important to support of course. But also it's emblematic of the strong opinions many js/ts projects will have about how they like to structure things.

[stefnotch]

Oha, so Lygia has invalid WGSL files?
https://github.com/patriciogonzalezvivo/lygia/blob/main/math.wgsl

I am so tempted to say that if the language server emits errors for those, then that is the correct behavior. 😆

But properly supporting Lygia is more important.

We could ignore those files by relying on shadowing (wesl prefers wesl files over wgsl files). That strikes me as somewhat brittle though.

So the better solution here would really be "use a exclude = ["*.wgsl"]". No wesl tool wants to work with them after all.

Though this raises a secondary question for me: If I do open a Lygia .wgsl file, what should the language server do. Should it try to check that one file? Or should it ignore the file, since it's likely to have invalid syntax.

[stefnotch]

Tagging as M2, since I'd like to get some consensus here for the language server impl.

So I'm currently leaning towards

Exclude is useful, and implies the extra strong semantics of "don't touch at all". So the test runner would ignore those files, so would the formatter, and the language server would never consider those files to be a part of the wesl project.

Include is overcomplicating it.

[mighdoll]

We started with just include, on the belief that include was the more typically expected api than exclude. I think that's right still, I keep the stuff I want to ship organized. The stuff I don't want is more haphazard to track.

Technically, all the patterns are expressible with just one of include or exclude. But some situations are more ergonomic with one or the other or even both at once. AFAIR that's why we decided to include both when last we discussed it. tsconfig supports both, package.json supports both. I'd guess that most people use include first and than add in exclude sometimes (if they need to change the defaults at all).