Add support for macro imports

[Gozala]

Macros are executed at compile time, while import / require execute at runtime, there for macro definitions in the imported module has does not affect requirer.

There are few possible ways to go about this:

1. Change import semantics, and read imported modules at compile time so that macros could be registered prior execution. Problem with this approach is tight coupling with node-js require semantics that may not be possible in browsers or be just overkill.
2. Implement subset of clojure ns that would
   understand clojurescript's :require-macros keyword. Problem with this is that it will look like clojure
   but will behave differently not to say that ns is overkill and it's own dsl.

[thehydroimpulse]

@Gozala Just wondering if you've started tackling this yet?

For the browser, things are super messy. Some use components, bower, AMD, etc... so it's a bit harder. But because this is just to share macros, you could pass the files to wisp:

Example:

wisp app/**/*.wisp

This would read in all the files, and perform the macro compilation, which subsequent files that are loaded can use. Then you'd expand the files back out to the filesystem.

The glob path is just an example. Just passing a set of files (like it is right now) works as well.

I'm thinking you could use the same approach to the Node runtime. You'd typically want to expand wisp to JavaScript so that you can inspect it, remove the wisp dependency for production, and works wonder for debugging (Perhaps a performance increase as well. Well, initially at least. Since modules are cached, the performance hit of compiling the language drops.)

Thoughts?

[h2non]

+1 for glob expand pattern support

[Gozala]

@xkxx I'm posting your email here (hope you don't mind) as I'd prefer to keep a conversation about this here rather than in private email thread.

I got some spare time this weekend and was working on macro import support. It is not difficult to do but requires some architectural changes to the compiler. It goes something like this:

Make macros unique to each source file (a current bug with the compiler)
Collect public (non ^private) macro definitions after the analysis stage and output them with (:macros output)
When requiring a module, find the path via require.resolve(), process it, get .macros and put them in macros.

To avoid overhead, I'd say we use a new keyword for macro import, like :use-macros (like ClojureScript).

Also, it's very likely that someone would reference the macros in a few files over and over again. To avoid having to recompile these files which causes massive overhead, I think we can do something similar to how Nimrod does this. We create a cache directory called wisp-cache in the export directory, and dump the generated ast in there. This provides two advantages: 1 it allows lazy compilation, and 2. It avoids overhead in macro imports.

And on the topic of lazy compilation: can we add a comment on every generated js file that contains the compiler version and source file hash?

[Gozala]

I got some spare time this weekend and was working on macro import support.

I'm glad you have time to hack on this! I'm more then happy to assist you to run this to a completion!

It is not difficult to do but requires some architectural changes to the compiler. It goes something like this

I think so to, although I think you would also need to fix a reader / analyzer to make sure that namespace info isn't lost (see #92)

Make macros unique to each source file (a current bug with the compiler)

This is a part that does require not loosing symbols namespace information so that macro name can be bound to specific namespace.

Collect public (non ^private) macro definitions after the analysis stage and output them with (:macros output)
When requiring a module, find the path via require.resolve(), process it, get .macros and put them in macros.

This may work but I think it will not address some issues, like macros should have access to functions or other bindings available in the context they were defined. What I had in mind was slightly different. I though macro definitions could compile to exported JS object (with a special macroexpand method) in the module they are defined with in. That way compiler could just import the module and then pass AST nodes to a macro objects to do the transformation. For example, let's say we are compiling file a.wisp that depends on a foo macro from file b.wisp compiler could just do something along these lines require("b").foo["wisp.macroexpand"](ast). That way compiler is no longer constrained with an IO delegating it to require, neither it needs to explicitly deal with compilation of dependencies as require("b") either will import either already compiled b.js or will live compile it. It would also address issue of having access to scope bindings from the macro definition. Finally this also makes distributing wisp packages through npm very easy as compiled .js files will be all that is needed for a compiler to do it's job.

One thing that I really like about this approach is that it does not introduce dependency on IO (with except of require of course) which is important to support in browser compilation.

To avoid overhead, I'd say we use a new keyword for macro import, like :use-macros (like ClojureScript).

I was thinking to support :use-macros as an alias for :require just to support a compatibility with ClojureScript.

[Gozala]

@xkxx let me know what do you think about my comments.

[Gozala]

Another alternative I was considering was to make macros not an object that are regular module exports but rather a functions stored as a properties of generated _ns_ object to export _ns_ instead.

[xkxx]

I thought about it and right now I definitely agree that it'd be easier to
save macros as JavaScript than to parse them from source every time. I'm
not too kin on polluting generated code though. Maybe we can do something
like generating a file__macros.js for every file.wisp?
The thing with having :use-macro is that we need to somehow distinguish
between function imports and macro imports. Since js is dynamic we can't
detect it at compile time.
On May 12, 2014 9:02 AM, "Irakli Gozalishvili" notifications@github.com
wrote:

Another alternative I was considering was to make macros not an object
that are regular module exports but rather a functions stored as a
properties of generated ns object to export `ns

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-42851418
.

[Gozala]

I thought about it and right now I definitely agree that it'd be easier to save macros as JavaScript than to parse them from source every time. I'm not too kin on polluting generated code though. Maybe we can do something like generating a file__macros.js for every file.wisp?

We could but I'm not sure it's necessary. If you don't want to pollute generated JS I think it's reasonable to have an option to exclude macros.

The thing with having :use-macro is that we need to somehow distinguish between function imports and macro imports. Since js is dynamic we can't detect it at compile time.

That is why I wanted to make macros not a functions but a special objects so that compiler could distinguish weather given foo in the module context is a function or a macro.