Yes you can - as soon as you start wrapping your arbitrarily complex macros into a custom syntax. I am easily doing this stuff with any language I am adding macros and extensible syntax to.
Of course I can. If my tools are communicating with my compiler, they know everything it knows. I have a single tiny generic Emacs mode (and a similar Visual Studio extension) that handles all the languages designed on top of my extensibility framework.
It's trivial. Any PEG parser I add on top automatically communicates all the highlighting, indentation data and all that to the tools (and it's inferred from the declarative spec, no additional user input is required). Underlying typing engines do the same, for the nice tooltips and code completion. The very compiler core doest the same with all the symbol definitions, dependencies, etc. Easy.
This sounds very interesting :) And I think Colin Flemming is doing something similar in Cursive? In any case, I'd like to see more of what you are talking about - do you have any more documentation of it, a writeup, blog post or video?
If I understand it correctly, Cursive is something different, they don't want to run an inferior Clojure image (unlike Slime, for example), but reproducing a lot of Clojure functionality with their massively complex static analysis tools. But I might get it wrong, all the information I have about Cursive came from one of its advocates who is very aggressively against the very idea of an inferior REPL for an IDE.
I've got some code published, but not that much in writing, planning to fix it some time later. See the stuff at my github account (username: combinatorylogic). Relevant things there are Packrat implementation, literate programming tools and an Emacs mode frontend.
Exactly! You end up defining your macros in a particular restricted subset of lisp, and your tooling for Emacs and Visual Studio has to know about that particular subset. Other people writing similar macros will no doubt have their own, subtly different subset, and their own integrations for their subset. But since your way of writing declarative specs for language customization isn't standardized, you can't use each other's tool integrations.
The way you express DSLs is something that needs to be understood by language tooling, so it belongs in the language spec.
No. Tools do not know anything about the restrictions. In fact they work with a wide range of languages, not just lisp. The only "restriction" is a protocol, built into the macro expander, syntax frontend and compiler core.
So, in your rot13 example compiler would rat all the new identifiers with their origins to the tools.
> So, in your rot13 example compiler would rat all the new identifiers with their origins to the tools.
How can the compiler know which identifier connects to which origin, unless because the macro complied with some standard/restriction/protocol? From a certain perspective all I'm suggesting is making these protocols part of the language standard - that is, define the DSL that's used to define DSLs, rather than allowing macros to consist of arbitrary code.
> Yes you can - as soon as you start wrapping your arbitrarily complex macros into a custom syntax.
Well, by that definition you get exactly the same if the host language of your DSL is statically typed and doesn't use macros. Custom syntax is custom syntax and whether tools/IDEs understand it has nothing to do with the host language.
Of course macro+syntax extension got absolutely nothing to do with what you can achieve in a language without macros.
And, no, you did not understand. Any custom syntax you're adding (if the right tools are used, like mine, for example) would automatically become available for your IDE and all the other tools, because they're reusing the same compiler front-end.
> And, no, you did not understand. Any custom syntax you're adding (if the right tools are used, like mine, for example) would automatically become available for your IDE and all the other tools, because they're reusing the same compiler front-end.
Just being able to execute the macro isn't enough for the IDE though. E.g. if a macro is "rot13 all identifiers in this block" then sure the IDE can run it, but it can't offer sensible autocompletion inside the block without understanding more about the structure of the macro.
IDE does not execute the macro - it knows the result of its expansion from the compiler. And compiler keeps a track of all the identifiers and their origins.
The IDE can autocomplete the rot13ed identifiers from outside, perhaps. But it can't possibly suggest rot13ed identifiers inside the macro block for autocomplete, because it can't possibly know that that's what the macro does.
Why? You know which macro made the identifiers. You know what this macro consumed. In most practically important cases this is sufficient.
But, yes, you cannot do it with the Common Lisp approach, where macros operate on bare lists, not the scheme-like syntax objects. The problem here is that the lists had been stripped from the important location metadata. For this reason I had to depart from the simple list-based macros and using custom syntax extension with rich ASTs underneath. Still, on top of a Lisp.
Even with location information, if the IDE's going to offer autocomplete inside the macro it would need to be able to invert the way the macro transforms identifiers, which is not possible to do to arbitrary code.
I agree that this is very rarely practically important - but if you think about it that's precisely the fact that a more restricted alternative to macros should be adequate.
