This is probably not the whole picture, and I have a very Rust-centric view of this, but I'll take a stab at it.
The correct analogue for Lisp macros is not C++ templates, but the C preprocessor itself. Specifically, a Lisp macro gets to take a particular section of code and change it as it wishes, with everything already conveniently tokenized for the programmer's convenience. Imagine if you could just write your own C preprocessor as part of your program and have the compiler automatically execute it on specific program areas that want your preprocessing.
Rust macros work similarly to this, the main difference being your syntax needs to be tokenizable as Rust instead of Lisp. But they're also rather powerful. So, for example, in Rust you only have one object system which has structs, traits, and very limited higher-kindedness[0]. But there's plenty of other object systems Rust would like to interop with: Objective-C, Swift, COM, and C++ among others.
The canonical way of doing this in Rust is to write a macro[1] that takes your class definition and converts it into a series of structs, traits, and/or function pointers that suitably interop with the foreign code. Code outside the macro then can reference the class created by the system.
If you don't have macros, your other options are:
- Metaclasses, which are the canonical way in Python of doing foreign object interfaces, though with an added wrinkle: multiple inheritance from classes of different metaclasses requires writing a combined metaclass that does both. In macros you usually just can't mix them like that, though I doubt you'd need to define a single class accessible from, say, both Objective-C and Windows COM.
- Write your own damned preprocessor. This is what Qt did with MOC (metaobject compiler) to get signals and slots[2]. If C++ had macros, Trolltech probably would have written MOC as a macro instead of a separate build step with a separate C++ tokenizer.
[0] A concept which I will not be explaining in this post, but it has to do with things like generic associated types which were needed for lifetime bounds on async traits
[1] Usually a "procedural macro", which is different from the pattern-matching macros Rust usually teaches in ways that don't matter here
it's "meta" programming, since Lisp macros do source transformations: code to new code. So one writes code which will rewrite code. For example some programming language lacks a control structure one would want. Instead of waiting for the benevolent dictator for life implementing this feature, one can do it by creating a macro, which implements the control structure.
For example, imagine that you want to program with state machines and you need a short notation for that in your programming language. In Lisp you could design a syntax for a state machine and the Lisp macro would transform state machine descriptions into the code used to implement them -> the generated code typically will be longer and full of implementation details -> in the state machine description one would only specify what's necessary. The Lisp macro will do the code transformation from using the new control structure to the implementation of the control structure.
Thus one can view Lisp as a programmable programming language.
Templates are not nearly as powerful as lisp macros and stitching together code is 100 time more complex (bordering on Turing tarpit territory), but almost arbitrary compile time code generation is still possible. See boost.lambda, boost.spirit, eigen or anything using expression templates for example.
Aside the complexity, the main thing C++ currently lacks is code introspection to modify existing code without using an ad-hoc DSL. I think that even the reflection proposals do not go as far as actually introspecting code.
Let’s say, you’re writing a web application. In most programming language you’d be using libraries or rely on a framework. With Lisp macros, you can program the archetype of a web application. And then use a simpler language to describe you application. Think of it as programmable snippets. Something like Ultisnips [0], but inherent to the language.
Think how much common code can exist in a software but cannot be refactored to a functions because it will have too many variables. Or the multiple problems with classes tree and overloading. Macros let you solve that.
One example is the ~use-package~ macro (Emacs plugin) [0]. Using packages in emacs is mostly the same code over and over. They've already been abstracted in functions, but you still find yourself juggling with so many utilities. You could write a bigger functions, but it will then have a lot of conditional branches. This macro selectively select the code it needs and transforming it if needs be and then the result will be evaluated.
It's a bit hard to explain for me (English is not my native language). But it's the difference between coding a solution with all the edge cases baked in and coding an archetype that let you add your own cases. With functions, you abstract common algorithms, with macros you abstract common architecture.
I think an example would be helpful, conditionals:
Within the "program dimension" there is just no way to run code conditionally without an if, no matter how much you move left and right, you are constrained. It is only possible by using the "higher dimension".
Lisp macros take zero [0] or more unevaluated forms, does something with them (which may be computing a value, see the factorial example elsewhere in this discussion and the submitted blog), and then returns a new form (which could be that computed value, again see the factorial example) which the Lisp system evaluates eventually.
One way to see this would be to do something like:
The value bound to a inside the macro is the unevaluated form (the first and third items printed out). In this case the macro itself is just the identity macro other than its print effect so it returns the original form, which is why we get 1 and 3 printed as well.
Here's an example of processing (very primitive) the unevaluated form to produce something new:
(defmacro infix (a op b)
`(,op ,a ,b)) ;; alternatively: (list op a b)
(print (infix 10 + 20))
;; output
30
Now we can get fancy (this is primitive still, but works):
This generates a valid (prefix notation) lisp form from a more traditional infix form from mathematics. If I were being more clever I wouldn't use the second item in the list (the operation) directly but restrict it to valid arithmetic operators and change how I walk the structure. That would remove the need to force explicit parentheses since I could add in a proper parsing step. This is a very primitive version of what loop and other complex macros do. They take essentially a different language, parse it, and emit a valid Lisp form which is then evaluated.
You could use this to get constexpr like behavior but once you do that you run into problems, you can't do this for example:
(defmacro foo (a b) (+ a b))
(foo (+ 1 2) (+ 3 4)) ;; error
(let ((a 1) (b 2)) (foo a b)) ;; error
(foo 1 2) ;; => 3
It only partially works because it only works, when a and b are both numbers. If they're symbols (second case) or other forms (first case) then the macro attempts to compute something that cannot be computed. You can fix the first case by doing:
(defmacro foo (a b) (+ (eval a) (eval b))
But that still leaves the second case erroring out. You could do something like what I did with infix which walks the forms and determines if they can be evaluated (no unbound variables) and then evaluate them conditionally, leaving expressions with unbound symbols intact to be processed later.
So C++ constexprs are less than Lisp macros, but if you want to use Lisp macros to do the same thing as constexprs you have to do more work. Check out On Lisp and Let Over Lambda for two books that go deep into macros in Common Lisp.
[0] I honestly don't know why you'd want to do this, but technically it's valid to do:
(defmacro foo () (some form to return))
(macroexpand '(foo))
;; => (some form to return)
I cannot think of a case where this would be useful, but someone else might think of one.
Speaking frankly, I firmly believe it’s all a matter of taste. I like lisp because it matches the way I approach problems. I like to think of the program as a jig I’m building rather than a static component.
Being able to write macros means I can write code the SHAPE that I want, regardless of underlying implementation, but I can also manipulate other equivalently meta forms as well as primitives, which sets it at a higher level than templates.
I’m terrible at explaining, but if you’ve never tried lisp I strongly and wholeheartedly suggest you give it a try. For learning, I’d recommend Racket. Try and get at least as far as syntax-rules and syntax-case.
I can visualize this metaphor just fine, but I can't tell why it's useful. Can you make this concept more concrete?