I'm a Lisper who's also programmed in Scala. (And for the record, I'm not a Lisp purist - in fact, there are a number of things I really like about Scala).

But I'm not sure how I feel about this change. Scala can already implement things like DSLs through creative use of type inference, etc.

> It's nice to use standard list operations in a macro instead of needing to learn AST magic.

The power of Lisp's macros come from it's homoiconicity, which is something I could never have appreciated before learning Lisp - in fact, I didn't even understand its significance until I wrote an implementation of a Lisp compiler for the first time. (For the record, it was a better learning experience than it was a compiler, but that was the whole point).

The best explanation I've seen of homoiconic macros is one written by a well-known Perl programmer. http://lists.warhead.org.uk/pipermail/iwe/2005-July/000130.h... (In a way, it makes sense that a Perl programmer would be the best at explaining what homoiconicity is, given that its extreme heteroiconicity is the underlying cause of many of the common criticisms of Perl to begin with).

That email is mostly accurate, and it includes this gem:

> Source code generation is unreliable and inadvisable in every language except Lisp.

This is where macros really shine. Macros allow you to generate source code reliably - or put another way, they let you refactor your code far more than you ever thought possible. (When you realize how easy it is to write Lisp programs that recursively define other valid Lisp programs, it's not hard to see why it's such a favorite of the AI community![1])

But here's the catch - for the kind of reliability that MJD talks about, you need to have the internal representation of the language be a fundamental data structure in the language, not just the textual representation.

Why's that? It's simple:

> It's nice to use standard list operations in a macro instead of needing to learn AST magic.

In Lisp, the internal representation is a list. The AST is a list. There's no 'instead of' here... macros are simply AST magic made simple....!

[1] http://en.wikipedia.org/wiki/Technological_singularity

     But I'm not sure how I feel about this change. 
     Scala can already implement things like DSLs through
     creative use of type inference, etc.

(-) there are some god-awful DSLs available for Scala ; the presence of a macros system would allow the reuse of syntax instead of coming up with operators that look like random noise

(-) LINQ becomes possible - I don't know how many people here used LINQ, but it's really awesome to express a query in terms of filter, map and flatMap, and depending on the underlying collection to issue a database query or to submit some code to the GPU for processing.

And LINQ is an instance where static typing really shines.

I think Lisp's success in early AI work had as much to do with its support for higher order functions, repl, dynamic dispatch etc as it did with the macro facilities. Lisp was way ahead in those respects.

But you don't see nearly as much use of Lisp in AI now that many other languages have learned all of Lisp's tricks except homoiconicity and that a lot of "AI" in the field these days is really statistics and linear algebra.

But if you want the best model for the GP language so that it can easily self modify, you will recreate a lisp (at least such was my case) because it looks like Lisp is the homoiconic language and is complete in the mathematical sense (how the reals are complete). If you look at stuff like decision trees, random forests and genetic programming; while implementation language doesn't matter, for execution they really come into their own in a language like lisp.

This has led me to believe that lisp is so far ahead of its time that it is not the language of humans but instead the natural tongue of AIs built on Turing machines.

I don't know what counts as "AI" anymore but in the Machine Learning community, it looks like Matlab, R, Python and Java dominate in terms of use and cutting edge packages. This is due to two things: modern ML does not focus on areas where Lisp gives a clear advantage and 2) what everyone else is using and reduced learning overhead are most important in determining favorites.

Thank you for that - I enjoyed reading it.

Also, is that our btilly?

To me this is like a crutch that the language itself cannot solve.

Ultimately, Scala will be held back because of Java. Scala deserves to live on by itself.

    (defun add-two (x) (+ 2 x))

    (defmacro add-two (x) `(+ 2 ,x))

Whereas in Scala the macro facility looks different enough from vanilla code to have to be learned separately.

Backquote-comma-splice is so useful for building symbolic trees that it's an example of what Whitehead famously celebrated:

http://introtologic.info/AboutLogicsite/whitehead%20Good%20N...

Off topic, but if you read that piece it's interesting how much of what he says applies also to programming, yet how controversial it remains in the realm of programming languages.

    (defmacro add-two (x) (quote ((quote +) 2 x)))

So yes, I really like the `,@-style notation, but also think the alternative is tolerable.

(It bugs me that I can't think of a simple example that must be a macro (and not a function), but you get the idea.)

It's interesting, though, that this is a usability question rather than a technical one, i.e. you can build the same expressions without that notation, just not as easily. But that's what's great about that Whitehead piece I linked to: he shows how fundamental this is. It seems like a surface matter but isn't, because it influences what other things one can use one's limited mental capacity to do. It makes subsequent thoughts possible that wouldn't otherwise be. (Edit: this is why programmers who talk about how language doesn't matter are deeply wrong. And also why programming languages have surrounding cultures. But I go off topic again.)

By the way, your macro example is off, since there's nothing to cause X to be evaluated; it's forever embalmed in that outer QUOTE.

> (It bugs me that I can't think of a simple example that must be a macro (and not a function), but you get the idea.)

My main three rules: 1) delayed, conditional, repeated, etc. evaluation of arguments 2) compile-time computation for massive optimization potential 3) running or inserting code before or after any of the argument forms (useful for tests--init clean env, run test body, destroy env) or changing the argument forms themselves.

Implementing IF in terms of COND or vice-versa.

  # without macro
  def addTwo(x: Int) = x + 2

  # with macro
  def addTwo(x: Int) = macro addTwoImpl
  def addTwoImpl(c: Context)(x: c.Expr[Int]): c.Expr[Int] = c.reify(x + 2)

  macro def addTwo(c: Context)(x: c.Expr[Int]): c.Expr[Int] = c.reify(x + 2)

  macro def addTwo(x: Expr[Int]): Expr[Int] = reify(x + 2)

  sub add-two   ($x) { 2 + $x }

  macro add-two ($x) { quasi { 2 + {{{$x}}} } }

http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Ref...

It's been a while since I watched this talk, so I may have the details wrong. It's a great talk, but at least read the abstract:

Reflection and compilers do tantalizing similar things. Yet, in mainstream, statically typed languages the two have been only loosely coupled, and generally share very little code. In this talk I explore what happens if one sets out to overcome their separation.

 The first half of the talk addresses the challenge how reflection libraries can share core data structures and algorithms with the language's compiler without having compiler internals leaking into the standard library API. It turns out that a component system based on abstract types and path-dependent types is a good tool to solve this challenge. I'll explain how the "multiple cake pattern" can be fruitfully applied to expose the right kind of information.

The second half of the talk explores what one can do when strong, mirror-based reflection is a standard tool. In particular, the compiler itself can use reflection, leading to a particular system of low-level macros that rewrite syntax trees. One core property of these macros is that they can express staging, by rewriting a tree at one stage to code that produces the same tree at the next stage. Staging lets us implement type reification and general LINQ-like functionality. What's more, staging can also be applied to the macro system itself, with the consequence that a simple low-level macro system can produce a high-level hygienic one, without any extra effort from the language or compiler.