I think it's inevitable that program synthesis will adopt deep learning methods, whether or not those look like large language models. There's been a bunch of work on this (that you're probably familiar with) from Dawn Song's group:
I think of large LMs trained on code as something that will sample strings from the space of "code humans might write". This isn't nothing, but as you say it's unlikely to produce programs that solve problems no human has solved before. However, assuming you have a rigorous enough spec, I don't see why you couldn't use something like reinforcement learning on such a model in conjunction with traditional verification tools...
Anyway, I agree it's unfortunate that program synthesis has been overlooked in the latest AI code generation craze -- I've mentioned it to a few journalists when talking about things like Copilot but it doesn't tend to end up in the final article.
I'm aware of the work on neural program synthesis that you link to but it would be unfair to say I'm familiar with it. I've read a few papers they've written, here and there. Interesting work, but as they say on their website, it's an extremely challenging problem.
I'm pretty sure that as you say deep learning will be used for program synthesis, in fact it already has - the article above mentions a few applications and startups etc. I'm really curious to see how all this will pan out a few years from now. But the applications that are developed commercially are really trying to implement programming aids, rather than solutions to the full program synthesis problem, which, in a nutshell, is to be able to generate arbitrary programs from a specification of some sort.
I'm skeptical that deep learning can really solve this more general (and much harder) problem. The reason I'm skeptical is that neural networks are great approximators but they're not known for their great precision, I guess. With program synthesis the point is usually to really get a program that does what the specification says, no ifs and buts and no 80%-there's. I don't know how neural netwokrs can do in this, but I don't think the answer is "very well".
>> Anyway, I agree it's unfortunate that program synthesis has been overlooked in the latest AI code generation craze -- I've mentioned it to a few journalists when talking about things like Copilot but it doesn't tend to end up in the final article.
Well, thanks for fighting the good fight :)
You've piqued my curiosity, of course. Can you share one of the articles you mention?
I think my optimism for deep learning methods here comes not from their ability to generate code from scratch but as a way to guide a search process in the space of programs. By analogy, Go and chess are also domains where approximations won't do, but DNNs have had excellent success in finding better search strategies for the game tree. That's the kind of thing that I think might be successful here.
Yes, there seems to be a trend of using deep learning to guide classical search. The first time I noticed it was in AlphaGo. I was initially skeptical but it seems to have picked up and be producing results.
Personally I'm not interested in that approach very much because my work is on constructive approaches, that completely avoid a search. Don't let me bore you with it (it's my doctoral research... you know how students get with their theses :) but there's links in my sig if you're curious.
> However, assuming you have a rigorous enough spec,
And thus the ball is moved back to the human programmers court, as it has since Cobols sentences and paragraphs were supposed to allow managers and business analyst to write code.
This is pretty much already assumed by program synthesis work AFAIK. There's also work on synthesis from I/O samples (e.g. FlashFill) but you don't get very good guarantees (and sometimes it makes hilarious mistakes: https://i.redd.it/c4lufe3m6jh21.gif ).
Actually, FlashFill is quite impressive- maybe it doesn't always get it right but it's true one-shot learning (rather than one-shot after billion-shot learning) that works very well most of the time.
I'm actually surprised about the Maruary, Apruary etc error. That is something that FlashFill should be able to handle very well. Strange.
Do you have a link to some post or comment etc with more context about that gif? Maybe it was an older version of the tool? It works correctly (generates month names) in my version of Excel (360 I think?).
Although I do see the Maruary etc error while I fill-in "February" (but not after I drag the cross-handle thingy down).
https://sunblaze-ucb.github.io/program-synthesis/index.html
I think of large LMs trained on code as something that will sample strings from the space of "code humans might write". This isn't nothing, but as you say it's unlikely to produce programs that solve problems no human has solved before. However, assuming you have a rigorous enough spec, I don't see why you couldn't use something like reinforcement learning on such a model in conjunction with traditional verification tools...
Anyway, I agree it's unfortunate that program synthesis has been overlooked in the latest AI code generation craze -- I've mentioned it to a few journalists when talking about things like Copilot but it doesn't tend to end up in the final article.