Aren't FPGAs mainly for design phase and the real crunching in the industry done on ASICs? At least whole automotive industry works that way - FPGAs to design/test stuff, ASICs for production and making money.
Nope, plently of people do heavy computing with FPGAs.
The speedup between FPGA -> ASIC is not that dramatic. Their real advantage is power draw and ammortized cost. FPGAs also have to initialize when powering up.
It mostly depends on how many you are planning to sell. For a given performance/functionality, FPGAs cost more per chip than ASICs. But ASICs come with a much greater upfront fixed cost.
What this means is that to implement the hardware the FPGA represents, you have to "program" it; this is typically done in one of only a couple HLLs (VHDL and Verilog, known as Hardware Descriptor Languages or HDLs).
At one time, Xilinx (Altera's competitor) made an FPGA which could be programmed "on the fly" very quickly; it (well, many thousands) were used for an interesting machine, of which only a few examples survive (the whole thing at the time was surreal, if you followed it - it seemed like a scam more than anything, but real hardware was shipped).
This machine was called the CAM-Brain machine, and was the creation of researcher Hugo de Garis (who is retired, and is a seemingly strange fellow in the AI community - but not as strange as Mentifex):
I encourage you to research this machine, and Mr de Garis, as the whole thing is fascinating (and I will also say, from a design perspective, the shipped CAM-Brain Machine was one of the "sexiest" looking boxen since the early Crays).
CAM-Brain meant "cellular automata machine brain" - it was basically an effort to evolve a neural network using CA and FPGA; the CA would evolve the HDL which described the hardware representation of the NN, which would then be dumped to the FPGA for processing. The process (from what I understand) was iterative.
I don't believe the "kitten" ever went past much more than some early 3D models (maybe some CAD, too) and a software simulator. At least, that's what you can still find out there today (images of the simulator running on Windows NT, iirc).
The effort was noble, but it didn't work for more than simple things. I think it was part of the "evolve-a-brain" NN dead end, which seemed to hold out some promise at the time.
That's just a bit of background, but it shows how Intel and FPGAs can be used for building hardware to represent neural networks (a GPU/TPU is not a neural network - it is merely a processor for the software representation of the neural network). Whether that's their intention, or something else (maybe something like Transmeta tried?) - only they know.
