It's certainly possible to do it in FPGA, I was involved in this project a few years ago: http://www.argondesign.com/news/2014/jun/23/ultra-low-latenc...

Although it limits the complexity of the model and makes it harder to iterate on the development of the model.