I have an easy wrapper to pocketsphinx here [1] - it has come in handy for me in the past. Another option is a gstreamer server with Kaldi [2].

[1] https://github.com/kastnerkyle/ez-phones

[2] https://www.reddit.com/r/MachineLearning/comments/3pr4v4/are...