This is exactly what I was looking for a few months ago. I wanted to build a voice activated gym assistant, mostly to help me track my sets, reps, and breaks. I was hoping to have it all on-device using a Raspberry Pi, but I didn't find anything great. It ended up being "easier" to use Google Cloud for speech recognition (easier in quotes, because dealing with Google's APIs is never easy).

Shelved that project while I was busy working on other things. This has me excited to give it another go!