Google is actually working on this. At this year's IO they announced that they are working with silicon manufacturers to include hardware acceleration for their TensorFlow Lite framework. This makes it possible to do on-device speech recognition and natural language processing while keeping the power consumption at acceptable levels.