I am and I would. I don't care if a computer is passively listening for a prompt — the computer has no consciousness and it doesn't do anything I'd even notice. I do care if somebody is trying to make my device do something without my OK.
This does not particularly concern me, as Alexa only transmits when you trigger it and stops transmitting soon afterward. If that's too much for you, I'm not going to try to change your preferences. All I'm saying is, I'm OK with a device that is constantly listening for a trigger, and I'm OK with a device transmitting messages at my request, but that doesn't mean I'm OK with advertisers attempting to exploit the device.
You really only have Amazon's word that the only data transmitted is for that particular command. Unless you tap the transmission, you only have Amazon's word that it only transmits when triggered.
