For the Echo, at least, it has to use your home network, so you could pretty easily run a packet capture to see if it's ever sending audio out when you don't want it to.
It's a cat-and-mouse game: What if it only sent the clandestine information when it picks up the "normal" word? The point is you don't control the device or its software.
Having all of the devices listening all the time would be a bandwidth and power nightmare, if not for the sender, for the receiver.