This is actually by design, I think, though it's arguably pretty surprising behaviour. Every audio output has an independent volume; it's just reverting to whatever the speaker volume happened to be.