Depends on whether you have a stage. If you're playing audio to go with a TV you don't want it drifting much. But when the scenario is playing music over an area you end up with a bunch of mutually-contradictory centers and it doesn't matter if they shift over a little. And music over an area is what sparked this talk.
If you sit on the left or right the sound should still come from "center stage" or (x,y,z)=(0,0,0) without being shifted or drift.