The thing about panic is that your brain associates the physical symptoms with that unpleasant state, such that feeling those physical symptoms in a different context might cause you to panic.

Had a bout with anxiety in the past, and an elevated heart rate from exercise would sometimes trigger it later.