Not at all, pure math is in part about exploring axiomatic systems that may or may not have a physical counterpart. The latter is immaterial.

Can you give some examples of axioms in pure math that run completely counter to our physical world? For example:

  It is NOT possible to draw a straight line from any point to any other point.
  It is NOT  possible to extend a line segment continuously in both directions.
  etc...

or

  Things which are equal to the same thing are NOT equal to one another.
  If equals are added to equals, the wholes are NOT equal.
  The whole is LESS than the part.

Note that the original forms of the above axioms "make sense" to us because everything in our physical experience agrees with them. So when you said that the "physical counterpart ... is immaterial", I was curious to see an example of a "physically impossible" axiom.

Most of large cardinal axioms.