the CMB is a bit mind bending; it's related to the question "if the universe is expanding, in the past it was smaller and smaller; so where is the center?"

Good read where you'll also learn about the "The Surface of Last Screaming":

https://ned.ipac.caltech.edu/level5/March03/Lineweaver/Linew...

I understand how there can be no center if the universe has positive curvature. But if the universe is saddle-shaped and non-infinite, surely there has to be a center?

I think not; consider pseudosphere, there is no obvious center. Its embedding in 3D space has infinite dimensions but the object itself has finite surface area.

https://en.wikipedia.org/wiki/Pseudosphere

If you mean object with finite dimensions and negative curvature and no center, there seems to be such a thing, see Fig. n066200e:

https://encyclopediaofmath.org/wiki/Negative_curvature,_surf...

EDIT

Rozendorn's idea seems to be still an open question:

https://mathoverflow.net/questions/52851/the-geometry-of-nad...

Rather than think "surface of a balloon inflating", think "surface of an (infinite?) rubber sheet being uniformly stretched". It does intuitively seem like if there is 0 curvature and no "center", then the universe must be infinite though ...