Well the natural numbers are the decatigorification [1] of the category of finite sets. I won't speak to other kinds of numbers but it's funny how category theory helps answer that question as well.

1 https://math.ucr.edu/home/baez/week121.html