Relative to what it was before. It has an intrinsic size. Think of it in terms of matter density if you find it more comfortable. The density simply goes down over time; distandes between galaxies increase.
Just because you can't measure something directly doesn't mean it doesn't exist. There's plenty of things we only know only through indirect measurement.