This the reverse, actually, you can use much smaller inductors the higher the switching frequency. That's why the GaN chargers are so much smaller, for example.
Smaller relative to the requirement for the frequency: you can cheap out both using lower frequency components _and_ using a small inductor than you should be for that lower frequency (or again, not using one at all at that lower frequency because it's still higher than the eye can directly perceive and you think that's all that matters)
