It's still distributed, just not entirely 100% distributed. It's like the difference in which IPFS nodes have which items pinned or how many seeders you have versus torrenters (and the subsequent file availability in that cluster) in BitTorrent.
In the reference implementation from Microsoft it most closely resembles a CDN: you give it a list of servers to back the git database when it needs to look up an object. You distribute those servers as you see need to, based on very similar logic to CDN distribution. For instance, you probably want at least one available per office to keep content close to the users that need it.
Even if a CDN is more "hierarchical" rather than P2P, it's still distributed, it's just distributed on a different axis than you are perhaps expecting.
Furthermore, to a very large extent that's an implementation detail. The GVFS protocol itself [1] is a very simple REST API, and there is absolutely nothing stopping you from building a GVFS "server" that is literally backed by IPFS or BitTorrent or some other P2P file system.
