You probably don't have network-based location enabled. Satellite-based location doesn't work well indoors and takes a while to get a lock. Network-based location support was recently added to GrapheneOS and is opt-in. See https://grapheneos.org/features#network-location and https://grapheneos.org/usage#network-location.