I get easily lost driving in my own city, I fear I would get lost in my own memory palace too if it was too big, like "did I turn left or right here when memorizing?".