Yes, that's commonly referred to as the Exploration-Exploitation Dilemma. Should the agent go deep or wide?

https://en.wikipedia.org/wiki/Exploration%E2%80%93exploitati...