Having a good prompt file ("memory") is an artform.
The AI hype folks write massive fan fiction style novellas that don't have any impact.
But there's middle ground where you tell the agent the specific things about your repo that it doesn't know based on its training. Like if your application has a specific way to run tests headless or it's compiled a certain way that's not the default average.