What matters in addition to grades is personality and people you know. Things stay anonymous in university during undergrad for the most part, but if you are really good professors will start to notice you and then you get invited to join seminars, do internships etc. Especially in intellectually very competitive fields like math and physics everyone recognizes that tests do not really matter, they are nescessarily trivial or test only a small subset of skills you need to do research.