I should probably qualify my statement more. There are certainly new and interesting problems in numerics. And even going from say O(n) convergence to O(n log n) convergence can lead to whole new classes of problems you can solve. I don't want to discourage anyone who loves numerics.
What I was trying to say was that as a graduate student you might be given a problem that already has many really good and smart solutions and be essentially told to find a better solution than all of these. How this goes will depend a lot on the specific problem, your advisor, etc.
But that "slightly more optimized version" may mean "one that does not quietly produce disastrously incorrect results for some input values".