You don't necessarily want the optimal solution, L2 regularization can give you a slightly worse solution on the training set but that will be better at generalization on unseen data.