I wonder if anyone has tried to approach the problem from the other end: start with the hand-tuned network and randomize just some of the weights (or all of the weights a small amount), and see at what point the learning algorithm can no longer get back to the correct formulation of the problem. Map the boundary between almost-solved and failure to converge, instead of starting from a random point trying to get to almost-solved.