Well I’m certainly no expert on this, but I would guess due to the previous comment the field is possibly too immature at the moment to have as much mathematical certainty as you might find with other methods and fields of mathematics. I recently read the beginning of the book Introduction to Mathematical philosophy by Bertrand Russell and in it he explains how at the times of the Egyptians though they invented geometry it wasn’t very formal and it was very much like the grandparent explained Machine Learning is where Romans used to test bridges they would just throw enough examples at something until they thought it worked. This didn’t mean geometry could never reach a system where they could know the surety of their theorems. The Greeks did just that by starting from basic assumptions or axioms and building a consistent and partially complete system that allowed them to prove many things that followed from there assumptions. There’s a possibility that at some future time (possibly future generations) we’ll have better mathematical tools to figure out the specifics of why neural networks and machine learning work and to the specific extent they do work.

I am also currently reading a Programmer’s Introduction to Mathematics by Jeremy Kuhn (excellent book by the way I would whole heartedly recommend to programmers who have some background in math or thinking in abstractions but who want to learn more math) and it has a quote that states learning Mathematics is a lot like walking into a series of dark rooms and feeling around and getting a feeling of what is in the room until you flip on the light switch but then you could always go to a new room and start all over. I think in that sense machine learning is a series of rooms some lit, but a majority still dark that we haven’t grasped yet.

I think there's a difference between pure mathematics and applied math in these discussions. Exploring the dark room of math is one thing while having a whole lot of great tools and figuring out how to use them in a rain forest to build a livable dwelling is quite another. Math being the abstraction of the world needs bridges to the problems we are facing. Hence the practice of trying a few real life examples (elephants) to test whether it (a method) works.