I am one of those without a PhD, but have taken the time to the learn the math & contribute quite a bit to this area.
That being said, I also don't think deep learning itself is not really a "science". The issue I have is you can't predict if a network will learn.
We're effectively testing deep learning networks the same way the Romans used to test bridges. Send a bunch of elephants over them, if it holds it's good enough.
There's obviously some indicators of success, but on a whole the overall interaction between components is very difficult to calculate and near-impossible to predict. While I think it's important to understand how layers interact and how a given function will impact your optimization, etc. it's not fully required to have a deep understanding of the mathematics, at least for most cases.
I also personally don't view anything on arXiv worth anything. I typically will read articles/papers myself if reviewing a candidate and / or would like to see their publications at conferences or journals. Otherwise, it's essentially a blog post (which IMO is fine, but will require me to review it).
The mathematics of neural networks is very well understood from a theoretical and scientific perspective. It is easy to say that a neural network will have predictive power given labelled training data.
Whether or not the predictions it is making satisfy unstated requirements is another problem altogether.
> We're effectively testing deep learning networks the same way the Romans used to test bridges. Send a bunch of elephants over them, if it holds it's good enough.
Well for human level tasks maybe, but what about other areas of research where we like to discover patterns not seen by a human? Like finding links between genomic interplay with external perturbation such as radiotherapy? It'll be very lucky to have such 'elephants' to test at all.
Well I’m certainly no expert on this, but I would guess due to the previous comment the field is possibly too immature at the moment to have as much mathematical certainty as you might find with other methods and fields of mathematics. I recently read the beginning of the book Introduction to Mathematical philosophy by Bertrand Russell and in it he explains how at the times of the Egyptians though they invented geometry it wasn’t very formal and it was very much like the grandparent explained Machine Learning is where Romans used to test bridges they would just throw enough examples at something until they thought it worked. This didn’t mean geometry could never reach a system where they could know the surety of their theorems. The Greeks did just that by starting from basic assumptions or axioms and building a consistent and partially complete system that allowed them to prove many things that followed from there assumptions. There’s a possibility that at some future time (possibly future generations) we’ll have better mathematical tools to figure out the specifics of why neural networks and machine learning work and to the specific extent they do work.
I am also currently reading a Programmer’s Introduction to Mathematics by Jeremy Kuhn (excellent book by the way I would whole heartedly recommend to programmers who have some background in math or thinking in abstractions but who want to learn more math) and it has a quote that states learning Mathematics is a lot like walking into a series of dark rooms and feeling around and getting a feeling of what is in the room until you flip on the light switch but then you could always go to a new room and start all over. I think in that sense machine learning is a series of rooms some lit, but a majority still dark that we haven’t grasped yet.
I think there's a difference between pure mathematics and applied math in these discussions. Exploring the dark room of math is one thing while having a whole lot of great tools and figuring out how to use them in a rain forest to build a livable dwelling is quite another. Math being the abstraction of the world needs bridges to the problems we are facing. Hence the practice of trying a few real life examples (elephants) to test whether it (a method) works.
That being said, I also don't think deep learning itself is not really a "science". The issue I have is you can't predict if a network will learn.
We're effectively testing deep learning networks the same way the Romans used to test bridges. Send a bunch of elephants over them, if it holds it's good enough.
There's obviously some indicators of success, but on a whole the overall interaction between components is very difficult to calculate and near-impossible to predict. While I think it's important to understand how layers interact and how a given function will impact your optimization, etc. it's not fully required to have a deep understanding of the mathematics, at least for most cases.
I also personally don't view anything on arXiv worth anything. I typically will read articles/papers myself if reviewing a candidate and / or would like to see their publications at conferences or journals. Otherwise, it's essentially a blog post (which IMO is fine, but will require me to review it).