That’s how we become numb to the progress. Like think of this in the context of a decade ago. The news would’ve been amazing.
Imagine these headlines mutating slowly into “all software engineering performed by AI at certain company” and we will just dismiss it as generic because being employed and programming with keyboards is old fashioned. Give it twenty years and I bet this is the future.
You're taking intelligently designed specialized optimization algorithms like the one in this article and trying to use their credibility and success to further inflate the hype of general-purpose LLMs that had nothing to do with this discovery.
A decade ago it wouldn't have been called AI, and it probably shouldn't be called AI today because it's absurdly misleading. It's a python program that "uses gradient descent combined with topological optimization to find minimal graphs corresponding to some target quantum experiment".
Of course today call something "AI" and suddenly interest, and presumably grant opportunities, increase by a few orders of magnitude.
That’s been called AI for about thirty years as far as I am aware. I’m pretty sure I first ran into it studying AI at uni in the 90s, reading Norvig’s Artificial Intelligence: A Modern Approach. This is just the AI Effect at work.
Hahah, if you're going to go that route you may as well call all of math "AI", which is probably where we're headed anyhow! Gradient descent is used in training LLM systems, but it's no more "AI" itself than e.g. a quadratic regression is.
Neural networks are on the hype now, but it doesn't mean that there was no AI before them. It was, it struggled to solve some problems, and to some of them it found solutions. Today people tend to reject everything that is not neural net as not "AI". If it is not neural net, then it is not AI, but general CS. However AI research generated a ton of algorithms for searching, and while gradient descent (I think) was not invented as a part of AI research, AI research adapted the idea to discrete spaces in multiple ways.
OTOH, AI is very much a search in multidimensional spaces, it is so into it, that it would probably make sense to say that gradient descent is an AI tool. Not because it is used to train neural networks, but because the specialty of AI is a search in multidimensional spaces. People probably wouldn't agree, like they don't agree that Fundamental Theorem of Algebra is not of algebra (and not fundamental btw). But the disagreement is not about the deep meaning of the theorem or gradient descent, but about tradition and "we always did it this way".
Gradient descent is used in machine learning, which is a field in AI, to train models (eg. neural networks) on data. You get some data and use gradient descent to pick the parameters (eg. neural network weights) to minimise the error on that training data. You can then use your trained model by putting other data into it and getting its outputs.
The researchers in this article didn't do that. They used gradient descent to choose from a set of experiments. The choice of experiment was the end result and the direct output of the optimisation. Nothing was "learned" or "trained".
Gradient descent and other optimisation tools are used in machine learning, but long predate machine learning and are used in many other fields. Taking "AI" to include "anything that uses gradient descent" would just render an already heavily abused term almost entirely meaningless.
It is a simple iterative algorithm that goes from one point to the next. It doesn't even have memory of previous steps (caveat, the authors used BFGS which approximates the Hessian with previous gradient iterates, but this is still not AI). There is no finding weights or any such thing.
If every for loop is AI, then we might as well call everything AI. Can you pass me the AI, please?
Not every for loop is AI but pretty much any complex equation with unknown parameters that needs to be trained by gradient descent is usually an ML problem the overwhelming majority of the time.
In that AI has come to mean data-driven algorithms, I don't see this as being AI. What they describe is a local-global optimization method with BFGS as the local optimizer (this is not AI) and a noised average weighted by local optimizer performance as the means to produce new starting points. This is simply a heuristic similar to particle swarm optimization. As far as I know, these are not called AI and funds for this type of research don't typically come from funds targeting AI.
This conflation of everything with AI is precisely why people say things like "gradient descent is most often used in ML" without evidence, and this likely being wrong. No, instead it is 1) ML is the currently most prominent (to the public) use of mathematical optimization and 2) everything else is called AI to the public so they conflate that with ML even when it isn't.
Take a random employee at an engineering or applied sciences (non experimental) lab, ask them if they ever use mathematical optimization, chances are a majority will tell you they do. The vast majority of these are not using or devising ML algorithms.
This matters because of what is clear from this thread. Some people devise a classic algorithm that requires intimate knowledge of the problem at hand, the press calls it AI, the public thinks it's AI, registers one more case of "AI as the tool to replace all others". The Zeitgeist becomes that everything else can go to the bin, and AI (by the more restrictive definition) receives disproportionate attention and funds. Note that funding AI research would not fund the people in the headline, unless they do like the minority of bandits that rebrand their non-AI work with AI keywords.
I’m talking about the application of gradient descent: ie When it is used it’s used on an equation that is too complex for analytic methods.
When the equation is too complex for analytic methods but good enough for gradient descent that equation is overwhelmingly the majority of the time characterized as AI.
A gradient descent is used to solve optimization problems, those arise in many many cases unrelated to ML. Please research a little the history of this field, and notice how it predates ML by decades (even 180 years in the case of the gradient descent specifically).
A great deal of applied mathematics is related to finding a minimum or maximum quantity of something. There are not always constructive methods, sometimes (often) there's no better way than to step through a generic optimization method.
Some quick examples clearly unrelated to ML, and very common as they relate to CAD (everywhere from in silico studies to manufacturing) and computer vision:
- projecting a point on a surface
- fitting a parametric surface through a point cloud
Another example is non-linear PDEs. Some notable cases are Navier-Stoke's equations, non-linear elasticity, or reaction-diffusion. These are used in many industries. To solve non-linear PDEs, a residual is minimized using, typically, quasi-Newton methods (gradient descent's buff cousin). This is because numerical schemes only exist for linear equations, so you must first recast the problem as something linear (or a succession of those, as it were).
By the way, I might add that most PDEs can be equivalently recast as optimization problems.
Yet another is inverse problems: imaging (medical, non destructive testing...), parameter estimation (subsoil imaging), or even shape optimization. Similarly, optimal control. (similar in that it is minimizing a quantity under PDE constraints)
To summarize, almost every time you seek to solve a non-linear equation of any kind (of which there are many completely unrelated to ML), numerical optimization is right around the corner. And when you seek to find "the best" or "the least" or "the most" of something, optimization. Clearly, this is all the time.
I think I've provided a broad enough set of fields with ubiquitous applications, that it is clear optimization is omnipresent and used considerably more often than ML is. As you see, there is no association from optimization to ML or AI, although there is one the other way around. (much like a bird is not a chicken).
Right but gradient descent is not used for non linearity. The neural net is linear. Gradient descent is used because of sheer complexity. That’s why you know it’s ai.
Imagine these headlines mutating slowly into “all software engineering performed by AI at certain company” and we will just dismiss it as generic because being employed and programming with keyboards is old fashioned. Give it twenty years and I bet this is the future.