It's interesting (to me) visualizing all of these techniques as efforts to replicate A* pathfinding through the model's vector space "maze" to find the desired outcome. The potential to "one shot" any request is plausible with the right context.
> The potential to "one shot" any request is plausible with the right context.
You too can win a jackpot by spinning the wheel just like these other anecdotal winners. Pay no attention to your dwindling credits every time you do though.
On the other hand, our industry has always chased the "one baby in one month out of 9 mothers" paradigm. While you couldn't do that with humans, it's likely you'll soon (tm) be able to do it with agents.
> Now, some might interject here and say we could, of course, train the LLM to ask for a calculator. However, that would not make them intelligent. Humans require no training at all for calculators, as they are such intuitive instruments.
Does the author really believe humans are born with an innate knowledge of calculators and their use?
I think he means that you don't need to train a human to understand that a calculator is useful, and in particular when a problem is hard enough that you need to bust out a calculator. That sort of logic is self-apparent in humans, but struggles to be consistently evoked in LLMs.
That said, I was using simple +*-/ calculators as a small child and I don't think I needed to be taught anything other than MC/MR. The tool is intuitive if you are familiar with formal written arithmetic (of course hunter-gatherers couldn't make sense of it).
I remember having several days of lessons on how to use calculators in elementary, middle, and high school. They are "intuitive", but not all the functions of them are, and if you fully rely on intuition, you might expect them to do things like respect order of operations, which they very well might not.
As a kid of about 5 or 6 years old I used my first calculator with no instruction whatsoever. We are not talking about scientific calculators. Addition, Multiplication. It does not require training or instruction, just a minute of exploration.
You did however need to be taught math first, you needed to learn how to pick things up, read numbers, interact with buttons, understand that a device might have an on and off state, and a zillion other things. It took about 5 or 6 years of training time to make that happen, and it was the result of parents, teachers, or others actively taking time to train you. That process didn’t involve parking you in a library at birth so you could just go figure it out.
Author is simply being obtuse and presumably has some axe to grind or is just ignorant of how LLMs are trained. For example, LLMs don’t learn to chat from the data, they have to be instruct tuned to make that happen. Every LLM chatbot you’ve ever used had to have this extra training step. Further, this is the exact same training process that can also train for tool use.
Trying to say “this should just happen from the data” is silly, it isn’t how any of this works. It’s not how you learned things, and it’s not how LLMs-as-chatbots work.
> Trying to say “this should just happen from the data” is silly, it isn’t how any of this works. It’s not how you learned things, and it’s not how LLMs-as-chatbots work.
Yes, that was the entire point of the article. We are in agreement.
I’m guessing this is the tool behind the Vision Pro’s Photos.app’s 2D-to-Spatial feature which produces excellent results. It truly improves most photos significantly.
