It’s how LLMs work - they are effectively recursive at inference time, after each token is sampled, you feed it back in. You will end up with the same model state (not including noise) as if that had been the original input prompt.
LLMs sure. My question is whether it is the same in practice for LLMs behind said API. I found no official documentation that we will get exactly the same result as far as I can tell.
And no one here touched how high a multiple the cost is, so I assume its pretty high.
It depends on how large the input prompt (previous context) is. Also, if you can keep cache on GPU with a LRU mechanism, for certain workloads it's very efficient.
You can also design an API optimized for batch workloads (say the same core prompt with different data for instruct-style reasoning) - that can result in large savings in those scenarios.
If you can pipeline upcoming requests and tie state to a specific request, doesn't that allow you to change how you design physical memory? (at least for inference)
Stupid question, but why wouldn't {extremely large slow-write, fast-read memory} + {smaller, very fast-write memory} be a feasible hardware architecture?
If you know many, many cycles ahead what you'll need to have loaded at a specific time.
Or hell, maybe it's time to go back to memory bank switching.
They are almost certainly doing this internally for their own chat products.
The simple version of this just involves saving off the KV cache in the attention layers, and restore it back instead of recomputing. It only requires small changes to inference and the attention layers.
The main challenge is being able to do this under scale, e.g. dump the weights out of GPU memory, persist them, and have a system to rapidly reload them as needed (or just regenerate).
Stealth startup | ML Engineer (edge inferencing) | Bay Area or LA (hybrid) | Full-time
We're a stealth startup building something unbelievably ambitious in the AI space that blends AI and gaming tech - co-founders are Andy Gavin [1] (co-founder of video game developer Naughty Dog) and myself (VP @ Microsoft, Credit Karma, previously Google and Naughty Dog). Venture-backed by top investors including First Round Capital and Battery. Get in on the ground floor and work directly alongside a living legend and a dream team of world-class talent.
We're looking for someone who has deep experience with CoreML and optimizing model inferencing for mobile usage (e.g. ANE on iOS).
If interested, reach out to me at: scott@persona-ai.ai
Full disclosure: I worked with Andy on Crash and am an angel investor. If you’re interested in AI and gaming I recommend you reach out. This is an extraordinary team and opportunity.
I had the pleasure of working closely with Jeff Slutter early in my career. Was the first really fantastic engineer I worked with - he’s at Santa Monica Studios (God of War) these days. I think Matt Toschlog is there too now. Glad to see these folks working on an awesome AAA franchise.
Monarchies usually result in long-term thinking? I’m sure we haven’t read the same history books. Most monarchs throughout history have been very self-interested and their decisions have been focused on their own personal interest - the definition of short-term.
I’m not claiming democracy is any better - humans are notoriously bad at long-term thinking unfortunately.
I mean, some of those personal interests were long term too.
"Peasants need food within the next few days or they'll die": short term interests of the people.
"I need aristocrats to take everyone's grain so I can make them be knights and keep people from overthrowing me": long term interests of the monarch.
the people are very good at identifying their short-term interests (not starving,) but it's hard to figure out long-term collective interests (exactly how much grain to give up for however many armored guys on horses you need.) this is why modern democracies are usually republics.
Hamas gained a ton of support for their cause by the tragedy inflicted in the counterattack. I believe they celebrate the deaths of Palestinian innocents as much as they do the Israeli ones so they can extract propaganda wins. This has been part and parcel of their strategy for decades - it’s why they embed their military activity in soft targets like schools and hospitals.
The Israelis know this by now, so the fact that Israel was goaded into a ground war speaks as much to the political situation as anything else, but either way it’s tragic.
I don’t think “goaded” is the right way to look at it. The Israelis simply don’t care about the optics, they are looking to end this back and forth once and for all
> they are looking to end this back and forth once and for all
You can't eradicate ideologies like Hamas through military means.
You do it by turning people against them and robbing them of legitimacy.
Instead of using the hostage situation as an opportunity to change Gazan's mind about Hamas, Israel killed ~30k of them, created a humanitarian disaster and bred an entirely new generation of Hamas supporters.
> Instead of using the hostage situation as an opportunity to change Gazan's mind about Hamas
How would that suppose to happen if the civilians celebrated en masse the great success of killing and harming the Jews?
Civilians in Gaza have zero sympathy for the Jews. Why would they change their minds when every cruelty against any Jew is a cause for applause for them?
Compare this to the reaction of the West towards the terrorist attack in Moscow. Many Europeans expressed compassion. Would you think relationship between Russia and the West was salvageable if the entire Europe cheered on the streets because some Russians died?
I'm not sure what made you think I believe that Jews have any more capacity to change their minds about Palestinians than Palestinians about Jews.
I also don't know why you think we should be destroying ideology of some Israelis (which would require killing them). I don't think we should be destroying ideology of Palestinians which would require killing them too.
However their ideologies are in confrontation and one has behind it a local superpower with plenty of conventional forces and even some nuclear and the other has rockets made from repurposed water pipes, aviation that consists of paragliders, human shields that lives neither side respects or values and zero allies, even willing to do as little as accepting civilian refugees.
It's clear which ideology will get broken or eradicated, along with the people subscribing to it, if they are stubborn enough.
You seem to make some arguments about morality and symmetry. There's no symmetry because force vastly differs and morality is an afterthough. Where morality solidifies it is history. And the history is written by the victors.
I am making those arguments because your response to a comment saying "You cannot destroy the ideology of Hamas through military means" was not "Yes you can, force vastly differs in this conflict and morality is an afterthought". Your response was instead about how cruel Palestinian civilians are and how much they celebrate the deaths of Israelis.
Anyone who read your original response would think you are saying "Gazans don't deserve to have their minds changed becasue they are evil in a way other enemies are not evil to each other" (you even included European reactions to the Moscow attack, cleverly forgetting Russia has not besieged and wholly blockaded any Europeans for multiple generations). So I also made that assumption
You misread my response. My response was "You can't destroy ideology of Hamas by non-military means". I wasn't making a point about cruelty of Palestinians and I especially wasn't making the point of unique cruelty of Palestinians. I was making a point that their minds are beyond change by arguments other than overwhelming lethal force.
I wasn't making a moral argument about who deserves to get what. I was making a consequentialist (?) argument about who will get what given where they are now.
Please read my earlier comment with that in mind and you'll see how much you have read into them beyond the words I have written.
If you define “goaded” as “put into a situation in which a different response wouldn’t have been possible”, then it’s the correct word. The Israelis could have decided to "end the back and forth once and for all" five years ago or five years from now, but they didn’t— they decided to do it now.
> What is the goal here? How can we "win" in Gaza?
People make this a philosophical question. It's not. Israel's goal is to eliminate Hamas. Not the "idea", the actual people and military material of the fighting force of Hamas.
That is Israel's goal.
Also, it's looking increasingly likely that there will be something of a re-occupation of Gaza, at least for the immediate future until some alternative governing entity will be found.
Note: I'm stating the goal without getting into whether it's a good idea or not, or even achievable or not.
But worth saying that Israel's government thinks it can achieve the goal of destroying Hamas, and even if not, it is possible that when the war was launched, it did think it could be achieved (and e.g. underestimated Hamas).
Also, we use "achievable" here as if it's one dimensional, but it's not - the question is actually "achievable at what cost", where the cost is in Palestinian civilian lives lost and the population displaced, in IDF soldier's lives, and in disruption of the Israeli economy.
Seems the plan is to invade Rafah and force all remaining Palestinians to flee into the Sinai desert/Egypt. Then there'll be no Palestine, and eventually the world will forget about what happened.
It would be nice if the children on both sides weren't involved or they were allowed to get out and the world could have cut off both countries completely like a Bug's Bunny cartoon with a saw.
No money from the US or Iran would flow in until this thousand year religious war was figured out so we could all stop caring about a tiny piece of land that only matters because of religious text.
Jews, Christians, and Muslims have been fighting over this and the surrounding area since before they wrote all their books to convince everyone they are in the right.
If that is your way of saying, that there was never peace in that region among the 3 Abrahamic faiths, then you are highly mistaken. Seems history loves to be forgotten and rewritten.
It seems like rather than adding a random amount to each sample (which lets them compute a mean by recreating the same audio and extracting out the differences), Safari could instead add randomness that is based on a key that rotates every hour. (Function of audio sample and key, so the noise would be the same in a given session, but useless for tracking an hour later).
If you averaged together ten such samples, you'd get something that approaches the true values from the device. The more samples you have, the closer it would get.
Fixing this would require removing the information leak entirely, not just masking it under a layer of random deviations.
The grandparent post accounted for exactly that criticism. By having the source of randomness fixed for a limited time period, a fingerprinting algorithm wouldn't be able to gather enough unique samples for averaging to be useful. And given the extremely fine differences in the floating point numbers, any injected noise would so overwhelm the data that you'd need hundreds, perhaps thousands of samples in order for averaging to be useful.
Wouldn’t it help if the noise added were deterministic based on origin? That way it can’t be averaged out by oversampling. So something like RNG_SEED = HMAC_SHA256(PERSISTENT_SECRET,Location.origin)
Stealth startup | Lead mobile engineer, AI Engineer | Bay Area or LA (hybrid) | Full-time
We're a stealth startup building something unbelievably ambitious in the AI space that blends AI and gaming tech - co-founders are Andy Gavin [1] (co-founder of video game developer Naughty Dog) and myself (VP @ Microsoft, Credit Karma, previously Google and Naughty Dog). Venture-backed by top investors including First Round Capital and Battery. Get in on the ground floor and work directly alongside a living legend and a small team of world-class talent.
Lead Mobile Engineer - Swift and C/C++, CoreML a plus
AI engineer - Finetuning/Retraining LLMs (LLama/Mixtral/etc), MLOps a plus
If interested, reach out to me at: scott@persona-ai.ai
