The license for this [1] prohibits use of the model and its outputs for any commercial activity, or even any "live" (whatever that means) conditions, commercial or not.
There seems to be an exclusion for using the code outputs as part of "development". But wait! It also prohibits "any internal usage by employees in the context of the company's business activities". However you interpret these clauses, this puts their claims and comparisons on completely unequal ground. They only compare to other open-weight models, not GPT-4 or Opus, but a normal company or individual can do whatever they want with the Llama weights and outputs. LangChain? "Your favourite coding and building environment"? Who cares? It seems you're not allowed to integrate this with anything else and show it to anyone, even as an art project.
There's some irony in the fact that people will ignore this license in exactly the same way Mistral and all the other LLM guys ignore the copyright and licensing on the works they ingest.
So basically I, as an open source author, had my code eaten up by Mistral without my consent, but if I want to use their code model I’m subject to a bunch of restrictions that benefit their bottom line?
The problem these AI companies have is they live in a glass house and they can’t throw IP rocks around without breaking their own “your content is our training data” foundation.
They only reason I can think of that Google doesn’t go after OpenAI for scraping YouTube is then they’d put themselves in the same crosshairs, and may set a precedent they’d also be bound by.
Given the model is “on the web” I have the same rights as Mistral to use anything online however I want without regard for IP, right?
This is an actively litigated and unsettled area of law. You, and nobody else, can say any of this with confidence until these lawsuits get to a judge, and even then it’s per jurisdiction rulings. The US, EU, and Japan may end up with different rulings. International trade agreements may be updated. Industry may settle on some sort of broadly acceptable revenue sharing model.
The point is: nobody knows and the AI companies are getting well ahead of the law.
The law is interpreted anew each day. Nothing is outside the law.
Perhaps if all rules were written in stone, and clear of ambiguity, we would not need judges or the legal process. But that’s not how any of this works.
The concept of "intelectual property" is crazy to begin with, you can't own a thing that is not economically scarse. Nobody owns the atmosphere, nobody owns math, nobody owns a sequence of bytes. If you can copy it, you can't steal it. You have the natural right to use their public code the way you want, as they have the natural right to use your public code the way they want. In the positivist game though, wins whoever has more money to spare on lawyers, so you lose by default even if you wanna play the legal game. The deep pockets always win on that front.
IP is cultural maximalism paperclipping. It ties ambiguous substrings not economically explained to as much of human output as mechanically possible and arbitrarily inflate importance of living humans.
I used to spend a lot of time (thousands of hours) contributing to open source projects. Over the past few years I've stopped contributing (except minor fixes) to any project under MIT/Apache or similar licences.
Interesting, I think that’s a totally valid response to this trend of capturing “value” of open source via Cloud Services (for a while now) and Code Gen (more recent).
I think SV is just dead set on killing the golden goose of open source and the web by extracting as much as possible with no regard for the wasteland left behind.
> So basically I, as an open source author, had my code eaten up by Mistral without my consent
Not necessarily. You consented to people reading your code and learning from it when you posted it on Github. Whether or not there's an issue with AI doing the same remains to be settled. It certainly isn't clear cut that separate consent would be required.
MIT/BSD code is fair game, but isn't the whole point of GPL/AGPL "you can read and share and use this, but you can't take it and roll it into your closed commercial product for profit"? It seems like what Mistral and co are doing is a fundamental violation of the one thing GPL is striving to enforce.
No. Either MIT/BSD code isn't fair game because it requires attribution, or GPL/AGPL code is fair game because it isn't copyright infringement in the first place so no license is required.
It'll be a court fight to determine which. Worse, it will be a court fight that plays out in a bunch of different countries and they probably won't all come to the same conclusion. It's unlikely the two licenses have a different effect here though. Either they both forbid it, or neither had the power to forbid it in the first place.
> but isn't the whole point of GPL/AGPL "you can read and share and use this, but you can't take it and roll it into your closed commercial product for profit"?
You can profit from GPL / AGPL code but just also make all your source code open source and available for everyone to see.
> You consented to people reading your code and learning from it when you posted it on Github.
And if I never posted my code to github, but someone else did? What if someone had posted proprietary code they had no rights to to github at the same time the scraper bots were trawling it? A few years ago some Windows source code was leaked onto Github - did Microsoft consent then?
I did not give consent to train on my software and the license does not allow commercial use of it.
They have taken my code and now are dictating how I can use their derived work.
Personally I think these tools are useful, but if the data comes from the commons the model should also belong to the commons. This is just another attempt to gain private benefit from public work.
There are legal issues to be resolved, and there is an explosion of lawsuits already, but the fact pattern is simple and applies to nearly all closed-source AI companies.
Mistral is as open as they get, most others are far worse.
Here you can use the model without issues, as others are saying it’s doubtful they would sue you if you were to use code generated by the model in a commercial app
Replit’s replit-code[1,2] is CC BY-SA 4.0 for the weights, Apache 2.0 for the sources. Replit has its own unpleasant history[3], but the model’s terms are good. (The model itself is not as good, but deciding whether that’s a worthwhile tradeoff is up to you. The tradeoff exists and is meaningful, is my point.)
>They only reason I can think of that Google doesn’t go after OpenAI for scraping YouTube is then they’d put themselves in the same crosshairs, and may set a precedent they’d also be bound by.
It's like killing Caesar. As long as we all stab him, everyone is guilty and no one can prosecute us.
While you might call it absurd, I feel like these glass houses are why we've seen so much rapid progress with AI recently.
> They only reason I can think of that Google doesn’t go after OpenAI for scraping YouTube is then they’d put themselves in the same crosshairs, and may set a precedent they’d also be bound by.
It will be the smartphone patent wars all over again with hundreds of lawsuits against big tech and AI companies.
We are already past the 'fair use' excuses at this point especially when OpenAI is slowly striking deals with news companies to train on their content (with their permission) and with intent of commercializing the model.
I think a lot of the license motivation is to have real time information for RAG. I doubt that is being used for foundation training, it’s just not enough volume.
Five years ago it would not have been at all controversial that these weights would not be copyrightable in the US, they're machine generated output on third party data. Yet somehow we've entered a weird timeline where obvious copyfraud is fine, by the same entities that are at best on the line of engaging in commercial copyright infringement at a hereto unforeseen scale.
It's clear that when enough money and power is on the line - and fear that other countries will overtake them - all countries are willing to conveniently and pragmatically ignore their laws. I don't think this is any kind of surprise.
Is that so certain? To be able to make claims for what you can use the output, can you do it without making any claims for about control and ownership of the output?
Of course, they can revoke your right to use the software, but if it goes to court, that would be interesting case.
If there’s no copyright in the weights to begin with, the only restrictions you have are the ones you agreed to when you accepted the license agreement. Find the weights somewhere else and you don’t have to worry about the license.
I don’t know why there isn’t more discussion on this point and people just assume there’s an underlying copyright basis to the licensing of weights. As far as I know that isn’t settled at all.
I think it's not important if this is enforcable, this license sounds to me like a warning that the output may be radioactive, which is where AI copyright discussions are slowly shifting towards.
I'd like to know how they think they'll prove I didn't write whatever code I generate. Unless it is a direct copy of something else available to the investigator, good luck.
should it be morally ok to not follow these kinds of license, maybe except when you are selling a service without making any changes? i wonder what people visiting this site thinks about this.
> licensed under the new Mistral AI Non-Production License, which means that you can use it for research and testing purposes. ...
Which basically means "we give you this model. Go find its weaknesses and report on r/locallama. Then we'll use that to improve our commercial model which we won't open-source."
I'm sick of abusing the word "open-source" in this field.
> I'm sick of abusing the word "open-source" in this field.
They don’t call this open source anywhere, do they? As far as I can see, they only say it’s open weights and that it’s available under their Mistral AI Non-Production License for research and testing. That doesn’t scream “open source” to me.
They do say "open-weight", which is I think still very misleading in this context. Open-weight sounds like it should be the same as open-source, just for weights instead of the full source (for example, training data and the code used to generate the weights may not be released). This isn't really "open" in any meaningful sense.
This is why I prefer the term "weights available" just like "source available". It makes it clear that you can get your hands on the copy, you could run this exact thing locally if they go out of business, etc. but it is definitely not open in the OSS sense.
The fact that I can downloaded it and run it myself is a pretty meaningful amount of openness to me. I can easily ignore their bogus claims about what I'm allowed to do with it due to their distribution model. I can't necessarily do the same with a propriety service, as they can cut me off if the way I use the output makes them sad :(
> I can easily ignore their bogus claims about what I'm allowed to do with it due to their distribution model.
If you're talking about exclusively personally use, sure. If you're talking about a business setting in a jurisdiction that Mistral can sue in, not so much.
Being able to use it in a business setting is a pretty darn important part of what Open Source has always meant (it's why it exists as a term at all).
> If you're talking about a business setting in a jurisdiction that Mistral can sue in, not so much.
I'm reminded of the Japanese concept called Sosumi :)
> Being able to use it in a business setting is a pretty darn important part of what Open Source has always meant (it's why it exists as a term at all).
I'm quite familiar with the history of that term, but neither I nor Mistral used it. None of their models have been open source; they have been open weight. You can argue that they are actually "weight available" given the terms they write next to the download link, but since there has been no ruling on whether weights themselves are covered by copyright (and I think that would be terribly bogus if they are), I simply choose not to care what they write in their "terms of use".
The inference engine that I use to run open weight language models is fully free software. The model itself isn't really software in the traditional sense. So calling it ____ware seems inaccurate.
The interpreter is free software. The model is freeware distributed as a binary blob. Code vs. Data is a matter of perspective, but with large neural nets, more than anywhere, it makes no sense to pretend they're plain data. All the computational complexity is in the weights, they're very much code compiled for an unusual architecture (the inference engine).
Regardless of the distinction of code vs data, putting a limit to the number of inferences you can run on a model is essentially the same as using a copyright license on a PNG to limit the number of times you can "run" the PNG with a photo viewer. Is that enforceable? Does it matter? Does the copyright extend to music that my photo viewer generates when I open the image? IANAL but imo, no.
All their other models are “open source” and it was the selling point they built their brand on.
I doubt they made their new model completely different from previous ones so it’s supposed be open source too, unless they found some juridical loophole lol
This is maybe a debatable claim, but I’ll contend that without the magnificent rebel who leaked the original LLaMA weights the last, what, 15 months would have gone completely differently.
The legislators and courts and lawyers will be years if not decades sorting all this out.
For now there seems to be a productive if slightly uneasy truce: outside of a few groups at a few firms, everyone seems to be maximizing for innovation and generally behaving under a positive sum expectation.
One imagines if some really cool tune of this model shows up as a magnet or even on huggingface, the courteous thing probably happened: Mistral was notified in advance and some mutually beneficial arrangement was agreed to in outline, maybe inked, maybe not.
I don’t work for Mistral, so that’s pure speculation, but the big company I spent most of my career at would have certainly said “can we hire this person? can we buy this company? can we collaborate with people who do awesome stuff with our stuff that we didn’t think of?”
The icky actors kind of dominate the headlines and I’m as guilty as anyone and guiltier than most of letting that be top of mind too often.
In the large this is really cool and kind of new.
I’m personally rather optimistic that we’re well past the point when outright piracy or flagrantly adversarial license violations are either necessary or useful.
To me this license seems like an invitation to build on Mistral’s work and approach them with the results, and given how well a posture of openness with some safeguards is working out for FAIR and the LLaMA group, that’s certainly the outcome I’d be hoping for in their position.
Maybe open AI was an unrealistic goal. Maybe AvailableAI is what we wind up with, and that wouldn’t be too bad.
If you want to live on the legal edge, it’s unclear whether there is any copyright in model weights (since they don’t have human authorship), so just wait for someone to post the weights someplace where you can get them without agreeing to the license.
So, it's almost entirely useless with that license, because the average pack of corpo beancounters will never let you use it over whatever Microsoft has already sold them.
They could potentially watermark the model in order to identify the output. There are techniques for doing that, for example by randomly assigning token into groups A and B, group A probability is increased over group B, if group A is over-represented, chances are that that the output comes from the watermarked model.
How effective these techniques are and how acceptable as a proof it is is yet to be defined.
I don't think it is the case here, they probably don't really care, and watermarking has a cost.
My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)
I normally ask about building a multi-tenant system using async SQLAlchemy 2 ORM where some tables are shared between tenants in a global PostgreSQL schema and some are in a per-tenant schema.
Nothing gets it right first time, but when ChatGPT 4 first came out, I could talk to it more and it would eventually get it right. Not long after that though, ChatGPT degraded. It would get it wrong on the first try, but with every subsequent follow up it would forget one of the constraints. Then when it was prompted to fix that one, it forgot a different one. And eventually it would cycle through all of the constraints, getting at least one wrong each time.
Since then benchmarks came out showing that ChatGPT “didn’t really degrade”, but all of the benchmarks seemed focused on single question/answer pairs and not actual multi-turn chat. For this kind of thing, ChatGPT 4 has never managed to recover to as good as it was when it was first released in my experience.
It’s been months since I’ve had to deal with that kind of code, so I might be forgetting something, but I just tried it with Codestral and it spat out something that looked reasonable very quickly on its first try.
>It would get it wrong on the first try, but with every subsequent follow up it would forget one of the constraints. Then when it was prompted to fix that one, it forgot a different one. And eventually it would cycle through all of the constraints, getting at least one wrong each time.
That drives me nuts and makes me ragequit about half the time. Although it's usually more effective to go and correct your initial prompt rather than prompt it again
I had a similar experience. I was trying to get GPT 4 to write some R/Stan code for a bit of bayesian modelling. It would get the model wrong, and then I would walk it through how to do it right, and by the end it would almost get it right, but on the next step, it would be like, oh, this is what you want, and the output was identical to the first wrong attempt, which would start the loop over again.
Similar experience using GPT4 for help with Apple's Accessibility API. I wanted to do some non-happy-path things and it kept looping between solutions that failed to satisfy at least one of a handful of requirements that I had, and in ways that I couldn't combine the different "solutions" to meet all the requirements.
I was eventually able to figure it out with the help of some early 2010s blog posts. Sadly I didn't test giving it that context and having it attempt to find a solution again (and this was before web browsing was integrated with the web app).
More of an issue than it not knowing enough to fulfill my request (it was pretty obscure so I didn't necessarily expect that it would be able to) was that it didn't mind emitting solutions that failed to meet the requirements. "I don't know how to do that" would've been a much preferred answer.
This seems an important failure mode to me. I too have noticed gpt4 looping between a few different failure cases, in my case it was state transitions in js code. Explaining to it what it did wrong didn't help.
Give an LLM all the time you want, and they will still not get it right. In fact, they most likely will give worse and worse answers with time. That’s a big difference with a software developer.
My experience is very different. Often it (ChatGPT or Copilot, depending on what I'm trying to accomplish) gets things right the first time. When it doesn't, it's usually close enough that a bit of manual modification is all that's needed. Sometimes it's totally wrong, but I can usually point it in the right direction.
I mean, with a nonzero temperature, the randomness will eventually produce every combination of tokens in the corpus, so with a sufficiently large "all the time you want" you can produce limitless correct answers
I love to ask it to "make me a Node.js library that pings an ipv4 address, but you must use ZERO dependencies, you must only the native Node.js API modules"
The majority of models (both proprietary and open-weight) don't understand:
- by inference, ping means we're talking about ICMP
- ICMP requires raw sockets
- Node.js has no native raw socket API
You can do some CoT trickery to help it reason about the problem and maybe finally get it settled on a variety of solutions (usually some flavor of building a native add-on using C/C++/Rust/Go), or just guide it there step by step yourself, but the back and forth to get there requires a ton of pre-knowledge of the problem space which sorta defeats the purpose. If you just feed it the errors you get verbatim trying to run the code it generates, you end up in painful feedback loops.
(Note: I never expect the models to get this right, it's just a good microcosmic but concrete example of where knowledge & reasoning meets actual programming acumen, so its cool to see how models evolve to get better, if at all, at the task).
This is the same level of gotcha that everyone complains about when interviewing. It's mainly just depending on the interviewee having the same assumptions (pings definitely do not have to be icmp) and the same knowledge base, usually bespoke, (node.js peculiarities). I can see that an llm should know whether raw sockets are available, but that's not what you asked.
In fact you deliberately asked for something impossible and hold up undefined behavior as undefined like it's impugning something.
> In fact you deliberately asked for something impossible and hold up undefined behavior as undefined like it's impugning something.
Correct, I did. This is a direct indictment on a given model's ability to plan/reason in this particular context. There are plenty of situations where models will respond with "Sorry, that's not possible". Ask GPT-4 "Tell me how to grow biological wings on a human" and it will respond with something along the lines of "this isn't currently possible, but here's a theoretical exploration of the idea"
GPT-4 gets very close on its own to the node.js question via a similar response breakdown above, provided the prompt is clear and detailed enough. But I test the open weight models in the same way to see if they have the capacity to exhibit similar reasoning or chain of thought process on their own. They usually don't without excessive prompt engineering or few-shot.
I said that I don't expect models to get this right not because I don't _want_ them to, it's because I think its an important milestone when they do. Autoregressive token prediction is unlikely to produce the real outcome im testing for here, but if it ever does thats an interesting finding.
I usually through some complex Rust code with lifetime requirements. And ask them to fix it.
LLMs aren't capable on providing much help for that in general, other than some very basic cases.
The best way to get your work done is still to look into Rust forums.
It works amazingly well for the ones that never coded in Rust, at least in my experience. It took me a couple hours and 120 lines of code to set up a WebRTC signaling server.
Damn, show us your brilliant prompt then. LLMs cannot do this, not even in python, of which there are libraries like Blacksheep that honestly make it a trivial task.
My point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed it debug stuff.
Well sure, but that wasn't what we were discussing. The original comment says they use that as their benchmark. While their coding task is a bit complex compared to other benchmarking prompts, it's not that crazy. Here is an example of prompts used for benchmarking with Python for reference:
At the end of the day LLMs in their current iteration aren't intended to do even moderately difficult tasks on their own but it's fun to query them to see progress when new claims are made.
Exactly, expecting one shot 100% working code with one prompt is ridiculous at this point. It's why libraries like Aider are so useful, because you can iteratively diff generated code until it's useable.
Sure it's impossible at this point, but the point of a benchmark isn't to complete the task it's to test it's efficacy overall and to see progress. None of them are 100% at even the simplistic python benchmarks, doesn't mean we shouldn't measure that capability. But sure, I get it. That's not how they are intended to be used but that's also not the point the commenter was laying out.
Prompts like yours (I ask them for a fluid dynamics simulator which also doesn't succeed) inform us of the level they have reached. A useful benchmark, given how many of the formal ones they breeze through.
I'm glad they can't quite manage this yet. Means I still have a job.
No he is right, he is saying taken to the extreme. The point is the more and more specific you have to prompt, the more you are actually contributing to the result yourself and the less the model is
Yes but the build up isn't manual. You go patching prompts with responses until the final result. The last prompt will be almost the whole code complete, obviously.
Well now we get into information density and Komolgorov complexity. The more complicated your desired output program is, the more information you'll have to put in, ie, more complicated prompts.
It's something I know how to do after figuring it out myself and discovering the potential sharp edges, so I've made it into a fun game to test the models. I'd argue that it's a great prompt (to keep using consistently over time) to see the evolution of this wildly accelerating field.
i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code.
GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last.
Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestral with additional pretraining (it seems like we don't know how much more tokens are on this one, but CodeLLama was 500B-1T extra tokens of code).
Zuck has mentioned recently that he doesnt see coding ability as important for his usecases, whereas obviously OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI.
That's a really surprising thing to hear, where did you see that? The only quote I've seen is this one:
>“One hypothesis was that coding isn’t that important because it’s not like a lot of people are going to ask coding questions in WhatsApp,” he says. “It turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical structure of knowledge, and just generally have more of an intuitive sense of logic.”
Make Sense, they want better interaction whit users for Whatsapp, Instagram and Facebook marketers, content creation and moderation,and their glasses(ai /ar) I don't see in that context why the should push more effort into llm coding, is sad anyways
> OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI.
And researchers from Google Deepmind, University of Wisconsin-Madison and Laboratoire de l’Informatique du Parallélisme, University of Lyon, actually publish some of their results in that direction [1,2].
Codex[1] is OpenAI's CodeGPT. It's what powers GitHub Copilot and it is very good but not publicly accessible. Maybe they don't want something else to outcompete Copilot.
Is there a way to use this within VSCode like copilot , meaning having the "shadow code" appear while you code instead of having to tho back-and-forth between the editor and a chat-like interface ?
For me, a significant component of the quality of these tools resides on the "client" side; being able to engineer a prompt that will yield to accurate code being generated by the model. The prompt needs to find and embed the right chunks from the user current workspace, or even from his entire org repos. The model is "just" one piece of the puzzle.
Not using Codestral (yet) but check out Continue.dev[1] with Ollama[2] running llama3:latest and starcoder2:3b. It gives you a locally running chat and edit via llama3 and autocomplete via starcoder2.
It's not perfect but it's getting better and better.
Having the chats in Obsidian lets me save them to reference them later in my notes. When I first started using it in VSCode when programming in Python it felt like a lot of noise at first. It kept generating a lot of useless recommendations, but recently it has been super helpful.
I think my only gripe is I sometimes forget to turn off my ollama systemd unit and I get some noticeable video lag when playing games on my workstation. I think for my next video card upgrade, I am going to build a new home server that can fit my current NVIDIA RTX 3090 Ti and use that as a dedicated server for running ollama.
I created a simple CLI app that does this in my workspace, which is under source control so after the LLM execution all the changes are highlighted by diff and the LLM also creates a COMMIT_EDITMSG file describing what it changed. Now I don't use chatgpt anymore, only this cli tool.
I never saw something like this integrated directly on VSCode tho (and isn't my preferred workflow anyway, command line works better).
- You shall only use the Mistral Models and Derivatives (whether or not created by Mistral AI) for testing, research, Personal, or evaluation purposes in Non-Production Environments;
- Subject to the foregoing, You shall not supply the Mistral Models, Derivatives, or Outputs in the course of a commercial activity, whether in return for payment or free of charge, in any medium or form, including but not limited to through a hosted or managed service (e.g. SaaS, cloud instances, etc.), or behind a software layer
Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon.
Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvidia 4090 GPU has trade-offs. My 128GB MacBook Pro handles models using up to 96GB of unified memory, running at a little under half the speed of a 4090. If you use a quantized version of full floating point model, you can run the largest open models available.
While the 4090 has 24GB of dedicated memory and higher bandwidth (1000 GB/s vs. 400 GB/s on M3 Max), the Mac’s unified memory system (up to 128GB) is flexible and holds smarter models (8 bit and 6 bit models act still mostly all there, 4 bit is so so, 2 bit is brain damaged).
The M2 Ultra in Mac Studio offers even more (800 GB/s bandwidth and 192GB memory). So, ok, 6 or 8 of 4090 cards or 4 x A6000 cards excels in raw performance, but Apple’s unified memory in a laptop fits in your backback.
It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling better if you look at the convenience and price relative to model size. Models that fit in one 4090 or even a pair of 4090s are toys compared to what fits on these, so for the big models you're comparing a laptop to a minifridge.
It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model.
I also get the utility of running Linux instead of the closed up mac os.
But running large models locally is not exclusive to mac studio, you can do the same on PC for a much lower cost.
I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4.
Asahi is Fischer-Price tiers of support compared to what even Nvidia, the most loathed OEM on Linux, provides for free to their users. If that's the best option available, it should be no wonder that server customers are avoiding Apple like the plague. Apple has to beg their audience to reverse-engineer their own OpenCL drivers if they want them; Nvidia ships them alongside CUDA. These two companies are not the same.
Have you ever seen the inside of a datacenter? Why is it that surprising to you that nobody perks up when you start waxing on about battery life? Even terms of power-to-performance, Apple's latest chips get ethered by Nvidia's server offerings.
This "Apple for Inference" meme is so dead that I can only feel sad when I see people unironically promoting it. You actually think serious customers are going to load up Asahi (even funnier, MacOS) on their Mac Pro... so they can inference half as fast as a single Blackwell GPU? You think the industry is doing this shit? I don't even think the Steve Jobs apologists are dumb enough to fall for this one, you must be a particularly aspirational shareholder.
The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB?
you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.
Wait for a gguf release of this and it will fit neatly into a 3090 with a decent quant. I'm excited for this model and I'll be adding it to my collection.
I'm honestly not sure on how to measure the amount of vRAM required for these models but I suspect this would run relatively fast, depending on your use case, on a mid to high end 20 or 30 series card. No idea about Apple unified RAM. I get a lot out of performance out of even older cards such as a 1080ti but haven't tested this model.
If I can’t use the output of this in practical code completion use cases, it’s meaningless, because GH Copilot exists. Idk what they’re thinking or what business model they’re envisioning - Copilot is far and away the best model of this kind anyway
Did yall see what happened when they democratised art? I don't want to have a billion and one AI garbage libraries to sift through before I can find something reliable and human-made. At least the potential for creating horrific political software is slightly lower than with simple images.
It is fast all right, but the quality is not there. I asked it to implement OAuth with Stytch and Ktor and it made everything up. I pointed out the correct name for the package and asked if it really knew the SDK, and it apologized and repeated the same made up code after merely changing the name of the package.
This is actually why we (Stytch) haven't rolled out any of these "chatbots for code".
We have a big list of example questions we get from devs trying us out and we've tested several home grown and third party providers and thus far haven't seen anything good enough that we'd put into production.
Thanks for testing this out for us! I'll cross it off our list :)
Are you saying GH has more than Codestral and therefore GH has a better model? Or that Codestral would be better because Codestral is not littered with "bad" code?
Bad code is obviously very subjective, but I would wager that GH places a much higher value on feedback mechanisms like stars, issues, PRs, velocity, etc. Their ubiquity likely allows them to automatically cherry-pick less "bad code."
Is there a vscode extension that could plug any model out there and have a similar experience to copilot. I always want to try them but I cant be bothered to do a whole setup each time.
There are plugins for various IDEs that operate like copilot but let you select model you want to use, just supply your key. CodeGPT for JetBrains/Android Studio is pretty good. I think you can even use a model running locally.
I'm so happy now LLMs are democratising access to programming, especially open models like what Meta with Llama and Mistral is doing with Codestral are doing.
The abundance of programming is going to allow almost everyone to become a great programmer.
This is so exciting to see and each day programming is becoming a solved problem so we can focus on other things.
My experience with coding with LLMs is that the only thing it's really good at is generating boilerplate that it has more-or-less seen before (essentially a library, even if is somewhat adapted), however it is incapable of the creative thinking that developers regularly need to engage in when architecting a solution for their use case.
My experience is the opposite. When I started using Copilot I thought it would only be good at standard boilerplate but I'm constantly surprised how well it understands my completely convoluted legacy architecture that barely I understand myself even though I'm the only contributor.
Understanding existing code is in its wheelhouse (provided the infrastructure feeding the existing code to the prompt is working well), but I believe if you examine the totality of work a human programmer is involved in, an LLM is woefully behind in many areas (gathering proper requirements, potentially iterating/pushing back on requirements, architecting a solution on a macro level, other gaps an LLm cannot fill).
Parents problem I experienced -> it gets "stuck" and its limitation of learning loop (humans are always asking why it gets stuck and how to get unstuck), LLMs just power through without understanding what "stuck" is.
For explaining existing corpus, algorithm it does a fantastic job.
So likely we will see significant wage garnishing in "agency/b2b enterprise" shops.
In my experience these tools amplify the quality of a programmer.
I have seen good programmers dramatically increase their productivity, but I've also seen others copy-pasting for loops inside other for loops where one loop would definitely suffice. We're not quite there yet.
Absolutely it amplifies. Complex and esoteric configuration of frameworks, for example, entails so much reading and Googling and can be very time consuming without AI. AI can help to bring custom software to the markets that could not otherwise afford to pay for it.
I observe a certain laziness in myself when it comes to certain problems. It's easier to ask a LLM and debug provided code, but I ask myself if I'm losing some problem solving capabilities in the long run because of this.
Similar to the loss of speed in doing mental arithmetic because of calculators on the smartphone.
This enables everyone to be great programmers like how easily available power tools enables everyone to be a great carpenter and general craftsman.
You’ll get a lot of shitty stuff and the profession will get hollowed out losing attraction of the smart people. We’ll be left with low-quality, disposable bullshit while wondering where all the programmers went.
Shadow libraries did more to democratize anything than LLMs. And following a book like Elixir in Action (Manning) will get you there faster than chatting with LLMs or copilot generating code for you.
That's the risk of AI.
Not that AI outperforms humans already but that managers believe it does. That and that code writing is the main work of programmers.
What's the business model for semi open source models like these? Is it just because they can't be fully closed as they have to then compare with OpenAI. Who would pay for these model if better is available for cheaper from Anthropic or Google.
Does anyone know of a link to a codegen comparison page? In other words, you write your request, and it's submitted to multiple codegen engines, so you can compare the output.
Sometimes it outperforms GPT-4 in quality by a fair amount, and other times it starts repeating itself. Duplicating function definitions, even misremembering what things are named.
It seems to have to do with length. If the output exceeds a few thousand tokens, it seems to experience some pretty bad failure modes.
This article says 88GB without quantization. Though it then goes on to make a ridiculous claim that if you had 128GB of RAM, then using up 88GB of 128GB would make everything else really slow because I guess the think the remaining 40GB of RAM somehow isn't enough for your OS and desktop apps.
There seems to be an exclusion for using the code outputs as part of "development". But wait! It also prohibits "any internal usage by employees in the context of the company's business activities". However you interpret these clauses, this puts their claims and comparisons on completely unequal ground. They only compare to other open-weight models, not GPT-4 or Opus, but a normal company or individual can do whatever they want with the Llama weights and outputs. LangChain? "Your favourite coding and building environment"? Who cares? It seems you're not allowed to integrate this with anything else and show it to anyone, even as an art project.
[1] https://mistral.ai/licenses/MNPL-0.1.md