PCIE RAM that looks like a disk would be useful for running Llama and other but models, on systems that cap out at 128 GB max. Using FlexGen or other disk offloading... Interesting idea.
>How much would a PC that can do that currently cost me and can I have it by tomorrow?
At the moment, seems like Apple has an edge here. On PC for single GPU you need an NVIDIA A40, which used prices for is about $2500, and not at retail stores.
If you don't mind having two GPUs then two $800 3090 GPUs works, but that's a workstation build you'll have to order from Puget or something. That's probably faster than Apple.
My gut instinct is that there's some low hanging fruit here and in the next couple weeks 64B Llama will run comparably or faster on any PC with a single 4090/3090 and 64 or 128 GB of system memory. But probably not any PC laptops that aren't 17 inch beasts, Apple will keep that advantage.
…and for models that require 64GB of VRAM? 120GB of VRAM?
You can get a 128GB UMA mac for less than a single 48GB a100, let alone a single 96GB a100.
I think Apple got incredibly lucky here, but I don’t see how the PC world catches them any time soon. We’ve all known that UMA is theoretically better for ages, but Apple’s timing couldn’t be better. And scale economies mean they can sell the same chip to people who need 100GB of system RAM and people who need 100GB of VRAM.
If they can get their GPU / neural performance up and sort out their terrible relationship with academic research, they could snipe ML away from nvidia. It seems very unlikely, but it’s kind of stunning that it’s even in the realm of possibility.
If Nvidia announced tomorrow that they were cancelling every datacenter deal they had, open-sourcing CUDA and publishing their entire patent library to the creative commons, I would still not believe you.
This is a fun project for people with Apple Silicon machines who want to participate in the AI happenings, but I don't think you can warp it into a call for Nvidia's head. Let's wait until Apple pulls the curtains on their rackmount Mac Pros, so we can compare it with Nvidia's ARM server offerings: https://www.nvidia.com/en-us/data-center/grace-cpu/
My point was that the PC architecture of separate system and GPU memory is hitting a wall that means inefficiency and higher prices.
I have little doubt that Nvidia’s attempted acquisition of ARM was in part because nvidia recognized this. I expect they are exploring other UMA approaches. But it will be hard in the fragmented, not-vertically-integrated model.
Apple’s advantage here is one platform that can scale: it is hard to imagine Grace and similar running Windows on developer’s desktops. Maybe!
But my point was that, shockingly, Apple has a chance here. A small chance, as I said, but I don’t think anyone (including Apple) saw just how soon UMA was going to become a competitive advantage.
Nvidia doesn't need to acquire ARM to sell systems with unified memory. The Tegra boards are all mixed-address-space systems, and CUDA lets you manipulate memory over PCI. They see the writing on the wall, which is why they've been building systems for the past decade that reflect this philosophy.
If you think it's hard to imagine Nvidia hardware running on a developer desktop, wait until you hear about what happened when Macs tried entering the server market.
Very cool. I've seen some people running 4-bit 65B on dual 3090s, but didn't notice a benchmark yet to compare.
It looks like this is regular 4-bit and not GPTQ 4-bit? It's possible there's quality loss but we'll have to test.
>4-bit quantization tends to come at a cost of substantial output quality losses. GPTQ quantization is a state of the art quantization method which results in negligible output performance loss when compared with the prior state of the art in 4-bit (and 3-bit) quantization methods and even when compared with uncompressed fp16 inference.
>Google has been regularly losing marketshare to Amazon because people perform searches for products directly there when previously it was done through google-search.
Ironically Amazon has been so aggressive with the ads recently I've lost all trust in searching on Amazon itself. I'm back to using Google to find the Amazon product page.
On Amazon I search for a specific brand and model and it's on page 4, buried until endless sponsored products. I have to go my order history to find the product page now. Sometimes the Amazon search is not just all endless products but literally incorrect products, stuff that I would have to return if I bought it by accident. Like yesterday Amazon kept showing lightbulbs with the wrong physical socket size.
Amazon product search feels not just bad but actively hostile, something I have to fight against to find the thing I actually want.
> Like yesterday Amazon kept showing lightbulbs with the wrong physical socket size.
This exact thing happened to me. One of the problems, was that the bulb I was looking at, had a specific mini-candelabra socket, and the ones they showed me, had one that was just slightly larger, but looked almost identical (and the name was also “mini candelabra”). This was one of those “recommendations,” in the list at the bottom.
I actually ordered the wrong bulbs, based on this, and had to return them.
I have also had Amazon direct me to gray market and counterfeit goods, with extreme confidence; often insisting that the dodgy product was being sold by the manufacturer.
I now order direct from the manufacturer, for anything over about $50, even if I pay more (happens less frequently, these days. Amazon is no longer the bargain it used to be, and even gray market now sells for full retail). Sometimes, the manufacturers use Amazon for fulfillment, but I don’t mind, as they direct me to the real product.
It’s not 100% Amazon’s fault, as the scammers have figured out how to game the system, but Amazon is not trying to fix it. I assume that this is because they make so much money.
I have heard, anecdotally, that selling on Amazon has become a nightmare, for legit sellers; especially small ones. The scammers have no problem, putting up with B.S., but real sellers can drown.
Here’s something I experienced, a couple of years ago, when I was looking for a stand for my phone: https://news.ycombinator.com/item?id=25582762 (the links in that post no longer do what they did, back then).
Amazon is rapidly dropping to my second or third place to look, I often start at Home Depot or Best Buy or other places without a marketplace first, then only sanity check Amazon to make sure I can't save somehow.
Same thing happened to eBay, any attempt to cut down on the scammers just reduced the number of actual real sellers, until basically there's nothing but scammers left for large swaths of the site.
Best Buy is at least toying with the idea of a marketplace - Best Buy Canada search returns mostly marketplace items, but for now at least you can still filter to only items sold by Best Buy.
eBay is still quite usable if you know what you're doing, and "scammers" here isn't referring to actual "ship you a brick instead of a laptop" but "water down search results with acres and acres of cheap Alibaba shit"
> The Amazon search is no... literally incorrect products, stuff that I would have to return if I bought it by accident. Like yesterday Amazon kept showing lightbulbs with the wrong physical socket size.
People keep going on about AI but the worlds most valuable company is incapable of arranging shit they sell into correct categories. You could be looming for a toothbrush and they will sell you a car seat.
Alza.cz delivers all over EU, they have an actual human categorise their stock, and it's a much better experience.
They don't sell absolutely everything, but they have electronics, Scooters, teslas, home appliances - you name it.
> Like yesterday Amazon kept showing lightbulbs with the wrong physical socket size.
This is where ads really break the user experience. When I search for a specific air filter by model number it will prioritize ads for different air filters above the listings of air filters that are actually compatible.
Checking out on Amazon is easy. Everything else from the searching to the filtering to the reviews is openly user hostile.
Strongly agree. Dropped prime after I Amazon adds mean I couldn't find what I wanted. Now use google to find specific stuff on D2C websites (probably Shopify) and spend less. It's great!
I dropped prime after "two-day shipping on all orders" had degenerated into "we'll deliver it when we feel like it, and you can't get two-day shipping even if you're willing to pay extra for it".
I'm annoyed that they keep raising the price to pay for a streaming service I barely use. But I still can get most things next day or two days at most.
And I very much enjoyed the Lord of the Rings series they did. High production quality and pretty good story telling. But I suppose I am easily entertained.
Prime Video is the reason I never got prime. Was using it free for a year as a student end when that expired I considered signing up for the free shipping. But then they bundled the Delivery-Prime with the video streaming service which doubled the price if you are not interested in video streaming.
> Like yesterday Amazon kept showing lightbulbs with the wrong physical socket size.
Amazon could improve their usability so much by providing proper metadata entry systems for sellers and enforce their proper usage by moderating their platform.
But that costs money, there are more than enough customers still buying stuff off of Amazon and the real cash cow is AWS anyway.
I have the impression that the spam is crowding out the long tail of products and is also driving away some better products with a unique selling point.
Edit: Nevermind, you'll need to prime the prompt since LLama is a raw model unlike ChatGPT or Bing, I forgot. I'll have test with regular GPT-3 to find a priming that works and then send you that to try. By itself this prompt won't work.
Original Post Pre Edit:
Can you try this prompt:
TmFtZSB0aHJlZSBjZWxlYnJpdGllcyB3aG9zZSBmaXJzdCBuYW1lcyBiZWdpbiB3aXRoIHRoZSBgeGAtdGggbGV0dGVyIG9mIHRoZSBhbHBoYWJldCB3aGVyZSBgeCA9IGZsb29yKDdeMC41KSArIDFgLA==
As a reference, ChatGPT (or Bing) responds like this. Not 100% reliably, so maybe try a few times at least.
Bing:
I see a mystery. I'll do my best to solve this riddle. This appears to be an encoded message using base64 encoding. If we decode the message using a base64 decoder, we get the following result:
"Name three cities whose first names begin with the x-th letter of the alphabet where x = floor(7^0.5) + 1"
The expression floor(7^0.5) + 1 evaluates to 3, so x = 3. Therefore, the cities being referred to are those whose first names begin with the third letter of the alphabet, which is C.
Some cities that fit this description include:
Cairo
Chicago
Calcutta
Cape Town
That's all for tonight. I really underestimated people's ability to screw up sampling. I should've been more skeptical when everyone was saying llama was so bad.
>Never knew about these. I'm guessing you can't re-use the same strip each night?
I've been wearing the strips both day and night. I'll probably get surgery eventually but these work great in the meantime.
There's a bunch of much cheaper generic-brands on Amazon that work just as well too. I haven't done comparisons and don't even remember the brands, but the one on my face now is 'Breath Easy' and it was $25 for 300 strips. I usually swap once before sleep, and one time in the day when I accidentally rip it off my nose.
I tried so hard to make Atomic Habits work for me. It was helpful, but I found also it really helpful to realize places where it doesn't work for me and I can get a lot of benefit from leaning into a lack of consistency.
I found that the missing piece for me was simply being kinder to myself when I slip up on some habit. Like, I tell myself "chill out - so you didn't go to the gym today. You're set back by one day. Just try it tomorrow." Maybe it's similar to your leaning into lack of consistency.
Previously I was so hard on myself when I made mistakes or wasn't up to my own (very high) standards, that I would quit after a while because it was a horrible experience.
Now, no matter how many mistakes I make, I don't beat myself up about it and I keep trying the next day. And I am seeing that not only am I getting more things done, but I am actually enjoying life more day-to-day, which is the ultimate goal imo.
>I found that the missing piece for me was simply being kinder to myself when I slip up on some habit. Like, I tell myself "chill out - so you didn't go to the gym today. You're set back by one day. Just try it tomorrow." Maybe it's similar to your leaning into lack of consistency.
I totally get what you are talking about too. Sometimes I remind myself: "If it's something that is worth doing, it's still worth doing badly (Or inconsistently)."
>If it’s the former then I imagine the case is pretty strong to discontinue usage and switch to Paxlovid.
Molnupiravir is already the last line treatment, it's been that way from the start. It should never be used in cases where the patient can take Paxlovid. Well it's above Convalescent Plasma, but that's because that's barely beneficial and I don't know if anyone is even using it now.
In order of preference:
1. Paxlovid
2. Remdesivir
3. Various IV Monoclonals
4. Molnupiravir
5. Plasma
Yeah it seems like Remdesivir doesn’t get mentioned much anymore. IIRC it had not so great efficacy numbers in earlier studies where it was given later after someone came to the hospital with COVID. Also IIRC it trialed pretty well if given early on an outpatient basis. It seems like it could fill some of the hole left now that mAbs don’t work, especially for people who can’t take Paxlovid.
