More realistically I think you'd need something like "Now write your post in the style of a space pirate" with a 10 second deadline, and then have another LLM checking if the two posts cover the same topic/subject but are stylistically appropriate.
I'm running the Q4_K_M quant on a xeon with 7x A4000s and I'm getting about 8 tok/s with small context (16k). I need to do more tuning, I think I can get more out of it, but it's never gonna be fast on this suboptimal machine.
you can add 1 more GPU so you can take advantage of tensor parallel. I get the same speed with 5 3090's with most of the model on 2400mhz ddr4 ram, 8.5tk almost constant. I don't really do agents but chat, and it holds up to 64k.
That is a very good point and I would love to do it, but I built this machine in a desktop case and the motherboard has seven slots. I did a custom water cooling manifold just to make it work with all the cards.
I'm trying to figure out how to add another card on a riser hanging off a slimsas port, or maybe I could turn the bottom slot into two vertical slots.. the case (fractal meshify 2 xl) has room for a vertical mounted card that wouldn't interfere with the others, but I'd need to make a custom riser with two slots on it to make it work. I dunno, it's possible!
I also have an RTX Pro 6000 Blackwell and an RTX 5000 Ada.. I'd be better off pulling all the A7000s and throwing both of those cards in this machine, but then I wouldn't have anything for my desktop. Decisions, decisions!
I thought paging was so inefficient that it wasn't worth doing vs using CPU inference for the parts of the model that are in system memory. Maybe if you have a good GPU and a turtle of a CPU, but still somehow have the memory bandwidth to make shuffling data in and out of the GPU worthwhile? I'm curious to know who is doing this and why.
I went looking for information on the oldest known lobster, and found this article about a 20lb lobster named George who was estimated to be 140 years old. Neat!
It doesn't have CUDA beacuse that's NVidia-only and it doesn't have OpenCL unless you use the binary-only drivers, which only work on a handful of very new cards.
I actually switched from a 7900 XTX to a 4090 BITD because I wanted CUDA, so I get that angle, but that doesn't mean I go around telling people "AMD isn't accelerated," because it's not true and it's a silly thing to try to claim.
This is a surprising opinion to encounter, given my experience with scaling on Windows, where simple things like taking my laptop off its dock (going from desktop monitors to laptop screen) causes applications to become blurry, and they stay blurry even when I've returned the laptop to the dock. Or how scaling causes some maximized window edges to show up on the adjacent screen. Or all manner of subtle positioning and size bugs crop up.
Is this more of an aspirational thing, like Windows supports "doing it right", and with time and effort by the right people, more and more applications may be able to be drawn correctly?
[edit] I guess so, I see your comment about setting registry keys to make stuff work in Microsoft's own programs. That aligns more closely with my experience.
Not sure about the underlying reason, but I use Windows for work and the only program I've encountered in the past two years with this behavior is the Eclipse IDE. Everything else deals very well with rescaling and docking / undocking to 4k displays.
I did this and was happily Windows-less for quite a few years. I ended up building a PC with a big GPU and so I switched back to PC gaming with a Windows installation alongside Linux, but I still think the console route is a great option.
reply