Funny thing is ‘Local ML’ tinkerers largely can’t afford GPUs in the US either.
The 5090 is ludicrously expensive for its VRAM pool. So is the 4090, which is all but OOS. Nvidia will only sell you a decent-sized pool for $10K. Hence non-techbros here have either been building used RTX 3090 boxes (the last affordable compute GPU Nvidia ever sold), EPYC homelabs for CPU offloading, or have been trying to buy those modded 48GB 4090s back.
The insane supply chain is something like this:
Taiwan GPUs -> China
China GPU boards -> US
US GPU Boards -> Smuggled back into China
Deneutered GPU Boards -> Sold back to US
All because Nvidia is playing VRAM cartel and AMD, inexplicably, is uninterested in competing with it when they could sell 48GB 7900s basically for free.
The issue is that nVidia are increasingly marketing their consumer grade GPUs to “prosumer” users. Whether that is small research groups working with “AI” or people farming the latest memecoin or… the other things you would need REALLY REALLY high bandwidth linear algebra from and let’s move on.
Whereas AMD are actually still targeting that consumer market. I think it was the nvidia 40x generation where their consumer cards had like no memory at all and AMD were pumping out 16 GB on their cheap(-ish) models? My brain can only remember card generations while I am actively shopping and… yeah.
And yeah. I would LOVE an AMD card with 32 or even 64 GB of even slower memory. But games are still going to target nvidia because people keep buying it and that means that you just won’t have much use beyond the 8 (or apparently now 16) GB that nVidia are going to let you buy. At which point… why waste money?
As for the prosumer and enterprise space? nVidia… have a long history of being assholes and previous GN videos have talked about the behind the scenes pressure they allegedly apply to system integrators and the like. And I will leave that there for Reasons.
But yes, many mid-tier and even high-tier companies could benefit from just buying AMD cards and there is very much a market for “high end” AMD cards… it is just that they have so few customers to make it worthwhile.
It’s not explicable why AMD is not breaking rank on VRAM and vGPU, same reason as failing Intel.
But the reason is not mentionnable in polite company and it relates to why AMD exists at all.
By all accounts, AMD should have gone under decades ago, instead they’re one of the only x86 platform licensee and they got that basically because of a fluke in history.
But here’s the real deal, because of the regulatory environment, monopolies are technically illegal.
Of course since 1980s enforcement of that has been a total joke as proven with the failure of the Microsoft anti-trust case.
Anti-trust is currently neutered and even back then it wasn’t really “anti-trust”, merely anti-monopoly.
That is the playbook for Intel and Nvidia that allows AMD to continue existing. They exist so that the other two aren’t monopolies.
They are kept alive as long, some market segments will have some competition and other will simply not be touched by Intel and AMD, dynamically decided, kind of like splitting territory, so that AMD can always survive.
So that Intel and Nvidia don’t became actually illegal monopolies.
This is a very conservative playbook as since the 90s they could have very well become monopolies and the neolibs wouldn’t have squeaked. They would have cheered !
That’s why you’re not getting that actually threatening to Nvidia amounts of VRAM and vGPU from AMD and why they’re dropping support for their 2018 datacenter GPU if they get a little too much of a good deal on the used market.
The solution is simple, destroy nvidia, not a calculated and gentle trust bust, no, break the company so it stops existing as a coherent entity.
Nvidia is a bunch of software and PNGs on top of TSMC. Break Nvidia, break Intel, break microsoft, break cisco, broadcom, break everything in silicon valley and put it all in a blender. And if anything grows too big again, break it the duck up again.
You could also buy the Apple Studio with its large amount of unified ram for a similar price of a 5090. Of course it’s not as fast but it could run a model that needs more ram.
The pricing for memory is still pretty bad. $4K for 96GB, $5.6K for 256GB, $10K for 512GB. One can get 128GB on the M4 Max for $3.5K, at the cost of a narrower bus so it’s even slower, but generally, EPYC + a 3090 or 4090 makes a lot more sense.
SOTA quantization for these are mostly DIY. There aren’t many MLX DWQs or trellis-quantized GGUFs floating around.
But if you want to finetune or tinker instead of just run, you’re at an enormous disadvantage there. AMD’s Strix Halo boards are way more compatible, but not standalone yet and kinda rare at this point.
Funny thing is ‘Local ML’ tinkerers largely can’t afford GPUs in the US either.
The 5090 is ludicrously expensive for its VRAM pool. So is the 4090, which is all but OOS. Nvidia will only sell you a decent-sized pool for $10K. Hence non-techbros here have either been building used RTX 3090 boxes (the last affordable compute GPU Nvidia ever sold), EPYC homelabs for CPU offloading, or have been trying to buy those modded 48GB 4090s back.
The insane supply chain is something like this:
Taiwan GPUs -> China
China GPU boards -> US
US GPU Boards -> Smuggled back into China
Deneutered GPU Boards -> Sold back to US
All because Nvidia is playing VRAM cartel and AMD, inexplicably, is uninterested in competing with it when they could sell 48GB 7900s basically for free.
The issue is that nVidia are increasingly marketing their consumer grade GPUs to “prosumer” users. Whether that is small research groups working with “AI” or people farming the latest memecoin or… the other things you would need REALLY REALLY high bandwidth linear algebra from and let’s move on.
Whereas AMD are actually still targeting that consumer market. I think it was the nvidia 40x generation where their consumer cards had like no memory at all and AMD were pumping out 16 GB on their cheap(-ish) models? My brain can only remember card generations while I am actively shopping and… yeah.
And yeah. I would LOVE an AMD card with 32 or even 64 GB of even slower memory. But games are still going to target nvidia because people keep buying it and that means that you just won’t have much use beyond the 8 (or apparently now 16) GB that nVidia are going to let you buy. At which point… why waste money?
As for the prosumer and enterprise space? nVidia… have a long history of being assholes and previous GN videos have talked about the behind the scenes pressure they allegedly apply to system integrators and the like. And I will leave that there for Reasons.
But yes, many mid-tier and even high-tier companies could benefit from just buying AMD cards and there is very much a market for “high end” AMD cards… it is just that they have so few customers to make it worthwhile.
It’s not explicable why AMD is not breaking rank on VRAM and vGPU, same reason as failing Intel.
But the reason is not mentionnable in polite company and it relates to why AMD exists at all.
By all accounts, AMD should have gone under decades ago, instead they’re one of the only x86 platform licensee and they got that basically because of a fluke in history.
But here’s the real deal, because of the regulatory environment, monopolies are technically illegal. Of course since 1980s enforcement of that has been a total joke as proven with the failure of the Microsoft anti-trust case. Anti-trust is currently neutered and even back then it wasn’t really “anti-trust”, merely anti-monopoly. That is the playbook for Intel and Nvidia that allows AMD to continue existing. They exist so that the other two aren’t monopolies. They are kept alive as long, some market segments will have some competition and other will simply not be touched by Intel and AMD, dynamically decided, kind of like splitting territory, so that AMD can always survive.
So that Intel and Nvidia don’t became actually illegal monopolies. This is a very conservative playbook as since the 90s they could have very well become monopolies and the neolibs wouldn’t have squeaked. They would have cheered !
That’s why you’re not getting that actually threatening to Nvidia amounts of VRAM and vGPU from AMD and why they’re dropping support for their 2018 datacenter GPU if they get a little too much of a good deal on the used market.
The solution is simple, destroy nvidia, not a calculated and gentle trust bust, no, break the company so it stops existing as a coherent entity.
Nvidia is a bunch of software and PNGs on top of TSMC. Break Nvidia, break Intel, break microsoft, break cisco, broadcom, break everything in silicon valley and put it all in a blender. And if anything grows too big again, break it the duck up again.
You could also buy the Apple Studio with its large amount of unified ram for a similar price of a 5090. Of course it’s not as fast but it could run a model that needs more ram.
The pricing for memory is still pretty bad. $4K for 96GB, $5.6K for 256GB, $10K for 512GB. One can get 128GB on the M4 Max for $3.5K, at the cost of a narrower bus so it’s even slower, but generally, EPYC + a 3090 or 4090 makes a lot more sense.
SOTA quantization for these are mostly DIY. There aren’t many MLX DWQs or trellis-quantized GGUFs floating around.
But if you want to finetune or tinker instead of just run, you’re at an enormous disadvantage there. AMD’s Strix Halo boards are way more compatible, but not standalone yet and kinda rare at this point.