My three NVIDIA cards are more power efficient than my one AMD card, both at idle and during usage.
Official ROCm is like pulling teeth with poor support for desktop cards. Debian, a volunteer led project, have better ROCm CI than AMD and support more cards.
Look at any benchmarks. NV midrange cards are faster than AMD and at least a generation in front. Owning a 7900XTX is an embarrassing disappointment.
I like AMD and want them to succeed, but they are way behind NV in this area.
> Official ROCm is like pulling teeth with poor support for desktop cards...
I agree with most of your post and fled the AMD ecosystem some time ago because of the machine learning situation, but their problem seemed to be more the firmware bugs and memory management of compute shaders than the higher level libraries.
The obvious solution to this one would be not to use ROCm. ROCm has always been a bit of a train wreck for small users and it doesn't seem to do anything special anyway. The way forward would be something more like Vulkan which the server that today's link points to seems to be using. The existence of a badly managed software package doesn't really imply that users have to use it, they can use an alternative.
It would be nice if AMD sorts themselves out though. The NVidia driver situation on linux is painful and if AMD can reliably run LLMs without the hardware locking then I'd much rather move back to using their products.
Yes, AMD themselves even use Vulkan tg numbers in their marketing material, because it's faster than ROCm on everything RDNA2 onwards (seems embarrassing).
However for pp, Vulkan is still nowhere near close to ROCm. That matters for long context and/or quick response. A lot of people really care about that time-to-first-token.
Have a Strix Halo 128 running Qwen 3.5 122b at 35t/s using Vulkan and kernel 7.0.0 on a 400w PSU. Pretty hard to beat for the price and power consumption IMO. But to be fair I compile everything myself so proprietary drivers required by nvidia are a non starter for me.
