Its theoretical 120GB/s (with soldered lpddr5, the fastest) is between a M2 (100GB/s) and a M2 Pro (200GB/s).
For a portable solution it can be a nice compromise (only considering LLMs here). The only way to get 64GB of Ram (or more) with apple is with a M1/M2 Max. It's way faster (400 GB/s) but the retail price is at least ~4000€. The M2 Pro doesn't get above 32 GB of RAM.
For a while I considered a macbook pro, but the price tag, mixed with the fact that the SSD is soldered (just with LLMs, I plan a lot of mileage on them, and the failure mode of these macs is to just never boot up ever again, even with an external drive), and then I heard about the GPD Win Max 2 with an AMD 7840U. It's a lesser known brand, and I'll have to wait for early october to get mine with 64GB LPDDR5, but seeing some people receiving small parts after breakage/malfunction for previous gen also tipped the scale. For LLMs, it should be about a third the speed than a portable M2 Max, but it's a quarter the price, and I like some stuff that it can do that no other laptop can, so I'm fine with the tradeoffs.
I’ve had decent luck with zen minipcs- though the earlier gen 4700u. Take 64gb but too slow for LLMs realistically. Plus only 8gb max assignable to gpu.
I don't know if it applies to older gen, but the team behind mlc-llm are suggesting that at least with steamdeck's APU you can go beyond the cap, hopefully it applies to other APUs:
Regarding the choice of device, I'm regularly in places where 24h electricity/internet is not guaranteed, so renting cloud gpus: nope, bulky gaming laptop that chews through battery in a couple of hours: nope (even though I came really close to getting one with a RTX 4090 for 2600€, but I'm done with space heaters), frame.work 13? RAM to slow...
Mobile devices with at least 64GB of lpddr5 with decent battery consumption? The choice is quite limited.
No, the reason macs are better on LLMs is memory bandwidth 800Gb/s on Ultra 2 . I couldn’t find a good source but it seems that Ally mem bandwidth is around 70GB/s
A combination of high memory bandwidth and large memory capacity is necessary for good performance on LLMs. Plenty of consumer GPUs have great memory bandwidth but not enough capacity for the good LLMs. AMD's Phoenix has a memory bus too narrow to enable GPU-like bandwidth, and when paired with the faster memory it supports (LPDDR5 rather than DDR5) it won't offer much more memory capacity than consumer GPUs.
A mini PC with that chip, 1 TB of storage and 64GB of ram (both replaceable) costs like 800€ and fits behind your monitor. Getting that much memory in a consumer GPU is definitely quite a bit more expensive. Also, for comparison an M2 Ultra with that amount of storage and ram is 4800€.
So I am not doubting that a 6 times as expensive computer is probably "better" by some metric, but for that drastic difference I am not sure that is enough.
While I 100% agree on the price comparison, you’ll need to reach some threshold for LLM performance to consider them as usable. As Someone not very knowledgeable at the topic, the pure difference in the numbers lead me to question if you could even reach that usable performance threshold with the 800€ mini PC
Note that when referring to memory capacity, I specified LPDDR5, because that's the fastest memory option. If you want to go with 64GB of replaceable DDR5, you'll sacrifice at least 18% of the memory bandwidth. (And in theory the SoC supports LPDDR5-7500, but I'm not aware of anyone shipping it with faster than LPDDR5-6400 yet.) So you could get to 64GB on the memory capacity with a Phoenix SoC, but only by being at a 10x disadvantage on bandwidth relative to an M2 Ultra—which doesn't make a 6x price difference sound outrageous, given that we're discussing workloads that actually benefit from ample memory bandwidth.
