200 tokens per second isn’t achievable with a 1.5B even on low-midrange GPUs. Unless they’re attaching an external GPU it’s not happening on a raspberry pi.
This article is disjointed and smells like AI.
Yeah, my computer also runs a game at 200fps. But I’m not saying if it’s Minesweeper or a recent AAA game…
Yea… it’s not quite the same thing to actually run DeepSeek R1, a 671B model, and for example DeepSeek-R1-Distill-Qwen-1.5B
A recent i7 on CPU only can manage qwen 1.5 in a satisfactory way, comparable to big online players. Curious about recent ultra Intel and snapdragons
My old computer and laptop run 13B models at 2 - 3 tokens a second on CPU only. That’s a bit slower than I can read. I wonder what my new phone does.
How. I though u needed huge amounts of vram on exorbitantly prices GPUs to run LLM with decent capacity? Are the just running a really small model or is it hyper parametrised? Or is the “thinking” process just that effective u can make up for a weak LLM?
Even though it is the smallest of the distilled models that model still outperforms GPT 4o and Claude Sonnet 3.5.
The 7B parameter models crush the older models on performance benchmarks. The 14 billion parameter model is very competitive with OpenAI o1 mini in many metrics.
Yea sounds like it’s their smallest model
I was using their 7B model and it was kinda poop. Gonna try the 14B one next when I get home
Just tried some of them today and they failed at trivial (for a human junior programmer) code modifications.
