Rendered at 22:30:28 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
LaurensBER 24 minutes ago [-]
I'm getting questions weekly about running AI in-house now, in Europe trust in both the US and China seems to have hit an all-time low and AI is already scary for a lot of people.
I have no doubt that local first is the future that we're heading towards. We're seeing "algorithmic progress" towards more efficient models and _some_ hardware being available (NPUs, this chip, etc).
At the moment it doesn't seem like an appealing offer since it's just a little bit too constrained and since internet is universally available but if we can run a DeepSeek 4.1 Flash class model cheap and local, it becomes a no brainer.
augment_me 3 hours ago [-]
Quirks from the article:
1) The performance numbers are simulated
2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
NewJazz 2 hours ago [-]
The chip is designed to run models with 20 billion to 100 billion parameters (approximately 3 billion to 20 billion active parameters)
Doesn't seem like 100B dense models are even on the table, only MoE models. Kinda misleading.
gunalx 1 hours ago [-]
Not really. Models of that size is reasonably often moe. It is also 100b even if it only has 20b active at a time.
sampton 1 hours ago [-]
13.5W is Jetson nano territory.
idiotsecant 2 hours ago [-]
I think the human brain uses something on the order of 10-20W so 13.5W seems like there's probably some room to improve on a 4b model!
dinfinity 1 hours ago [-]
Statements like these often overlook that a human brain is pretty slow for a lot of tasks. It's hard to make direct comparisons, but for language (related cognitive work) we're at maybe a few tokens per second.
I wonder how even a team of humans (a meat mixture of experts) would perform on the AI benchmarks, measured in energy use per intelligence index/benchmark point.
egberts1 56 minutes ago [-]
"Few tokens per seconds"?!
Human brain is an advanced Next Concept Model.
danuker 8 minutes ago [-]
Back in July, I timed myself manually looking at some data files to create an index out of them, and out of curiosity asked some AI to do the same.
It took me an hour (61 minutes) and I made 4 mistakes in about 80 entries. It took DeepSeek-V4-Flash 3 minutes, it cost $0.034, and only made 2 mistakes. There is no contest. Humans have lost.
delichon 2 hours ago [-]
423. Could y'all please start prefixing comments with the number of joules you consumed in generating it, so that I can decide if it's worth reading?
Terr_ 2 hours ago [-]
Flashbacks to dial-up internet era, where any website would, parenthetically, indicate how big a linked file was with an estimated download time.
snapplebobapple 7 minutes ago [-]
Eleventy billion:watch snl celebrity jeopardy.
^what every advertiser will do if we actually do this lol
PaulKeeble 2 hours ago [-]
80 TOP/s and 150GB/s memory gives us the fundamentals of where this sits performance wise although its not said if the TOP/s is for INT4 or FP16 or what which changes the prefill performance considerably.
A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.
This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.
ilaksh 2 hours ago [-]
If it can work with 100B parameter models then stopping with edge devices makes no sense. This kind of compute-in-memory technology being so much more efficient would make most or all existing data centers obsolete. Or so it seems?
Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.
It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.
nomorewords 3 hours ago [-]
This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices
tintor 24 minutes ago [-]
autonomous hunter drones running 100B models
baalimago 3 hours ago [-]
> "large large language models"
tensward 4 hours ago [-]
Wonder how it would feel because the details shown seems a bit slow to edge device use
I have no doubt that local first is the future that we're heading towards. We're seeing "algorithmic progress" towards more efficient models and _some_ hardware being available (NPUs, this chip, etc).
At the moment it doesn't seem like an appealing offer since it's just a little bit too constrained and since internet is universally available but if we can run a DeepSeek 4.1 Flash class model cheap and local, it becomes a no brainer.
1) The performance numbers are simulated
2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
Doesn't seem like 100B dense models are even on the table, only MoE models. Kinda misleading.
I wonder how even a team of humans (a meat mixture of experts) would perform on the AI benchmarks, measured in energy use per intelligence index/benchmark point.
Human brain is an advanced Next Concept Model.
It took me an hour (61 minutes) and I made 4 mistakes in about 80 entries. It took DeepSeek-V4-Flash 3 minutes, it cost $0.034, and only made 2 mistakes. There is no contest. Humans have lost.
^what every advertiser will do if we actually do this lol
A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.
This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.
Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.
It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices