Rendered at 23:30:07 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pinyen_cheng 14 hours ago [-]
That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
alansaber 11 hours ago [-]
Exactly. I used to think new tokenizers for domain-specific models was the way (makes intuitive sense, better representations etc), but the added overhead of essentially training from scratch made it much more expensive than a FT.
OOTY00TYOO 11 hours ago [-]
If we cant' build the hardware fast enough. Can someone explain to me why there's apparently 50% of the gpu and memory that hyperscalers buy sitting in warehouses due to lack of data centers? The hyperscalers are hoarding tech and can't even use it. It's ridiculous.
Lerc 9 hours ago [-]
As far as I can tell, because that is not happening.
The suggestion seems to have come from the claim that to maintain a priority queue, you have to be buying consistently. That might mean buying more than you need to ensure supply when you need it. The assumption that the excess goes into a warehouse seems to have come from someone's fevered imagination.
If there are GPUs built and not yet installed, and people who want to be using them now, money can be made getting the supply to find the demand. Most large scale companies know how to do that sort of thing. If someone has a data center and no GPUs arriving for six months, paying a few million to trade the GPUs that will exist in six months for the ones that exist now gets them an additional six months of operation, while ensuring the original buyer will have GPUs when they need them, plus a few million.
torginus 8 hours ago [-]
Yeah. Don't know anything about data center GPUs, but it's common knowledge you can get like 80% of the performance of a gaming GPU at half the power or even less. That's basically how Nvidia's mobile GPUs are. So if you had a warehouse full of GPUs, you could just go ahead and install them, run them at low power, and still get like 2/3rds of the performance.
bigbadfeline 1 hours ago [-]
> So if you had a warehouse full of GPUs, you could just go ahead and install them, run them at low power, and still get like 2/3rds of the performance.
This doesn't add up. First, it's still hoarding, I'd call it lukewarm GPUs as opposed to dark GPUs. Second, nobody would do that because it needs more space and maintenance, more racks, along with the added wear, tear and power supply losses that come with them. How do you people come up with these genius ideas, justifying the documented hardware hoarding [1]?
Here's another fun case, where some GPU hoarders simply forgot about a truckload of GPU's and didn't notice it was missing for full two weeks after it was stolen. It doesn't look like they need GPU's for "urgent installation" [2].
There are reports and rumours that NVIDIA are forcing the data centre companies to buy the GPUs upfront. Though there isn't a lot of clear evidence that this is the case.
I dont understand how this comment connects to the linked research paper. Anyone?
tomrod 5 hours ago [-]
There as a Youtube overview that tried to tie the research paper and the consequences on the market that I saw on this topic.
I suspect other influencers/popularizers are parrotting the same claims. Its not directly related IMHO.
IanCal 11 hours ago [-]
That depends on how long it’s sitting there, given they are two different things to manage it makes sense that you’d either have gpus sitting unused while a dc gets built or a dc sitting empty while you wait for gpus.
alansaber 11 hours ago [-]
They could probably rush them to production, but it's added logistics and would lead to the parts degrading faster.
owebmaster 9 hours ago [-]
Lack of customers*. SpaceXaitwitter has some big beautiful idle datacenters with all GPUs ready to use.
idiotsecant 11 hours ago [-]
Because slowing down compute purchase is collapsing the hype train which is the same as death. Vibes are a wave that must be maintained or crash down onto reality.
dormento 10 hours ago [-]
Also, tinfoil alert, if they buy it means you can't (or are eventually priced out of the hw market). Their only "moat" (if you can call it that..) is the price of local inference capacity, which is still kinda prohibitive for the 99%.
swiftcoder 15 hours ago [-]
Who's going to train a nice big coding model based on this? I don't have the GPUs to do it...
LoganDark 14 hours ago [-]
I hope Apple Silicon will one day be competitive with GPUs. It dominates much of the CPU market, and it has a GPU, but it's still not even remotely comparable to Nvidia for inference (factor of >5 in some cases, especially in prefill) or even for gaming ("edge computing" in Nvidia terms).
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
mycall 9 hours ago [-]
Depends what you are comparing. Apple Macbook Pro M5Max out performs NVIDIA RTX N1X laptops for about the same cost.
LoganDark 8 hours ago [-]
Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090.
Not comparing price or efficiency or value or anything like that. Just the platforms as a whole.
swiftcoder 8 hours ago [-]
> Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090. Not comparing price or efficiency or value or anything like that. Just the platforms as a whole. So M5 Ultra vs RTX 5090
The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
If you want to buy the 16x RTX 5090s you need to get to the same 512 GB of RAM, we're talking $100,000 just in GPUs, industrial levels of power/cooling/noise, and the interconnect bandwidth is still going to kill it for normal inferencing workloads.
Of course, that 16x cluster is an absolute beast in terms of raw compute - it's just not all that useful for the things people are trying to do with such machines today.
LoganDark 6 hours ago [-]
> The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
I agree that Apple silicon is far better in terms of memory capacity, which is part of why I'm rooting for it. But an SoC like Apple silicon is roughly analogous to a CPU+GPU when it comes to compute, and the GPU side is quite lacking compared to today's "traditional" GPUs, even though the CPU is somewhat industry leading.
Mainly, I'm disappointed with the performance of dense LLMs on Apple silicon compared to Nvidia, due to the large disparity in both memory bandwidth and GPU compute. Apple's latest chips are starting to push into just over a terabyte per second of memory bandwidth, about two-thirds of what the 5090 achieved nearly two years ago now. They are catching up quick and I do hope they pass, but they haven't quite yet.
I also hear that Apple silicon is pretty bad at prefill compared to Nvidia even when memory bandwidth is not an issue, but haven't verified. It would make sense though due to the much greater compute.
siamaksalman 15 hours ago [-]
I'm trying to run it, so far no luck
cubefox 11 hours ago [-]
This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
johntb86 7 hours ago [-]
Maybe if it's worse than next-token-prediction it could still be used as a drafting model, similar to how DFlash works.
cubefox 6 hours ago [-]
What's that?
johntb86 6 hours ago [-]
https://arxiv.org/abs/2602.06036 - they use a faster (single-pass) diffusion model to generate predictions for 7 to 15 tokens in a single step. Then if they're accepted that really speeds up the autoregressive model because it doesn't have to generate tokens one at a time.
I keep getting clickbait youtube videos about how "NCM" obsoletes "LLMs" and "Nvidia memory chips" (yes, it's that bad).
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
hsuduebc2 13 hours ago [-]
I think they think an LLM is just whatever another LLM tells them it is when they ask it to generate a script.
alansaber 11 hours ago [-]
Hey, me too.
12 hours ago [-]
totetsu 14 hours ago [-]
Can we apply any insights from image and video compression and partitioning to this kind of dimensional masking ?
clilog_d 16 hours ago [-]
[flagged]
logicalnerds 10 hours ago [-]
[dead]
shiandow 13 hours ago [-]
I'm not sure why but using X instead of the word "times" annoys me. Probably because one of the first things I got taught at university is that mathematical notation is not prose and shouldn't be used as such.
spiderfarmer 13 hours ago [-]
But what about “got taught” instead of “was taught” and the missing comma before “but”?
shiandow 13 hours ago [-]
Well they didn't teach me English. Or in English for that matter.
And I don't think I've ever used the Oxford comma.
There was probably some rule about not beginning sentences with Or and And but you probably know better than I.
wccrawford 11 hours ago [-]
Different cultures have different norms and rules. I think you're saying that, in your native language, they taught you not to use mathematical notation in prose.
They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.
sorenjan 9 hours ago [-]
The first time I heard anyone pronounce 2X as "two ex" instead of "two times" was Steve Jobs in one of his presentation.
shiandow 10 hours ago [-]
It's a more general rule about when and where to use mathematical notation. Though style rules do differ in places.
For what it's worth I've not seen "12x" in formal contexts, but if that is proper then the times symbol still shouldn't be styled as part of the maths formula imho.
jabron 13 hours ago [-]
The Oxford comma is the comma before the conjunction in a list of three or more terms, e.g. "one, two, and three". I think your sentence actually didn't need a comma anway.
cubefox 12 hours ago [-]
I wonder whether "10" in "10×" also counts as mathematical notation. Or if just "ten times" is allowed.
shiandow 11 hours ago [-]
Technically yes, but quoting mathematical constants is definitely allowed. You could also say O(N) times for instance. The kind of language that is discouraged is writing "therefore efficiency increases by O(log(N)) times" as "∴ efficiency increases by O(log(N)) *".
For one it simply looks like you forgot to finish the formula.
cubefox 6 hours ago [-]
I particularly dislike Greek letters, because I don't know how to pronounce them in my head. For "×" you at least know it's supposed to read "times".
The suggestion seems to have come from the claim that to maintain a priority queue, you have to be buying consistently. That might mean buying more than you need to ensure supply when you need it. The assumption that the excess goes into a warehouse seems to have come from someone's fevered imagination.
If there are GPUs built and not yet installed, and people who want to be using them now, money can be made getting the supply to find the demand. Most large scale companies know how to do that sort of thing. If someone has a data center and no GPUs arriving for six months, paying a few million to trade the GPUs that will exist in six months for the ones that exist now gets them an additional six months of operation, while ensuring the original buyer will have GPUs when they need them, plus a few million.
This doesn't add up. First, it's still hoarding, I'd call it lukewarm GPUs as opposed to dark GPUs. Second, nobody would do that because it needs more space and maintenance, more racks, along with the added wear, tear and power supply losses that come with them. How do you people come up with these genius ideas, justifying the documented hardware hoarding [1]?
Here's another fun case, where some GPU hoarders simply forgot about a truckload of GPU's and didn't notice it was missing for full two weeks after it was stolen. It doesn't look like they need GPU's for "urgent installation" [2].
[1] https://www.datacenterdynamics.com/en/news/microsoft-has-ai-...
[2] https://www.tomshardware.com/pc-components/gpus/six-pallets-...
[1] https://tech.yahoo.com/gaming/articles/did-nvidia-manufactur...
[2] https://www.reddit.com/r/Games/comments/1xdjbd1/multiple_rep...
[3] https://www.techradar.com/computing/gpu/is-the-gpu-stock-cri...
[4] https://www.businessinsider.com/amazon-nvidia-ai-gpu-data-ce...
I suspect other influencers/popularizers are parrotting the same claims. Its not directly related IMHO.
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
Not comparing price or efficiency or value or anything like that. Just the platforms as a whole.
The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
If you want to buy the 16x RTX 5090s you need to get to the same 512 GB of RAM, we're talking $100,000 just in GPUs, industrial levels of power/cooling/noise, and the interconnect bandwidth is still going to kill it for normal inferencing workloads.
Of course, that 16x cluster is an absolute beast in terms of raw compute - it's just not all that useful for the things people are trying to do with such machines today.
I agree that Apple silicon is far better in terms of memory capacity, which is part of why I'm rooting for it. But an SoC like Apple silicon is roughly analogous to a CPU+GPU when it comes to compute, and the GPU side is quite lacking compared to today's "traditional" GPUs, even though the CPU is somewhat industry leading.
Mainly, I'm disappointed with the performance of dense LLMs on Apple silicon compared to Nvidia, due to the large disparity in both memory bandwidth and GPU compute. Apple's latest chips are starting to push into just over a terabyte per second of memory bandwidth, about two-thirds of what the 5090 achieved nearly two years ago now. They are catching up quick and I do hope they pass, but they haven't quite yet.
I also hear that Apple silicon is pretty bad at prefill compared to Nvidia even when memory bandwidth is not an issue, but haven't verified. It would make sense though due to the much greater compute.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
And I don't think I've ever used the Oxford comma.
There was probably some rule about not beginning sentences with Or and And but you probably know better than I.
They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.
For what it's worth I've not seen "12x" in formal contexts, but if that is proper then the times symbol still shouldn't be styled as part of the maths formula imho.
For one it simply looks like you forgot to finish the formula.