How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.
Are you thinking of fundamental architectural changes (like the Transformer)? Or incremental (like MoE or GQA)? Are there specific neolabs or techs you are following that lead you to this prediction?
Indeed it seems quite possible we are one architecture breakthrough away from existing chip stock driving us all the way to ASI.
Nobody has done AFAIK the information theory to prove it’s not possible.
The fundamental issue is that for these systems bigger is essentially always better (if affordable). So if we can squash something like Kimi K3 down to run on a 'normal' system, that just incentivizes devs to increase the model size until once again we're at the limit of what can be run.
Hyperscalers are screwed, data center mania couldn’t even be completed during this massive spending spree, all while the people seemingly sitting on the sidelines are working on getting these things baked into the OS and chipsets in a way consumers wouldn’t notice
I have nothing to say about this company's long term potential, but I'm curious to know whether this will become an enormous "tik stok" like SK Hynix became in South Korea. I've watched a short documentary about the investment mania in SK and it's really scary.
If the golden stick of memory sitting on Trumps desk has its intended effect - and the US allow Apple to buy memory from CXMT then another 500% surge might happen.
The more interesting question is: why does the rest of the world not increase their production prowess? I am aware of most of the reasons, but I am getting tired of the "China is better than the rest of the world". It is more that China did a lot of things that the others either failed to do or no longer wanted to do. Case in point is Germany: from "our cars are best" to "hey, chinese cars are now better" (or, cheaper, for a comparative quality). This constant complaining is so strange. Would that energy not be better spent to try to become better?
It is because in the west we suffer from self inflicted obstacles that make it impossible to get out of our own way. The Chinese gov must think they are playing on easy mode.
> Would that energy not be better spent to try to become better?
The problem is, shareholders have bled Western companies dry in the chase for returns on investments. There is no money left in the companies to retool them to be competitive with China.
Meanwhile, the CCP just pours dollars into their companies.
Unfortunately, there is no punishment in Western societies if you run a company into the ground as long as you pay your debts on the way out.
It takes 2 years in EU to get approval to build a shed. Imagine how long it takes to get approvals for a chip factory which uses a lot of water and very toxic chemicals.
UK has been trying to get approvals for a new runway at Heathrow airport for over 20 years !!! now.
Because the capitalist west doesn't have strategic view, beside maybe on the defense sector.
Everything else must produce profits next quarter not in the next 3 years.
It's this. Wall Street won't allow american companies to make long term decisions and long term planning that cost money in the short term, because they want all that money immediately now. Like, right now now. If an executive promised they could deliver big profits in 5 years by spending the next two investing their profits, Wall Street would fire them and put in someone who will happily spend all the profit (plus debt) to do stock buybacks and dividend disbursals.
Of course CXMT, Micron, and family will still be profitable, but maybe not 'surge 470% from IPO' profitable.
Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.
Some groups are baking models into silicone, Deepmind has an example, it gets 18,000 tokens/sec on Llama 3.1, not sure about parameter size
Indeed it seems quite possible we are one architecture breakthrough away from existing chip stock driving us all the way to ASI.
Nobody has done AFAIK the information theory to prove it’s not possible.
It's like people saying mobile chips are going to be better than PC chips... until they realize PC can be made by mobile chips too if that comes true.
Hyperscalers are screwed, data center mania couldn’t even be completed during this massive spending spree, all while the people seemingly sitting on the sidelines are working on getting these things baked into the OS and chipsets in a way consumers wouldn’t notice
So others did spend energy on becoming better and now you see a different picture when you look at the world.
The problem is, shareholders have bled Western companies dry in the chase for returns on investments. There is no money left in the companies to retool them to be competitive with China.
Meanwhile, the CCP just pours dollars into their companies.
Unfortunately, there is no punishment in Western societies if you run a company into the ground as long as you pay your debts on the way out.
UK has been trying to get approvals for a new runway at Heathrow airport for over 20 years !!! now.
how much more do they need to ship to satisfy you?