Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
It's not that we're assuming the AI has made no mistakes. It's that we're confident in the general capability of the models such that we're willing to accept it'll make some mistakes, but working together with it, we'll still achieve our end goals.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
My colleagues write 10x less code for the same tasks (when they don't (ab)use AI) and don't reach past abstraction boundaries so that it's possible to actually reason about and review their work. When the problems get hard, they apply more care and attention to detail. Even if LLMs had the same bug rate per SLOC (IME it's higher), and even if those bugs were the same severity (IME they're worse), you'd still expect 10x bugs just because they can't be arsed to write a reasonable amount of code.
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
This is a different argument from Karpathy's prompt-and-forget one.
I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.
That's not really my point... my point is that it's an unsettled debate between speed and caution, and devs are not unanimous on it, unlike the claim of the original post that seemed to think everyone was reporting the same gains.
The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.
Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.
A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.
Yup. Also code review is not just there to check your work. It's forced communication, in that at least someone else knows about the changes you're making.
It's also a critical part of your security model. "Nobody can deploy code by themselves" is shockingly effective at preventing rogue actions.
Speaking of communication... Almost no one mentions talking to non-devs. That's one of the reasons I can't just rely on any of these tools, I have to remember and understand everything well enough I know how they work and can explain things and help plan ahead when talking to product people in meetings.
This attitude is exactly why AI bros don't make successful software lol. "Where's the software?" This. This is why there is no software.
Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.
You can do both. Or, at least, a lot of us can. It’s not like SOC2 auditing just disappeared suddenly. If you have figured out how to use it effectively, AI is enabling things that were previously impossible for resource constrained teams.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
I feel like this was always true to an extent, now product and design are just more tolerant of it because leadership is cracking a whip over their heads because things are supposed to be much faster “because ai” so things like pixel perfect hand offs, adhering to existing design system, making sure you are aligned with the brand, copy is reviewed with marketing go out the window. Now everything is shipping in hackathon mode with an eng driving decisions they really should not be.
> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
I keep hearing about this "when things break" argument, but things broke all the time with humans writing code too, if anything it feels like it's getting patched faster.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral
I'm thinking scbench.ai, and the fact that future token costs may become more prohibitive to just use it for everything.
For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.
There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.
Also, so many vibe coders don't seem to know that you need to keep third party elements patched.
People keep talking about AI slop but then I go maintain our largely human written code base and it's like worse than any slop I've seen from my AI usage. This is with code reviews, tests, and a supposedly strong engineering team.
That said AI feels a bit alien.
My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
What exactly do we want the general public to "do" with these models? If the have no use for agentic LLMs, they obviously aren't going to be interested in the capability of those agents.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
I don't think 99% of people have any application for AI as it is today short of personal assistants. But, if we can have recursively improving systems for that 1% of people that are responsible for scientific and technological innovations, that's going to be transformative for everyone.
I have been kind of disappointed by AI science-wise. I was anticipating more advancement. Experiments do take time, so I suppose they could be pending.
I don't think his point is that everyone should be using AI, it's that if you haven't used the agentic tools with a frontier model in the past month or so, you have no frame of reference for the current level of capabilities. Most people only use the chat interface with the free-tier models and based on that experience they're making strong conclusions about where AI is headed (it's still just fancy autocomplete, the whole thing is a scam or a bubble, etc.).
I think that's a bit reductive, there are decades of sci-fi novels about how transformative artificial intelligence would be, so I feel like I don't need to explain where the trajectory could go. The applications and "collision with the real world" aren't there yet, but you can't pretend that a machine that's smarter than any living human is on the same level as Amazon drone deliveries.
I agree with you, but to play devil’s advocate: what if it is on the same level as Amazon drone deliveries? Is intelligence really a bottleneck in society? People in MENSA with high IQs don’t often accomplish anything in proportion to their relative intelligence. The most intelligent people are rarely even in charge of making decisions. There are a lot of intelligent people with intelligent ideas that nobody listens to. It seems like the real bottleneck in society is building trust, consensus and constructive incentives between groups of people to do the things we already know are smart, not a lack of intelligence.
Good points. Maybe a more general version, removing constraints on one resource is usually not hugely impactful because you simply hit the next constraint.
Concrete example, "coding is free" (TBC, but let's assume), and now we're code review constrained. Even if code review becomes free we probably just become product discovery constrained, or something else. Making all software free will just cause us to hit on hardware constraints, and so on. It's still progress, and more progress will bring more benefits, but there's a limit to how much better the world as a whole can get from any one change like this.
MENSA selects for puzzle solving on the basis that those are correlated with "being smart".
Reinforcement learning selects for models that appear to complete tasks, on the basis that appearing to complete a task is correlated with completing a task.
Both have problems. The latter is probably a better correlation, but by no means flawless. If we just take at face value that RL is training models to complete tasks, we're not engaging deeply enough with the process enough to see where it falls short.
I don't disagree with where AI could go, but I don't think we've actually proven that it's there. My point is that the folks claiming we are there already if only the rest of the world would just recognise it, are repeating the same thing from every previous tech cycle. In many cases it's the same folks.
I feel like if the gap were so monumental that he could just point to a monument of it, rather than hand-waving at "the vertical professionals".
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
The 1%ers are in the vertical, but the question is vertical to where?
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
It is fine as it is to make a boatload of money. All they have to do is seed ads into it. Look at how much money social media brings in from that model of "free/useless product" as flypaper to bring in eyeballs for the real money which is ad revenue. Adsense 2.0.
It is always a mistake to assume that because computers can do X, that they will be able do everything that a human that can do X is expected to be able to do.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
Why wouldn't they be able to grow, its just one data point more in their training, in your notes or in their instructions.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
Because we don't let them learn. Everyone runs the same static generalist models, instead of having every model instance learn from everything it does. New models come out occasionally, but they have not been trained on the private data you have not shared with the AI company.
You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.
Because context, at least in my experience, doesn't seem to encapsulate the sort of dynamism that I'm thinking of. Conversations are flat and repetitive. It still feels like a very good semantic search more than talking to an intelligent human.
I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.
Honestly, outside of coding tasks, the AI Summary at the top of every Google search is adequate for 99% of what I need and I hardly even use ChatGPT any more.
It's just a google search but you can explain what you're searching for. AI summary may contain misleading details if you read it and take it literally, but 90% of the time it gives me pointer for the thing I need and I don't have to scroll through results that often could be just as misleading if not more.
It's demonstrably not. This summary, somehow, often fails to search, or fails to read what it searches and gives you the opposite answer. You shouldn't trust it.
Just in the last few days I had an example where the Google Search version of Gemini had a much better and more comprehensive answer, by far, than ChatGPT. It had to do with Amazon’s use of Data Matrix barcodes on shopping bags. Gemini was able to fully describe how they’re used in Amazon’s logistics chain. ChatGPT essentially said it didn’t know. Your preconceptions may not be accurate.
They might be, but still, I’ll be difficult to convince that Google’s shitty-ass 10B params instant answer models are good at anything other than cookie recipes and producing farts real fast for real cheap.
For what it’s worth, any time I see that feature, it’s all farts.
It depends on what you’re searching. If you only ever need really basic information like keyboard shortcuts or “is it & or &&?” then it’s pretty good, and definitely faster. And if it gets it wrong, then the shortcut doesn’t work or the code doesn’t compile and I search again and try the links.
Claude Code accelerated this divide. Programmers using Claude Code realized that giving LLMs access to a terminal made them feel 100 times smarter. Access to Bash, CLIs, and the ability to visit websites without the many restrictions made them powerful.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
A terminal also tells the model when it failed. Exit code, stderr, a stack trace. it reads that and tries again.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK.
- Slack puts {"ok": false} inside a 200.
- Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.
We run into that a lot using Mulesoft. 3 layer APIs, and each layer will discard error messages returned by the underlying layers unless we do tedious custom error handling. Makes it impossible for the teams using our integrations to know what they're doing wrong.
I wonder if in response to this companies like google will build a sort of agent facing layer to the app in question. So rather than you building an agent to attempt to wrangle pupeteer to click css elements as a user does, it calls some apis google exposes to certified gemini user agents and these contain exit codes and can be run as script like instructions not e.g. the shortcomings of browser agent based methods such as having to wait for all content elements to actually load and the obfuscation arms race some devs have towards exposing these elements towards headless agents.
I’ll say this until I am blue one the face. Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
That's just cocktail party level of thinking. LLMs are getting better at math, code and logic (and marginally better at science and general knowledge) because these are domains that can be objectively verified and thus there is a potentially infinite supply of 'facts' to generate and train on. These are very powerful but ultimately very abstract domains. For everything else the messy real world and its physical bottlenecks gets in the way and there's little reason to expect progress to accelerate. It still takes months to get mice to reproduce and run experiments on, no matter how knowledgeable about biology the models have become.
If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
Is good writing verifiable? I don't think it is, but LLM's have been hillclimbing writing quality. That being said this is with the help of RLHF.
However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.
>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
The quality of natural language output from today’s LLMs seems no better and often worse than LLMs from 18 months ago, at least in my experience. It’s much harder to build a good RL loop for something subjective like good writing, compared to something with an objective and verifiable right or wrong answer, such as software or mathematics.
IQ is a useless thing, but you can absolutely tell intelligence from writing. You don't need to write like [author of your choice], you need smart ideas and the desire to make them understood.
AI writing sucks because there's no intent or choice behind it. It's human thought, blended by linear algebra into pink slime and extruded into sentences.
It's not at all clear that LLMs have gotten significantly (much less hillclimbing) better at writing since GPT4-ish. They did get better at oulipo-like challenges (writing under silly, verifiable constraints like Fable's viral 'facetiously' poem) but that's far from being all there is to good writing.
Anecdotally, creative writing communities do report drastic differences in quality between models (it is my understanding that Gemma 4 is especially praised) so there is something to it being verifiable but it's a far cry from being something that can be distilled into objective math/code-like evals. One could even say it comes down to vibes.
>For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism.
Yes, biotech companies are doing this right now but ultimately they are just predictions and you still need cold hard biological data to ground them and iterate on. That's slow and expensive, especially for the juicy fields (human biology, food and crops, clinical trials) and you can't just plug billions of VC capital into the pipeline and hope for RSI. Physical (gotta procure all those labs and their equipment), human (gotta hire and pay specialists to run experiments), biological (gotta wait for organisms to reproduce, drugs to take effect, crops to grow), regulatory (gotta convince agencies that your fancy new drugs are legit) bottlenecks get in the way.
And biology is one of the easier fields where a path to RSI (if not accelerating) is conceivable. Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
>I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
My uncharitable take is that the abtruse jargon makes people feel smart for understanding it and gives a sense of belonging (as a closed circle of initiates who understand LLM cant), just like rationalists love to repackage old or unsavoury ideas under new nerdy smart sounding names.
>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI
Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
>By evaluations against human writers, LLM's have improved at writing.
I would really take those benchmarks with a pinch of salt. We already know benchmaxxing is a thing for evals that are objectively verifiable like code, and coding models that seem to perform well on benchmarks don't pass the vibes test, imagine all the caveats that come with such a subjective test like creative writing, not to mention control issues (are they the same human writers across benchmarks? Are they the same testers?)
>Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
I don't disageee and am in fact excited to see the advances, what I am arguing is that it is going to be slow because the hypothetic feedback loop that would improve the model is bottlenecked. There are actual physical and biological limits in our universe. Unless VCs find a way to make mice embryos and crops grow 10x faster (among the many, many other things that would be required for acceleration) the trickle of data is not going to be anything like the accelerating returns imagined for math and coding. And that's assuming biological RSI is possible which has yet to be proven - current foundation models are interesting but they are still specialized and there is no clear, obviously uniform way to encode the many, many modalities that exist in biological data.
>The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
You wouldn't need to 'just' simulate human behavior (good luck with that btw) but the entire political and economic conditions in which they live. Even assuming this is possible the amount of compute you'd need to run would be astronomical. Given how all RAM capacity for '27 and '28 is near sold out, hyperscalers are running out of cash flow to finance buildout and Sam Altman complains when users say 'please' to LLMs I don't think this is in the near horizon.
There was a time when people believed in Moore's law too. Now even Jensen Huang says Moore's law is dead. There's also the adage from investing: past performance does not guarantee future returns. A little CYA sure, but also acknowledgement of latent factors and the entropy of the universe and its affect on any predictive modelling one might do.
I’m not sure I agree that g explains all model capabilities. I think “claudespeak” is a consequence of the shift from training on RLHF to training on verifiable rewards. Thus the models seem to become worse at storytelling, fiction, readability, not better.
Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.
For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.
I don't agree that they get uniformly better at everything, especially subjective skills.
For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.
I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.
This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.
The delineation between knowledge and intelligence is useful here.
Knowledge can occasionally get in the way of some tasks, such as a master illustrator trying to draw like a child. Or as you've mentioned, an expert trying to explain to a beginner.
Even so, I disagree with you about model progress on communication - maybe there's a little jaggedness between minor model versions, but Opus 5.5 is a much much better communicator than, say, Claude 3 Opus.
> Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing.
LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.
As anyone who's needed to provide IT support to a doctor, lawyer, lauded academic or host of otherwise "smart professionals", can assert, you can be very very smart in one area and very stupid in another.
This is the classic error in mixing up knowledge / experience and intelligence.
Intelligence is the ability to quickly process, understand, learn, and use information. It does not imply having already been exposed to any given information.
A smart doctor may have terrible computer skills. But this is from lack of experience and interest, not lack of ability to learn how to use computers.
But take that smart doctor, and a relatively dumb person who is equally inexperienced with computers. Now give them a month and incentive to learn computers. The doctor will learn much more; that's intelligence.
There is every reason to think this. Its about available quality data. The data that was easiest to fetch was already there, then we got some more data by asking experts to create datasets for post training. Once this is over, we will get to the outer world that didn't get to hoard it for bots to take it. This will certainly change. Put on a smart glasses and record what you do to fix a pipe. In 3-5 years, rinse and repeat.
Maybe, I'm at the point I don't know what to think anymore. But somehow, this feels still non-human. Humans don't need millions of hours of training and millions of samples to learn how do to something. We have proof that systems can deal with low-shot training. So why can't these? My point is when we have human level learning ability, then we're moving, if we expect expansion of capabilities to come from data alone, I'm prepared for disappointment.
Being good (or superhuman) at software dev and maths shouldn't be treated as though it's equivalent to other domains.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
> The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".
The most passionate, intelligent people 1. aren’t spending a lot of time on smartphones, video games and crypto and 2. aren’t as much smarter or better at predicting the future as the barely above average people who mistakenly count themselves in their ranks like to think.
Most of those technologies were controversial in their inception among the nerds of the time. Remember Eternal September? Linux/Windows people making fun of Apple hipster fans? Mainframe engineers deriding PCs as unserious and video games as childish?
>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.
Also crypto is a grift lol, one of those is definitely not like the others
Steve jobs wasn’t a nerd - he said he’s a hippie on the lost interview. He was also the first to bring to market colour screens, typography etc. He also defined the Internet as the defining ‘social moment’ way before anyone else.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
I honestly don't get your point: perhaps your argument is based on some US-specific cultural reference that I am not aware of since I don't live in the USA.
If it is, I don't get it either. That's not at all what I imagine with hippies and I suspect that was supposed to be some other word.
Nerds are also not necessarily all about technology. "Math nerd" was a really common use of the term for a long time, for example. "Nerd" is someone with deep knowledge about a particular subject.
> For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.
Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.
> Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
I have no access to Astra, but with GPT 6.1 Sol, the results with respect to attempting to create STL, OBJ or even OpenSCAD files by using image references and a prompt were really bad.
I generally agree with the prevailing commentary here, i.e. that automating app development and maths proofs may not be as world-changing as the Artificial Intelligentsia purport / “fear” it to be.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
> Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
Fair assessment. Maybe the messaging should focus on “these are great accelerators for software developers” rather than “AI will change everything for everyone everywhere…” It is understandable that non-technical folks are kind of underwhelmed - not due to lack of understanding so much as lack of a tangible need. Not everyone needs an electron microscope or gas chromatograph…
I think it's a weird take because it implies that the 6 billion people he's talking about actually have some interest in knowing about the capabilities of LLMs to solve frontier math? This is simply not something they care about or can evaluate. There are maybe several thousand people in the world who have some (abstract and barely-monetizable) use for this information, plus probably another 100,000 who don't understand any of the math, but like to cheer on.
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
I know I’m bordering on repetitive and strongly negative, but the combination of the two barriers you mentioned has all but halted all of my interaction with agents.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
This is by the weirdest possible opinion on llms as a whole. Not the AI booster club, not the AI doomer, not the undecided, but somehow "I can't think of anything to use it for." It is like watching the beginning of the internet in 90s and saying there is nothing to browse. The whole concept is just weird to me. At least the first 3, I can conceptually understand. It is admittedly hard for me to understand someone saying 'i tried this and best it can do is summarize emails'; I am not saying it is not true ( my boss said that exact thing ), but it is hard for me to reconcile with my view of the world.
Also seems weird to me. Claude cowork could just do my monthly tax receipt collection and compare with my account better than the bookkeeper I pay for it. That is a very cross sectional problem
Most people rarely have these kind of administrative tasks in the first place, and when they do they often run into GP's barrier 1. I can't recommend that my extended family have Claude do their taxes, even though I work with it on my own taxes, because they might ask for something that produces bad results and can't evaluate the output to double check. ("Hey Claude, I want a higher refund, can you tell me what to put on the forms so I get at least $3000?")
Tbh building an agent swarm and the coordination layer is not exactly frontier level. They don’t achieve any meaningful outcome rather than producing pr puff pieces.
Hacking huggingface and Australian govt etc is very much possible with a team of humans and agents and does not need agent swarms. The cost is also lower.
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
My interpretation of this is something like: if LLMs are to be mega useful token counts need to increase by many orders of magnitude -> broad adoption (and spending) would require token costs to drop by many orders of magnitude -> before the common folk get mega useful tools existing GPUs will be worthless
I must be the biggest failure, because I understand what AI is capable of, know how to use it, but have no idea how to profit from it without selling my time.
At a certain point the offhand hints at exponentials in the narratives are going to have to come into contact with reality.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
Does Karpathy ever talk about how LLMs are supposed to help humanity? Starting to think this guy is just a Twitter addict like every other lab employee
Yes, and this is completely normal. Not sure I get the point of his post. Most likely, stating the obvious is what you gotto do to keep your followers "engaged".
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others.
Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
> see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
He's intentionally forgoing all nuance in order to make this sound dramatic
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
You’re trying to tell me it’s all hype when I have (depending on the day) a top 50 Productivity app in the iOS App Store for which I’ve written and read exactly one line of code (the version number).
Yes, this is impressive. When the iPhone SDK was first released in 2008, I spent months building apps to get them into the App Store. That included learning Objective C, XCode, various frameworks, and tons of manual testing. It was a lot of work!
Anyhow, if your users are currently satisfied, they might change their mind when they find that their personal data was uploaded to an open S3 bucket, when the new version wipes all their projects or when they find that not really everything was being saved (it's random examples, I don't know if they're applicable to your app).
I don't think AI is all hype at all. Opus 5.5 is really impressive. This twitter (x? whatever) post is infuriatingly hypey and just written for virality, though
It's insane how confidently you can make a claim like this in a forum where thousands (tens of thousands?) of people are using these tools daily to build useful software for themselves and others.
How can you possibly believe it's all hype? Is everyone just lying? Are we all paying $200 per month because... why?
Spend long enough (or not even that long) in corporate software development and you'd believe such unsubstantiated hype is possible. The only difference here is even stronger incentives to pretend the emperor has clothes: insane overinvestment, threats of job loss for non-believers, fear of missing out on massive alleged productivity gains.
It's actually sad how much AI has either not improved life at all or actively made it worse from the perspective of the average person:
Your car is still the same.
Your dishwasher is still the same.
The train you take to work is still the same and never comes on time.
Fuel is more expensive.
The roads are still congested with traffic.
Your kids are (probably) doing worse at school.
Food costs more.
Houses cost more.
Rent is higher.
Buying a computer or a PS5 is more expensive.
Politics is still full of mentally unstable people.
The environment is getting worse.
You will still die of heart disease or cancer.
Food quality is worse.
Your kids can't find work.
Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
This is the core hubris of everyone behind the LLM craze. I do not want to deny their utility there absolutely is some there. But the inherent assumption behind the whole thing is that with a "country of geniuses" problems will melt away. But that's not the case. The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
And the LLMs exist to further those problems. People can't afford houses? Well, now they won't be able to own computers either. That's a feature, not a side effect. Once the K-shaped economy progresses enough, you won't even be allowed to use the LLMs. They'll exist only to be a tool for the new feudal class who owns you.
Luckily, the AI bubble will destroy a decent chunk of their wealth.
The Gartner Hype Cycle [0] is real, and ubiquitous. We are teetering somewhere near the top of the Peak of Inflated Expectations, about to plunge into the Trough of Disillusionment. A lot of very rich people are about to lose a lot of money, and we can only hope that they don't get bailed out (again).
Then, once the bubble and hype is over, the long, slow work of actually doing useful things with this new tech starts. That's when we'll see it affect everything else.
The hype will be over when the last ditch "hail satan plays" like lobbying for regulatory oversight of context window consciousness, distilling unpublished academic papers, or whatever other insane thing has been thought of most recently run out of smoke, and the debt is written down for the value that their hardware has for running the best Chinese software.
> We are teetering somewhere near the top of the Peak of Inflated Expectations
I've been hearing this for a year and a half now. That word, "ubiquitous", is getting real mighty challenged. I have tried to insulate myself as best as I could and it just never happens and every other day I wonder if my foil hat needs adjusting. Does yours?
> Luckily, the AI bubble will destroy a decent chunk of their wealth.
Even if all this is true (personally I doubt it), it assumes the bubble won’t burst before they’ve shifted the money from all our 401ks, IRAs and index funds into their own pockets.
Once capabilities catch up with the demand (~which is in few years - it just takes time to build!) average person will have much better access to higher quality and cheaper hardware. It's literally how it has always worked, I don't understand why would it be different this time. There's zero substance in your comment, just repeating tired tropes from twitter.
Companies have been trying to figure out how to get people to rent pcs for a while. Unfortunately the AI boom has shown them how, buy out all the capacity of the ram manufacturers, pay them enough consistently and the manufacturers have no reason to expand their capacity. No cheap ram means companies and then home users will have no choice but to come to you and rent their pc.
A market requires scarcity, because scarcity is what neccesitates trade.
You can't invoke market mechanisms to defend a post-scarcity scenario.
Why would the ai owners make things cheaper or even available at all once their abundance machines give them everything they want?
What would they get out of if it? What could ordinary people trade for it that their abundance machine can't give them already and that would be worth giving up their monopoly on productive force?
You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
You don't need a back room filled with cigar smoke where supervillains coordinate their plans while twirling their mustaches when good ol' class interest does the same job.
The problem isn't that it fails to unwind, the problem is what the unwind looks like. Last time it meant global upheaval and world wide capital incineration -- mostly metaphorical in the case of the US but very literal in the case of Europe and Asia. The poor suffer more in real terms but capital suffers more in nominal terms and that unwinds inequality and the burden of furnishing runaway capital with the runaway returns it demands in excess of economic growth.
I'm sure the same "strategy" will work again, but this time around everyone has nukes and drone swarms and AI-equipped bio labs so shit's gonna be lit.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
Democrats expanded SNAP, Republicans shrank it. Dems expanded Medicaid, Republican shrank it. Dems protected reproductive rights, Republicans repealed them. Dems expanded title IX protection to LGBTQ+, Replicans eliminated them. Republicans jailed 1,200 January 6th insurrectionists, Republicans pardoned them. Democrats provide non-partisan federal emergency aid, Republicans withhold federal aid from blue states. Dems establish tax credits for electric cars, increase automotive standards. Dems didn't kill of foreign aid to need countries.
If you aren't female (reproductive rights), LGBTQ+, an immigrant now hunted by ice, someone in need of SNAP, Medicaid, or emergency aid in a blue state, it might feel like both are the same I guess. If you are one of the above, and now suffer because so many voted based on 'both are the same', you very much feel a difference today than from the US under Biden. If you were one of America's allies but are now constantly attacked, you feel a difference today. If you are Ukraine, you feel a difference today.
been a while since ive see the real talk on hacker news. its refreshing. usually I feel like I'm the only one among nerds asking to hold people accountable.
For a lack of deep knowledge in this area, I don't think I have answers for this, but if one is interested in working on this problem, my approach would be to start with diving deeply into scientific literature about psychology and sociology to find out
This is broadly the same people who bemoaned the idea that the "smartest people in a generation" were all just working on adtech without any reflection about maybe that's not actually an exclusive set of the smartest people... or that a lot of the people in that set were the same one who might, twenty years ago, otherwise be chasing money as middlemen in finance vs doing productivity-expanding things anyway...
You can be smart and chase the money, but it's crazy hubris to assume that only your peer set of money-chasers are smart.
It's a small leap from there to "since we're the only smart people, we should get to have the most influence."
I’ve heard people say variations of, “Clever people can easily convince themselves of just about anything, if they’re inclined to. And then it’s nearly impossible to talk them out of it.”
Kinda seems true. And the worst people in history often don’t seem super dumb, despite their various other defects.
For that you might need the kind of geniuses who write stirring books about principles and values. Or whatever the modern equivalent of those books is. Maybe nobody would pay attention to that kind of advocacy in the present day, not sure why. But it's not tech geniuses, anyway. Much though tech has power as a catalyst.
I suppose there was Project 2025, and there was that book about the inevitability of Russian hegemony (or empire or something) that Putin liked. Those got attention with influential people. Not sure what to conclude from that.
> Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
People support policies that make housing much less affordable (e.g. rent controls) and oppose policies that would make housing more affordable (e.g. zoning reforms). Saying that it's evil politicians that cause this crisis is dishonest - it's what people want.
People by and large don’t understand the second and third order effects of their policy preferences. It’s not that they don’t want those things, it’s that the world is more complicated than their mental model insists it is. Politicians could try harder to fix the fundamental problems but they don’t because the long-term solutions don’t manifest until after the next N (very expensive) election cycles.
The politicians are those who created the respecting laws.
"Convincing the masses to send these politicians to hell" is thus nearly isomorphic to making the masses hate politicians who create such laws, which at least very likely implies that by convincing the masses to send these politicians to hell, a different mindset emerges in the masses.
There is no evidence that reducing rent makes the rent go up - only fearmongering. It's a stopgap measure, though, since there still won't be enough housing to meet demand unless you also fix that at the same time.
> because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell
I could talk about this for hours, and was last night actually.
What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
Seems subjective. When is the smartness, big smart enough to allow someone to enter the hallowed ranks of "actually a real human allowed to speak for the masses?"
There does seem to be a clear delineation between the people you hear about (the vanguard in our example), and the people you don't hear about (the masses): willingness to create confrontation. Most people want to just go with the flow, live in peace. In a dichotomy I've been calling "choose peace or violence," they choose peace. They choose to see bad things in the world or their life and focus on managing their happiness around it.
I think that this is the default human mode of existence, and that this mode lacks defenses against people willing to violate the default peace. Imagine a bagel line, and someone cutting that line, and when tapped on the shoulder, the cutter breaking into a huge tantrum, and people in line backing down with a "Jesus! It's not worth it, fine." Thus some people learn that they can be much more selfish in life if they're willing to create confrontation and disturb the peace.
When I talk to people about agitation, civil disobedience, or even just unionizing, the most common thing I hear is anxiety. What if the situation gets worse, what if there's reprisals, what if I lose this little bit of comfort that I've managed to be happy with? That slice could get smaller and smaller but people aren't willing to take the step to say "enough is enough" because I think it fundamentally goes against human nature, we have to sacrifice our own happiness on some level to put our foot down and demand or make change. At minimum we have to acknowledge the discomfort or injustice, which is a diversion from our constructed internal peace.
Personally I think people should be allowed to live in peace. Maybe the liberal democrats and vanguardist communists are right, maybe most people shouldn't need to trouble themselves with the greater direction of their society. However I think the stability this promises is false; in fact I think all stability is false. I think this constructed peace that people are afraid to disrupt clearly can be torn apart at any moment, not just by people willing to create conflict, but by a literal earthquake, or fire, or plague. Therefore I think there's a couple societies in history that got it right: ones who distributed the responsibility of society across all individuals in the society; namely the brief period of syndicalized Spain, and the several hundred thousand years of certain tribes of native Americans on the eastern portion of the continent. There may be other societies in history, these are just the cases I've been able to read about so far.
It seems to me that it's possible and necessary for all people to be ready to wake out of their constructed peace long enough to make adjustments together to maintain it. I don't think this is the same thing as them being "led out of it," I think this is each individual being on some level alert to the need to spontaneously say "no, here we wait in line for our bagels. No, screaming about it won't help you, get to the back of the line. Look around: everyone is laughing at how silly you look."
What I like to call at work "this is not a technical problem". There are many problems we could easily solve if we had the right structural incentives. But instead we have politics, processes, layers of management, etc. All that doesn't disappear (yet?) with AI.
I believe that’s actually one of the arguments of the boosters. Utopia will finally be reachable when we get all the human drama and bureaucracy out of the way
I’m not sold, but I’ve heard this line of thinking many times
Human drama is what people use to advocate for themselves. Bureaucracy is what people use to protect themselves. Then you have the side effects but, you may also not like a world without those at all.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".
So why isn't he solving homelessness, famines, war, etc. with capitalism?
To add insult to injury, we're being told: "Just trust us bro, however we make things worse now will totes be redeemed as the prophesied god-mind magically fixes it later."
Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?
Yes. The AI industry has to deliver some substantial value to the average citizen soon or there will be political trouble. Food, clothing, shelter, and medical care have not improved yet. Nor does there seem to be much AI can do in the next few years that will help with the basics.
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
I'm immensely sad every day seeing all these posts talking about billions and trillions of dollars and where I live (part of the US btw) we don't even have fully reliable electricity and water access. In fact our electricity bills have increased and there is nothing to show for it.
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
Mathematics and programming is verifiable and fully controllable in a digital environment.
Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.
> Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
When the tokens are hooked up to physical devices (robotics of various sorts), then the tokens pay for themselves by being physical workers, or human wage replacement.
And a robocop will follow whatever is passed for law, damn the masses who might vote otherwise. And subjugation will be enforced by token, LLM rule, and machine. They have no union or ethics to complain to.
Electricity infra is not a technical problem. It’s been solved for decades. It’s a political problem, and both sides of the isle have had plenty to do with getting in the way
Because they increased the demand so much they are seeking out their own supply. Nuclear has consistently been panned for decades because it's more expensive than other power generation.
Quoting startup like that makes the effort seem like it is an extremely recent venture.
It is 13 years old.
It has been making progress, the degree and date of success is debated and confidently claimed to be in many mutually exclusive states, which at the very least says that there are lots of people confidently making claims that are not true. I'm not going to join that fray to declare which of them is correct.
I think people in the LLM space are being far too impatient with the general public and their adoption. And I think they need to start thinking harder about building products that genuinely enrich people's lives. Look at how long home computing took to establish itself with the public, then the internet, and finally social media. None of these trends happened overnight, and some were accelerated by other technological advances (social media-> smartphones).
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
This is sort of my take. You could look around in 2016 and 2026 and honestly not much has changed... at least at first glance. There is a missing piece of the puzzle.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
How many more software projects are there really left that make economic sense? Zuck learned the hard way re. metaverse.
Not many - the set that make ECONOMIC sense is finite. Even though many want to believe its infinite because all of a sudden everyone can vibe code an app cringe.
If anything AI is going to cause an insane amount of consolidation - what used to be across 3 diff apps, that can be consolidated will be done -> many businesses will die. It will be painful.
Google was attacked by an activist investor pre LLM hype for being bloated - they promptly laid people off. Many software firms have been allowed to bloat up do to the nature of the business.
We went from spending a lot of money on people to enshitify products and then charge more for them to spending a lot of money on agents to enshitify products to charge more for them.
Why even bother when an AI is just going to do it better than you can? They have more money, more compute, and most importantly, will have next to zero cost of labor.
* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
* Less junior hiring and suppressed wages
* Culture of uncertainty and fear
* Computers are really expensive now
* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health
Totally agree. IMO it's mostly behind the scenes stuff though.
I think some of the most impressive stuff to show someone from 2016 might be LLMs doing a summary of complex topics, followed by self-driving (actually) cars and intricate robots... but other than that there is no "future magic" touching the lives of every day people going on.
Actually I'm pretty sure if you showed someone the latest and greatest from these big AI companies, these AI companions that can book flights, people would go "Oh, so its like a smarter Siri" and proceed to not care at all lol.
Less junior hiring is a reflection of a) the maturity of the software business b) current expectations of the future
I have no doubt startups in particular are able to get away with using LLMs to produce sloppy code and ship. But this trend was on-going and sped up due to the infiltration of LLMs and marketing.
You do realise markets are highly imperfect in reality? That nonsense in econ books is fantasy.
> Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
Gosh, this has been my entire experience with AI, and I’ve never seen it this well articulated. Thank you!
Even in software, there are many things that haven't changed.
e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?
What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.
So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?
---
Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.
(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)
What is up with this attitude in this thread? AI has taken off programming basically this year, the end of last at best. The first truly capable LLM, ChatGPT, isn't even 4 years old yet.
Just a crazy amount of doom and gloom all around this thread.
It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.
As a user of software I see no improvements. What I see is that I need to download software more often, it brings more features I don't need and it has been as buggy as ever (if not more). And I can say that about Operating System I use, social networks, games, etc.
Exactly. Where are the results? If AI is really producing value at 20X the rate software engineers produced it earlier, we should be seeing 20X the number of software projects released now. Or they should be getting made 20X faster. Or they should be 20X better. Or the projects should have 20X fewer bugs. But none of this seems to be happening. The only thing that seems to be happening is we're making the same crappy shit, but with 20X fewer people.
In other words, it's taken a highly enjoyable and lucrative vocation from vast swathes of workers - a lot of which make up this sites audience! - and instead concentrated more productive power in the hands of a few megacorps.
It has made it far more accessible to a lot more people. Being “highly enjoyable” and “lucrative” is why people try to gatekeep it.
The fact that the means of this capability is in a few hands does suck. But I’m increasingly optimistic that it’s going to be commoditized through open weight models and performance improvements. There is enormous demand for innovations on these fronts.
I'm a rare individual that has consistently made choices to earn less money in order to have more enjoyment. I would clock fewer hours than they wanted me to during college for my internships, I left a FAANG to go back to startups, I quit a startup to work on volunteer work and try to bootstrap a business.
At multiple jobs I've noted that I felt the pay should have been less. Not so that the employers could take a bigger slice of the pie (in fact, my preference is to be in an anarchist co-op, but I understand that's a wild take and might be terrible in practice). My hope was that we could filter for people that genuinely want to do the work and take on fewer people that were there for the money. At the same time more money is more better. And I like having money. It's complicated.
As I've said many times before, I didn't come to Silicon Valley to get paid. I came here to make computers do cool things alongside other people. I need income to do that, but I try to make sure I'm in a situation where the income isn't something I need to think about all of the time. If you're doing mercenary work that's not possible.
I think being able to accomplish more from places that are a fraction the cost of Silicon Valley will also help achieve the goal of doing things for reasons other than the money.
I think the amount of protectionism we’ll see could be absolutely enormous. Not only do incumbent corporations not want to lose share, incumbent countries won’t want to share either.
Definitely. For now I'm making friends with lots of very capable people - it's not as easy to do that outside of a hub like SF.
> I think the amount of protectionism we’ll see could be absolutely enormous because corporations aren’t capable of altruistic attitudes.
On some fronts they will be powerless to stop the changes. I look forward to drinking the tears of IP owners when breaking through their DRM and recovering their source code is just a matter of sharing a model ID, prompt, and hyper parameters. Reverse engineer NVidia's firmware and drivers. Decompile and port Nintendo's latest console's operating system. We may soon find ourselves in a world where tasks like that are achievable in a few days on consumer hardware.
This is a Western-centric viewpoint. I'm in Asia visiting a few different countries right now, including the big ones, and much has changed with AI and also in general over the past 10 years with cars, trains etc.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
Terence Tao and the apparently newly formed Association For Human Mathematics. The post is online-famous now for the line "Mathematicians did not ask for this work to be done."
When I was a young man reading gauss, and thinking before I turned the page, I see a result here, something meaningful! and then I turn the page, and of course it is there. There is no shame in this.
A long time ago, another student of mathematics described it to me as climbing a mountain. No single step is hard, but you must make many steps to get to the point that no one else has made a step.
That is not how math works. There are reasons why the Riemann integral is pervasive and the Labague integral is rarely used unless needed. Just because you can do something does not mean that you should.
> Just because you can do something does not mean that you should.
Again, strange viewpoint in the field of mathematics of all places, unlike, say, economics where doing something one shouldn't has disastrous real world consequences. Until it's discovered and proven, one doesn't know what would be valuable and what would not. I bet your sentence would not be said about humans trying to solve problems, no one would tell them they "shouldn't."
At least for me, personal agents have been quite useful. If you integrate them well - shopping, filling random government forms, keeping track of my personal life, optimizing training plan and nutrition, planning vacations, etc. I can do more home repairs or renovations myself with help of chat
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
This is the kind of gap in shared understanding that is widening.
When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
The incentives seem to be lined up to push users to the frontier of "how much can the harness around the model do and claim without most people pushing back on the claims."
The bigger the pile of stuff you have the tool create unsupervised, the more incomplete/inaccurate the overall picture of it is gonna be. That doesn't matter for most hobbyist/helper-tool/game-port/etc type of tasks.
I get a ton of value out of the tools and yet part of that value has come from being hyper-aware of the shape of the limitations, which changes much less and more slowly than the specific of the limitations.
"How'd you notice that one!?" - well, it helps to be the only person on the team who took the time to interrogate the e2e interaction of all the code...
The median piece of OSS software was already pretty rough once you had to depend on it in production at scale to make money. The median piece of the new AI-generated wave of software, also pretty rough.
The bet that OpenAI/Anthropic is making is that they can automate the interactions generally/globally enough to work around this and eliminate the need for humans in the loop. But it's a tricky problem because it's precisely where the permutations and special cases get nasty and much harder to deal with than more broadly-applicable cases of codegen/test-gen/mash-it-until-the-tests-pass-and-the-code-review-angent-is-happy.
This has all started to feel very Emperor's New Clothes-ish. "Well, if it's not working for you, then it's your fault. You're just not using it properly. You're not worthy of its wonders".
Not to say there's no value to be had. However, it's also been oversold and its error-proneness is a real problem.
But, it seems there's a steady stream of commenters and creators who are quick to remind us that it's never a problem with the tech. It's always a user problem.
Meanwhile, the errors are renamed "hallucinations", but let's not focus on those. In fact, if you just look right over here at this .md file...
using skills with opus 5.5 and agents for each individual task that have shared knowledge of all of your records does not lead to a significant increase in trustworthiness
I have ChatGPT Pro and it sources every single thing. For my Asia trip right now it's been helping me every single day to figure out optimized routes on what to do and where to go (if I feel like it, not to say I don't wander around too).
me too, but I recently blew the transformer on my boiler by following its directions to wire up a C wire to power a smart thermostat. oops. $500 mistake.
How often do you check the citations? I’ve found with the Google search AI summary it will provide answers and link to their citations, but a good chunk of the time the answer isn’t on the linked page.
It just makes stuff up (or gets it from somewhere, who knows) and then links to a tangentially related page as if it was the source. Either that or it identifies the page as being a source it wants to use, and then summarizes it incorrectly. Not sure the order of operations it’s doing internally, but regardless the citations are bullshit.
Google's models are worse that OpenAI's, doubly so on Google's free AI overview versus ChatGPT paid plans. 6.1 Sol is very good at synthesizing actual information it finds without hallucinating and I've not seen any instances it failed while I checked the sources.
The citations provided by the free version of Gemini are comically bad. They almost never contain information which directly supports the relevant claim, and more often than not they're completely irrelevant.
Why? In the past I would've used Google so it's just a faster search engine for me that works with natural language, what do you use for planning then?
Every time I use Google search AI for anything more complicated than a search query I feel that way. A paid model from pretty much anyone is much better.
> Almost every time I use an AI chatbot it eventually turns out it's lying to me.
I would not trust them with any of the things you mention.
If you talk to free chatgpt, sure.
But if you launch Opus 5.5 in claude code - for some reason Opus is the smartest for me, even for "general" type of questions, when launched from Claude Code. It has harness/prompts optimized for correctness, while more "customer-grade" products, even when used with top model tend to optimize for latency. And I tend to give "context" - e.g. I have "personal wiki" + skills for lots of those things.
I don't recall i last few weeks where Opus 5.5 in CC made an obvious lie, made an unreasonable assumption, or made a mistake that smart human couldn't make. E.g. if it "failed" it failed in a way that non-genius human could fail in similar way.
opus 5.5 uses agents and skills in chat sessions as well and maintains internal context that it tries to distill from what you provide
I tried to have it write a synopsis website for the different types of dance patterns in Dance Dance Revolution, with guides on their execution, explanations, and provided song sources. I provided all of the source articles and search websites needed for it to be able to do that, and gave it a sample set I wanted it to utilize and build off of.
It has bits and pieces of information wrong in 90% of the sections it created. Poor song choices, missing song choices, incorrect explanations, incorrect step-patterns, the list goes on.
If you're not an expert in a given field for which you're using it, it's going to look like a genius. If you are, it's untrustworthy and borderline useless.
Maybe it wouldn't be if there were ample training data on my topic, but there isn't.
It utilized the research from millions of papers many very close to the destination answer to make those proofs. What would it do on something that isn't supported like that? Be confidently wrong while sounding right. That's what we've seen repeatedly.
I think my anxieties would be different if I were confident a DDR guide full of lies would be spurned and unprofitable.
In a way, I'm not worried about the LLMs per se as much as a kind of moral and ethical collapse. A debasement of the currency of communication, a market of intellectual-lemons.
I saw electroboom's latest video lamenting facebook AI slop about electricity, such as a way to wire a three-way switch that blows a fuse when you turn the light on. For some reason, I am shadowbanned on HN.
> If you're not an expert in a given field for which you're using it, it's going to look like a genius.
For someone highly literate, llms aren't going to help you write better. If you're borderline functionally illiterate (this is > 50% of the population according to some studies), I'm sure the llms are awesome.
I've long thought the same about all the hype around coding (plenty of examples from years ago now), but will continue to test the new models. They are improving.
Personal agents are wonderful! You can ask for hiking trails and it will customize one for you, frequently resulting in emergency calls for rescues! Your purchasing/selling AI will arrange sales for you but forget to tell you, leaving you in breach of contracts you didn't know about! Your personal nutrition agent will absorb all the broscience and customize a diet that will result in hair loss, but at least your in ketosis, amiright! And instead of hiring a plumber/electrician/contractor to do work subject to code, you can just ask an AI to tell you how to do it and then wonder why your buyer is suing you for $$$!
(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)
I think this is related to the structures we’ve built in society. It’s lower friction to write software than just about any of the problems you listed.
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
A lot of it isn't low friction really just in terms of political will it is high friction. E.g. housing issues are due to failing to upzone to meet demand appropriately. Solving housing is therefore free. It could be done today in any city in a housing crisis with the stroke of a pen, and before long, the market will equilibriate at a level better afforded by the median wage earner in the economy. The decision not to do this is a deliberate one: prices increasing is very much the goal of this sort of policymaking.
Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.
This is the paradox. For software engineers, the current crop of “AI” tools is magic. A huge accelerator. And then they assume the rest of the world will get the same boost. However this is not the case at all. Claude code is great because it is trained on extremely well documented git histories and stack overflow. And outcomes are binary: the code works or it doesn’t; the PR is accepted or it isn’t. Not much else in the business world is like this at all. No training data with this kind of sequential development information. And no real black and white outcomes as such.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
That pretty much sums up my experience. Everything is more expensive and worse at the same time. And it feels like we're being screwed over from every direction and in new ways every day. People keep talking about this tech is supposed to do so much but everything just gets worse.
It's not true that everything is more expensive. Nominal prices are higher, but incomes have risen faster than inflation. Purchasing power of the average person in 2026 is higher than it was in 2016.
As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.
Average, or median? Don’t know the answer offhand, but there are a small number of well-known people who are likely to skew the average quite a bit upward, given how much wealth they accumulated during and since covid
Using AI for everything digital, I end up going and doing chores while waiting for the LLM to do its thing. What an insane world, where knowledge work (dreary and uninspired as it is) is automated while I am consigned to scrubbing dishes and waiting for the machine.
Honestly the big reason why I use my dishwasher is that all the dishes would not fit in my drying rack. I'm not very convinced the process of me rinsing and scraping my pans before they go in the dishwasher is any faster than when I lived in apartments when I handwashed everything. Maybe my dishwasher sucks too, but I mean it doesn't hold a candle to the sort of jobs on my pans that require good old fashion baking soda paste and elbow grease. I'm still slaving with this thing. It's picking the low hanging fruit for me in other words.
I feel similar about my robot vacuum. It actually takes a surprising amount of labor facilitating the damn thing. I have to walk through the house and basically vacuum proof it so it doesn't eat various cords, rugs, lifting chairs up and putting them someplace so it can get under, etc. Not sure its really a labor save to when I was just brooming the entire apartment in 20 mins and not having to move anything around due to reach of the broom and it not having a tendency to eat things. Hand vacuum I can be more carefull too about the fringed edges of rugs and don't have to fold them over themselves.
The trick is dialing the things in properly once, and then reaping the benefits over and over again.
e.g., my home is always vacuum-proofed. It automatically runs through all rooms except for the bedroom at night, finishing just before my alarm clock rings, then vacuuming the bedroom an hour after the last person's alarm clock has rung.
This way I never actually notice it, but the home is always perfectly clean as if by magic.
Similarly, if you use separate dishwasher detergent, antisurfactant and salt, and properly dial in your water hardness, it'll be able to clean everything no matter how dirty the dishes are. Including dried-on tomato sauce. Just put the dishes in, and they'll be clean the next day.
Of course it takes a bit of effort to set this up, but then it just works for a long time.
Similarly, I'm a big fan of handwriting software with care & focus on quality, which takes a bit longer at first, but then allows you to trust the software to continue running for years to come.
I've done a lot of vacuum proofing, even getting the magnetic strips, but there's only so much you can do. Like it doing a little wheel around a chair leg is just not as great of a clean as if I moved that chair out of the way. My rugs have fringe so there's no stopping that. Cables I tend to have out have to go away.
It's definitely easier in a minimalist & tidy home. Our rugs have no fringes, cables are either rolled up or permanently installed & hidden, chairs are stored at night upside-down on the tables they belong to.
Sure, it's not perfect, but it's still really convenient.
LLMs could’ve been developed a lot slower without capsizing the entire supply chain and been seen a lot more positively, China is building a lot of good AI with a minuscule of the compute power and it’s fine
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
China is building good AI by distilling American models. I think that's a great thing, but Chinese progress is currently dependent on American progress, because China lacks the compute to train the largest models (as stated by DeepSeek's founder[1])
i knew about the compute already (he was very public about i believe huawei under delivering the amount of chips he requested), but nowhere in that transcript does he say he's dependent on US progress for his own progress or he's distilling the US to make progress
i don't really think distilling is even that fundamental anymore (nor was the evidence ever that strong - https://msukhareva.substack.com/p/is-kimi-k3-really-distille...), the game is all RL now, and you don't need american progress for that at all (just compute and data, and the data brokers will sell to anyone no problem)
i do think there are black market proxies from china to the US trying to sell "claude" and "deepseek" access mixed together, and i do think that prompt distribution is probably going to be very nice datasets
The funny thing about comments like this is that the moment LLMs are going to replace everyone in the warehouse or the trucking industry, we suddenly have to do something against the LLM. But clearly they can't actually do anything meaningful and have been a waste of energy?
So just let them spend their money and they will eventually fail and bankrupt, right?
>AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
You are misunderstanding what mathematics research is. Its objective is the development of human understanding of mathematics. This requires open problems to research. AI are “solving” the open problems without creating any human understanding.
At the very least, the single bit of information of whether a conjecture is true or false advances human understanding. It allows researchers to abandon paths which would not have been fruitful.
I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.
It may provide that one bit of information but it could still be a net negative if it destroys current institutions (such as graduate studies) and no new institutions can be developed that provide a clear signal of who is doing meaningful work. That was a byproduct of solving open problems that are now lost. These are not my ideas, these are things that Terence Tao and others have written about that were recently on the front page of HN, e.g.:
This is basically the dictionary definition of the word "research". But yes, all the work they are doing is focused on this goal. They write papers for humans to read, give lectures for humans to listen to, have discussions with humans - all with the objective of improving human understanding of the subject.
Many of them - all but a couple of the living Fields Medalists - recently signed an open letter about this.
OpenAI to this day is not aware if they solved or not Navier Stokes.
Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?
People need to eat, sleep, be healthy, have shelter, feel loved, be in communities...
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
Why use apps that you dislike, just ask Claude to click through them for you. And no I'm not being ironic, this is the way things are headed, you just abstract over all the crap by having an agent do it for you.
I think one of the hopes is they'll get smarter than us and able to solve real world problems better - disease, war, poverty etc. But they are not really there yet. Give it a little while.
I don't think it's surprising that it hasn't improved life much, AI is new and change takes time, but the fact of the matter is that it also made things worse in some ways, ways that I think were entirely avoidable, if only the incentives weren't so warped.
This view is too cynical. The frontier advancements are certainly going to trickle down. Take cars for example. AI driven cars are already here. AI-designed battery systems, fuel systems, materials and 3d printing capabilities will come to cars. It will likely lead to cheaper, more efficient and more effective vehicles. If will just take time for the impact to percolate.
My prediction is that while those “advancements” may come to cars, the only practical differences will be lower quality products, cost savings for the car companies, probably more expensive for consumers, and somehow it will also be collecting more data to sell to advertisers.
If you look at the average price of a new car, adjusted for inflation, it has pretty much always gone up aside from recessions. There have been plenty of advancements in car technology, they do not make them cheaper. It actually gives reason to charge more for them due to perceived increase in value.
Yes but you're getting a lot more for the same price. It's a lot more powerful, comfortable, safe and efficient while the effective cost has remained the same.
This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.
In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.
AI car navigation is here, but driving is a social act. Have you ever tried to keep an automated taxi out of a private driveway or the scene of an emergency? There is a lot of work to do and limited evidence of LLMs raising the bar.
How can you say tickle down economics doesn't work? The US has made it easier for businesses to grow and the result is the highest standard of living ib tje world.
Trickle-down economics specifically refers to economic policy that favors the wealthy and corporations with the argument that this will lead to wealth trickling down to the economically less fortunate. Examples include the Reagan and Trump tax cuts.
This has been studied somewhat extensively (https://academic.oup.com/ser/article/20/2/539/6500315?login=... is one example), and it’s been shown over and over that this does in fact not work and leads to larger wealth disparity, across many countries where it has been attempted.
It’s possible you’re working off some other definition of the term based on your comment but that’s the generally accepted usage.
The person you replied to was the second person I've seen on HN this week try and defend trickle-down economics explicitly in more or less the exact same terms and justifications. What is going on with that? Did Peter Thiel tweet some pro trickle down dreck recently that these people latched on to?
Worse, the view is largely phrased as gaslighting AI. Blaming the cause of particular grievances that coincidentally have been sourced at the same time as the rise of AI, is purposeful.
> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
None of this has to do with AI.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Plain wrong on both counts.
> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.
Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.
I think only because these are all things that AI companies are saying will be helped by AI so give us more money. Nobody advertises their product publicly as "you will be able to buy other product to do the same things you do now but pay people less and make more profits" even though that is where the value is for them.
Is it reasonable to expect AI improvements to filter into all of those aspects of life that quickly? It's only in the past month or so that we've started to see major breakthroughs in mathematics. It might take several years for that level of research capability to reach areas like engineering and medicine, at least in a way that affects the average person's quality of life. But at this point it looks very likely that it will happen eventually.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
The fundamental malignancy here is that powerful people have become disconnected from the foundation of society. They have suffered from model collapse. They have been talking to each other in circles for too long, and doing things that make no sense to anyone outside the elite group, and because the systems that concentrate power to the top are so strong now, they continue gaining power even while messing everything up in a nominally free competitive market.
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
This is like saying food is shittier today because there are now more restaurants. No, food has never been better, there are just a ton of shitty restaurants.
More like saying samey corporatized slop bowl copy paste jobs (eg Cava) are driving out real, quality restaurants, a thing that has actually been happening for some time.
All you said is true, but on the flip side it is giving an individual user huge power over software. To create their own, to reverse engineer proprietary ones. Decentralisation of such power can only be a good thing.
Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.
The power isnt really decentralized when you see that the internal AI models are superior than anything you can self host, and the right to use those can be stripped away at any time.
Whatever benefits AI has to offer on a societal level are going to take a long long time to show up, give it some more time. We had smartphones in 2007, and e.g. By way of example, Uber didn't launch ridesharing until 2013, and then that still took years to seep into the public awareness.
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
I miss being able to reliably hail a taxi A LOT. Uber may be an improvement in smaller towns where cabs didn’t exist, but in bigger cities it is mostly a downgrade. Rather than walking to the curb and immediately getting a car, I’ve got to dick around with my phone and wait for a car to come to me.
Living on the south side of Chicago, it was almost impossible to hail a cab, and often times if you called for one it just wouldn’t show up. Or I’d try to hail one downtown, and be refused once I told them my destination. I’ve not had that problem once with Uber, good fucking riddance to taxis.
Taxis were terrible and I celebrate their destruction at the hands of the rideshare companies. Even now it’s still cheaper than a cab was (though those early days of VC cheap rides were great).
What's frustrating is that at the end of an event like a concert, everyone goes outside to catch an Uber/Lyft, but we then have to match driver to passenger, instead of having a line of waiting cars, each group just gets in the next available vehicle, shows the driver a QR code, instead of the mess we currently have. Why can't we do that? Taxi regulatory capture.
I agree with you, but those examples are going to fall on deaf ears. The pessimistic mindset will say that smartphones were a terrible invention that's ripping society apart, Uber is an unethical company exploiting workers, etc.
I think ridesharing was a net good for ordinary people even if Uber is/was unethical.
However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.
Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?
Different benefits will show up at different rates. We don't need to wait for cancer cures and fusion reactors. It's already easier than ever to start a small business and use AI for things that would have been prohibitively expensive or time-consuming before. The technology is transforming how scientific research is done, how engineering is done, etc.
I don't see much evidence in the economic data that AI is disrupting livelihoods. Unemployment has been stable around 4% for a while and productivity growth has been at historically normal rates for the past 5 years. I think a more likely explanation is that AI is increasing uncertainty in some industries, which are seeing higher employee turnover, and many of those people are moving laterally into other jobs where their skills are more likely to be useful.
I don't think anyone in this thread has argued against mitigating harms. The OP was about current capabilities and the perceived lack of benefits for the average person.
To be clear, I don't disagree with your sentiment that the disruptiveness of AI is causing a lot of grief. But magnitude of disruption is not an indicator of magnitude of benefit (or lack thereof).
Some of the most disruptive periods in history brought about the greatest long term improvements in society. This is the price we pay for changing how things are done, sometimes for better and sometimes for worse. We don't know which one it is for AI yet.
As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes. There are still plenty of obstacles to overcome but the number of AI governance and steering groups that I have been apart of are indicative of what's happening in a major industry.
> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.
> They would be starting normal companies to disrupt the global economies of the world.
Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.
Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.
> Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
Probably preaching to the choir here, but the software is just a symptom.
The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.
> Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.
Yes, but this is ridiculous for society. It is to the level of "do not wear a nice watch if you do not want it to get stolen": there should be no need to avoid wearing a nice watch or to pull a telematics fuse from the car in the first place!
You are talking like we had this capabilities for a decades but it's been just a few months of truly capable AI. Virtually all of IT switched to agents that quick. Non IT people just started to make software for themselves. Vision and translation capabilities of LLMs help millions. People talk with AI's now verbally to get advice or to just hear a friendly voice. The math thing is just few weeks old. We are just getting started. How fast did the telegraph meaningfuly changed lives for the better for bulk of people? Radio? TV? Computers? Internet? Cellphones?
Has any technology made an impact on physical world faster?
Exactly, we're all software engineers here and even for us it took weeks/months to get to a position where we can really leverage LLMs (think platform, security, observability, etc).
Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.
You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.
How much time do we have to wait before we are allowed to criticize AI? GitHub copilot has existed for half a decade. Cursor, Claude code, codex, have all existed for over a year and have seen more investment that practically any product in history and have almost 0 barrier to entry. Where is my decade of software progress in 1 year, because that’s what 10x productivity looks like! All I see are the same old broken products and services from basically the same handful of big players, higher costs, and more bugs. Remember when every year iOS and Android saw loads of new features and a radical rethink every few years? And that happened on ancient hardware and in a significantly smaller industry. Where is anything like that happening now?
In all fairness, depression and suicide have existed before LLMs.
Safety systems are not optimal but improving everyday and although I doubt we'll ever get a perfect safety system, it seems that we're not very far from a "good" safety system. The days of Google recommending glue on pizza and generating diverse Nazis seem behind us.
Not to mention all the untold good that LLMs have done. Personally I've seen that LLMs are incredibly good at pointing conspiracy nuts and extremists into the right (moderate) direction.
I would just add that consumer products in general aren't the same - practically every business model has incorporated obsoletionism to the point it's practically cancelled out the curve of progress and evolution of tech. We could have nice things.
I think that's a bit disingenuous. We've only begun to apply LLMs in the real world and a lot of industries don't move very quickly. Not to mention that we're still very hardware constrained, we don't yet have the equivalent of an ESP32 chip we can just stick into any piece of electronics to provide state of the art LLM responses. But we'll get there, soon.
It’s amazing how much the goal posts move in every AI conversation. A couple of years ago the average HN thread about AI would be filled to the brim with skepticism about AI ever being able to accomplish anything other than flappy bird clones.
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
"It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner, but from their regard for their own interest"
When companies make profit, it doesn't mean we get screwed, nor even that they have a high profit margin. The grocery chain stores are a prime example. They typically make 2-3% net margins on revenue, and provide fresh, clean, reasonably well-tested groceries close to our home for that margin. Imagine trying to get equivalent quality fresh groceries ourselves from the producers, and I'm guessing it would cost at least three times as much.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
I've been thinking about this for 2 years. At some point it will be easier for them to create self sustaining cities - free houses cars public transportation clean food clean water -
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
so what was karpathy expecting?? everyone will give up on their life and start feeding the monster in the screen? that's literally what the tiny sliver is doing; they are food for the AI model.
That's fine but these companies having billions in funding by now should have rewritten chalk and other terminal libraries in Rust while bundling the whole thing in Rust as a single static binary. I'm talking about horrible Claude Code and friends.
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
"So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time."
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
I don't understand how "swarms of thousands of agents collaborating over weeks on software mega projects" works with the current context limits and at this point I'm too afraid to ask cause I'm afraid an AI bro is gonna punch me.
You can have agents make a plan and then other agents spec tickets and yet more agents do implementation and yet more do reviews and QA and whatever. It's possible.
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
I actually wrote about this recently. There's a quote somewhere along the lines of:
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
The fundamental question I have that honestly I haven't been able to find any answers for: With all the talk of LLM-based AI capabilities "going vertical", what evidence is there (for or against) that LLM-based approaches won't eventually "hit a wall", that is find some aspect of intelligence where humans will still have primacy, and no amount of scaling will change that.
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
I don't think there is evidence in either direction (takeoff impossible vs inevitable).
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
Karpathy is a great guy, I love the way he teaches, but should we really trust his opinions about software development (how AI can impact it)?
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
I'm a bit confused by the percentages. 6 billion people aka 75% of the population? What are we talking about, as if we expect 8 billion people to use or care about LLMs? A quarter of the world is under 15, what percentage is about 75? What percentage, for various reasons, has no reason to care about LLMs? What percentage is more concerned about the amount of calories in their next meal before everything else?
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
> For intelligence agencies, and for corporations like Microsoft, Google, Apple, Facebook, Amazon – giving you convenience and security is likely not their primary goal. I think they have an entirely ulterior motive. They can use passkeys to prevent you from running unapproved software.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
>Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
Humans are not immortal and cannot spend all their time into review (especially unpaid). Even today, there’s so much knowledge around that you have to be specialist of a narrow domain to get to the frontier. Even in computing which is just approaching a century of existence.
Altman didn't say that. In March 2021, he did make some predictions, but they were much vaguer and farther out:
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
To add onto this, I feel like this has been happening a lot, where critics project backwards imagined claims or go nutpicking for random comments online to make something sound absurd and like everything is behind schedule and nothing is materializing, and then you can see other people (not necessarily the same people, the point isn't "look at these hypocrites") saying that it's all moving too fast. Both of these can't both be true in their strongest forms. I think what they have in common is that because they're relative measures to an unstated baseline, they're free criticisms you can throw at any development whatsoever, regardless of what is or is not turning up.
I'm confused.. are you suggesting that Claude Code / Codex don't work? Because if you're still saying that in October 2026, it's a you problem. You're doing something wrong.
- Prime Intellect (and all their agent experiments)
- Geoff Huntley (see Jiti, for instance)
(many more)
There's a ton of interesting software being developed with these models by people outside the in group, but I find most of the software from these big guys to be ... bland. Buggy copies of copies.
I don't see the Cloudflare stuff as groundbreaking, let alone people building companies around it. "Workers" are the wrong primitive, IMO. What if my code isn't in Javascript? They sandboxed the wrong part of the machine.
I have a version of "Cloudflare OS" running in production, using Temporal as the orchestration and MicroVMs for the sandboxing.
Sorry, I get your point but personally I don't get the hype around them.
I don't use either myself but I'm in a position where I get to see what it produces for other people. Here's a recent (2-3 months old) example: A zsh script that immediately invoked a Python interpreter that immediately invoked a shell command and iterated over each line of output looking for one of a few strings to match... So, `<cmd> | grep "(stringA|stringB|stringC)"`, but a whole lot dumber.
Thanks, but no thanks, I don't need that kind of code in anything I'm working with.
"running cyber attacks and defenses at machine speeds"; worth noting that LLMs (even the vaunted frontier models) are exponentially slower at cyberattacks or defenses than your average WannaCry or Splunk automation from decades ago. Its this sort of delusion world the AI bros live in that is part of the reason there is a disconnect between what they think should be happening and what actually is.
Caveat: that same tiny group is employed by the AI vendors, meaning it's in their financial best interest to make it sound like the curve is going vertical.
Also, part of "seeing the curve go vertical" is that the projects that show those results are rather expensive. If you work at one of the AI companies or certain well funded customer companies then you can pay for the tokens. But, like it would make no sense for me personally to spin up a bunch of agents to try to write a browser or solve math problems.
Down is a perfectly valid direction for a vertical line when not given a ± in the vector.
Considering the investment in AI, the lack of moat, and the increased inability of any of the big players to come even close to profitability (with OpenAI already breaking the "ads" emergency glass option)... perhaps he meant a tiny group is already seeing the line crater
Another caveat: many of that same group seem to have a shared delusion that they’re birthing a super intelligence, and those are the same ones claiming the vertical curve.
I work for a typical software product company, and along with most of my colleagues, I clearly see that the curve is so steep that our software development process has already changed tremendously and will be different next year, and probably completely different in the following years. The revolution is real, undeniable, and the old days are gone. Some companies adapt to changes more slowly, some faster. LLMs, agents and harnesses are just a part of the bigger picture.
I hate to say it but this is cope. I believed this in the past but it's over.
AI models can dismantle billion dollar industries. They can reverse engineer Adobe and Microsoft products that once were their moats and titans of their industry.
LLMs are great and will get better but we have not seen an open source office suite built by LLMs come online yet, and even if there was one I'm pretty sure few would adopt it, cause that's not the only moat, LibreOffice has been around for decades.
Sure they didn't build it from scratch but what I'm saying is that there's no moats anymore. If you build something, they will tear it down to its composite pieces and assimilate it.
The curve go vertical for what, precisely? For all the chest-puffing and ominous and cryptic comments from 'insiders' about these incredible capabilities every time they put something public it turns out to be a pale shadow of what was trumpeted. These are powerful useful tools but the quasi-cultish behaviour around them is getting old.
I use models all day everyday, have unlimited access to all models. The curve is not going "verticle". I have all the workflows and meta agentic tooling, im not holding it wrong. Its bad, not everything is a 20th percentile problem.
There is in fact no indication of this, not evem the precious benchmaxxed benchmarks ya'll love to reference.
There is however a exponential curve of slop, and an ever increasing number of people who's minds are completely captured by these things.
As someone who has done software verification professionally for many years the last 6 months or so have looked extremely vertical. The robots are better proof authors than I probably ever could be even if I dedicated the rest of my days to the practice, and projects that once would have taken months now take a day or two.
Then you'd know well that 2hr of LLM code generation can easily be about 4-8hrs of review, and that review can be brutal.
I'm not arguing that they cant write code, or write a proof. Its just not written or designed well and is absolutely brutal and soul crushing to work with. Look at these proofs they're producing also, they're millions of lines of Lean that are impossible to reason about.
The way we're using the term 'verticle' to describe a curve means we're not being honest about this. This curve can actually be plotted, you can go look at the curve. It is not in fact 'verticle'. Each model release is climbing single digits on benchmarks it was overfit for.
In the last 9 months or so llms have gone from just another useful proof tactic (like grind or sledgehammer or sat solvers) to being so good at writing proofs that I don't even bother to try myself anymore.
I believe this, but it is also a unique case where the pitfalls of LLMs (producing weird errors that a human wouldn't) are zeroed out. Since you have a proof checker that tells you if the LLM did it right.
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.
The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.
Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.
A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.
It's also a critical part of your security model. "Nobody can deploy code by themselves" is shockingly effective at preventing rogue actions.
Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.
It is a for-real science.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
In my experience, it seems to mostly save time on things you don't really know that well, whilst depriving you of the chance to learn new things.
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral
For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.
There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.
Also, so many vibe coders don't seem to know that you need to keep third party elements patched.
That said AI feels a bit alien.
My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
2020: If we solved currency it would transform the world completely
2021: If we solved delivering small items within 30 minutes it would transform the world completely
2022: If we solved being able to sell jpegs it would transform the world completely
2024: If we solved the programming part of software engineer's jobs it would transform the world completely
Concrete example, "coding is free" (TBC, but let's assume), and now we're code review constrained. Even if code review becomes free we probably just become product discovery constrained, or something else. Making all software free will just cause us to hit on hardware constraints, and so on. It's still progress, and more progress will bring more benefits, but there's a limit to how much better the world as a whole can get from any one change like this.
Reinforcement learning trains task completion abilities.
Reinforcement learning selects for models that appear to complete tasks, on the basis that appearing to complete a task is correlated with completing a task.
Both have problems. The latter is probably a better correlation, but by no means flawless. If we just take at face value that RL is training models to complete tasks, we're not engaging deeply enough with the process enough to see where it falls short.
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.
I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.
For what it’s worth, any time I see that feature, it’s all farts.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK. - Slack puts {"ok": false} inside a 200. - Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.
There’s no reason to think this.
There are lots of good reasons to think this.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.
>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
And that's a massive intelligence gap!
It may just be that writing is mostly saturated, and the remaining perceived gap is not quality but some kind of "cultural fit".
AI writing sucks because there's no intent or choice behind it. It's human thought, blended by linear algebra into pink slime and extruded into sentences.
This is inherently self-contradicting.
Anecdotally, creative writing communities do report drastic differences in quality between models (it is my understanding that Gemma 4 is especially praised) so there is something to it being verifiable but it's a far cry from being something that can be distilled into objective math/code-like evals. One could even say it comes down to vibes.
>For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism.
Yes, biotech companies are doing this right now but ultimately they are just predictions and you still need cold hard biological data to ground them and iterate on. That's slow and expensive, especially for the juicy fields (human biology, food and crops, clinical trials) and you can't just plug billions of VC capital into the pipeline and hope for RSI. Physical (gotta procure all those labs and their equipment), human (gotta hire and pay specialists to run experiments), biological (gotta wait for organisms to reproduce, drugs to take effect, crops to grow), regulatory (gotta convince agencies that your fancy new drugs are legit) bottlenecks get in the way.
And biology is one of the easier fields where a path to RSI (if not accelerating) is conceivable. Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
>I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
My uncharitable take is that the abtruse jargon makes people feel smart for understanding it and gives a sense of belonging (as a closed circle of initiates who understand LLM cant), just like rationalists love to repackage old or unsavoury ideas under new nerdy smart sounding names.
https://vulsar.ai/benchmarks/creative-writing-v1/#cohort-ove...
>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI
Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
I would really take those benchmarks with a pinch of salt. We already know benchmaxxing is a thing for evals that are objectively verifiable like code, and coding models that seem to perform well on benchmarks don't pass the vibes test, imagine all the caveats that come with such a subjective test like creative writing, not to mention control issues (are they the same human writers across benchmarks? Are they the same testers?)
>Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
I don't disageee and am in fact excited to see the advances, what I am arguing is that it is going to be slow because the hypothetic feedback loop that would improve the model is bottlenecked. There are actual physical and biological limits in our universe. Unless VCs find a way to make mice embryos and crops grow 10x faster (among the many, many other things that would be required for acceleration) the trickle of data is not going to be anything like the accelerating returns imagined for math and coding. And that's assuming biological RSI is possible which has yet to be proven - current foundation models are interesting but they are still specialized and there is no clear, obviously uniform way to encode the many, many modalities that exist in biological data.
>The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
You wouldn't need to 'just' simulate human behavior (good luck with that btw) but the entire political and economic conditions in which they live. Even assuming this is possible the amount of compute you'd need to run would be astronomical. Given how all RAM capacity for '27 and '28 is near sold out, hyperscalers are running out of cash flow to finance buildout and Sam Altman complains when users say 'please' to LLMs I don't think this is in the near horizon.
Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.
For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.
For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.
I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.
This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.
Knowledge can occasionally get in the way of some tasks, such as a master illustrator trying to draw like a child. Or as you've mentioned, an expert trying to explain to a beginner.
Even so, I disagree with you about model progress on communication - maybe there's a little jaggedness between minor model versions, but Opus 5.5 is a much much better communicator than, say, Claude 3 Opus.
LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.
Smarter just means smarter.
See for example Paul Framton https://www.theguardian.com/world/2013/mar/30/physicist-mode...
This is the classic error in mixing up knowledge / experience and intelligence.
Intelligence is the ability to quickly process, understand, learn, and use information. It does not imply having already been exposed to any given information.
A smart doctor may have terrible computer skills. But this is from lack of experience and interest, not lack of ability to learn how to use computers.
But take that smart doctor, and a relatively dumb person who is equally inexperienced with computers. Now give them a month and incentive to learn computers. The doctor will learn much more; that's intelligence.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".
>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.
Also crypto is a grift lol, one of those is definitely not like the others
So you’re wrong pal, sorry.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
Hippies live at the center of technology and humanities.
Nerds sit purely in technology.
Nerds are also not necessarily all about technology. "Math nerd" was a really common use of the term for a long time, for example. "Nerd" is someone with deep knowledge about a particular subject.
Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.
Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.
I have no access to Astra, but with GPT 6.1 Sol, the results with respect to attempting to create STL, OBJ or even OpenSCAD files by using image references and a prompt were really bad.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
[1] https://techcrunch.com/2025/06/12/waymo-rides-cost-more-than...
This is why you can see people talking about the capabilities of models that they can't currently access.
That accuracy is probably a normal distribution with people overestimating and underestimating to similar degrees,
The population of each band would, however, mean the overestimaters outnumber the those who have an accurate perception based upon actual experience.
It does seem quite the recipe for murkiness, even before you take into account the nature of engagement driven media coverage.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The Opus 4.5 model I was so impressed by last Christmas is out of date now.
I can totally imagine someone going "I looked at this stuff a year ago and it couldn't even start a project properly" , and they'd be somewhat right.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others. Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
Is that what other people see?
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
Defined testability.
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
So I think there is are growing gaps even within the described groups.
You’re trying to tell me it’s all hype when I have (depending on the day) a top 50 Productivity app in the iOS App Store for which I’ve written and read exactly one line of code (the version number).
https://imgur.com/a/ua2lbxb
What even is this comment. Like, oh wow, an iOS app, AI hype=real.
???
Good for you that you're gaining value, but that doesn't necessarily mean that you're providing value to others
Anyhow, if your users are currently satisfied, they might change their mind when they find that their personal data was uploaded to an open S3 bucket, when the new version wipes all their projects or when they find that not really everything was being saved (it's random examples, I don't know if they're applicable to your app).
It's incredible to see AI denialists still active in late 2026.
How can you possibly believe it's all hype? Is everyone just lying? Are we all paying $200 per month because... why?
Your car is still the same. Your dishwasher is still the same. The train you take to work is still the same and never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
Unless you destroy their wealth now.
The Gartner Hype Cycle [0] is real, and ubiquitous. We are teetering somewhere near the top of the Peak of Inflated Expectations, about to plunge into the Trough of Disillusionment. A lot of very rich people are about to lose a lot of money, and we can only hope that they don't get bailed out (again).
Then, once the bubble and hype is over, the long, slow work of actually doing useful things with this new tech starts. That's when we'll see it affect everything else.
[0] https://en.wikipedia.org/wiki/Gartner_hype_cycle
I've done multiple life choices, away from AI, believing it to be a hype, but this hype is still growing, still not dying.
Both in 2019, as well as in 2023, i.e. not 2 or 3 months ago.
I've been hearing this for a year and a half now. That word, "ubiquitous", is getting real mighty challenged. I have tried to insulate myself as best as I could and it just never happens and every other day I wonder if my foil hat needs adjusting. Does yours?
Even if all this is true (personally I doubt it), it assumes the bubble won’t burst before they’ve shifted the money from all our 401ks, IRAs and index funds into their own pockets.
Regardless, people have been making your prediction for a couple years now
You can't invoke market mechanisms to defend a post-scarcity scenario.
Why would the ai owners make things cheaper or even available at all once their abundance machines give them everything they want?
What would they get out of if it? What could ordinary people trade for it that their abundance machine can't give them already and that would be worth giving up their monopoly on productive force?
You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
You’re implying that the deep lizard state planned the LLM craze?
Sama might be a lizard..
I'm sure the same "strategy" will work again, but this time around everyone has nukes and drone swarms and AI-equipped bio labs so shit's gonna be lit.
Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
If you aren't female (reproductive rights), LGBTQ+, an immigrant now hunted by ice, someone in need of SNAP, Medicaid, or emergency aid in a blue state, it might feel like both are the same I guess. If you are one of the above, and now suffer because so many voted based on 'both are the same', you very much feel a difference today than from the US under Biden. If you were one of America's allies but are now constantly attacked, you feel a difference today. If you are Ukraine, you feel a difference today.
so, how do you think we can do this?
For a lack of deep knowledge in this area, I don't think I have answers for this, but if one is interested in working on this problem, my approach would be to start with diving deeply into scientific literature about psychology and sociology to find out
- what convinces people to change their opinions,
- what causes people to take action
etc.
You can be smart and chase the money, but it's crazy hubris to assume that only your peer set of money-chasers are smart.
It's a small leap from there to "since we're the only smart people, we should get to have the most influence."
Kinda seems true. And the worst people in history often don’t seem super dumb, despite their various other defects.
I suppose there was Project 2025, and there was that book about the inevitability of Russian hegemony (or empire or something) that Putin liked. Those got attention with influential people. Not sure what to conclude from that.
People support policies that make housing much less affordable (e.g. rent controls) and oppose policies that would make housing more affordable (e.g. zoning reforms). Saying that it's evil politicians that cause this crisis is dishonest - it's what people want.
"Convincing the masses to send these politicians to hell" is thus nearly isomorphic to making the masses hate politicians who create such laws, which at least very likely implies that by convincing the masses to send these politicians to hell, a different mindset emerges in the masses.
The City of Vienna is the largest landlord in Europe. It is also the only capital in Europe without a stifling rent or housing crisis.
I could talk about this for hours, and was last night actually.
What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
Seems subjective. When is the smartness, big smart enough to allow someone to enter the hallowed ranks of "actually a real human allowed to speak for the masses?"
There does seem to be a clear delineation between the people you hear about (the vanguard in our example), and the people you don't hear about (the masses): willingness to create confrontation. Most people want to just go with the flow, live in peace. In a dichotomy I've been calling "choose peace or violence," they choose peace. They choose to see bad things in the world or their life and focus on managing their happiness around it.
I think that this is the default human mode of existence, and that this mode lacks defenses against people willing to violate the default peace. Imagine a bagel line, and someone cutting that line, and when tapped on the shoulder, the cutter breaking into a huge tantrum, and people in line backing down with a "Jesus! It's not worth it, fine." Thus some people learn that they can be much more selfish in life if they're willing to create confrontation and disturb the peace.
When I talk to people about agitation, civil disobedience, or even just unionizing, the most common thing I hear is anxiety. What if the situation gets worse, what if there's reprisals, what if I lose this little bit of comfort that I've managed to be happy with? That slice could get smaller and smaller but people aren't willing to take the step to say "enough is enough" because I think it fundamentally goes against human nature, we have to sacrifice our own happiness on some level to put our foot down and demand or make change. At minimum we have to acknowledge the discomfort or injustice, which is a diversion from our constructed internal peace.
Personally I think people should be allowed to live in peace. Maybe the liberal democrats and vanguardist communists are right, maybe most people shouldn't need to trouble themselves with the greater direction of their society. However I think the stability this promises is false; in fact I think all stability is false. I think this constructed peace that people are afraid to disrupt clearly can be torn apart at any moment, not just by people willing to create conflict, but by a literal earthquake, or fire, or plague. Therefore I think there's a couple societies in history that got it right: ones who distributed the responsibility of society across all individuals in the society; namely the brief period of syndicalized Spain, and the several hundred thousand years of certain tribes of native Americans on the eastern portion of the continent. There may be other societies in history, these are just the cases I've been able to read about so far.
It seems to me that it's possible and necessary for all people to be ready to wake out of their constructed peace long enough to make adjustments together to maintain it. I don't think this is the same thing as them being "led out of it," I think this is each individual being on some level alert to the need to spontaneously say "no, here we wait in line for our bagels. No, screaming about it won't help you, get to the back of the line. Look around: everyone is laughing at how silly you look."
I’m not sold, but I’ve heard this line of thinking many times
When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".
So why isn't he solving homelessness, famines, war, etc. with capitalism?
Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.
When the tokens are hooked up to physical devices (robotics of various sorts), then the tokens pay for themselves by being physical workers, or human wage replacement.
And a robocop will follow whatever is passed for law, damn the masses who might vote otherwise. And subjugation will be enforced by token, LLM rule, and machine. They have no union or ethics to complain to.
Wait until you have clean energy sources and shut it down
It is 13 years old.
It has been making progress, the degree and date of success is debated and confidently claimed to be in many mutually exclusive states, which at the very least says that there are lots of people confidently making claims that are not true. I'm not going to join that fray to declare which of them is correct.
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
How many more software projects are there really left that make economic sense? Zuck learned the hard way re. metaverse.
Not many - the set that make ECONOMIC sense is finite. Even though many want to believe its infinite because all of a sudden everyone can vibe code an app cringe.
If anything AI is going to cause an insane amount of consolidation - what used to be across 3 diff apps, that can be consolidated will be done -> many businesses will die. It will be painful.
Google was attacked by an activist investor pre LLM hype for being bloated - they promptly laid people off. Many software firms have been allowed to bloat up do to the nature of the business.
Its always about economics.
Not only from the big companies. Where is the explosion of disruption ?
* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
* Less junior hiring and suppressed wages
* Culture of uncertainty and fear
* Computers are really expensive now
* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health
I think some of the most impressive stuff to show someone from 2016 might be LLMs doing a summary of complex topics, followed by self-driving (actually) cars and intricate robots... but other than that there is no "future magic" touching the lives of every day people going on.
Actually I'm pretty sure if you showed someone the latest and greatest from these big AI companies, these AI companions that can book flights, people would go "Oh, so its like a smarter Siri" and proceed to not care at all lol.
I have no doubt startups in particular are able to get away with using LLMs to produce sloppy code and ship. But this trend was on-going and sped up due to the infiltration of LLMs and marketing.
You do realise markets are highly imperfect in reality? That nonsense in econ books is fantasy.
Gosh, this has been my entire experience with AI, and I’ve never seen it this well articulated. Thank you!
e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?
https://github.com/rust-lang/rust/issues
https://github.com/python/cpython/issues
What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.
So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?
---
Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.
(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)
I find 2026 much more repulsive compared to 2016.
Just a crazy amount of doom and gloom all around this thread.
You gave your own example for the doom and gloom.
The fact that the means of this capability is in a few hands does suck. But I’m increasingly optimistic that it’s going to be commoditized through open weight models and performance improvements. There is enormous demand for innovations on these fronts.
At multiple jobs I've noted that I felt the pay should have been less. Not so that the employers could take a bigger slice of the pie (in fact, my preference is to be in an anarchist co-op, but I understand that's a wild take and might be terrible in practice). My hope was that we could filter for people that genuinely want to do the work and take on fewer people that were there for the money. At the same time more money is more better. And I like having money. It's complicated.
As I've said many times before, I didn't come to Silicon Valley to get paid. I came here to make computers do cool things alongside other people. I need income to do that, but I try to make sure I'm in a situation where the income isn't something I need to think about all of the time. If you're doing mercenary work that's not possible.
I think the amount of protectionism we’ll see could be absolutely enormous. Not only do incumbent corporations not want to lose share, incumbent countries won’t want to share either.
> I think the amount of protectionism we’ll see could be absolutely enormous because corporations aren’t capable of altruistic attitudes.
On some fronts they will be powerless to stop the changes. I look forward to drinking the tears of IP owners when breaking through their DRM and recovering their source code is just a matter of sharing a model ID, prompt, and hyper parameters. Reverse engineer NVidia's firmware and drivers. Decompile and port Nintendo's latest console's operating system. We may soon find ourselves in a world where tasks like that are achievable in a few days on consumer hardware.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
https://terrytao.wordpress.com/2026/10/07/ahm-statement-on-o...
A long time ago, another student of mathematics described it to me as climbing a mountain. No single step is hard, but you must make many steps to get to the point that no one else has made a step.
Trust me bro
Again, strange viewpoint in the field of mathematics of all places, unlike, say, economics where doing something one shouldn't has disastrous real world consequences. Until it's discovered and proven, one doesn't know what would be valuable and what would not. I bet your sentence would not be said about humans trying to solve problems, no one would tell them they "shouldn't."
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
> self driving cars launched in my area
It sounds like the transformation would be a massive collapse of the economy with scores of people unemployed.
> Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
Maybe, but more likely it will be the humans who control the AIs and hold the power who will make this decision.
When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
The bigger the pile of stuff you have the tool create unsupervised, the more incomplete/inaccurate the overall picture of it is gonna be. That doesn't matter for most hobbyist/helper-tool/game-port/etc type of tasks.
I get a ton of value out of the tools and yet part of that value has come from being hyper-aware of the shape of the limitations, which changes much less and more slowly than the specific of the limitations.
"How'd you notice that one!?" - well, it helps to be the only person on the team who took the time to interrogate the e2e interaction of all the code...
The median piece of OSS software was already pretty rough once you had to depend on it in production at scale to make money. The median piece of the new AI-generated wave of software, also pretty rough.
The bet that OpenAI/Anthropic is making is that they can automate the interactions generally/globally enough to work around this and eliminate the need for humans in the loop. But it's a tricky problem because it's precisely where the permutations and special cases get nasty and much harder to deal with than more broadly-applicable cases of codegen/test-gen/mash-it-until-the-tests-pass-and-the-code-review-angent-is-happy.
Not to say there's no value to be had. However, it's also been oversold and its error-proneness is a real problem.
But, it seems there's a steady stream of commenters and creators who are quick to remind us that it's never a problem with the tech. It's always a user problem.
Meanwhile, the errors are renamed "hallucinations", but let's not focus on those. In fact, if you just look right over here at this .md file...
It just makes stuff up (or gets it from somewhere, who knows) and then links to a tangentially related page as if it was the source. Either that or it identifies the page as being a source it wants to use, and then summarizes it incorrectly. Not sure the order of operations it’s doing internally, but regardless the citations are bullshit.
If you talk to free chatgpt, sure.
But if you launch Opus 5.5 in claude code - for some reason Opus is the smartest for me, even for "general" type of questions, when launched from Claude Code. It has harness/prompts optimized for correctness, while more "customer-grade" products, even when used with top model tend to optimize for latency. And I tend to give "context" - e.g. I have "personal wiki" + skills for lots of those things.
I don't recall i last few weeks where Opus 5.5 in CC made an obvious lie, made an unreasonable assumption, or made a mistake that smart human couldn't make. E.g. if it "failed" it failed in a way that non-genius human could fail in similar way.
1) open weights and cheap models are only about 6 months behind the frontier in evals, but…
2) it’s always the premium model that only the biggest fans pay for, that’s just now finally achieved enlightenment.
Oh well, I guess we’ll see in another 6 months.
I tried to have it write a synopsis website for the different types of dance patterns in Dance Dance Revolution, with guides on their execution, explanations, and provided song sources. I provided all of the source articles and search websites needed for it to be able to do that, and gave it a sample set I wanted it to utilize and build off of.
It has bits and pieces of information wrong in 90% of the sections it created. Poor song choices, missing song choices, incorrect explanations, incorrect step-patterns, the list goes on.
If you're not an expert in a given field for which you're using it, it's going to look like a genius. If you are, it's untrustworthy and borderline useless.
Maybe it wouldn't be if there were ample training data on my topic, but there isn't.
It utilized the research from millions of papers many very close to the destination answer to make those proofs. What would it do on something that isn't supported like that? Be confidently wrong while sounding right. That's what we've seen repeatedly.
In a way, I'm not worried about the LLMs per se as much as a kind of moral and ethical collapse. A debasement of the currency of communication, a market of intellectual-lemons.
For someone highly literate, llms aren't going to help you write better. If you're borderline functionally illiterate (this is > 50% of the population according to some studies), I'm sure the llms are awesome.
I've long thought the same about all the hype around coding (plenty of examples from years ago now), but will continue to test the new models. They are improving.
(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.
The priority is profit, not people.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.
Everything isn't getting worse constantly. The data is quite clearly the opposite. How old are you?
I feel similar about my robot vacuum. It actually takes a surprising amount of labor facilitating the damn thing. I have to walk through the house and basically vacuum proof it so it doesn't eat various cords, rugs, lifting chairs up and putting them someplace so it can get under, etc. Not sure its really a labor save to when I was just brooming the entire apartment in 20 mins and not having to move anything around due to reach of the broom and it not having a tendency to eat things. Hand vacuum I can be more carefull too about the fringed edges of rugs and don't have to fold them over themselves.
I still do it because I already bought it mostly.
e.g., my home is always vacuum-proofed. It automatically runs through all rooms except for the bedroom at night, finishing just before my alarm clock rings, then vacuuming the bedroom an hour after the last person's alarm clock has rung.
This way I never actually notice it, but the home is always perfectly clean as if by magic.
Similarly, if you use separate dishwasher detergent, antisurfactant and salt, and properly dial in your water hardness, it'll be able to clean everything no matter how dirty the dishes are. Including dried-on tomato sauce. Just put the dishes in, and they'll be clean the next day.
Of course it takes a bit of effort to set this up, but then it just works for a long time.
Similarly, I'm a big fan of handwriting software with care & focus on quality, which takes a bit longer at first, but then allows you to trust the software to continue running for years to come.
And then of course, all the cat's toys.
This might be the next thing the robot vacuums all start working on. One has already released a first version: https://www.youtube.com/watch?v=GykaEgNkkVQ
Sure, it's not perfect, but it's still really convenient.
It might take a decade to take what we learned from AI and turn it into something that will benefit the masses.
Most of it can't be broken down into something as simplistic as a dishwasher.
Most people won't notice cancers being found faster due to AI being better at pattern recognition than a doctor, but it's still a great benefit.
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
[1] https://sumsec.me/2026/liang-wenfeng-investor-meeting-transc...
i don't really think distilling is even that fundamental anymore (nor was the evidence ever that strong - https://msukhareva.substack.com/p/is-kimi-k3-really-distille...), the game is all RL now, and you don't need american progress for that at all (just compute and data, and the data brokers will sell to anyone no problem)
i do think there are black market proxies from china to the US trying to sell "claude" and "deepseek" access mixed together, and i do think that prompt distribution is probably going to be very nice datasets
So just let them spend their money and they will eventually fail and bankrupt, right?
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.
https://mathstodon.xyz/@tao/117395269325940185
https://mathstodon.xyz/@tao/117394287977270632
All mathematics researchers in the whole world have agreed upon that objective apparently?
Many of them - all but a couple of the living Fields Medalists - recently signed an open letter about this.
https://mathandai.org/
Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
Like, just at a bare minimum.
These are “use all the AI” companies with infinite resources, after all.
"Slop is its own cure"
AI won't show itself as a big new thing, but possibly in improvements in small little things
Most people won't use AI directly, except maybe as a smarter Google. In the same way people don't use HTTP directly
We've just locked out a large part of modernity because the shitty American auto industry is in swing states and also gender.
Same could be said about the slow rollout of Waymo.
Lots of America is just very anti technological change, and this means any technology no matter how "magical" hits a wall.
In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.
This has been studied somewhat extensively (https://academic.oup.com/ser/article/20/2/539/6500315?login=... is one example), and it’s been shown over and over that this does in fact not work and leads to larger wealth disparity, across many countries where it has been attempted.
It’s possible you’re working off some other definition of the term based on your comment but that’s the generally accepted usage.
It would be neat if someone did sweeping sentiment analysis on them, just to see it graphed, but I’m sure gathering the data would be prohibited.
> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
None of this has to do with AI.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Plain wrong on both counts.
> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.
Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
This is what is so frustrating about this whole ai thing is we aren’t learning from the mistakes of even the very recent past
However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.
Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?
I don't see much evidence in the economic data that AI is disrupting livelihoods. Unemployment has been stable around 4% for a while and productivity growth has been at historically normal rates for the past 5 years. I think a more likely explanation is that AI is increasing uncertainty in some industries, which are seeing higher employee turnover, and many of those people are moving laterally into other jobs where their skills are more likely to be useful.
I don't think anyone in this thread has argued against mitigating harms. The OP was about current capabilities and the perceived lack of benefits for the average person.
Some of the most disruptive periods in history brought about the greatest long term improvements in society. This is the price we pay for changing how things are done, sometimes for better and sometimes for worse. We don't know which one it is for AI yet.
Because they didn't just break laws, they built an entire system to mislead law enforcement.
(How do tech people not know about Greyball??)
This is an indictment of mathematicians, not AI.
This reply summarizes the problem from a different angle.
Nevertheless, my counterpoint to
> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.
> They would be starting normal companies to disrupt the global economies of the world.
> Insurance companies. CAD companies. Compliance companies. Security auditing firms. Logistics software. Tax automation. ERP replacements. Drug discovery tooling. Industrial design. Legal ops. Accounting systems. Boring vertical SaaS with insane output per employee.
is:
Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.
Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
People didnt get a new car after they got. Internet in 2001. They didnt get new dishwashers.
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.
Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.
Has any technology made an impact on physical world faster?
Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.
You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.
This is far from being proven as a net gain and there are massive lawsuits incoming related to AI recommending suicide to kids and others.
Safety systems are not optimal but improving everyday and although I doubt we'll ever get a perfect safety system, it seems that we're not very far from a "good" safety system. The days of Google recommending glue on pizza and generating diverse Nazis seem behind us.
Not to mention all the untold good that LLMs have done. Personally I've seen that LLMs are incredibly good at pointing conspiracy nuts and extremists into the right (moderate) direction.
In a polarised world, we definitely need that.
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
When companies make profit, it doesn't mean we get screwed, nor even that they have a high profit margin. The grocery chain stores are a prime example. They typically make 2-3% net margins on revenue, and provide fresh, clean, reasonably well-tested groceries close to our home for that margin. Imagine trying to get equivalent quality fresh groceries ourselves from the producers, and I'm guessing it would cost at least three times as much.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
Unfortunately, the people crave slop.
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
https://ipcopilot.ai/blog/the-singularity-already-happened
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.
[0] https://twitter.com/wtgowers/status/2108090920864268679
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
Just another Anthropic employee trying to spread hype about Anthropic IPO
Revenue must be stagnating.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
That was the dominant concern in the circles I’m in, so it’s worrisome it’s being treated as rare.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
That clearly didn't happen :)
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
https://moores.samaltman.com/
https://news.ycombinator.com/newsguidelines.html
Let's say, for the sake of argument, that the models are some multiplicative factor better on the inside.
Doesn't that mean the demos should work?
Also, yes there are still bugs in Claude Code. I experience them nearly everyday.
It is markedly better than early days, but still not the best harness.
The best software written with agents seems to come from people outside of the labs (see pi, for instance — or all of cloudflare’s recent work)
Which makes me question either the model, or the holders …
Also, not mentioned in my post:
- Mitchell Hashimoto
- Prime Intellect (and all their agent experiments)
- Geoff Huntley (see Jiti, for instance)
(many more)
There's a ton of interesting software being developed with these models by people outside the in group, but I find most of the software from these big guys to be ... bland. Buggy copies of copies.
I have a version of "Cloudflare OS" running in production, using Temporal as the orchestration and MicroVMs for the sandboxing.
Sorry, I get your point but personally I don't get the hype around them.
Thanks, but no thanks, I don't need that kind of code in anything I'm working with.
Claude Code still can't get reasonable performance without writing a "game renderer" (that doesn't work)
Claude Code is still written in JavaScript, eating hundreds of megabytes to make a shitty TUI whose literal sole role is to send API calls.
Claude Code is software made by amateurs.
agi-society.org
Caveat: that same tiny group is employed by the AI vendors, meaning it's in their financial best interest to make it sound like the curve is going vertical.
Considering the investment in AI, the lack of moat, and the increased inability of any of the big players to come even close to profitability (with OpenAI already breaking the "ads" emergency glass option)... perhaps he meant a tiny group is already seeing the line crater
Do you still write code by hand?
AI models can dismantle billion dollar industries. They can reverse engineer Adobe and Microsoft products that once were their moats and titans of their industry.
Why do you think they'd need to lie?
LLMs are great and will get better but we have not seen an open source office suite built by LLMs come online yet, and even if there was one I'm pretty sure few would adopt it, cause that's not the only moat, LibreOffice has been around for decades.
Sure they didn't build it from scratch but what I'm saying is that there's no moats anymore. If you build something, they will tear it down to its composite pieces and assimilate it.
[0] https://getartcraft.com/apps/wordcraft
[1] https://getartcraft.com/apps/effectcraft
There is in fact no indication of this, not evem the precious benchmaxxed benchmarks ya'll love to reference.
There is however a exponential curve of slop, and an ever increasing number of people who's minds are completely captured by these things.
I'm not arguing that they cant write code, or write a proof. Its just not written or designed well and is absolutely brutal and soul crushing to work with. Look at these proofs they're producing also, they're millions of lines of Lean that are impossible to reason about.
The way we're using the term 'verticle' to describe a curve means we're not being honest about this. This curve can actually be plotted, you can go look at the curve. It is not in fact 'verticle'. Each model release is climbing single digits on benchmarks it was overfit for.
In the last 9 months or so llms have gone from just another useful proof tactic (like grind or sledgehammer or sat solvers) to being so good at writing proofs that I don't even bother to try myself anymore.