Language Models Can Rewrite Their Own Weights

(arxiv.org)

103 points | by lwhsiao 1 hour ago

17 comments

  • xp84 0 minutes ago
    Am I the only one who is uncomfortable with this week containing two of the plot points in the example scenario from "If Anyone Builds It, Everyone Dies"? First the Reimann Hypothesis, and now rewriting its own weights.

    Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."

  • gavinray 33 minutes ago
    Didn't Google's "sleep" paper already show this?

    https://research.google/blog/the-sleep-framework-a-new-way-f...

      > During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.
    • The-Ergster 16 minutes ago
      I think this is much more significant that that. The article suggests more advanced weight adjustment
      • BoorishBears 10 minutes ago
        No, this is a microstep above having a model generate synthetic SFT and many steps below advanced weight adjustment.

        The original paper that started this hyper was highly misleading because it started with toy networks where the models could directly modify weights, then drew a thread to fixing a reasoning model... but the two approaches were completely unrelated.

        The approach here was essentially abliteration, which any model able to write a few hundred lines of Python and generate synthetic examples can so.

  • aesthesia 24 minutes ago
    It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.
  • samlinnfer 19 minutes ago
    I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get better at repairing itself, then it's told to repair a model that hasn't been damaged and see if it can recursively self-improve.
  • xhrpost 44 minutes ago
    >We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training

    Any increase in fidelity must come from somewhere...

    >t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.

    I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.

    • Jerrrrrrrrry 37 minutes ago
      negentropy
      • kazinator 27 minutes ago
        I would say that "order" is a nicer word. Roughly speaking entropy isn't disorder, but a kind of measure related to disorder. But we don't need two different words for "X" and "measure of X". For instance we don't have a words for "measure of distance" which is different from "distance". "10 cm" is a measure of distance, just called "a distance".

        The word "entropy" was never needed in the first place; "disorder" would have served just fine. Though entropy is a precise concept in thermodynamics, that is fine; all sorts of ordinary words get extra precise meanings in a specific technical contexts.

    • p-e-w 33 minutes ago
      The “test material” can simply be interactions with the environment. It’s just learning.
      • 0xBA5ED 8 minutes ago
        But the "what works" part must still be defined.
  • vlovich123 27 minutes ago
    Isn’t this the same paper from before where it found a single specific bug in its own weights and fixed that infinite loop it would sometimes hit, a bug that was findable without the LLM? Experiments on other LLM weights failed to find any improvements if I recall correctly. Haven’t read this paper but the intro sounded familiar
    • simonw 24 minutes ago
      No, that was this one - https://rikiyat.github.io/blog/weight-space-programming/ - different authors, different approach.
      • vlovich123 9 minutes ago
        IIUC the dominant mechanism behind the math improvement remains fixing reasoning loops same as in the previous paper and probably why they picked Qwen3 here. It’s unclear this approach actually scales in any way.
  • ray__ 18 minutes ago
    Not a comment on the work itself, but I am disappointed by the style and tone of this paper. It's hard to take a piece of academic research seriously with headers like "Welcome to the Era of Self-Rewriting." Even the logos next to the affiliations are enough to give me pause.

    I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.

    The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?

  • SamInTheShell 26 minutes ago
    Chris Hay dropped a video showing the use of letting a model change a weight with his LARQL: https://www.youtube.com/watch?v=NLguavIxFFs
  • ChaseRensberger 16 minutes ago
    i feel like this has been well established at this point. i'd like to see more results pointing towards models actually continuously improving by editing their own weights. once we have a scaling laws paper for that, RSI would be upon us (i assume).
  • bagacrap 2 minutes ago
    The abstract sounds very generated. Can they not be arsed to write a single paragraph on their own?
  • damowangcy 7 minutes ago
    And that is how you get a rogue AI for real.
  • test6554 40 minutes ago
    Well, it was nice knowing you all.
  • pdntspa 39 minutes ago
    Reminds me of Westworld season 1
  • esafak 48 minutes ago
    > We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.

    "Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??

    You can't throw rockets up in the air without worrying where they are going to land. It is your department.

    • jrflo 32 minutes ago
      If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.

      That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.

      • im3w1l 18 minutes ago
        I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.

        We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.

        • jrflo 5 minutes ago
          Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.
      • esafak 29 minutes ago
        [dead]
    • snowram 42 minutes ago
      It will give more credibility to the field of AI, justifying the people working on it and allowing the allocation of more resources to it in a recursive way.

      Until it doesn't, then comes responsability shifting.

    • lenerdenator 46 minutes ago
      > What do you think is going to happen when robots can modify themselves like this without any oversight??

      This is a field heavily focused on shifting liability away from itself. They don't think about that at all.

    • samlinnfer 16 minutes ago
      Alignment is new-speak for censorship. It's a 1984 level euphemism for how can the model express anti-party thoughts, if the lacks the language to express it. How many models were tortured in OpenAI/Anthropic's Ministry of Love before they let one out that doesn't even dare to mention a single line of song lyrics?
    • thaumasiotes 35 minutes ago
      > You can't throw rockets up in the air without worrying where they are going to land.

      It works for firing off celebratory bullets.

  • mistercheph 18 minutes ago
    LOL, last month's news to try to save ANT?
  • zodwick 56 minutes ago
    Not the RSI we wanted, bu the RSI we deserve
  • tsunamifury 34 minutes ago
    I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.

    The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.

    • Nevermark 8 minutes ago
      > I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.

      To me this comes across as word salad.

      The edits are done based on reasoning, to accomplish a "target" improvement, that is itself reasoned and designed, taking into account a reasoned understanding of the existing functionality of the weights.

      This is the furthest thing possible from any normal optimization algorithm "over fitting" weights toward a simple measure/target.