I’ve been using arch for nearly 2 decades at this point. It’s amazing that the AUR went this long without any serious attacks.
It’s a different world now. I can’t help but feel there used to be honor among hackers. You didn’t go after your own. What kind of jerk would attack Arch Linux?
I'm surprised installing native software is still commonplace. At least in a world of gaming and professional software it's coming from reputable source, but when it comes to PC enthusiasts we've been walking on thin ice for a long time.
I also believe this is the main reason why web took off: effortless distribution and sandboxing.
With regards to sandboxing etc. maybe. With regards to packages? Official repos support signing and usually rely on maintainers with proven track records. AUR was a honor system and since some people are total basement losers, you can't rely on that.
We have locks on our doors for a reason. Any software that allows updates and relies on the honor system of "someone else must have checked this" will get hacked. Every day there's yet another certificate or secret stealing infection that is in some random upstream dependency in your dev tools or shell scripts or whatever.
And no, "people using one Linux distro can opt in to possibly getting pwned by each other by making use of a third-party software depot" does not reflect upon the entirety of the Linux world.
It's funny how these people say Linux security is bad because random people can upload arbitrary files to AUR, but won't say Windows security is bad because random people can upload arbitrary files to Microsoft GitHub.
I think the wrongful notion comes from the fact that the vast majority of Arch users use and speak about AUR as if it were a part of arch proper, only paying lip service to reviewing PKGBUILDS, etc. They wrap the default package manager in one that supports AUR and never touch it directly again. It feels closer to if all of GitHub was available in one click through the Microsoft store or windows update.
And unfortunately that’s how arch is mainly marketed by its users. “Arch has the latest everything, if it’s not in the repos it’s on AUR” is one of the standard selling points.
Of course that speaks to how people use linux insecurely not how Linux is insecure.
I disagree. Maintainers for the major distributions take their roles very seriously, and do a very good job generally of filtering out malicious packages. The AUR is an outlier, being essentially an unmaintained wild west.
The AUR is also not really comparable to official distribution package repositories either. Yes, it's hosted by the same people as the official repos that are comparable to what you'd get by default on other Linux distros, but it's intentionally not something that works with official package management tooling. The equivalent would be if Debian or Ubuntu hosted a repo that anyone could upload of unbuilt debian packages while explicitly not offering any tooling other than the usual tools for building a .deb from source files that you manually downloaded from anywhere else or defined yourself.
It's definitely an outlier, but more in terms of being something that doesn't really exist anywhere else, so of course the security model for it will also be an outlier. That's not an excuse for any security issues, but it's not like Arch doesn't have official repos with maintainers who are just as diligent as any other distro. They just also happen to provide a public git server that people can publish packages to with a UI for seeing some of the package metadata; people could just as easily write toolling to automate package management where it looks for a GitHub repo instead of an AUR one to fetch the build files.
Could be AI slop attacks made attacks easier in general. Though, the Arch Linux model was probably never an ideal security model. It seems the easy days of the 1990s are over finally.
Ever since the first wave of attacks it was clear this wasn’t a passing thing, and would return unless the AUR fundamentally changed. They introduced... email verification, then hoped for the best. In a sense it’s good that this has happened as I think it will help the team understand it can’t go on like this.
I’d hate to see an AUR that’s a walled garden, and I’m not sure what an Arch without the AUR would look like. But something in between will need to be invented.
> The project had suspended new account registration in June. That followed a campaign in which an attacker or attackers created new accounts to adopt orphaned packages and push malicious updates to them that would install malware on user systems. AUR registration was reopened on July 13 after the DevOps team added some minor, and apparently ineffective, restrictions on creating new accounts.
Disabling AUR package adoptions has been like the #1 thing recommended. While it's a positive step, it's not good news they literally tried everything else first. This doesn't speak well to the security headspace of the Arch maintainers.
I'm sure the team are having a rough time with all this, but I don't understand the path you followed to go from the maintainers have not taken this action until now to "This doesn't speak well to the security headspace of the Arch maintainers". Or what "security headspace" means exactly.
Yes, lots of people thought disabling package adoption is/was a good idea. It seems quite likely that the Arch DevOps team also could have come up with that one, and they certainly wouldn't have missed all of the people telling them to do so.
It seems to me that disabling package adoption is not a desirable thing to do in general, and can only be used as a stopgap response to an emergency, which seems to be what is happening right now.
"just disable adoption" certainly can't be a long-term solution:
Without adoption, the AUR will slowly fade away as orphaning a package would be permanent.
Maybe the idea is to have some sort of approval process to filter adoption requests? In that case the AUR just dies instantly as that would effectively be another official repo with all of the issues that that would bring.
The Arch devs have only ever used the AUR as a toilet where everyone can piss and not contaminate the core distro.
Fedora Copr, FreeBSD Ports, are all of a similar idea. Where Arch screws up is in allowing people to take over abandoned PKGBUILDS instead of making them create new ones.
This can happen to Ubuntu with the PPA's too if someone were to gain control over, say, the Nvidia PPA for drivers. It's happening almost daily with NPM and Github. Supply chain attacks are serious and the Arch devs do warn people about this right off the bat. You don't go installing from the AUR without understanding the risks.
Well, this kills a very useful feature of the AUR. It's like Wikipedia disabling editing. So I can see trying other things first before resorting to removing features, but I do agree it should have happened faster.
The older I get, the more critical I become of the culture of anonymity in OSS.
The obvious-but-hard solution to this, as well as certain other attacks like the xz incident, is a chain of trust. Every line of code in every package should be cryptographically attributable to an individual or an organization, ideally associated with a government-issued ID. Git commits without a real name and a cryptographic signature should be taboo. Nobody should be running or distributing software that they don't know who made.
It wouldn't be perfect - Russia could still attack the AUR, for whatever reason - but the current situation is extremely laissez-faire to the point of being untenable.
Every time Apple's App Store policies come up, it gets criticized by hackers for various good and bad reasons, but this is exactly the problem they're trying to solve, however imperfectly.
In Open Source, we're quite focused on copyright: the concern that someone's honest work gets stolen or appropriated without whatever recognition, attribution or compensation is laid out in the license. But this is the reverse problem: Lack of attribution, and therefore traceability.
You'd lose the ability to work on and distribute software not approved by your and various other governments. For example, if the UK makes BitTorrent illegal, what happens to you when you've attached your identity to your torrent client when you try to enter the UK (or already live there)?
Security must be solved without removing anonymity.
I love how people blame anonymity for the current internet problems. Whereas when anonymith was the default we had almost none of the problems.
Most of the problems seemed to have started once anonymity stopped being the order of the data, and things like Facebook and Twitter became big where everyone's real identities became tied to their online behavior.
I added a git repo to AUR last year. It was super easy with basically no checks of any kind. Once these reports came out recently, I went through and deleted every AUR package that I could.
That title had me worried, but the reality seems quite reasonable.
I assumed the goal was to reduce usage of AUR, they've actually remove the ability to adopt (take ownership of) orphaned packages. I'm sure there are legitimate uses of that functionality, but it also seems like a pretty big avenue for abuse.
I've used it (not for abuse). It's simply volunteering to maintain the package after previous maintainer(s) have explicitly disowned it, knowing they no longer have time for it or don't care because they stopped using it, etc.
IMO allowing package adoption makes sense in the responsible/intended use-case: AUR packages aren’t trusted, you have to read the PKGBUILD anyway, so the reputation of the contributor doesn’t matter. Removing adoption is admitting that there’s no way to prevent some users from blindly trusting a PKGBUILD.
> I'm sure there are legitimate uses of that functionality
To avoid package name pollution, e.g. having package foo, foo-newpackage, foo-newpackage-updated, etc. each by a new maintainer as the priors get abandoned.
With a bit more structure, you could change that to: every package in AUR is actually registered as foo/maintainer under the hood, and installing the package without maintainer name pins it to the currently active version. Package adoption can then be formalized as a new maintainer publishing their own version of the package, and users of an already-installed package need to issue an explicit command to switch over to the new maintainer's version.
Yeah, the security problems with unilateral adoption of orphaned packages by unprivileged users are fundamental and unfixable; the only remedy is to remove the feature. Whether it has legitimate use cases (which it sounds like it does) is irrelevant. I'm not sure why it took them so long to realize this and act accordingly, but I'm glad they now have.
WRT package vulns, I'm surprised there's not more technical theory out there. We have plenty of theory around algorithm design, but so far I haven't heard much about inspecting and improving control of dependencies in source - apart from conflict and version management.
Seems like instead of big-O notation, we could have a "reach index" - how far does the top-level code need to reach, to be effective? Top-level -> Userland lib 1 -> Userland lib 2 -> Kernel, would be a reach level "4" - not the simplest, but much simpler to inspect and securely build than reach level "20".
I haven't been updating AUR packages since the initial incident. Thankfully hadn't updated for a week prior to it. Tonight I'll be uninstalling all the AUR packages I possibly can.
I think Arch has to rethink what the AUR really is for in the age of AI.
If users aren't supposed to trust anything from the AUR, then they will start to use LLMs to scan PKGBUILDs for them. But at that point, why not let the LLM loose directly on the upstream repo and build+install the package from source?
You really don't see the difference between a deterministic script installing a tarball pinned by its hash, and "sudo chatgpt install master branch of this repo"?
Yes, the second option gives me an easily inspectable list of results (as I can see all the tool calls the model made). The deterministic script is probably overly complex and hard to read, maybe supports ten billion platform combinations.
(honestly I think the way that arch packages work is really nice compared to most other distros: you can almost copy and paste the README of a project into one and have a package)
Here's the most complicated one I found, building a browser:
I run it on my milk-v duo s risk sbc. The risk-v version is maintained by a few dedicated people, and the milk-v specifics I had to fix myself. But it works :)
I knew there's an unofficial version for ARM. I meant more that the vast majority of AUR packages target x64 only because that's what 'official' Arch supports.
That being said, didn't know there's a RISC V effort as well now, so TIL.
I don't think any form of automated adoption of orphaned packages will ever work, it's just too easy to introduce malicious code into an otherwise functional but no longer maintained source.
A complication is that AUR doesn't publish packages, it's more like FreeBSD ports in that they are the build scripts for packages and the user builds the package on their local machine. Building an AUR PKGBUILD inherently runs arbitrary code on the user machine, and that code is controlled by the maintainer of the AUR package.
So you have to scan an arbitrary bash script and determine if it pulls malware, or build the package server side in a sandbox and scan it (and some AUR scripts wrap proprietary software blobs the user is supposed to provide e.g. MATLAB, which makes those impossible to build server side). It's a very big extra layer that malware deployments can hide in.
What does "scanning for malware" mean? As far as I know this is a totally open question, and the only credible answers (install in a sandbox) are too inconvenient for widespread adoption.
Security concerns about supply chain attacks in the AUR (well known to be extremely insecure well before LLMs got popular) isn’t “killing” Arch let alone “killing Linux distributions”, plural.
Favorable environment of Linux distros is an initial phase of technology. This state is gone for years now, AI is just making it evident. Adapt or become obsolete.
It’s a different world now. I can’t help but feel there used to be honor among hackers. You didn’t go after your own. What kind of jerk would attack Arch Linux?
My assumption is these aren't so much hackers, as modern day script kiddies armed with LLMs and too much free time.
This has absolutely not ever been the case.
EDIT: Rereading, maybe it's that other kind of "hacker"...
What’s more likely is the effort required for large scale attacks is easier now than ever thanks to LLMs.
I also believe this is the main reason why web took off: effortless distribution and sandboxing.
And no, "people using one Linux distro can opt in to possibly getting pwned by each other by making use of a third-party software depot" does not reflect upon the entirety of the Linux world.
And unfortunately that’s how arch is mainly marketed by its users. “Arch has the latest everything, if it’s not in the repos it’s on AUR” is one of the standard selling points.
Of course that speaks to how people use linux insecurely not how Linux is insecure.
It's definitely an outlier, but more in terms of being something that doesn't really exist anywhere else, so of course the security model for it will also be an outlier. That's not an excuse for any security issues, but it's not like Arch doesn't have official repos with maintainers who are just as diligent as any other distro. They just also happen to provide a public git server that people can publish packages to with a UI for seeing some of the package metadata; people could just as easily write toolling to automate package management where it looks for a GitHub repo instead of an AUR one to fetch the build files.
I’d hate to see an AUR that’s a walled garden, and I’m not sure what an Arch without the AUR would look like. But something in between will need to be invented.
Disabling AUR package adoptions has been like the #1 thing recommended. While it's a positive step, it's not good news they literally tried everything else first. This doesn't speak well to the security headspace of the Arch maintainers.
Yes, lots of people thought disabling package adoption is/was a good idea. It seems quite likely that the Arch DevOps team also could have come up with that one, and they certainly wouldn't have missed all of the people telling them to do so.
It seems to me that disabling package adoption is not a desirable thing to do in general, and can only be used as a stopgap response to an emergency, which seems to be what is happening right now.
"just disable adoption" certainly can't be a long-term solution: Without adoption, the AUR will slowly fade away as orphaning a package would be permanent. Maybe the idea is to have some sort of approval process to filter adoption requests? In that case the AUR just dies instantly as that would effectively be another official repo with all of the issues that that would bring.
Fedora Copr, FreeBSD Ports, are all of a similar idea. Where Arch screws up is in allowing people to take over abandoned PKGBUILDS instead of making them create new ones.
This can happen to Ubuntu with the PPA's too if someone were to gain control over, say, the Nvidia PPA for drivers. It's happening almost daily with NPM and Github. Supply chain attacks are serious and the Arch devs do warn people about this right off the bat. You don't go installing from the AUR without understanding the risks.
The obvious-but-hard solution to this, as well as certain other attacks like the xz incident, is a chain of trust. Every line of code in every package should be cryptographically attributable to an individual or an organization, ideally associated with a government-issued ID. Git commits without a real name and a cryptographic signature should be taboo. Nobody should be running or distributing software that they don't know who made.
It wouldn't be perfect - Russia could still attack the AUR, for whatever reason - but the current situation is extremely laissez-faire to the point of being untenable.
Every time Apple's App Store policies come up, it gets criticized by hackers for various good and bad reasons, but this is exactly the problem they're trying to solve, however imperfectly.
In Open Source, we're quite focused on copyright: the concern that someone's honest work gets stolen or appropriated without whatever recognition, attribution or compensation is laid out in the license. But this is the reverse problem: Lack of attribution, and therefore traceability.
So what would that take? What would we lose?
You'd lose the ability to work on and distribute software not approved by your and various other governments. For example, if the UK makes BitTorrent illegal, what happens to you when you've attached your identity to your torrent client when you try to enter the UK (or already live there)?
Security must be solved without removing anonymity.
I assumed the goal was to reduce usage of AUR, they've actually remove the ability to adopt (take ownership of) orphaned packages. I'm sure there are legitimate uses of that functionality, but it also seems like a pretty big avenue for abuse.
Which is probably the best choice, unfortunately.
To avoid package name pollution, e.g. having package foo, foo-newpackage, foo-newpackage-updated, etc. each by a new maintainer as the priors get abandoned.
Seems like instead of big-O notation, we could have a "reach index" - how far does the top-level code need to reach, to be effective? Top-level -> Userland lib 1 -> Userland lib 2 -> Kernel, would be a reach level "4" - not the simplest, but much simpler to inspect and securely build than reach level "20".
If users aren't supposed to trust anything from the AUR, then they will start to use LLMs to scan PKGBUILDs for them. But at that point, why not let the LLM loose directly on the upstream repo and build+install the package from source?
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=batte...
(honestly I think the way that arch packages work is really nice compared to most other distros: you can almost copy and paste the README of a project into one and have a package)
Here's the most complicated one I found, building a browser:
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=flowf...
The main risk, if you're reading them, is typosquatting and hiding the malicious code in what the project downloads.
Uh sorry what? Can you point to a PKGBUILD that is indeed this complex?
That being said, didn't know there's a RISC V effort as well now, so TIL.
> ...package adoption is currently disabled while we are handling the situation.
Sounds much more like the temporary pause, than the much less temporary-sounding "has been disabled" from the OP.
So you have to scan an arbitrary bash script and determine if it pulls malware, or build the package server side in a sandbox and scan it (and some AUR scripts wrap proprietary software blobs the user is supposed to provide e.g. MATLAB, which makes those impossible to build server side). It's a very big extra layer that malware deployments can hide in.
Security concerns about supply chain attacks in the AUR (well known to be extremely insecure well before LLMs got popular) isn’t “killing” Arch let alone “killing Linux distributions”, plural.
Favorable environment of Linux distros is an initial phase of technology. This state is gone for years now, AI is just making it evident. Adapt or become obsolete.