This describes my life as an open source maintainer at the moment!
In the first 10 years of the rclone project we received about 20 security disclosures through GitHub. We had to deal with over 40 in the last month! That has taken a huge amount of my time, even using AI tools to triage and come up with fixes for review.
The hit rate for those security disclosures is pretty good - about 75% of them have a nugget of something which needs looking at. The configurations for rclone have got increasingly unlikely so I'm hoping they will dry up eventually.
I was considering just merging the fixes straight to master just to make my life easier rather than holding a dozen independent security fixes on branches and merging them at the point release and hoping not to have too many conflicts to fix up. I've decided to stick with the process for the moment.
GitHub assigns CVEs for the advisories. Before the AI apocalypse they took 2-3 days for an assignment but now it they are running at 3-4 weeks so I have to send the point releases out with CVE-PENDING in the changelog which isn't ideal.
Not sure what the solution is, but it is definitely a problem for us.
Possibly some sort of ai agent code review system that churns through code looking for these vulns before the code is published. It feels like it's all about who has the resources to find bugs at the moment but that it should be a standard to catch issues before prod moving forward..
Yea, AI bug finding over entire projects, at least on SOTA models is super expensive, hence those with the resources setup to find the most bugs in an automated way.
That said there are a number of people and companies working on more focused means of driving the LLM to look were bugs would be the most dangerous and in doing so reduce the token spend of each bug found.
In some ways the better you are at security stuff the more you can reduce your spend by better driving the LLM to problem spots.
> GitHub assigns CVEs for the advisories. Before the AI apocalypse they took 2-3 days for an assignment but now it they are running at 3-4 weeks so I have to send the point releases out with CVE-PENDING in the changelog which isn't ideal.
A strange bottleneck; anyone know why that would be so slow?
I feel like someone somewhere is working on an opensource GPU compute resource pool you guys could pull from as part of a donation type framework. I got a bunch of local gpu resources just relaxing, and if I could load up a a binary to provide compute for X hours a day overnight or whatever, that'd be cool.
http://github.com/buzz/block is quickly becoming a reasonable approach to shared agent compute, though it's still a bit more hands-on than the older 'compute donation' projects like SETI or Folding@Home.
I can’t comment on if it applies to your workflow, but one process I’ve used is to aggregate and land ~10 security patches at a time. Eg,
- grab a group of (related) bugs/defects/vulns
- fix them on a branch like bug-batch-XXX
- run that group through the verification, landing in main, CI/CD flow to amortize process cost
- repeat as needed to process backlog
My experience is that process often has irreducible time (eg, two days due to reviews by various parties); but that time slot can be shared between several bugs in a single PR — especially if you have several related to the same feature.
It's easier to find bugs, fix them, yet there's less will than ever. My bosses just want speed and will give me a 30 minute lecture on why I don't need to solve a bug that Claude solved in 5 minutes, I've verified, and it's already in an open PR. All the while we're pushing out bugs faster and faster.
No matter how good AI gets at fixing bugs we'll never fix them when there's no will to fix things. Software will never be good if there's no will to make good software. The problem has always been about will. To many better products. It's insane that in a time where we can do better on speed and quality we still choose speed and tell ourselves it's velocity
Something that might come out of all of this is that companies that previously slow rolled security fixes will now be forced to fix them quickly, because the speed of AI and the liability of being insecure.
Yea, when their cyber insurance policies start biting them because AI found a bug months ago and it was never fixed I figure we'll start seeing a change.
With that said there are numerous companies that are very concerned about the situation. They know AI is finding bugs in their software at an accelerated rate, one they are having difficult times keeping up with because they want human understanding and review of the fixes to avoid introducing new bugs.
Depends on the jurisdiction. Where I live, companies are not liable for insecure software, if sold to a private person for example. Enter huge botnets of crappy home routers. Hopefully this is going to change.
Most places I've worked have been infuriatingly uninterested in fixing bugs, and would release software with major known bugs and a vague plan to fix them later. Of course, when "later" came, there were more features to cram. No time to fix those bugs.
I'm hoping one of the unintended side effect of it being essentially free to find and exploit (and fix) software bugs is that companies become less cavalier about shipping bugs in their software. Unlike most of the industry I don't believe "bugs are inevitable." Bugs are a choice developers make when they're rushing and careless and when all of their incentives are to ship quickly. You can ship bug-free software but it takes (or used to take) a really long time and a lot of care, care that commercial software developers just don't ever seem to muster.
Maybe when their software is getting 0wned over and over and 30 security issues are published a day, they'll start caring and taking their time.
If your boss asks for X feature and you split it into 4 PRs, 3 that implement X and a 4th that fixes a security bug, how would they know? Someone that uses those 30mins like that isn't reviewing your PRs themselves. Just fix it and be done with it. You only needed to argue with the boss when you needed to make time for it which would delay something else. If it just appears done they'll just react with clap emoji later when you announced you also fixed this extra thing.
You haven't worked at conservative software shops I guess. The last place I worked at had this huge song and dance with you trying to raise a PR that looked irrelevant to the task at hand would lead to meeting with your lead dev, your manager and your skip level as you tried to explain why the bug needs fixing...
The worst one was where I fixed a datetime bug and although it had been sending out false alerts, I was asked to dry run the 5 lines of code I changed, like a coding interview. In all this pressure I forgot what the code was meant to do, and was dismissed and asked to set up another meeting with an explanation of all the various cases that could happen...
Let's spend $5m to have a meeting about why we can't spend $50.
Penny wise, pound foolish
The best is the explanations are in the PRs. They even have Claude these days to answer their questions... I swear, these people are more addicted to spending money than they are about making money.
That's mostly how I'm shifting to doing things but let's admit, this is metric hacking. And let's also admit, it takes longer to go through that song and dance than it actually takes to do the work.
Very low signal information, preemptively trying to cover every rebuttal despite nobody ever planning on making one, in the few times someone does he plays devils advocate endlessly
Like bro just let us babysit these agents, everything’s going to happen
You will see, they will change both laws and expectations, to say its normal for software to always have terrible bugs. You can always solve a problem by lowering your expectations. :-)
Well, they at least say that one is obligated to address bugs in a timely manner (where "not widely exploitable; won't fix" is a perfectly valid resolution).
!!! You described my experience in the workforce to a t! So frustrating that tech managers look at tech debt as a thing to be maintained at a certain level instead of allowing us to achieve perfection
> tech managers look at tech debt as a thing to be maintained at a certain level
What they don't seem to understand is all debts have interest. In my current project our entire team is tripping over that interest created by one person. Though my manager is frustrated with me because I'm "spending too much time trying to understand". Btw, that is a few hours here and there, maybe a day for a rabbit hole which results in me creating a dozen or more issues. How is that "slow"?
> allowing us to achieve perfection
Impossible
Look, I'll defend high quality code all day long. Good code allows you to move fast just like taking 30s to tie your shoe laces allows you to run faster.
But we can't write high quality if we pretend that perfection exists. Globally optional solutions are the exception, not the norm. There's almost always tradeoffs we need to make. The high quality code, the high quality engineering, is understating, tracking, triaging, and minimizing those tradeoffs. It is writing code that allows you to change those decisions as quick and effectively as possible. But perfection doesn't exist. It's a good thing to chase, like Utopia, but ultimately unobtainable. A good engineer knows how to triage.
Advocating for perfect code will be a losing battle. But we should not, even for a second, let that be interpreted as meaning quality doesn't matter. I'm still frustrated with people who say "don't let perfection be the enemy of good enough" as most of the people that say that believe "good enough" is "it looks like it is working" and call a "demo" a MVP
with agent loops, the cost of exposing bugs should be cheap now. every pen tester should be using agent loops and on the flip side, an agent could easily see this issue once filed and a fix immediately done. it's incredible how cheap code is now.
I don't think this is new with LLMs (finding an exploit based on a few words offhand has always been a fun part of exploit development), but it's scaled and democratized to mass exploitation of low value targets. Backing exploit PoCs out of patches, commit messages, and random overheard or over-read sentences is a practice as old as vulnerability research. The difference with LLMs is that an explosion in actors "skilled enough" (human or not) has enabled sloppy / low-skill "exploit the whole Internet" actors in a way they weren't previously enabled.
I do agree with the author's ideas, though; most of these are things that should have been done much sooner, and I suppose it's good in a sense that there is a forcing factor now.
> Backing exploit PoCs out of patches, commit messages, and random overheard or over-read sentences is a practice as old as vulnerability research
True, but it used to take days or weeks of research, testing, and RE to get those PoCs.
Today the entire chain - reading commits, RE patch binaries, building exploit, scripting exploit scan, $profit - can be fully automated and happen in minutes or hours.
I think roll-out and deployment are even bigger issues. Who updates their software stack within 10 minutes? Most CI runs take longer to verify the business logic is still working.
Add to that the danger of supply-chain attacks where you don't even want automatic updates.
I do wonder what the hit rate is in general for Claude finding a successful exploit when prompted with a rumor that leads it to assume the bug is there.
“I’m told there’s a path traversal exploit in this package. Can you find it?” - probably a reasonably high chance of it finding one, even if you just made that rumor up.
Sounds like the next step is to start rumors about vulnerabilities so that someone else will find them and you have plausible deniability. "Hey, did you hear? Someone told me <competitor company> might be leaking their logs from a publicly accessible endpoint..."
Transferring ideas in their abstract form is entirely possible now. What this blog post describes I believe even undersells the current reality. I've had success with finding crits by pasting in extremely high level or abstract concepts. I recently went from a talk abstract (3-4 sentences) to crits lol (finding that vuln type).
No mention of memory safe languages? Sure, it does not help existing projects, and sure, of course you can still have logic bugs (or memory bugs using escape hatches like unsafe). But they do help significantly in reducing the number of exploitable bugs.
Yeah, but there's already plenty written on that topic elsewhere; the OP is making a distinctly different point and everything else in it still applies, I think. Even if you are using a memory safe language -- if you are developing security sensitive software, you need to understand that exploits can now be researched, developed, and deployed in hours (or for stupid bugs, minutes!) instead of days or weeks or months. And this process can now just be massively scaled up with money and compute. It was not like this even 24 months ago.
In the first 10 years of the rclone project we received about 20 security disclosures through GitHub. We had to deal with over 40 in the last month! That has taken a huge amount of my time, even using AI tools to triage and come up with fixes for review.
The hit rate for those security disclosures is pretty good - about 75% of them have a nugget of something which needs looking at. The configurations for rclone have got increasingly unlikely so I'm hoping they will dry up eventually.
I was considering just merging the fixes straight to master just to make my life easier rather than holding a dozen independent security fixes on branches and merging them at the point release and hoping not to have too many conflicts to fix up. I've decided to stick with the process for the moment.
GitHub assigns CVEs for the advisories. Before the AI apocalypse they took 2-3 days for an assignment but now it they are running at 3-4 weeks so I have to send the point releases out with CVE-PENDING in the changelog which isn't ideal.
Not sure what the solution is, but it is definitely a problem for us.
I felt that. The problem with doing that is you hate yourself afterwards so not a good solution either, gotta do it properly.
Thanks for working on rclone, Nick!
Possibly some sort of ai agent code review system that churns through code looking for these vulns before the code is published. It feels like it's all about who has the resources to find bugs at the moment but that it should be a standard to catch issues before prod moving forward..
That said there are a number of people and companies working on more focused means of driving the LLM to look were bugs would be the most dangerous and in doing so reduce the token spend of each bug found.
In some ways the better you are at security stuff the more you can reduce your spend by better driving the LLM to problem spots.
A strange bottleneck; anyone know why that would be so slow?
- grab a group of (related) bugs/defects/vulns
- fix them on a branch like bug-batch-XXX
- run that group through the verification, landing in main, CI/CD flow to amortize process cost
- repeat as needed to process backlog
My experience is that process often has irreducible time (eg, two days due to reviews by various parties); but that time slot can be shared between several bugs in a single PR — especially if you have several related to the same feature.
Can you discuss this? I might be able to help.
No matter how good AI gets at fixing bugs we'll never fix them when there's no will to fix things. Software will never be good if there's no will to make good software. The problem has always been about will. To many better products. It's insane that in a time where we can do better on speed and quality we still choose speed and tell ourselves it's velocity
With that said there are numerous companies that are very concerned about the situation. They know AI is finding bugs in their software at an accelerated rate, one they are having difficult times keeping up with because they want human understanding and review of the fixes to avoid introducing new bugs.
I'm hoping one of the unintended side effect of it being essentially free to find and exploit (and fix) software bugs is that companies become less cavalier about shipping bugs in their software. Unlike most of the industry I don't believe "bugs are inevitable." Bugs are a choice developers make when they're rushing and careless and when all of their incentives are to ship quickly. You can ship bug-free software but it takes (or used to take) a really long time and a lot of care, care that commercial software developers just don't ever seem to muster.
Maybe when their software is getting 0wned over and over and 30 security issues are published a day, they'll start caring and taking their time.
The worst one was where I fixed a datetime bug and although it had been sending out false alerts, I was asked to dry run the 5 lines of code I changed, like a coding interview. In all this pressure I forgot what the code was meant to do, and was dismissed and asked to set up another meeting with an explanation of all the various cases that could happen...
Penny wise, pound foolish
The best is the explanations are in the PRs. They even have Claude these days to answer their questions... I swear, these people are more addicted to spending money than they are about making money.
Very low signal information, preemptively trying to cover every rebuttal despite nobody ever planning on making one, in the few times someone does he plays devils advocate endlessly
Like bro just let us babysit these agents, everything’s going to happen
FFS, we're living in a world where Linux has more than doubled in popularity mainly due to Microsoft actively fleecing their customers. It's crazy
/I use arch btw (and have been on it for over a decade)
Look, I'll defend high quality code all day long. Good code allows you to move fast just like taking 30s to tie your shoe laces allows you to run faster.
But we can't write high quality if we pretend that perfection exists. Globally optional solutions are the exception, not the norm. There's almost always tradeoffs we need to make. The high quality code, the high quality engineering, is understating, tracking, triaging, and minimizing those tradeoffs. It is writing code that allows you to change those decisions as quick and effectively as possible. But perfection doesn't exist. It's a good thing to chase, like Utopia, but ultimately unobtainable. A good engineer knows how to triage.
Advocating for perfect code will be a losing battle. But we should not, even for a second, let that be interpreted as meaning quality doesn't matter. I'm still frustrated with people who say "don't let perfection be the enemy of good enough" as most of the people that say that believe "good enough" is "it looks like it is working" and call a "demo" a MVP
Huh?
I do agree with the author's ideas, though; most of these are things that should have been done much sooner, and I suppose it's good in a sense that there is a forcing factor now.
True, but it used to take days or weeks of research, testing, and RE to get those PoCs.
Today the entire chain - reading commits, RE patch binaries, building exploit, scripting exploit scan, $profit - can be fully automated and happen in minutes or hours.
Add to that the danger of supply-chain attacks where you don't even want automatic updates.
“I’m told there’s a path traversal exploit in this package. Can you find it?” - probably a reasonably high chance of it finding one, even if you just made that rumor up.
https://sgnt.ai/p/terrible-mistake/
Even dictators have to share.
LLMs are extremely good at finding corner-case vulnerabilities in C bindings even within a memory safe language; see for example the fixes in an OCaml crypto library here: https://discuss.ocaml.org/t/the-series-of-mirage-crypto-rele...