I know Mastodon hates LLM's and AI. So here goes!

I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.

The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.

I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.

I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.

The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.

I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.

I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.

Please run an LLM over your code base if it's internet facing or something critical, we thank you!

Can't wait for the discussions on this!

@drwhax I think we need a bit of nuance here.

As much as I am no fan of LLMs, I can see the value of running them over one's own codebase in order to find vulnerabilities.

But I remain unconvinced that vibe-coded fixes are a good idea, or ever will be. I think this needs to be clearly said.

The reason is that these are fundamentally different problems.

@drwhax finding vulnerabilities can be stochastic because it has a very clear and effective verification function: either the exploit works or it does not. Exploit code can be messy and convoluted, as it is not going to be maintained after the vulnerability is fixed.

Vibe-coding fixes does not have that kind of verification function: the fix must not only close the specific vulnerability, but *also* not introduce new ones or re-introduce old ones, and it has to be maintainable in the future.

@drwhax and in the end this will come down to managing limited resources and incentives, as always.

If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.

@drwhax and just to drop some data on vibe-coded software quality:
https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways

The report tries to be very "AI"-positive. But the numbers don't lie: +861% code churn, +242% incidents per PR, +54% bugs per developer.

@rysiek human in the loop etc :)

@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.

"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.

Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.

@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.

I almost puke writing this lol

@drwhax I am very very doubtful we will in any meaningful way.

In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.

We will be able to automate some things slightly better, though. But then the question is: at what cost?

@rysiek everyone's mental capacity? :)

@drwhax that too, but I meant it even in the purely monetary sense of token costs.

@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.