I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.
The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.
I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.
I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.
The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.
I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.
I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.
Please run an LLM over your code base if it's internet facing or something critical, we thank you!
1. Acceptance. There's the saying "you just need to be more secure than your neighbor". It's not fully true but biz people hold it so. It's flip side is a cynic commentary on security, and a true one at that: If your neighbor is not secure, you don't need to be either. This is still true if you're getting pwned. When everyone's getting hacked regularly, society will accept this new reality. Only the defense sector will still care for security. We've seen similar developments in different industries: Fast fashion isn't an adequate replacement for garment that stands the test of time. But it's getting impossible to buy good clothes, even if you want to. But also with other goods: If everyone produces low quality items, it's financially more reasonable to do the same.
2. Centralization. Coz we won't do the right thing and reduce complexity, we need to find a way to make complexity "secure": Cloud, Big5, etc. Corps that have the resources to host an Exchange server securely... or in compliance with regulations. Every tool you do not let a big house host is a liability. Not coz they're necessarily more secure, but because nobody will blame you if something gets wrong. We had that with "buying IBM" once.
I hate this outlook but I seriously doubt that we will correct course.
@drwhax finding vulnerabilities can be stochastic because it has a very clear and effective verification function: either the exploit works or it does not. Exploit code can be messy and convoluted, as it is not going to be maintained after the vulnerability is fixed.
Vibe-coding fixes does not have that kind of verification function: the fix must not only close the specific vulnerability, but *also* not introduce new ones or re-introduce old ones, and it has to be maintainable in the future.
@drwhax and in the end this will come down to managing limited resources and incentives, as always.
If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.
@drwhax in a way this is a question of how soon we finally get out of the Gartner hype cycle and people get to focus on figuring what these tools are *actually* useful for.
@drwhax But are you maling a tech bro richer to vibe code everything you do and just pressing Y without thinking, wasting tokens as much as possible because who cares? I have beef.
Are you actively (and disgustingly so), trying to replace every human in the loop just to create ghibli styled art slop or disgusting political videos so you can further your agenda? I have beef.
Are you actively numbing your brain, stop thinking about anything and becoming a reverse centaur on purpose? I have beef.
@drwhax I personally dont and will never vibe code because, in a very personal position, I love my brain and I like to think, and a brain without friction will cease to function (use it or lose it).
I love people making art (and yes, this includes code). And I deeply hate the contempt for people paying a tech bro to use genAI to dismiss and degrade those artists with a sort of vindictive glee, especially since most artists are already treated so badly. It is disgusting.
@drwhax But I don't mind the underlying tech as much as I dont have a beef against the, let's say Microsoft Kinect (a tool made in essence the same way).
So are you getting a Chinese open weight, AI (people won't say it, maybe because of fear of retaliation from the US, but most companies will do this) stuffing it in your server and having 90% effectiveness to find vulnerabilities and doing static analysis in code locally?
That sounds like an actual use case. Just one the tech bros don't like.
@drwhax i have personally helped 3 different fediverse server projects by sharing exploits with them that LLMs found. Most software is full of holes. Running an LLM to hunt for bugs is good advice. If you're not doing it, someone else will.
@drwhax do you have any pointers on *how* to run an LLM over the code base? Just tell Claude to "look for vulnerabilities" or is the process more elaborate?
There's a couple things you can do, one of them is indeed as basic as, this is app XYZ, written in PHP, review the codebase for security problems, then adversarially review your findings before writing them down in findings/ per bug in markdown. Also make a root cause and look for variants of the same class of issues across the code base.
@drwhax@can great pointer, thanks. Going to try this tomorrow on a couple of our codebases (one 100% human coded, one originally human but with progressive LMM development over the last couple of months) - might be adjusting some development schedules depending on what I find!
@drwhax Maybe the advice of the wiser among us will be heard after all these decades: focus on robust mitigations and attack surface reduction, because the moles can't be whacked anymore.
@buherator@drwhax I sometimes advise investors (poor life choices I guess) on how to deal with the increased risk of Mythos & others. And this was the bottom line.
I don't even need AI specifically to argue for that. The time-to-exploit has been dropping forever. We've built better catalogs and more versatile infrastructure and complex automations. It's very clear that this includes vuln research as well.
We had this already with static analysis hitting the field. We answered with "shift left": Building security into the dev process, since the earlier you do it, the less issues you'll end up with. Every step earlier has more downstream impact.
We've finally reached the stage where security as part of your design & architecture is not only best practice but becoming a hard requirement. Our automations (including LLMs) pushed us there and we need to deal with the outcome.
Or should. Because I'm a cynic and I've seen what's happened, I doubt that engineering will follow through with this. I see two paths and we'll move along both, and I don't like either: Acceptance and few centralized Oligopols.
The nice thing about open source is that there's always *someone* doing whatever obscure thing the mainstream finds too niche. I'm sure there are people right now who have for decades been building smaller software, and maybe for a certain non-corporate group there can be a pivot to that stuff. To avoid an endless edge of new code to harden, you need to forgo compatibility with new software & hardware, which has until now meant relative uselessness. But given how the AI bubble is driving up hardware prices, and the fact that we've seen standards like USB mature to a point that feels like it could be stable for a decade or more, maybe the endless cycle of new hardware stuff just slows down in certain backwaters. Of course this is anathema to corporate strategies, but I wonder if given the fact that the churning edge will be nearly impossible to secure, maybe a chunk of consumers will find ways to be satisfied with a smaller more stable core.
@tiotasram@buherator@drwhax Absolutely. I believe that, for personal use, low-power devices, cloud-less with P2P collab features are a viable route. I fear however, that using such devices becomes ever harder w/o locking yourself out of the rest of the world: Requiring apps for concert tickets, public transports, online banking, or event finding events through instagram. The latter can possibly be fixed through better communities, the former is hard to address, except by having a personal device and one just for carrying such credentials.
@ljrk@drwhax Re: 1., the usual question we got from CISO and above on our reports is "how does it look compared to similar companies?". I agree that this is in part the "you don't have to outrun the bear" logic, but also that people in position don't want to look incompetent in front of their peers.
Another thing is that you don't get fired if you didn't follow the hackers advice, but you do get fired if you don't pass compliance which is one part BS, and the other part is easy to cheat.
@buherator@drwhax Yup same. It's what we do at $dayjob to motivated customers, partially. I'm waiting for a cohort to realize that, if nobody does anything, nobody will look too bad.
@ljrk@buherator@drwhax Part of the source of this is the lawyers. In business, companies are typically shielded from liability if they follow "standard and customary business practices". So if "everyone else" is doing it that way, you can too without taking on the liability.
Of course, that takes you only so far. If your business is unable to function for two months, legal liability may well be the least of your concerns. But the problem with that threat is that it's a gray rhino risk. Those are really really hard to get businesses to spend on mitigating.
For better or worse, LLM-based attacks may remove most of the gray from those risks and reduce the resistance to mitigation.
@mweiss@buherator@drwhax Totally, I'm pretty sure that risks that "end/pause business" will be addressed soon-ish. However, risks that are... less so, e.g., data loss, data privacy stuff, ... we'll just get used to it. Or the many downtimes of GitHub.
@rysiek For sure and the other part is guardrails that currently aren't aligned with the maintainer from these parrots either. Patching is hard or it'll fail because "CYBER"
@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.
"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.
Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.
@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.
@drwhax I am very very doubtful we will in any meaningful way.
In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.
We will be able to automate some things slightly better, though. But then the question is: at what cost?
@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.
@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.
In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.
That said, it is still immensely expensive for them to run it on their own infra.
@rysiek@drwhax I remember reading an article about shit hospital admins wanting to ditch radiologists all but one who would check the AI dx...and if anything went wrong, can blame the hoo-man! And fire that one, hire another at an even LOWER rate! All for your Hellth.
@rysiek@drwhax "...the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding)"
This. Good at finding vulnerabilities, decent at creating an exploit blueprint, mediocre at verifying actual severity, and very much meh at creating a fix. Not the best combination out there, but alas, the first half necessitates the discussion.
@rysiek Human in the loop is important here, vibe-coded patches are most of the time not very good. However, the models do tend to improve, but what really helps in this case in lets say an C/C++ code base, alright parrot, make me an fuzzer that fuzzes these paths and we'll let it run on every commit so we can catch things.
Last night, well, eherm,... 2am, my parrot was adding a fuzzer to code that hadn't been fuzzed and it showed, bugs came falling out.
The human needs to stay in the loop with all these things of course, i'm not advocating for a pure AI/LLM/Parrot solution. But oh boy, are they good at pattern recognition.
@drwhax yup. All I am saying is that this distinction is important and needs to be constantly stressed, because people will end up creating more problems by vibe-coding.
@drwhax The bottleneck seems to be a lack of maintainers, so I don't really see how running LLMs on codebases will fix that. It'll take a maintainer to make the right decision based on LLM output and we're clearly running out of those :P
@drwhax@png I mean, it would not kill you to also submit a patch fixing the issue you found, would it. Certainly isn't the silver bullet, but I am sure it could help many of those overworked poor souls.
So.. have you submitted patches together with your reports?
@infosecdj@drwhax You're right in that it would help but good samaritans aren't enough to fix the problem. It would be best if there was put some actual funding into open-source security (given how important it is) but that's a political problem well out of our control as individual developers.
@drwhax a lot of us at Red Hat have been working at 100% for the last 6 months on basically this. I personally spend ~2h/day on AISLE and Mythos reports and getting fixes upstream.
@hughsie I'd love to hear more on the maintainer side of this. What i'm ultimately scared of is that everyone will burn out from the immense amount of stuff maintainers have to clean up.
@drwhax I'm paid by Red Hat to do this, and am given access to all the resources I need -- but it must be soul destroying for an open source contributor just doing OSS for fun as a hobby.
I keep telling myself that all this security stuff isn't much fun, but it probably makes open source stronger in the long term.
@hughsie@drwhax I think a lot of projects aren't clear where they stand between professional and hobby development and how much responsibility they should be taking on. Just because you donated some work to the world as FOSS and a company picked it up and distributes it as part of a product, doesn't make you responsible for the "supply chain" your hobby project finds itself in. IMHO any company which ships FOSS (which is _all_ of them) needs to vendor their dependencies, or have a contractual agreement for maintenance, and not just rely on the goodwill of the original author or volunteers for things like security, QA / CI testing, etc. I feel like a number of devs have gotten suckered into a culture of emulating commercial professional development (using an issue tracker, reviewing merge/pull requests, having numbered version releases, using CI and test suites, etc.) for things which are fundamentally non-commercial hobby projects and end up deeply unhappy burning their nights and weekends doing software maintenance because "people depend on them", like a martyr, and it feels rude to tell those people to get stuffed.
I think a fundamental way we should structure human systems is to limit the responsibility (and correlated damage) any one person can have on their own because our poor little meat-brains aren't really built for it.
@raven667@hughsie@drwhax EU is attempting parts of this. We'll see about how the realexisting version of this will turn out.
Other than that: Ensure awareness for said corporates by enforcing the existing rights on the components they use. There'll probably have to be some sort of enforcement threshold or staggered requirements.
And a large part of it is probably to just answer to requests for features/fixes/... with "pull request welcome" or "here's my hourly rate" to any corporate requests. Unfortunately, this is not trained for.
@drwhax It's not that you're wrong but you're not specific enough. Which AI, in what way? Plus, though I see you have enough technical expertise you come out with that OpenAI/Claude vibe that I detest. We should use LLMs, but in our own way, after stripping away the hype. All the LLM technobros have been stealing our code and other people's copyrighted work, and now want to sell it back to us with added bias, lack of provenance and old data. Excuse me if I'm not wildly enthusiastic about their for-profit products.
@drwhax I like to see it less as defeatist and more standing our ground. We don't need to rush. This will stop projects, hopefully before people burn out. That's the environment OpenAI and techbros have created. It's not given just because someone trashes your house you have to live there. You can build something new.
@hajovonta I don't think so, that's how LLM's trick you. LLM's are just really good at pattern recognition and have been trained on an endless stream of CTF writeups on how to exploit vulnerabilities. This is why they shine in finding vulnerabilities generally.
@drwhax I meant, many people argue that LLMs make many mistakes when writing code, but in reality, nobody compared code produced by an LLM to the average code base written by humans.
@drwhax On the flip side, when the subsidy for compute ends, edge computing will become affordable again. And given that things like rudimentary voice recognition now has been reduced to models that fit in an ESP32, I am inclined to think that special-purpose LLMs may be a cat that has left the bag or at least to be in a different league than frontier models.
@whvholst maybe things will turn cheap(er) again? I have some doubts about this and there's all kinds of ways to keep prices for components high by the companies and they've done this before and will do it again and they seem to get away with it mostly.
@drwhax I am old enough to remember RAM prices fluctuating between "having to sell a kidney" and "oooh, I get to max out my motherboard if I collect the deposit on these empty beer bottles" several times.
@drwhax Look, many people have different reasons to be against AI.
Mine is a very specific position against vibe coding, generative AI and ghoulish tech bros and their wannabes wanting to actively destroy excellence, art, nature and human work.
If you have a tool that does not contradict these specifics, then I have no issue. A static analyser ML-like code tool that works on premises, doesn't make a tech bro richer and dry a lake? This is OK for me (and I suspect, the future or much of this)
> I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure
You're missing the point. Even strong "AI critics" have already been seen to use these tools for these scans themselves. Just like other static code analysis tools and so on.
The main point and what people despise the most is when randoms use it and file bug reports they don't understand causing high workload validating their crap.
@drwhax Or, or, hear me out : we push back aggressively on this bullshit, stop financially supporting the culprits, and wholesale reject this nonsense.
The problem is that "we" is bearing a burden here that it can't actually sustain. It sounds great, but "we" does not include the adversaries, so "we" can't actually do this.
The "we" that includes people who think data centers are a bad idea can and should push back on them, just like we can and should push back on jet travel because of its carbon footprint. But if we unilaterally disarm, that makes us less, not more, likely to succeed.
@abhayakara@mastodon.nl @drwhax@infosec.exchange Maybe you should shake off your defeatism and stop being critical of those who are actually fighting back IRL on your lazy behalf. How many oligarchs are you bankrolling while you pontificate?
@wolf480pl sometimes, sometimes they don't, sometimes one finds a better way to chain vulnerabilities to achieve a certain objective. It all depends a bit on the harness as well, lots of small knobs to twist.
@drwhax Folk here hate AI probably because major players are evil, pirating copyright and exploiting privacy. But this just like Google being evil doesn't mean search engine, email, and cloud drive are evil.
There are open source LMs (Yes, open source, not only open weight) respecting copyright. We should use tech to add value, not letting bad actors harm us with it.
@drwhax This (which I do mostly agree with) is why LLMs feel like an attack on OSS and a massive centralisation push for control of computing, to me. They sell the attack and the defence.
@drwhax a lot of people have already said a lot of smart things in your replies, so i'll skip to a new direction:
why did you recently get access to these models? from a "media literacy" point of view, i feel like i don't know how to read your statement without knowing what your context is.
are you working with these models as part of a third-party, independent evaluation / audit? or are you working within a contract with either of these companies, do you receive money from them?
@drwhax@catileptic There should be a publicly funded project for a major foundation model, with both scientists AND humanists going over the training data with a fine comb.
Sadly, even the EU is too neoliberal and corporate to fund this (and always was).
you're legitimizing the marketing stunts of these companies, which both aim, in the current media hype cycle, to make themselves the main coveted technology of armies around the world
all the while erasing responsibility and accountability
i want these companies held legally liable. for their involvement in bombing a school full of young girls, for example
i don't want to read an ex-Amnesty Security Lab employee praise the same company that contributed to war crimes
@drwhax I don't want to debate, just one point: "and be creative enough". Creative is the wrong word. It's not at all creative what they do. It's pattern recognition, combining etc.
@drwhax not simply open source models, but inference chips that specialise on one model and car run them hundreds of times faster and cheaper, like the Taalas project
I honestly can't wait for the meltdown of my own sector. Imagine if making an internet service business suddenly becomes impossible, because if intelligent cyber attacks!
@drwhax we also recently got the access and its insane. We are debating when the open source models getting close / similar to it to setup a cluster and provide resources to smaller open source projects and ngo security related folks to make sure these ones are not left behind. But this requires a lot of effort and funds and also training for people involved so for now its just on our list of possible projects.
@drwhax I think LLMs get the low hanging fruit of vulnerabilities, so you shouldn't trust an LLM's security audit as a definitive audit. As models get better, they'll catch higher up fruit, but still.
@edward I do think that's the case at the moment, cross-file vulnerabilities are sometimes not found. What I feel they're mostly good at is, pattern recognition, e.g, there's a specific bug class fixed from past git history that it found repeated or as a variant. After all, LLM's is just a stochastic parrot and it shows
On “Mastodon hates LLMs”: I think almost everyone has ambiguous feelings about generative AI and rightly so. Bizarrely, this includes the CEOs selling it to us.
The ones here who excommunicate you for sharing results that help the community ride out the storm… I hope are just a vocal minority
@drwhax What would the estimated cost be for these 7-9 hours, assuming no more subsidies? Just looking for a rough ball park figure, as almost everything I've heard up till this point is that while the LLMs are effective, the cost is significant enough that you could be paying someone to do the work directly.
the reality is that LLMs are incredibly powerful tools, but mastodon is so into reality denial and while denying reality, the same people make fun of Trump supporters for denying reality (eg vaccines don't cause autism)
@failedLyndonLaRouchite They are, but they have flaws and the way these models are created is quite shit and there's a lot of things to say about that. It's not that I say, anthropic good, or openai good. It's more to say, we're entering an era with a lot of shit going to be thrown at people, companies and maintainers and I don't think people realize what's about to hit them!
@drwhax And, to be honest, if any of the OSS/FOSS software I use dares to make ANYTHING with LLMs (not AI, but GenAI) especially relying on companies like Anthropic or OpenAI, that software goes to the trash bin.
I know is not rational, but one cannot hate these companies, and what the hell, the underlying technology has very few non-evil use cases.
@chroma@drwhax Sadly, I'm afraid that sentiment means you should switch off your computer and walk away, never to look back. No matter which OS you use, there is 100% guaranteed already LLM generated code all over the place.
@drwhax What will any of this matter when we have turned the planet into a desert and caused our own extinction level event? There is an assumption that we will have access to electricity for any of this conversation to matter; when every day people have to deal with over 100° hot weather, rolling black outs, and drinking water restricted for anyone not rich/wealthy?
But let's keep protecting the people that made these problems and continue to make them worse. Justifying another technology built on theft and slave labor, I don't understand how anyone can be okay with that. We cannot separate the technology from its history, negative environmental impact, and that this technology is another scam to hoard even more wealth. The companies are using Enron's tactics for money and the packaging of debt into bad deals like 2008 in the USA is leading us to a very difficult time, where once again, these conversations will not matter as people will br trying to just survive.
@atraidez I think you're missing the point of my post, your anger is right and it's only downhill from here on out. It's also not going away, while people will also getting hurt with it if we don't find ways to patch whole classes of vulnerabilities. It's a perfect storm
@drwhax@atraidez It is 100% going away. It was never financially sustainable in the first place. Also, the planet will be reduced to cinders in the next 10-20 years; good luck hosting a web service when we're all foraging for nuts and berries.
@Netraven@drwhax You will, just through thin clients to access the actively monitored cloud VMs and charged by the month. That does of course presume a bunch of things that'll allow for that so it's very optimistic to me.
@drwhax You could patch faster but as the recent HF / OpenAI cyber attack stunt shows the big models and their cyber crap filters refuse to even talk to petty maintainers about security stuff, but will happily slopify code with myopic seds and greps. Case in point, even the Anthropic CVP does not apply to Fable, which writes code all the time that it can't review thanks to the zealous security filter, and is therefore useless where it would be needed most.
@SDRHoernchen Yes, I think this is a huge problem and it shows a huge gap. Sadly, I don't think this will be fixed as the line between offence and defence is too thin these days.
@drwhax I've been very concerned about the emergence of Mythos-class AI that can weaponize vulns. I'm personally responding by adding formal memory safety proof coverage to critical libraries in C (notably: OpenSSL). Testing - which can identify many but not all bugs - I no longer consider sufficient. Instead we need formal verification to prove the *absence* of bugs/vulns. #aicoding#security
@davidfstr I liked Halvar Flake's analogy, which i was sometimes a long the lines of, a bug every 10 years instead of 1 year. I think we can do a lot to prevent it, but there's always going to be bugs. I also think Mythos is way too hyped and these capabilities already exist in modern frontier models.
@koehntopp@drwhax The idea is to use the AI to defend against exploits found by AI. No matter what Anthropic will be paid, you can't break the tools of the oppressor with his tools.
@koehntopp this parrot is really good at pattern recognition, it can code and fix some I think given enough constraints and a feedback loop of adversarial review, but it still might not be up to the style of the codebase or it might even introduce new vulnerabilities. I still think we also haven't it fully figured out yet? I think this needs more benchmarks that are reproducible in some fashion
@koehntopp this parrot is really good at pattern recognition, it can code and fix some I think given enough constraints and a feedback loop of adversarial review, but it still might not be up to the style of the codebase or it might even introduce new vulnerabilities. I still think we also haven't it fully figured out yet? I think this needs more benchmarks that are reproducible in some fashion
@drwhax I'm glad you've invited discussions on this, because my immediate thoughts and questions diverge from the discussion I'm primarily seeing here. When you mention vulnerabilities, yes you mention they exist in code as well as in projects. When I think of vulnerabilities, I think of the method I am using to make and maintain a record of something or else to achieve a goal or state of physical safety. For example, if my household accounting spreadsheets and forecasts are handwritten then the vulnerability is risk of loss if the dog eats them, but risk is mitigated when I keep an offline digital file. When a family member starts a small business, I ask the questions learnt in my accounting and auditing classes because I believe the risks involved include thinking about how to prevent losses of whatever is important. I only had a couple of BASIC programming classes in 1984, but it seemed to me that in those days the instructor felt we should understand the process so that we could consider risks. I now wonder if the machine-assisted searching allows you to consider the assorted practical situations of each business whose vulnerabilities are being sought.
@MossyQuartz Apologies that this is just talking about computer vulnerabilities, but yes, you're right about risk modeling in the away from keyboard world!
@drwhax Computer vulnerabilities, alright that's fair. When I took the Computerized Accounting class in 1985, students were told the business should continue the manual accounting methods concurrent with the new processes for two years, or more, because certain daily functions could look alright while other annual functions might not be obvious until after the annual reports are printed and reviewed. That was how us old people were taught about computers, they said the data needed to be confirmed as accurate before we could trust it.
@drwhax so disclaimer, I’m not a coder or understand much about coding, but I have to mention this because it did make me laugh (hope someone finds this funny too) When you mentioned your “parrot” at work breaking things apart & finding “vulnerabilities” (yes I know you were referring to the program 😹) my first thought was : “if you could make a program like that I think the best name for it would be a Kia parrot” (use scientific name probably 😹) (From their propensity to break things apart 😹)
@drwhax telling maintainers to run LLM based bug searches and start writing fixes, while not providing them with money, free time, or other required means and support, can be seen as a form of agression ...
@drwhax Nah, i don't care anymore. I'd rather let them break everything than get blackmailed. Fuck it, honestly, the internet is a pile of shit these days anyway, let it burn. I'mm done feeling bad about this while everyone keeps holding down the pedal as we rush towards the wall.
What does it matter, seriously? Soon it will be too expensive for any normal person or project to run anyway and then only bad actors will use it. So what's this? A bandaid for the break dam? Pointless.
@drwhax This may be one of the only niche use cases where LLM technology may actually be useful, and it may be what Linus Torvalds was talking about when it comes to AI and the Linux kernel. Directly deploying AI code in the kernel could violate the GPL.
But as for auditing existing code for potential vulnerabilities, and the pen-testing stuff, this may be a viable use case.
But that's not what the AI companies promote. They promote the AI-powered shovelware development use case instead.
@alice_pea_3526 the parrots are pretty good at pattern recognition which is where I think they shine. Some smaller projects as well, but architecture wise, you'll have to handhold them a lot. Its almost like having a junior that's good in some incredibly niche things, but you'll be iterating a lot over code.
I think curl maintainer said something similar, the reports used to be bad from LLM's and they suddenly got a lot better. It still requires a pair of human eyes to understand if its not hallucinating.
I wish it was better at eliminating whole bug classes tho, that'd be the ideal situation?
@drwhax One thing I don't see anyone addressing is how we got into this situation in the first place, or how we get out of it, and I'm not talking about the LLM or any of the already well known and discussed downsides of that.
How did the software get so insecure in the first place? How do we prevent security vulnerabilities before they occur? Most of the risks aren't unknown or even novel, so why haven't we prioritized the solutions?
@drwhax I think the biggest lesson here is that Blue Team can't rely on patching to keep everything safe.
We have to actually do the rest of the stuff that security calls for, and not just keep everything patched but do more to minimize the risks when one of these unsleeping tools gets turned on us.
And so many companies don't even have a proper inventory of what-all needs to be kept up to date.
