luciferofastora

It’s exploiting a vulnerability (unsecured API) without permission of the gym running the software, to the detriment of whoever arrives and finds their reservation cancelled and probably also of the gym who now (unjustly) has to deal with the (justly) upset customer.

The gym’s software should be secured better, but that doesn’t make it less of a hack.

I mean, running around and foraging is their job. They don’t have to forage anymore, but they have to keep themselves occupied, so they pick up the stuff they foraged from their food bowl, carry it all the way home in the wheel, then bury them fuck-knows-where until you clean the cage and replace the bedding and find the all the caches they hid and forgot.

The amount of text they learn from is so big and their structure is complex enough that they can actually learn underlying patterns about WHY words are usually arranged in a certain pattern.

Quantity of training material doesn’t confer new abilities. It makes the resulting weights more representative of the language of the materials, but it doesn’t give something the text doesn’t have.

This fallacy is why I say philosophy should be more widely taught: the relationship between symbols and semantics isn’t quite so trivial. In the specific context of computational conscience, the Chinese Room is a well-discussed argument that demonstrates how command of language doesn’t necessarily require or produce understanding of the same. We can argue about the implications for human consciousness (I’d rather not), but the critical part is that processing a foreign language doesn’t translate it into mine.

For a more practical example, consider the issue of legal arguments citing made-up or irrelevant precedence cases. It is trivial to check whether a given case reference actually correlates with an actual case. It is critical that your citation both refers to an actual case and correctly reflects the contents. Someone who understands the nature of legal arguments knows why a certain arrangement needs to be an extant item in a finite set of instances of that arrangement (namely, a topically relevant subset of all legal cases in history, which is also a finite set).

Yet LLMs get it wrong. They get the shape right, but the filling is a game of Russian Roulette. When drawing a semantic connection between the current context and a related case, it should be a no-brainer to correctly write down the reference to that related case, but they don’t do that. They draw on the trained set of symbol correlations to produce something likely.

The same goes for essay prompts in exams where a human should recognise that an instruction to include references to Madagascar, in white font on a white background, isn’t actually part of the question but rather a trap to catch blind copy+paste into AI. The AI doesn’t understand that context or that it should disregard that part. It doesn’t actually know why the instruction is there, it just processes it into part of the context.

Funnily, the mistakes seemed similar to mistakes humans usually make, like forgetting carryovers in additions or subtractions.

That is a damning verdict for a machine literally invented for computing. If there is one thing a computer should be good at, it should be the thing it was built for. Carryovers (overflow flags) are part of the most fundamental ALU design. The fact that it reproduces human error shows that it doesn’t actually understand the assignment, it just imitates the training material. If it understood that the reason a certain pattern is there is because the human writing it made a mistake, it should be able to correct it instead.

Otherwise, it is a parrot, or perhaps a really studious child that’s great at imitating adults, without any care for the actual semantics. A computer making as many or even more mistakes than humans is useless (for that task; we agree that they can do some tasks just fine). If AI should be useful universally, it needs to understand these semantics.

Language is a tool for communicating thoughts and perceptions, but that doesn’t work the other direction. Words do not imply reasoning.

And most men in my life regularly spit out false knowledge with a confidence only a mediocre white man and an LLM can have. ;P

Yeah, I wonder what material LLMs developed by companies with white, male CEOs are dominantly trained on…

So maybe trying to somehow decelerate technological progress really can be a reasonable way to take pressure out of that system.

I’d say it’s less about deceleration itself, more about diversifying the efforts and exploring alternate avenues to achieve the things they’re lacking rather than pouring those resources exclusively into LLMs. The deceleration of LLM development doesn’t have to mean a total deceleration of progress.

I think discussions like this are important.

I find that writing out and defending my position helps me refine it. Sometimes, my perspective on things shifts just by trying to explain it. Discussions are the whetstone by which arguments are sharpened.

I’m not a native English speaker so some of this is a bit hard to put into words for me.

That makes two of us :D
Sometimes, these discussions help expand my Engish skills too.

So I do think it is a valid point along the other ones - AI does create value.

Yesn’t. It’s a text generator that predicts a likely series of words, based on the language patterns it learned from its training material. It doesn’t so much create value as aggregate the value of other people’s work into a weighted reproduction.

However, those weights are biased by quantity, not quality. It has no way to assess which responses are good, only which ones are likely. The average value it (re-)produces is an average of the value of the training material. Models trained on a wide variety of content (such as ChatGPT) will inevitably include a lot of material of little value to specialised topics.

Hence my argument: It can get lucky and predict a high-value response, but the problem is that it isn’t guaranteed to do so. A layperson doesn’t have the expertise to tell the difference. That not only dilutes the value, it invites false confidence in the results. In cases where accuracy is critical, this may actually produce negative value if people consult an unreliable model rather than a specialist.

And that’s the critical difference to human advice: human experience is shaped by a number of factors beyond just language. We create semantic connections to abstract concepts and attach specific meaning to certain words and patterns.

LLMs don’t have that abstraction. They can predict sequences of text that sound plausible and might coincide with something describing reality, but they can’t tell whether it’s accurate.

Advice has to be grounded in reality to be useful, and that’s what LLMs are missing.

They’re perfectly suitable for tasks where correlation is enough, but if the task requires actual understanding, LLMs aren’t equipped for it.

Otherwise there would be no hype.

Hype doesn’t always need a solid reason. In this case, a lot of hype stems from the hope that we may one day have the type of Artificial General Intelligence that SciFi has long dreamed of, the illusion that they may be able to do our work for us and the promise to company managers that they may be able to save money by replacing human employees with AI.

The various executives of the big AI vendors are obviously capitalising on that, stoking the hype with grand and utopic visions because they want to sell their product.

It’s not that it’s useless. It’s that the utility is far less than the lofty promises made by people milking the hype for all it’s worth. And that little utility comes at a terrible social, economic and ecologic price.

We might have different perspectives on this btw because I live in the EU. People don’t lose their home or water supply here because of AI companies.

I’m in the EU too, I just read a lot of US news. I don’t think we should dismiss the consequences our use of US infrastructure has. Worse yet, I don’t think we should dismiss the political dependencies that creates.

We need to be specific about what the problem is: the problem is not AI, the problem is that some people are getting super rich on the expense of almost everyone else.

I think that’s only a part of the problem. The second part is the lack of understanding you mentioned, and the resulting mis- and overuse. I’ve seen people trying to argue with experts because “ChatGPT said” because they genuinely do not understand that ChatGPT is a parrot, not an expert.

And the third part, again, is the disastrous effect on our world.

You can’t fight technological and scientific progress […]
I’d say we might actually in an “industrial revolution” kind of situation.

I’m not fighting it. I’m trying to pull it out of the pit that the current obsession with imitation has dug. When college students, the next generation or scientists, trades their scientific understanding for the convenience of high-tech parrots, that is the opposite of progress. It’s stagnation, fostered by those few people that are happily trading our future for their present profits.

That’s the mirage of this “industrial revolution” analogy: They’ve built something in the shape of a steam engine, promised the functionality of Spinning Jenny, sold fabric factory owners on the idea who fired their workers to buy these machines. The people they fired were promised that they’d have to work less, but weren’t told that they would be paid less too.

Now these machines turn out to not actually provide the smooth, spinning motion required for spinning thread. Factories are facing expensive production outages, compounding the expenses of getting these machines installed and can’t afford to hire all the workers back.

The machine shops built to produce these Sloppy Jennies or their parts will eventually find it harder to sell their iventory. Once they go under, their workers will also become economic casualties.

We’re at the point where we need to reinforce worker protections. We need to push for a system where our livelihood isn’t contingent on the amount of work we do. Only then can that utopia even manifest. And to actually make progress:

We’ll need all the social sciences now that we’ve cut funding for over the last decades

Yes.

We’ll need to understand the social dynamics of such technology to better prevent the disastrous side-effects. We’ll need to compare historical developments with present circumstances and plans to account for future developments. We’ll need to study the psychological effects of interacting with human-like machines to be able to correct course where needed.

Whatever control mechanism is supposed to prevent machines from producing harmful output will need heuristics based on those disciplines to assess the dangers of that output.

I’d include philosophy too. We don’t need neural networks that are a lesser version of human ones, nor just scaled-up variants that also replicate all the human inefficiencies, errors and biases. Those are the pure MINT approach to the problem, and it’s clearly coming up short.

We need to find a rigorous mechanism for representing semantic knowledge in digital systems that don’t just imitate, but surpass human cognition. We need a workable philosophical grounding for logical and mathematical approaches to modelling knowledge of concepts rather than just language.

We need to get over the error that LLMs are “thinking” or “just like humans”. They’re not, but as long as we’re stuck on the idea, we can’t fix it.

Also, we really, really should spend less time thinking about whether we could and more about whether we should.

I have no idea why my comment gets downvoted so much.

Because your comment seemed to add an anecdotal “it also helped two people” to a long list of evils as if it were equivalent. “Sure, people are suffering because of this, but two friends of mine are better off for it” – is that really worth it?

I use them a lot and I see people around me making their lives better using them.

Aside from the nebulous question of just how it’s supposedly making lives better, and whether that actually is an improvement, the question repeats: Is a fee people’s lives getting better really equivalent to the people losing their homes, water supply or job over it? It’s not that we’re unaware that it might have benefits, it’s that those benefits are massively dwarfed by the evils.

Are they overhyped? Probably.

LLMs in particular, and dangerously so. People treat them as virtual humans, use terms such as singularity and trust them with cognitive work a linguistic model plainly isn’t capable of actually doing. They deliver a convincing imitation, but there have been too many instances of that imitation not holding up to any reasonable standard. For a more benign example, Ford had to hire back its fired engineers because it turned out that GenAI can’t actually do their job. Less benign are things like giving harmful “health advice” or reinforcing their users’ delusions and psychoses to the point where they commit violence against themselves or others.

They’re still being hailed as the future, despite evidence backing up what theorists have been warning for a while: A text generator cannot generate meaning and should not be used for any purpose where the content actually matters. That your friends’ word salad happened to resemble useful advice is nice, but it wasn’t actually advice, just the product of a high-tech advice imitator that got lucky.

Is AI a technology with massive potential to change the world for the better? Also yes.

This is where we need to specify just what we mean by AI.

Language models are useful for linguistic correlation tasks, but the actual utility of that is hard to gauge, because it heavily depends on the mode of the output and on the recipient. Specialised models may make decent assistants for certain types of specialists.

They certainly aren’t capable of the type of “nobody has to work any more” utopia that conmen like Musk and Altmann are selling to the gullible. They’re fundamentally unreliable and their consumption of resources dwarfs their sloppy utility.

So while some form of AI might hold a key to a radically better world, LLMs and generative AI aren’t it.

Does AI have the potential to completely wreck our society and economy if we do the whole thing wrong? Probably also yes.

Probably? Have you taken a look around or read the whole list you replied to? The hype already is destroying lives, and there’s a slew of experts calling out the fraudulent scheme they’re pulling on the stock market.

The one point I’ll concede to the scammers is that it has fundamentally altered our world.

Just not for the better.

I hope we can find a way to turn this into something positive.

It starts with being honest and clear about what “this” actually is, understanding what it isn’t and not making excuses for a deeply destructive technological complex just because the mathematical parrot emits some useful responses now and then.

That’s awesome! Imagine if we took all the resources spent on pushing AI for stuff it isn’t good at and invested them in training models for detecting, classifying and correlating medical indicators and using that to anonymously connect people with a network of physicians and psychiatrists for further evaluation and preliminary diagnostics to help with getting actual treatment.

We could bring a more reliable version of that help your friends happened to get to more people, with less of a random chance that the model will reinforce the issues or misdiagnose people with dangerous consequences. We could improve the world.

Instead, they burn books, fire people and abuse eminent domain to displace people so they can run power lines to feed data centers fucking up the water supply for residents that got no say in whether they want oversized heaters in their backyard.

I’m glad your two friends got some use out of this. Is that really worth the price?

Cory has been that “crazy” guy warning of all sorts of grift, exploitation, privacy fuckery and security forever like a modern-day Kassandra. Now that we start to see the direction the technology sector is headed more clearly, his warnings finally start to ring true.

If we can pull it off before they manage to manipulate us into fucking up our world, sure. If we let their slopgenerators produce the drafts for the station, I’m sure they’ll love the breathtaking result.

Me, please. I’m wilting under the sun, and I’m pretty sure wilting under your insults can’t be worse.

and made way for one that will.

I think that’s the problem. Have you seen how hard it can be these days to rally volunteers to a common cause? Even if people were to back up and preserve all the content of Wikipedia, WM Commons, Wiktionary, Wikiquote, Wikidata etc., getting enough of the original contributors on board to found some new organisation without a thousand splinter factions pulling away over some issue or other is a long shot.

And in the meantime, for-profit corporations now bereft of their largest, free competitor would rush to fill the void and try to become the “new Wikipedia” (now with ads, spyware, premium features and blatant corruption, selling control of truth to the highest bidder).

WMF is a behemoth, kept moving through its own momentum. Letting it break in hopes of building something new is a dangerous gamble, unless we can get the successor up and ready to step in beforehand

…but no less popular

it’s still harder to live in space or on Mars than to try to fix Earth

Hush, unbeliever. We just gotta invest more in the technologies that let us escape from the consequences of our greed. Well, some of us. The most greedy, actually. The rest still get to bear the consequences, but they don’t matter.

Point is: pour more money into training machines to imitate all the things we don’t actually understand the complexity of, and surely we’ll no longer need those people when we finally escape to a better future where our serfs don’t need such petty things as rights and payment. And if you help us get that, you just might get a ride on our escape too!

(You won’t, but I’m hoping you’re gullible enough to fall for it.)

It is run by a complete idiot.

And enabled by other idiots. Musk doesn’t exist in a vacuum.

Pretty sure turning off supply is easier than getting an arrest warrant against a person whom the corporate system intentionally makes it hard for them to be held criminally accountable.

I applied to them once, six-seven years ago I think. They had an open position that fit my profile.

I have never been so retrospectively delighted about a rejection. Not only do I enjoy my current job a lot more than I expected, I’m also glad I dodged that particular bullet.

It’s a French instance, so that would make sense

Yeah, my phone clock doesn’t display exact seconds by default and I didn’t feel like gambling with the posting delay anyway, and I also didn’t feel like specifying a timezone offset because I hate timezones (guess what field I’m working in), so I just converted it to UTC and omitted the seconds.

This comment was posted 2026-07-29 06:14

But then how are you gonna extract profit from rubes just hoping for kindness instead of the eternal grind of squeezing pennies from everything?

Aww, I was hoping for… gestures at your username

I left a long time ago. Don’t think they were as trigger happy back then, so they never got the chance. You think I should open an account and start posting reasonable criticism of disinformation so they can ban me and I get to be one of the cool outcasts rather than a boring old emigrant?