disconnect […] from the internet

And these guys really think that this is doable as easy as snapping your fingers?

hmm…

And these guys really are national advisors on things related to computers and internet??

hmmmmm…

So what do you think they should do? Just ignore it?

The specific advice they give is to disconnect all PLCs and only access them through a VPN, enable password protection and change the default passwords, and only allow IPs from known engineering laptops and other critical assets.

This is absolutely doable and any IT professional could manage this change. The only barrier that prevented this from happening before is that you need buy-in from someone with access to the purse strings

Yup, Sleepy-Don really do have the whole Iran situation under control.

it actually depends on it to keep distracting the msms, from talking epstein.

I am repeating what I said elsewhere on a different comm:

I call FULL bullshit on this being Iran.

I personally worked on the cybersecurity incident that happened to St Paul municipal and water systems last year as the Incident Response Technical lead.

The threat actors were not Iranian then. They were Russian.

I personally rebuilt most of their networks, VMware hosts, storage, and servers. The majority of the VMware environments did not have config backups and were running 5 and 6 branches of VMware, only two had 7+.

Their network was insecure. They had IPSEC VPNs between the various location, but there was no further security in that respect. Their firewalls were not being kept up. AAAAND… I warned them that we thought they were still compromised.

In fact, the forensics team called them out as being clean on a group call one week into the incident, but while we were on the call with leadership, they got RE-infected.

I had my team compile a massive set of recommendations for them and I wrote the report. Water Systems specifically. They did not listen to us, apparently. The forensics team also did this.

They had FBI, national guard (fucking incompetent), and law enforcement (useless) in all meetings.

If they got into Water, they’ll get back into city and municipal again. I guarantee it.

This is not Iran. It’s the fucking incompetence of city government and Russian threat actors.

Minor nit: Russia and Iran are already working together on shaheds and targeting

No surprise at all if they also coordinate cyber

Whatever. I worked the fucking case. The infiltrators were Russian. After that, it’s a political or financial negotiation.

Yeah and Russia shares intel with Iran.

No doubt so trump can make up some bullshit excuse to try to suspend the midterms.

We already know he’s going to try, I hate how obvious this shit is from a thousand miles away, but still nothing is done about it.

If budget weren’t an issue, how many clones of your teams would it take to do that for all the water systems in the US in five years?

What are the odds that the average muncipality understands the security implications of just existing on the internet these days?

10 people per site, per incident, at 50-70 hours/week.

It’s thankless work and the people in control don’t typically take on contractors. They underpay the actual IR staff. That’s why I left the industry. It’s a big buddy-system network. Really fucked.

So to sweep all of the water systems in the country in advance isn’t doable in any time frame?

IR isn’t an acronym I’m familiar with.

Incident response, basically getting pulled in to clean up someone else’s fuck up that got them owned.

The hacks target internet-facing programmable logic controllers (PLCs)

Why the fuck
is your PLC
facing the fucking internet!

Jesus Zombie Christ an a pogo stick. Has no one been paying attention for the last two decades? Seriously, we learned this sort of lesson in Two Thousand and fucking Three. Your critical assets do not get public IP addresses.

It usually happens in 2 steps.

First step is that everything gets connected to the LAN and you can only access the PLC network from within the building’s network. Then some time later, management finds out that keeping someone on-call to go out costs a fortune, so they request that access be made so they can make the change from anywhere.

The IT team argues security, but no new hardware can be provisioned and a developing a new process is too hard. Then the magical phrase is uttered: “Just make it work”. So IT punches a hole in the firewall, adds a NAT rule, and job done.

Sadly yes, this is exactly what happens. And it ends up being IT holding the bag at the end, with the managers having long since cashed out their stock options and left.

“Well, we just bought this new monitoring platform from my cousin’s company and they need access to it. No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. Ok, then just open all ports to the network so they can connect. No, they don’t know what port or protocol, it also runs out of my cousin’s house so it’s on a dynamic IP so you can’t create some type of access rule for it. Just put an any/any rule in and it’s good. What do you mean we already have a monitoring platform? Well, this one is better!”

“No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. […]”

They won’t know that much. They don’t understand that their job isn’t running a water department, it’s running the computers that maintain the water department and that includes security. For someone familiar with computer security it’s easy to recognize negligence, but until the law sees it as criminal negligence nothing’s going to change.

Small town municipal water departments aren’t going to be able to afford the guy holding the shield against international state actors.

Literally this. It starts with “everything needs to be on an air gapped LAN so we can control it.” And then suddenly that air gapped LAN isn’t air gapped, because someone got tired of paying the call-out fee every time the on-call person had to drive to the site to push a few buttons on a keyboard. But figuring out a proper VPN (or at least a reverse proxy with access control) was too difficult or expensive, (management thinks the point is to save money on call out fees, not spend money on proper hardware) so they just poked a bunch of port-forwarded holes in the firewall and called it a day.

What’s extra ridiculous is VPNs are not that hard. I am fantastically awful with networking and even I can wrap my head around a WireGuard tunnel.

The issue is doing one at scale, with proper access controls. Also, let’s not skip past the whole “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene, etc… All without any additional IT support or hardware, because management is doing this as a cost cutting measure.

I can pretty much guarantee that the systems that get hacked aren’t the ones that do everything right. They’re the ones who have a single shared VPN password, haven’t updated their firewall since it was installed 8 years ago, and they haven’t even changed the default passwords for their control systems. Because they got tired of answering phone calls about what the password was, so they just made every username and password something stupid like admin/admin.

Even when a company is doing things pretty well with a VPN, all it takes is one mistake and the cat is out of the bag. At one of my previous employers, we had a fairly good setup with VPNs using single sign on via our IdM provider. IdM enforced SSO and the VPN had a number of policies setup which kept good control over what users could login. And then someone had the brilliant idea to take an old VPN appliance, reset it to factory defaults and configure it just enough to work on our network and then hung it out on the internet for some sort of test (our configuration management left a lot to be desired). Of course, that led to it being left there for a couple months and an attacker finally found it. They also had a valid username/password combination which got them in the door.

Security is hard and often expensive. But still, compared to “we put out OT devices on the internet” a basic VPN is a huge step up.

Fine. New rule, anything that’s your fault and costs a company money you get to pay for out of your golden parachute, your pension, your savings, anything. You caused a problem to save money this quarter forsaking the next quarter when you aren’t there? Screw off.

“network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene,

This is all shit they should be doing now. They have to have a firewall to get to the net, so they should be maintaining and patching it. Managing users is standard AD shit, and password hygiene is t as easy, but setting reqs in AD for complexity goes a long way (as does not forcing passwords to rotate on an arbitrary basis).

And yes, I know it doesn’t happen in a lot of places, which is why I said ‘should’. Having been on the cleanup side of multiple security incidents, doing the basics is harder than you’d expect for a lot of places …

Except you skipped the part where IT is ridiculously underfunded and treated like crap ("We’re a utility, not a tech company!")

With constant turnover, the person receiving the “Just make it work” call is about 6 layers removed from the person who originally set up the infrastructure, and probably completely unaware of all the previous security holes put in “because it’s air-gapped, it doesn’t matter.”

I did some work for a company that owned a /16 public IP range and used it on their internal network. I had to put a safety rated control system on it that ran mining machines that weighed 60 tons and cost a couple of million bucks each.

They gave me a subnet range for the system that was clearly non-private and I was like, “ummmmm, is that on the Internet?”

Apparently it wasn’t routed to the rest of the Internet but it always felt like it was one misconfigured router away from disaster.

2003??? We were learning this shit from Ferris Bueller back in the early 80s!!!

Not only that but it said many of them were using the default login credentials

You’d be surprised. Especially on city utilities.

A lot of this is often down to shitty IT practices by the vendor who has little IT knowledge let alone decent security insight.

I helped support a new PCL installation (technician control PCs and software upgrade) in this exact scenario many years ago. The vendor came in expecting to treat our system as their own. Direct remote access to several over-specced, physical servers from the outside world, available on our core network, shared account names (and passwords!) across all devices, every account a local admin and their own domain admin account so they can make changes as required… Needless to say our security guy almost broke a rib from laughing so hard!

What they got was very different. I ended up doing fairly basic troubleshooting to get the shitty software working without it needing admin rights to run - XP era software running on Win7 clients so it was meeting basic folder and reg key write permissions - everything else was locked down as tight as we could. They didn’t even get admin on the servers.

A few years later I heard they’d reported that our site was their most reliable. Probably due to the fact that they (or the techs) couldn’t fuck with anything without our permission. Apparently their original configuration is normal in the industry… Shudder

why is any critical control equipment internet accessible.

I understand some machines might need internet to send reports and make document filings and shit. But it should not be the critical control equipment that makes the system function. And there should be a massive physical gap between the internet machines and the critical machines.

Jesus christ, even by the abysmally low bar I have for humanity, it still finds ways to dig itself a path under it.

The entire tech sector screamed about the vulnerability of IoT devices, plenty of which are now participants in zombie botnets.

Does anyone remember in 2015 when YESCO billboards by the thousands were hacked to show Goatse instead of an AT&T ad? Pepperidge Farm remembers. IoTs are still vulnerable, including big municipal ones and, yes, Flock Safety cameras.

And CISA was severely broken had its staff severely reduced by DOGE.

In unrelated news the CDC, also a victim of DOGE, is entirely unprepared for the next epidemic.

So are we just kind of admitting that there exists no way to expose any networked device to the internet securely? Because if it’s not possible for PLCs I don’t see why it would be possible for any device. If water utilities have to take these offline, then how does that advice not apply for every internet-capable device in every commercial and industrial facility worldwide?

I don’t work directly with PLCs but we do have them at my work. The main thing that makes these different from any other devices is that they typically control physical machines. Which means there is a real danger that them becoming compromised could lead to damaged equipment or even death.

Additionally, many PLCs use older OS versions like xp because their software is notoriously out of date (if it’s working why rewrite it when that could introduce safety concerns)

This “urgent” message from CISA does not prove that any of these devices are internet accessible just that if they are then they should be removed. If CISA wanted to require this then they would have released a binding operational directive (BOD) or an emergency directive (ED). Both of which are publicly viewable on their site.

There are ways, but they are costly and complicated and do not stop all attacks. So the golden rule is that if critical infrastructure doesn’t need to connect to the internet, it shouldn’t.

So are we just kind of admitting that there exists no way to expose any networked device to the internet securely?

It’s done all the time. It’s not a product you can buy, though, it’s a process. There’s probably a good “go by” for how to start the process for citywide infrastructure, but it’s just one more document for people to ignore.

Why in the everloving fuck would scada ever be connected directly to even the regular lan, let alone the Internet!?!

Laziness. Poor training. Stupidity.

Why wouldn’t you want a key utility controlled from around the global?

This has been talked about for years and about how exposed SCADA is. I just assumed that it was fixed/isolated years ago. The fix appears to have been to stop talking about it.

That’s why NIS2 in Europe demands that OT is on a separate network from IT. My old workplace (well, the head of FO Technology and his gang anyway) was up in arms about it because it “was a waste of time” (and completely ruin the ideal architecture he had in his mind).

According to our peerless leader, I blame it on Minnesota because they are grossly incompetent

But then he tells California (and Canada) we are to blame for wildfires for our failure to rake our forests.

not so funny when it isn’t happening in the middle east but right at your doorstep

Went on the public tour of our local waste-treatment facility. Really fascinating stuff. Highly recommend it if you’re even slightly curious.

There are definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.

But there were also office equipment and personal devices everywhere. All it takes is for someone wanting to monitor the state of a digester on their laptop from home.

You just crossed the beams.

Automation engineer here who does a lot of SCADA

There are soooo many good ways to securely monitor stuff from home. But especially in critical infrastructure it’s just usually air gapped which is unbeatable in cyber security.

Tours of any local facility are super cool! I find it all so interesting, seeing how the things we use work!

I worked on new SCADA for a water company across all theirs sites. Previously they were a mismatch of different systems and all airgapped.

They standardised everything and put them all on VPN. Suddenly you could access the secure water treatment works with 24/7 security guards and the control room was a bunker, from the unmanned rural sewage works with a portacabin for the server, the key hidden under a brick, and wifi for the PLCs.

The sewage works had always operated like that because it was deemed low risk. No one considered the VPN changed the threat model.

definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.

You’re talking a massive nation with more systems like this than you could ever imagine. Probably more than all of Europe combined.

They are in every state of repair and disrepair that you could imagine, and many have been receiving government funding for at least at decade now (def more, but I can only speak to the last ten years or so) to replace these systems, including SCADA and PLCs.

So you have a patchwork of countless systems, in every level of tech modernity that you can imagine.

The newer systems usually have an internal network of sorts, and they might have a portal for someone outside the network to log in to remotely monitor.

The problem seems to be that these idiots kept the default credentials, etc.

But to answer you’re question, they’re not all 20-30 years old, and network connectivity (without being open to the outside internet, obviously), is a very useful feature when replacing those old systems with newer ones.

That’s why.

Not saying I ever accessed it, but there’s a generator in Germany open to the web without security. It makes a few MW. My guess is hospital or any other critical infrastructure.

I will take things that never should be connected to Internet for $200 Alex.

Exactly! Whenever I see one of these stories of a utility being hacked that’s my first thought.

Remote monitoring? Sure. Remote managing? Fuck that. These facilities shouldn’t be unstaffed for days at a time.

Oh for sure, air gapped sensors that relay information are going to be necessary. We should be building these systems with absolute security because they are critical to infrastructure.

Some people will say it’s too hard to retrofit onto legacy equipment, but all it really means is adding a secondary system on top of the legacy system. Just a sensor array that has no interaction with existing systems.

Yup. Fancy architecture diagram:

                  One Way Radio
                   Transmitter
[Nuclear Reactor] ------------> [Internet connected receiver] -> tHe ClOUd

It’s called paying people to watch the systems during off-hours. It’s often way cheaper than setting up firewalls, setting up alerts, maintaining patches, paying a VPN company to manage connections, paying another company to handle authentication, and so on. Security constraints are getting so bad at my office I’m seriously thinking of how we can do more things with pen and paper.

Security constraints are so bad at home I want to just buy paper books or go to library.

The things you mentioned probably aren’t the problem. It’s not hard or expensive to setup secure remote access for systems.

The real problem is that a lot of this very expensive and very specific infrastructure equipment is also very old, and frequently does not support newer and more secure protocols.

Source : I’ve been trying to get gas generators to fire off email alerts using modern authentication with conditional access for about a week now.

Changing tech is usually a lot harder than adoption. Better methods and protocols will keep emerging to fight new digital threats but with physical threats usually don’t change. Only exception is social engineering (e.g. attacking a locked door by tricking someone into opening it). Even then a physical presence is required and someone can’t enter a locked room from 2000 miles away.

A simple 300 bps connection over POTS to relay the information from the process controller is all you need. But even POTS is disappearing.

Right. Why would you do this? Monitoring being available on the Intranet, of course. But controllers on the open internet?

Might as well really focus on ease of use and just make a live website with one big button that says “cut off all electricity and turn all the water into poison, lol”

It is also very easy to airgap monitoring.

And guarded by default passwords

And the user support AI the website automatically opens in in popup can just tell you if you tell it you’re the King of Water and you forgot the account info.

It’s so fucking dumb. They are getting attacked, and they still need to be told to unplug? What the actual fuck?

yeah, but like…that won’t happen to me.

security through obscurity is nothing but a fantasy.

It is much more like yeah but like… That isn’t my job. I just scoop out the human poop.

But if they do that then how will one guy monitor and manage 20 plants at the same time?

OT networks are utter shit and everyone has known that for at least a decade.