Shifting gears: A cybersecurity journey for MSPs (Part 2)

How would the bad guys use my MSP tools against me?

Let’s say that one day, your early morning service desk technician comes in to work his shift. He gets in around 6 am, and his routine is to go through any stale tickets, review alerts, and make sure things are smooth for the day. He normally gets a few “failed backup” alerts. Most of them are for known issues, and he gets tired of clearing them out each morning. Because his email is so cluttered each morning, he didn’t notice that around 30 days ago, the “failed backup” emails stopped coming. If he did happen to notice, he would assume someone finally fixed the underlying issue. About time! What he doesn’t know is that the bad guys have been lurking in his network for around 60 days. A month ago, they turned off backups for all the MSP’s clients and disabled alerting.

Around 7:30 am, they get the first call. A client is complaining about a weird popup on their screen asking for Bitcoin. Then, they get another call. By 8:00 am, the phones are getting slammed with clients all calling with the same issue.

The bad guys have used the RMM platform to push, automatically and undetected with valid credentials, a ransomware payload out to all their client’s systems. While this is horrible, the MSP briefly takes solace in the fact that they had upgraded their backup solution last year and have solid backups for all clients from the night before they can roll back to. Until they login to the portal and realize they’re all missing…

While this may sound like the story of MSP nightmares, this is a true story, and it’s playing out far too often inside many MSPs.

MSPs have struggled over the years with scaling their business. They’re often in a position where they must decide on how to continue offering a high-touch, high quality-service while turning a profit. The answer to every question seems to be, “we need to hire more people!” The savvy MSPs have really embraced automation in their service delivery model in order to scale. They have utilized a combination of tools to resolve tickets automatically, scale environments, deploy patches, and generally make repetitive tasks easier.

The bad guys have seen this and recognized an opportunity. I like to use the analogy of a 500-unit apartment building. Is it easier to break into 500 individual units or to break into the manager’s office and steal the master key and simply unlock all the units? It’s much easier to break into the manager’s office.

(Perch calls this a “buffalo jump” attack. More can be found here and here.)

The bad actors can go big game hunting, where they look for one big ransomware payment, or they can aggregate the clients of an MSP and get a similar payout spread across a bunch of smaller, and more vulnerable, customers.

While most MSPs burn a lot of cycles worrying about how to protect their clients, how many cycles are you burning thinking about your own systems? What are some quick, reasonable, and necessary steps that an MSP can take to mitigate this risk?

• Password review:
  • This sounds so elementary, but a lot of automation is built on admin credentials and a singular password scheme. You may break some automation while you come up with a better solution, but it’ll just have to be broken. The risk is too great.
• Employee exit protocols:
  • Maybe you have had support desk personnel leave. Maybe you had to downsize because of COVID. You obviously turned off their user access, but did you change any administrative passwords they may know? We’d like to think our old employees are above this type of treachery, but “inside jobs” are common because the ransoms are so rich.
• Monitor your tools:
  • MSPs have standardized over the years on a few ecosystems. You may be a ConnectWise shop or an Autotask shop or one of the others. What that means is that the bad guys are familiar with the toolsets that typically MSPs love to use. It’s been well-publicized that the bad guys have found exploits in these tools and leveraged them to access downstream clients. You need to monitor your own toolset for breaches, as that may be the most vulnerable point for the bad guys to access all your clients.
• Security culture:
  • Recycling day at my house is on Tuesday. I want my kids to remember, on their own, to roll the recycling to the street. That is commitment. If I have to remind them to do it, I’ve gained their compliance. Similarly, you want your employees to be committed to a culture of security. They must have the discipline and rigor to think about “what’s the worst that could happen” at every turn. This is different than you telling them what you want them to do.

While the purpose of this blog wasn’t to scare you, I hope it did cause you to think about your own security position relative to your tooling and automation. You’ve worked hard to build automation and operational efficiency. Don’t let the bad guys use that against you. Harden your environment. It’ll be less convenient in many cases, but more secure. It’ll cost you some time now, but it’ll save you some serious potential anxiety later!

We'd love to hear your thoughts. Find us on Twitter, LinkedIn or write in to hello@perchsecurity.com

Next: Compliance 101: CMMC has entered the chat