Many MSPs feel awkward discussing data backup, disaster recovery, and business continuity with their clients. But having that difficult conversation these days is more critical than ever to be ready for the next major system snafu.
John Joyce
The CrowdStrike bug that inadvertently disrupted IT systems worldwide added a new disaster scenario that standard backup, DR, and BC tools didn’t fully address.
“We had a 90-minute cyber event that ground Earth to a halt,” said John Joyce, co-owner of CRS Technology Consultants near Fort Myers, FL, which supports about 150 companies with 21 team members. Working on the Florida Gulf coast, clients understand disasters like hurricanes and cyberattacks, but not a problem at the business supply chain level, said Joyce. “This turned into an operational business continuity issue.”
Looking ahead, there are chances something like this could happen again. So, what are the best ways to minimize the effects of the next snafu, and what client processes need to be improved?
And what do you do when your computers won’t boot? “Like Mike Tyson said, ‘Everybody has a plan ‘till they get punched in the mouth,’” said Scott Beck, CEO of BeckTek in New Brunswick, Canada, near the Atlantic Coast. “Many people never considered that.”
What other critical business elements have been overlooked? “The time to run a fire drill is not during the fire,” Beck quipped.
Dawn Sizer
Creating a Comprehensive Incident Response Plan
Your incident response plan is critical. Creating a plan is the first step, even if it’s only one page that details what you do if systems go down, noted Rob Burton, CEO of PreparedEx. “IT guys may get things running again, but often the business continuity part is lacking. So, you can’t switch to manual processes, go old school, and work until tech gets back online.”
Of course, systemwide snafus cannot be completely avoided, but you can mitigate them, said Dawn Sizer, CEO of 3rd Element Consulting. “Make alternate plans for continuity of operations. Vet all your vendors so all your proverbial eggs are not in the same basket.” Your vendors have some functional overlaps, so if your main, say endpoint protection, goes down, you have an alternative.
The Role of Tabletop Exercises in Crisis Preparation
The best process to put in place for the next systemwide snafu, said Burton, is an incident response plan accompanied by tabletop exercises. His company runs tabletop scenarios to prepare companies for such events.
Rob Burton speaks to attendees at ChannelPro DEFEND East.
For small groups, Burton trains the IT and executive teams together. Often, they undergo crisis testing and training separately because each is so complicated, Burton added. “We put them through the paces and test their premade plan with procedures as a team. Then, we help them improve their capabilities for any crisis event.”
Sometimes, companies have to go manual during difficult times, and then reevaluate after the crisis. Many of Burton’s clients book an exercise year after year, even quarter after quarter.
Regular Plan Reviews, Safeguards, and Budget
For most MSPs and their clients, it’s important to go through the IR plan at least yearly, Sizer said, “And preferably more often.”
Scott Beck
Clients tend not to panic when they know what the process is, even if the finer details aren’t right in front of them. Both customers and MSPs techs will perform better during a crisis if they’ve done it before, even as an exercise, she explained. “Having a good plan with incidents and outcomes you’ve seen before gives you both ideas on how to remediate issues, as well as a repeatable process.”
Clients also need the proper processes and safeguards in place as part of their MSP support package, emphasized Joyce. “We have nonnegotiable details with clients, including data recovery and business continuity tools and processes. We have these for partners as well.” If you promise your customers that you can support them, make sure your vendors can support you.
Affordability matters, of course, said Joyce. “Everything’s easy until it’s time to make the budget.” Customers want the most cost-effective options. “We’re stewards of their budgets. Our tools must work, but we look for those with the most value for the dollars.”
Leveraging High-profile Incidents
The recent CrowdStrike snafu and the cyberattack that hit CDK Global and affected car dealers made major news across industries and nations. Although they were crisis moments, Beck joked that he can appreciate the results.
“The media scares my clients for me,” he laughed. “We’ve already adjusted some contracts and made sales to assist in business continuity for the next event.”
Key Facts
Here are several things to remember about cybersecurity mishaps:
- It’s not a matter of if, but when.
- The more one vendor provides, the more pain when they fail.
- Create an Incident Response plan, even if just one page.
- Plan when you should call your insurance company and/or lawyer.
- Test your IR processes yearly at a minimum. More practice yields better results.
- Customer IT and executive teams must both be involved.
Image: iStock
