SHORTLY BEFORE THE PANDEMIC, Texas-based Juern Technology signed its second-ever manufacturing client, and the team was excited. Soon afterwards, an HR staff member at the client ignored email attachment warnings and clicked on a resume, which launched TrickBot, a nasty piece of malware that moved laterally within the company and damaged many PDF files. Despite excellent security tools, it took Juern Technology quite awhile to track down the source—so long that the MSP eventually lost the client.
That was a turning point for CEO Neal Juern, who decided that his company couldn’t go a month longer without developing incident response (IR) plans, not only for the company itself, but for its customers.
Neal Juern
As Juern found out somewhat painfully, every company needs some sort of IR plan today. And more often than ever, MSPs are either requiring them or developing them for their clients.
Incident response is the process of detecting security events that affect network resources and information assets and then taking the appropriate steps to evaluate and clean up what has happened, explains Kevin Beaver, a security consultant with Principle Logic. An IR plan is a written document that describes what should happen during and after a confirmed or suspected security incident.
While incident response plans have long been part of the cybersecurity response for large companies and those in heavily regulated industries, many believe every company needs one today. Here’s why:
- More attacks, more often. In the last 12 months, 24% of midsize businesses either suffered a cyberattack or thought they experienced one, according to research from Huntress. The situation with SMBs is even worse; 60% of SMBs experienced at least one in 2022, according to Devolutions’ State of IT Security in SMBs in 2022-23 Survey, and 18% experienced six or more cyberattacks.
- Hackers are getting more creative. Hackers are continually coming up with new ways to infiltrate networks, install harmful software, and activate botnets using everything from artificial intelligence to SAML extractors to get what they want.
- Vulnerabilities aren’t going away. The 2022 HackerOne report found that ethical hackers discovered more than 65,000 vulnerabilities in 2022 alone, up by 21% over the previous year.
- Attacks are more expensive than ever. A single attack cost companies a median of $18,000 in 2022, up from $10,000 in 2021, according to the Hiscox Cyber Readiness Report. At the same time, PWC found that 34% of companies in North America have suffered a data breach that cost more than $1 million during the past three years.
While issues like these clearly illustrate why IR plans are needed, some companies don’t even have a choice—because others insist on it. For example, supply chain partners concerned about third-party cyber risks often now require the companies they work with to have such a plan. The same is true of companies that apply for cyber insurance.
Kevin Beaver
Jay Stampfl, managing director of Alliant Cyber, an insurance broker that works with many different insurance companies, confirms that. “Having an incident response plan demonstrates that a company is well prepared to respond to a potential cyber event,” he says. Insurance carriers also are getting more stringent on what they are looking for every year, he adds. Many even want companies to have an incident response provider on retainer.
Some MSPs require that customers have IR plans, citing protection for both themselves and their clients. Dave Gruber, a cybersecurity principal analyst at Enterprise Strategy Group, goes so far as to say that MSPs should require new customers to have an IR plan before signing them, and if they don’t have one, MSPs should take steps to create one for them. “It should be a basic operating requirement,” he says.
The most important reason to require your customers to have incident response plans? They work. Scott Beck, CEO of New Brunswick-based BeckTek, says that while the company hasn’t been saddled with too many catastrophic cyber events, there have been times when invoking an IR plan helped resolve malicious incidents quickly. In one case, a penetration test uncovered an issue on a user’s computer, which turned out to be malicious. With the help of the incident response plan, BeckTek was able to respond and remove the computer from the network only 20 minutes from the time the issue was uncovered.
The Many Routes to an Incident Response Plan
Yet despite these valid reasons to implement an incident response plan, Huntress found that less than 50% of midsize businesses have one; research from CNBC and Momentive found small businesses are similar.
That’s where MSPs are filling the gap. They understand the importance of having an incident response plan—virtually every MSP has one for their own infrastructure—and they understand the value of customers having them as well.
Anthony Oren, CEO of New York-based Nero Consulting, started developing IR plans for his customers in a roundabout way, but he’s glad he did.
“During the pandemic, ransomware became more prevalent and I had some downtime, so I started reaching out to customers to persuade them to get cyber risk insurance. We found out pretty quickly that you have to prove to the insurance company that you have an incident response plan,” says Oren.
As a result, Oren began developing incident response playbooks for Nero’s customers. Today, Oren requires all customers to have one, and to ensure that they do, Oren puts them together himself.
Anthony Oren
The way Oren does it—customizing a plan for each customer—is just one option. Some MSPs flat-out require all customers to have one, and some only require formal plans for certain highly regulated companies. Others use the same IR plan for their customers that they use internally, with some modifications.
For MSPs that standardize on one IT stack for all customers, it’s often easiest to develop one broad incident response plan as a basis for all, even if it means customizing parts of the plan for some customers. BeckTek’s smaller clients, for example, operate with an incident response plan adapted from BeckTek’s own internal IR plan; Beck does adjust the plan as necessary. “All of our clients are running the same solution we run, and we manage their infrastructure, so a lot of it would be the same,” he explains. “But we also note what they consider critical applications and workloads, and we document the order and priority of restoration separately for each customer.”
Juern Technology operates in a similar fashion, adapting its umbrella IR plan to most customers with few changes. Yet both Beck and Juern know when a customer requires a fully customized IR plan. These are often customers in heavily regulated industries, as well as larger enterprises.
Steps to Developing and Documenting an IR Plan
While the contents of an IR plan or playbook will differ, the plan typically outlines the who, what, where, when, and how of addressing cybersecurity incidents. These plans usually contain the following:
- Incident classification by severity and impact. The highest level of severity might be a critical incident with a very high impact, followed by major incidents with significant impact and minor incidents with low impact.
- Playbooks or checklists for common scenarios.
- A detailed list of incidents requiring actions. This section should outline specific threats and situations that require formal IR actions.
- Detection, investigation, and containment procedures.
- Eradication steps.
- Recovery tasks.
- Breach notification details.
- Roles and responsibilities: Identifying personnel responsible for managing the incident response initiatives and taking actions detailed in the IR plan. This section should also include contact information. Gruber advises making sure everything is written down in hard copy. “Usually that information is online, but when you can’t get online, you have no way of reaching anybody,” he notes.
- Service level agreements (SLAs).
An easy way to get started with developing an IR plan is to utilize one of the many vetted and validated templates. Some of the most popular include NIST’s Computer Security Incident Handling Guide, CISA’s guidance on developing an IR plan, and the Cloud Security Alliance’s Cloud Incident Response Framework.
With both adapted IR plans and customized ones, it can be hard to make sure you are covering all bases for your customers. An umbrella plan is a start, but there are no shortcuts to getting it right.
Dave Gruber
“MSPs should deeply understand not only what systems customers are running, but the purpose of those systems,” Gruber says. “What parts of their business operation do those systems support? Who are the people who have pieces of sensitive information or IP-related things that would become big targets for adversaries? What are you doing to make sure that that data is best protected? You have to know every piece of the IT infrastructure and how it’s related to the business.”
That’s the way Oren tackles it. For example, a customer with a more basic version of Office 365 would require a different approach than a customer with Office 365 E5, the most feature-rich version. In the first case, the plan would need to spell out more security measures, while the second may require fewer because E5 comes with MDR (managed detection and response) or XDR (extended detection and response).
Customizing the plan also means getting deep into the stack—down to the level of which server supports specific critical apps, and even addressing SaaS subscriptions that the MSP may not support.
“There are servers and there are cloud-based SaaS applications, and then there are hybrid situations. In hybrid situations, data could be sitting both online and locally. You have to get very granular,” Oren says.
Understanding customers’ priorities and the order in which systems should be brought back online after an event can also make a difference in the speed of recovery. Juern recalls a client whose ERP system had a hardware failure, but the MSP was able to bring it back up quickly because the IR plan spelled out the order that everything had to be put back in place.
Getting IR Right
It’s one thing to create a point-in-time incident response plan, but it’s another to make sure it works and keep it up to date. The best way to do so is by testing it at least once or twice per year. Yet despite the importance, only 23% of companies admit to testing their IR plans twice a year or more, according to WSJ Pro Cybersecurity Survey.
“A real-world test is the best experience, especially tabletop exercises,” Beaver says. “I work with clients quite often on such projects and they can be quite eye-opening, especially for executives who have been largely disconnected from security.”
Testing also can save money. IBM Security’s Cost of a Data Breach Report 2022 found that organizations that tested their IR plans saved an average of $2.66 million in data breach costs.
It’s also imperative to revise and update plans as things change. “We’ve found that with some clients six months is fine because their environments are stable, but others experience more change, so it should be done more often,” Juern says. His MSP updates IR plans every six months, but Juern is considering moving to a quarterly schedule.
Along the way, Juern learned a few valuable lessons about developing and updating IR plans. Getting too specific, he says, means that plans became dated too quickly. As a result, the plans now point to current, easily accessible data instead of having it all in the IR plan. “We get specific on some things but just point to others,” he says.
In addition to protecting clients, developing expertise in creating incident response plans can be a real competitive advantage for MSPs. Oren notes it’s a great selling point, one that the company often leads with these days. Juern agrees that it’s a huge opportunity for MSPs: “If you’re an MSP and you’re not providing [an IR plan] to clients, they are getting underserved.”
But sometimes, despite your best efforts, you just can’t convince customers that having an incident response plan is in their best interest. When that happens, natural consequences usually change their minds sooner or later.
“When I met with a potential client and brought it up, they told me they had been fine for 15 years and didn’t want to spend the money,” Beck says. “Six or seven months later they called and asked for our help, and we did the best we could. That’s a hard lesson.”
KAREN D. SCHWARTZ has written hundreds of feature articles, hard news pieces, white papers, case studies, and book chapters on a variety of technology and business topics. She resides in Potomac, Md., and can be reached at karen@karendschwartz.com.
Image: iStock/ Urupong
