How familiar are you with Bandwidth.com?
If you’re like a lot of channel pros, you’d probably never heard of the company until this week, when your VoIP customers started complaining that they can’t make or receive calls. A check with your vendor then revealed that the source of the problem was a competitive local exchange carrier (CLEC) named Bandwidth.com that plays a critical if largely unseen role in the VoIP ecosystem, connecting IP-based networks to copper-based local phone systems.
An estimated 40% of VoIP providers in North America, including companies ranging from Microsoft, Google, and Zoom to Vonage, RingCentral, and 8×8, rely on Bandwidth to deliver services. So when unknown attackers launched a distributed denial of service assault on the company last Saturday, the ripple effects led to slowdowns or outright service outages for VoIP vendors, their partners, and end users.
“They didn’t go down, but the traffic coming into their IP addresses was getting flooded and then intermittency started happening,” says George Bardissi, CEO of BVoIP, one of Bandwidth’s many downstream users. “Either calls weren’t connecting all the time, and you’d have to try multiple times, or maybe the call quality wasn’t good.”
Bandwidth restored network stability within hours of the first strike, but the attackers returned the next day and have been waging a pitched battle with Bandwidth’s IT team involving continually shifting tactics and techniques ever since. The last major incident occurred Tuesday, but there’s no telling yet if the larger DDoS campaign is truly over. On Wednesday, a company spokesperson told ChannelPro that Bandwidth is “seeing some intermittent disruptions in service and working around the clock to restore.”
That’s consistent with what Cytracom, another Bandwidth partner, has experienced. “Monday was bad, and Tuesday,” says Zane Conkle, the vendor’s CEO, in a conversation with ChannelPro yesterday. “Today we’ve seen traffic back above 80% of what we’d expect to see.”
Bandwidth has said little publicly about its plight to avoid making a bad situation worse. “That’s always the challenge with these types of attacks,” says Richard Craighead, Cytracom’s vice president of engineering. “You want to provide enough information to enable the defenders to defend themselves, but you don’t want to arm the attackers with information that will allow them to circumvent whatever you’re trying to do to defend yourself.”
On Tuesday, however, Bandwidth CEO David Morken posted the company’s first public acknowledgement that a DDoS strike was underway. “We will not rest until we end this incident, and will continue to do all we can to protect against future ones,” he wrote.
And have no doubt, there will be more incidents like this one in the future. Bandwidth, in fact, isn’t even the first big carrier to come under assault recently. Two VoIP operators in the U.K. were struck roughly a month ago in what appears to be a connected series of attacks that subsequently struck VoIP.ms, a major Canadian provider.
The same kind of cybercriminals who targeted Colonial Pipeline because it’s critical to distributing fuel, it seems, have begun targeting Bandwidth and companies like it because they’re critical to distributing communications. Explaining that to SMBs, however, isn’t easy.
“All they know is I can’t make a phone call,” notes Cytracom COO John Tippett. “Fix it.”
If only it was that easy. “Toll-free numbers can be moved pretty easily to other places without much fuss, and outbound calling can be re-routed across multiple carriers without much effort,” Bardissi explains. “When it comes to inbound calling, the number has to be on a carrier, and that carrier of record basically owns it. You can’t really just shift it around.”
Or at least not quickly, as anyone who has switched cell phone providers knows. Porting a number to a different carrier typically takes days, if not weeks.
“You can’t just switch providers,” says Lance Condray, Cytracom’s vice president of customer experience. “It’s not like AWS is getting attacked, let’s spin up Azure.”
Cytracom could re-route its outbound traffic around Bandwidth, however, and did so almost immediately after the attack began. To mitigate the attack’s impact on inbound traffic it leaned on the redundancy built into its backend infrastructure. “The calls seem to be essentially back to where we would expect to see them, especially compared to the other providers that have been hit that have been down for an extended period with no calls transiting their networks,” Conkle says.
BVoIP, for its part, works with Tier 2 carriers one layer below Bandwidth in the telecom hierarchy, and those companies have multiple Tier 1 relationships. “They realized after a couple of days that this might go on for a while, so they started doing what they call ’emergency porting’ to non-Bandwidth network providers,” Bardissi says. Within roughly 24 hours, all of BVoIP’s MSP partners except the roughly 20% who maintain their own carrier relationships were no longer impacted by inbound calling constraints.
For channel pros affected by Bandwidth’s travails and those untouched by this week’s events, the big question now is what next? Here are five recommendations on what to do and more importantly not do from Cytracom and BVoIP.
1. Don’t single out Bandwidth.com
Yes, they got hit this time, but it could be anyone with similar importance in the voice communications supply chain next time. If anything, says Tippett, Bandwidth should be commended for how quickly and effectively they responded.
“They’re very prepared and have moved very quickly,” he says.
Condray, who before working at Cytracom was a C-level executive at Bandwidth, expects the company to be even better prepared for the inevitable next assault. “They’ll bring in even more experts, and the next time they get attacked—because it will happen again—maybe they’ll deflect 99% of it and we won’t actually see anything by the time it reaches us.”
2. Don’t ditch VoIP
If this week’s events have you thinking of steering your customers away from VoIP, don’t kid yourself, Conkle says. These days, even voice traffic from traditional phone companies transits IP-based networks at some point or another.
“At the end of the day, it’s going to touch Bandwidth or it’s going to touch one of these Tier 1 providers no matter which way you go,” he notes.
3. Don’t take a do-it-yourself approach to delivering VoIP services
Channel pros looking to maximize margins on voice services are often tempted to establish direct relationships with upstream carriers like Bandwidth rather than resell a VoIP vendor’s product. According to Tippett, though, this week’s developments highlight the dangers of flying solo rather than partnering with a company that has VoIP specialists on staff and a sophisticated infrastructure to leverage.
“You don’t have anybody to lean on,” Tippett notes.
Unless you have an enormous client base, moreover, you probably won’t get the kind of responsive support that Tier 1 and 2 carriers provide their largest customers. “You’re just sort of the bottom of the totem pole,” Tippett says.
If you insist on working directly with upstream providers anyway, Bardissi counsels, make sure at a minimum that you have several such relationships, so you can re-route outbound traffic if your primary carrier goes down.
“It’s going to cost money, but if you have it set up you’re in a better position than if you don’t,” he says.
4. Don’t shop for VoIP services on price
In fact, don’t make pricing the top selection criteria for any telecommunications-related services. “In the telecom world, it costs a lot of money to do things right, and you get what you pay for,” Condray says.
Craighead agrees. “If you go to an underfunded solution that doesn’t have the expertise, doesn’t have the right staff, an attack like this is going to put them out of business in short order,” he says.
5. Do vet VoIP providers carefully
If you haven’t before, now is the time to quiz current or prospective VoIP vendors hard about their security policies and preparedness. Do they have redundant resources? Do they have the expertise and financial heft to defeat a determined adversary? Who are their upstream carriers, and how many do they work with?
“If you don’t ask the question, you don’t have the answer,” Bardissi observes. And the middle of a DDoS attack is a bad time to get it.
