Lessons from the trenches of a cybercrisis rapid response team

When it comes to cybersecurity, almost every company wants to avoid the spotlight.

That’s why, if criminal hackers take down a company’s website by overloading it with traffic or encrypt a company’s files and hold them for ransom, many of them dial “Cyber 911.”

That’s the nickname for the hotline to reach IBM’s global red team of emergency responders for cyberattacks. Across the world, IBM’s teams are ready to dispatch quickly, to almost any location, to investigate Internet-related malfunctions facing its customers or those who cold call in a panic.

The goal is to “reduce the amount of data leaving the organization, isolate the bad people, preserve the information you have," says Phil Kibler, director of IBM’s Cyber Security Intelligence and Response Team.

IBM is just one of a growing number of major companies offering security and incident response for cyberattacks – for a price. Across the private sector, companies in energy, finance and even retail are finding they need protection from digital attacks but don’t have the resources or in-house technical know-how to do it themselves. So they are turning to vendors, including IBM or Dell SecureWorks and Mandiant, to make sure they’re prepared in the event of a cyberattack and help them respond if they’re hit.

It isn’t cheap. Engaging IBM’s emergency response team starts at $30,000, and costs vary depending on the time IBM needs to address the issue and size of the breach. Yet as hacks proliferate, business booms.

“Our joke is, you could almost swap the logo out because many vendors have this offering,” said Rick Holland, principal analyst at Forrester Research. “If you have services in your portfolio and you’re in the cybersecurity space, you’re adding incident response, because there’s so much money to be made.”

In the wake of high-profile attacks on companies such as Target and Home Depot, the number of requests for IBM’s security services have tripled in the past year, Mr. Kibler says.

Top executives are starting to request assessments even before they’ve been breached, adds Lance Mueller, senior incident response analyst at IBM Security Services. “Companies are saying, ‘Come to our environment, take a look, see if we’ve been breached but haven’t realized it – or what we can improve so we don’t end up on CNN.’”

So how, exactly, can companies avoid that nightmare scenario? Even among those savvy enough to call a the hotline, there are still some mistakes companies can easily avoid. Passcode spoke with members of IBM’s team to go behind-the-scenes of one major cybercrisis response center to hear lessons, trends and case studies of data breaches – from those who tackle them every day. 

Have a plan – before you need one

Turns out, according to Kibler, that more than half those who call IBM's hotline line do not have a satisfactory plan. “It’s not best of breed; it hasn’t been tested in a year; it hasn’t been updated in six months; or it’s never even been pulled out of the drawer.” IBM and other vendors can help companies develop them even before hackers strike.

This is becoming an increasingly attractive option for companies growing more wary of the embarrassment that would come with a breach, according to Mr. Holland, the analyst, who says companies are less hesitant to pay retainer fees they found undesirable just a few years ago. He tells clients to identify their vendors well ahead of time, to avoid a company’s employees “running around like a chicken with its head cut off at the time of the actual breach. The flashing lights are going, stress is high, the scope of the breach is unknown, the board is asking questions you don’t know the answers to.”

At that point, Holland adds, “trying to figure out who you’re going to use and the sourcing components is not something you want to do – you just want to be able to say, ‘Here’s the plan’ and execute that plan. Not come up with your plan.”

If you're under attack, don't send e-mails about it

Often, employees’ first reaction when their networks are compromised is to send e-mails about the crisis.

That’s not smart, Kibler says. The first thing the attackers will do to find out how a company is reacting to the attack is compromise the e-mail system to stay one step ahead of them. “I tell people, ‘Panic is your worst enemy,' " Kibler says. Response plans should address the method of communication when a breach happens. When in doubt: Pick up the phone.

Don't try to fix the problem alone if you're not a specialist

When a manufacturer in Mexico noticed one of its devices malfunctioning, they dialed Cyber 911. IBM’s team dispatched, quickly, to the site, to find the device in Mexico had hacker tools on it, including a password cracker.

But the Mexico manufacturer’s onsite employees accidentally destroyed a lot of potential evidence as they tried to fix the problem themselves, said John Brown, a senior incident response analyst at IBM’s Emergency Response Service. As a result, the incident response team was unable to reconstruct what happened and determine who was behind the attack once it arrived. “It’s really unclear if this was a target of opportunity, or if this was a targeted attack,” Brown said.

Test your systems to find out what’s vulnerable

The manufacturer in Mexico believed the data on its device was behind a firewall and untouchable to any outside hacker – and therefore that the system’s compromise was an inside job. That wasn’t the case. IBM found the critical data was actually not protected and the proprietary information was up for grabs.

“Through regular testing and assurance,” Brown said, “they should have known those files were exposed.”

Ransomware is almost always avoidable

Ransomware is malware that encrypts victims' data until they pay money to get the key. Victims are essentially faced with a choice: Pay the ransom to get the data back, or learn to live without it. A popular variety known as CryptoWall infected an estimated one million victims and garnered some $1.8 million in ransom.

Those victims should take a close look at their behavior. “Ransomware almost exclusively starts with someone inside the company doing something stupid,” Kibler says. “Meaning it was avoidable. If they had not visited a website they shouldn’t have, opened a file from somebody they shouldn’t have, if they did not suffer a spear phishing attack and were duped into clicking something they shouldn’t have.”

Back up your data

Sometimes the emergency responders can reverse engineer the malware to recover their files. But increasingly complex malware means there’s no guarantee that’ll work.

The responders have some more basic recommendations to avoid having your company’s files seized: Change your password often. Put your cursor over a hyperlink to determine where it’s taking you before clicking it; don’t assume it’s safe. If asked to go to a website, determine whether the sender someone you trust, or really them. Consider putting extra tiers of security in place to allow access to certain high-value data only to privileged users.

The clearest way to defeat ransomware, however, the cyberemergency responders say, is to backup the data so you can afford to lose it if it’s locked up.

Attacks can begin with a phone call

The IBM team sees hackers trying to get the financial information about the parent company by social engineering their way in. They might call the smaller shop on the phone and say, according to IBM's Mr. Mueller, “‘I belong to XY help desk, and want to help with your computer’ – but in reality that’s just an attacker trying to get in.”

The lessons: If you’re a franchise, do your diligence. Verify the caller’s identity on the phone. Report any suspicious behavior up the chain. If you’re a big company, make sure your leadership and your franchises understand the risks that aren’t always so obvious.

“Most security organizations are really sensitive and conscious about a forward facing threat, what’s coming through the front door, attacking our Web servers and main presence, not necessarily looking at backdoor and franchise,” Mueller said. “That’s exactly what happened with Target.” In that breach, attackers used credentials stolen from a refrigeration and HVAC contractor.

Designer malware on the rise

“This year, we saw malware that has become so specialized it only operates within that customers’ environment,” Kibler says. This makes it much harder for the emergency responders to combat. “If I take that malware from Korea and bring it to Singapore and have my team work on it, they can’t recreate it. Even if they take it to another environment in Korea they can’t recreate it.”

Since the only way to build malware like this is to have a lot of inside knowledge about a company’s network, Kibler recommends changing it frequently to make sure that can’t happen. “It’s a cat and mouse game to stay ahead of them – and changing things will help avoid giving them an easy target.”

Build security infrastructure

Brown has been working with a retailer that “got religion” after a credit card breach. But they were so far from their goals of building adequate security infrastructure within the company that IBM put in place an interim Chief Information Security Officer to help the company hire people and choose the right security solutions. Lesson: Company structure matters.

“Generally, you can say that most companies need a CISO,” Brown said. “What is really important is they have an incident response plan that reflects reality. And within that, you have someone who is going to manage the incident at a tactical level … regardless of title, someone who is responsible to respond to a computer security incident of one sort or another.”