Most incident response plans don't survive first contact

This is not technically a war story, however, it is an experience that I would like to share. I recently attended an event featuring a speaker from a large company that had experienced one of the most high profile and extensive breaches in recent history. For the sake of the company I will not name them in this blog, but I do want to stress that the company is very large and the breach was extensive, affecting millions of customers and their entire network. What was interesting is that the speaker was from the company’s legal department, and as such, is not a “technical” person. This provided a brand new perspective to incident response.

In my line of work as an incident response analyst, working in a Managed Security Services Provider company, I routinely help companies that suffer from security incidents. I have first-hand knowledge as to how devastating such an event can be to a company. This speaker stressed that their company lost well over a billion dollars in revenue as a result of the breach. Not including the costs associated with the investigation, response, payroll and other fiscal issues -- just the loss of revenue. Can your business handle such an event and such a loss?

The speaker stressed that the incident was a “seismic” type event. It was “devastating” and “draining” to all of their resources and personnel. They were physically ill, lost sleep and worked long hours in an effort to identify what had happened and how widespread it was in their network. Are your employees ready for such an incident? Do you have the resources available to deal with such a large event? Many think they are, only to find out that they had no idea what was going to occur when an event actually happens.

The speaker quoted, “Most plans don’t survive first contact with the enemy.” He stressed that this is entirely accurate. The company in question had an excellent security program in place and an incident response plan. They had an internal forensics unit established – and even had their own Security Operations Center. Yet, all of this preparation failed them when the actual breach occurred.

So, how can you better protect your data and that of your customers? I have a few takeaways from the speaker to share with you. His experience can help you better prepare for the inevitable breach.

1. Taxonomy becomes extremely important. The speaker, being non-technical by his own admission, specifically mentioned this as a failing in their response. During an actual incident, it is extremely important that everyone understand what is happening. This doesn’t require an in-depth technical knowledge, but knowing the basics can greatly aid everyone involved.

  • Invite and expect disparate members of your incident response team to attend information security meetings.
  • Include executive staff, legal staff, media relations and anyone else who might not be highly technical, but would be involved in any incident response effort.
  • This process will allow them to learn some of the taxonomy involved in information security and better equip them to deal with an incident when it happens!

2. Find out where your data resides and if you can access it during an emergency. With so many companies switching to Cloud-based solutions, this becomes vitally important to the incident response process. As the company involved in this article found out, when an incident happens, legal curtains come down and access is immediately restricted. This is done to protect all parties but it can create insurmountable hurdles to the investigative actions of your IR team.

  • Review all of your Service Level Agreements (SLAs) before an incident happens and find out what access you have and where your data is stored.
  • Engage with the Cloud Service Provider before an incident. Set up calls, exchange emails, conduct tabletop exercises - do whatever is necessary to learn how to access your data during an incident.
  • Ensure your security team is aware of the Cloud and provide them with the training needed to properly conduct IR in the Cloud environment of your choice.

3. Ensure your security personnel have the ability to interpret threat intelligence properly and apply it in the correct manner. One of the catchwords recently has been “threat intelligence” and it is a great idea, but only if applied correctly.

  • Identify threat intelligence that is applicable to your industry/business. There are many so-called threat intelligence outfits that can provide you with millions of threat indicators, none of which mean anything to you or your business. Narrow down your scope and engage with companies, which provide targeted threat intelligence, such as NTT Security.
  • Do not settle for “threat feeds.” These do provide you with actual intelligence, but with an overwhelming “feed” that is more often than not ignored and provides no substantial input into your security program.

4. Exercise all aspects of your incident response plan. Failing to prepare is preparing to fail. All aspects of your plan should be tested and evaluated to ensure that they are operating correctly when an incident happens.

  • During the incident, the company found that their phone system wasn’t up to the challenge. Their entire headquarter’s phone system crashed once the breach was made public and they had to stand up an entire call center.
  • Plan for the unexpected. Bring in an outside company to test your team, your plan and your response in surprise scenarios. The NTT Security Critical Incident Response Team offers proactive services that can meet this need for your company.
  • Use the experiences of others to train yourself. This can help you avoid the same mistakes or missteps during an incident. When you hear about a breach, learn all you can about it. Review your own security program to see if you are lacking anything that could prevent the same type of incident from happening at your own company.

I know this is a variation on a war story, but it holds important teaching points gleaned from those who experienced a life changing breach in security. Take the lessons to heart and implement change in your own incident response program.