Incident Response Management Plan Best Practices

An actionable incident response management plan for your IT teams

An incident response management plan defines the posture and actions IT operations teams take in order to effectively respond to incidents impacting customer experience. Given that 90 percent of large businesses say they experience major IT incidents and IT downtime several times a year, one begins to understand the importance of having incident response teams. However, for IT response teams to be effective at responding to issues such as security threats, site outages or degrading of site performance, they need to have the proper training, tools and mindset.

Unfortunately, most organizations do not have an incident team that is supported by these resources. Instead, as one source reported:

[M]any organizations do not have an incident response team, or have one it is under supported. According to a survey by the Ponemon Institute, most respondents agreed that the best thing their organization could do to mitigate future breaches was to improve incident response capabilities

Fortunately, we believe that effective response teams can easily learn the management practices and actions their teams need to take. As such, the goal of this blog is to highlight best practices modern IT teams should pursue.

Best Practices for an Incident Response Management Plan

1. Actively track what effects the customer

For proper alerting to occur, you need to make sure you have the proper monitoring in place. For monitoring, your team can use tools like Datadog, Solar Winds or one of many other monitoring tools. The goal is to also have confidence in the thresholds you have created. You want to make sure that your monitoring tool does not create false positives or create a high priority alert for an event that could be handled tomorrow morning at 9 am.

2. Ensure there’s an incident response management plan in place

Ensure that there is an incident response plan template in place of how your incident response team will be alerted. Know the answers to questions such as who will receive alerts and how will they be alerted. Ideally, you will want your alerts tied to a digital on-call schedule so that the proper engineer is alerted in case of a disaster. You also want to make sure there are escalations in place to ensure that back-up teams are notified if primary incident responders are unavailable.

Ideally, as part of the process, team managers will have runbooks at their disposal so that teams can manage incidents as independently as possible. Through information-sharing and judgment skill development, escalations are less likely to occur.  Effective incident management relies on having access to information on similar incidents which happened in the past. With this access, IT support can streamline resolution and reduce the risk of implementing a new plan.

With runbooks, engineers are clear on what steps need to be taken to effectively handle incidents and what precautions to take in responding to the situation.

3. Your incident response management plan requires a communications strategy

While effective communication can be challenging in the best of circumstances, it can be especially trying during an outage or when an external customer is facing an issue. The first goal of the incident management process is to restore a normal service operation as quickly as possible and to minimize the impact on business operations. To achieve this end, there are a number of tools that IT engineers should have at their disposal in order to expedite resolution of the issue.

• The fist maxim of true incident management actually suggests a tool to not use. This rule is to never use email to effectively escalate and manage an event. Escalation can easily generate an overwhelming number of emails notifications which can effectively derail the incident management process. According to Harvard Business Review:

The only way to keep productive energy flowing through this [email] network is for everyone to continually check, send, and reply to the multitude of messages flowing past—all in an attempt to drive tasks, in an ad hoc manner, toward completion.

Email becomes the platform where all tasks get dumped – including important IT incidents whose speedy resolution is key to keeping customers happy and the business running. As such, teams should look to communicate with their colleagues on a separate messaging application that has immediacy as well as priority settings.

• The second component of effective communications is the use of a critical messaging application with priority messaging. Engineers need to have the ability to instantaneously communicate with one another when attempting to resolve issues. Critical messaging applications should come with alerts so that individuals can ensure messages are recognized when they arrive and encourage a quick response.

Critical messaging applications can better ensure communications if the application comes with a method for creating persistent and actionable alerts and minimizing alert noise. That is, teams want alerts that will continue to notify individuals until the alert is answered. Some technologies like OnPage continue to notify individuals for up to 8 hours until the recipient responds to the alert. OnPage also has to send messages based on the priority of the alert. This helps filter out the high priority alerts from the low priority alerts.

• The third component of an effective communications strategy is to enable attachments so that IT teams can amplify explanations through documents or screenshots. Often these items are much better at explaining an issue than a much longer text.

Conclusion

These insights highlight the components you need to have in place to ensure your IT team is ready for proper incident response. You need to make sure you have the proper forethought, the right tools and the right procedures in place that can help your team grow.

To learn more about how to get started with incident response management, please contact us or download our whitepaper on  Incident Response Management for IT Teams.