Improving Incident Response Management

How to ensure proactive communications during an IT outage

In the world of IT outages and IT operations, incident response management plays significantly into how quickly the issue is resolved. The cause of the outage could be the result of a network configuration change, software upgrade, scheduled maintenance, surge capacity failure or simply a code change. Any one of these issues could cause hours of downtime. Knowing that an hour of IT downtime can easily cost over $100,000, it is important for every IT team to have a preconfigured, well-considered incident response plan to minimize downtime and keep key stakeholders informed.

The goal of this blog is to highlight the steps teams should take to effectively manage critical outages of their IT, no matter the cause. To that end, we will look at:

• Organizing teams
• Communication channels and stakeholder management
• Reporting

Organizing teams for proper incident response management

Armies are effective due to the strength of their leadership and their soldiers. Similarly, the incident response management team that responds to an IT outage will be successful if they have a strong leader along with a strong team to manage the outage. The IT outage team must have a preassigned leader as well as an assigned team whose job is to manage the outage. Otherwise, the resulting disorder will take away critical time from responding to the incident. Time will be wasted trying to figure out who should handle which part of the outage. If however these leadership and management roles are preestablished, then the team members can get to work right away and start resolving the issue at hand.

To ensure that there is no guessing as to who will be notified, the response team needs to have their names listed in a digital scheduler so that as soon as the outage occurs they are notified of the issue.  This team should also have back-up responders included in the digital schedule so that if anyone is out sick, they are sure to have a backup.

Alerting of the IT outage team should occur on multiple channels such as SMS, email, phone call and smartphone application. As such, getting an alert via smartphone should be the first step. The smartphone alert will be the most effective way of grabbing the engineer’s attention. SMS, email and phone calls probably best serve as effective backups. The goal is to provide primary and secondary forms of alerting to make sure that there is virtually no chance of letting the team members remain unaware of a brewing situation.

Communications channels and stake holder management

Once the IT outage response team has been alerted to the issue, IT needs an incident response management tool through which they can communicate and make sure that their messages are received immediately and prominently. In the high stakes game of managing IT outages, emails and SMS are not effective tools. Email is really a form of communication focused on an exchange between two people. As soon as multiple people get involved in an email thread, the communication gets muddled. Additionally, email is not good for real-time communications. There is inevitably a delay which prevents rapid resolution of the issues at hand.

Similarly, SMS faces similar issues to email in that it is not meant to be a collaborative tool or a tool that enables work to get done. The text messages are not integrated into the work thread and so they remain separated. In addition, and this is also an issue faced by email, it is impossible to query databases or execute functions from the command line of SMS or email.

More importantly, these tools don’t encourage collaboration, which is exactly what the team needs to effectively manage outcomes. Instead, if a team has an incident management system that allows for real time chat, they will be much more effective. Indeed, the strength of an IT team revolves around its ability to quickly resolve incident, chat plays a crucial role in rapid incident resolution.

Instead, teams should grab onto a smartphone application that elevates high-priority communication and separates it from the standard chat that occurs on straightforward ChatOps platforms. A technology like OnPage has the ability to link into standard apps like Slack and ensure that messaging continues on a separate, high priority channel.

At the same time that the members of the outage team are communicating, the incident commander needs to make sure that any updates are reported to important stakeholders. These individuals can receive an alert on their smartphone application that keeps them apprised of important developments. Later on, when the situation is resolved, the senior management can view the reporting details to learn about the meeting of SLAs and how well the team performed.

Reporting

For incident response managers to further ensure success, the actions taken by the incident response team need to be documented and measured. This type of visibility can only really occur when there is effective reporting attached to the critical alerting platform.

The reporting tool should provide summaries and insights through data. This information should highlight a team’s effectiveness across multiple shifts and time zones. Team leaders can thus easily see trends, performance, productivity and understand how well their team is doing.

Conclusion

The next time an IT outage happens, IT teams don’t need to feel like they have jumped into an abyss. With proper incident response management, team members will know their exact role in managing and resolving the crises.

This blog has outlined the four key components of an effective IT response stance. These are components that we have seen as effective across numerous teams. If you want to learn more about how OnPage’s incident alert management platform can help you with critical alerting, contact us.