The Human Side of Postmortems

Imagine you had to write a postmortem containing statements like these?

"We were unable to resolve the outage as quickly as we would have hoped because our decision making was impacted by extreme stress."

"We spent two hours repeatedly applying the fix that worked during the previous outage, only to find out that it made no difference in this one."

"We did not communicate openly about an escalating outage that was caused by our botched deployment because we thought we were about to lose our jobs."

While the above scenarios are entirely realistic, it's hard to find many postmortem write-ups that even hint at these "human factors." Their absence is, in part, due to the social stigma associated with publicly acknowledging their contribution to outages. And yet, people dealing with outages are clearly subject to physical exhaustion and psychological stress, not to mention impaired reasoning due to a host of cognitive biases.

This report focuses on the effects and mitigation of stress and cognitive biases during outages and postmortems. This "human postmortem" is as important as the technical one, as it enables building more resilient systems and teams, and ultimately reduces the duration and severity of outages.