Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again.
Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all.
> Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again.
And if this happens again? They advertised they had failover and mitigations for this in the RAREST of cases:
> Notices will be posted here when we re-route traffic, upgrade hardware, or in the extremely rare case our network isn’t serving traffic. - status.fastly.com
The extremely rare case happened for an hour, which is a very long time in internet time.
I think what you said is exactly why people have different opinions on this topic: what counts as "gross negligence" and what doesn't? Different people draw lines at different places.
There's, to me, no obvious clear cut line. But here are some indicators that make me consider someone was being grossly negligent and/or even malicious:
- ignoring warnings
- acting against known-to-them best practices
- repeating a previous mistake
But, again, these are just indicators, not a checklist.
Interestingly, any of these can happen also due to stress, burnout and generally broken company/team culture. Including something like a CYA culture where if they don't do something fast, they will be blamed for it, and thus they need to move fast and break things.
Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again.
Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all.
We are humans - don't forget that.
edit: forgot some words.