NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲Microsoft 365 outage leads to chlorine gas poisoning of 15 hotel guests (www1.wdr.de)
N19PEDL2 58 minutes ago [-]
This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
DanHulton 5 minutes ago [-]
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.

- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.

I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.

ashleyn 26 minutes ago [-]
This seems like the kind of thing which should have an audible alarm mounted on the wall in the pool room.
croes 22 minutes ago [-]
You want a panic? That’s how you get a panic
s0rce 13 minutes ago [-]
Facilities I worked in that had chlorine gas had audible alarms if leaks are detected because everyone could die.
windowsrookie 15 minutes ago [-]
So what if people panic? People panicking because of an alarm is better than being poisoned...
skeledrew 46 seconds ago [-]
So it's better to have deaths rather than panic?
mastax 19 minutes ago [-]
How do you feel about fire alarms?
greggsy 33 minutes ago [-]
Exactly. Email is also a ‘zero guarantee’ service, and should never be relied upon for life or death solutions.
mikeryan 3 minutes ago [-]
Not just that, in this case there was no way for the alert system to even know that the emails were delivered. I can’t speak to everyone but for my company emails were not bouncing, I just got them 12-24 hours late. At least SMS and the like know if the message reached the actual end recipients inbox.
cbarnes99 46 minutes ago [-]
It's the fault of the company that designed the automatic alarm. Not the hotel. But it is accurate to say that the cause of the issue was the outage.
buran77 34 minutes ago [-]
> It's the fault of the company that designed the automatic alarm. Not the hotel.

The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.

> But it is accurate to say that the cause of the issue was the outage

The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.

zardo 28 minutes ago [-]
Email going down is the proximate cause. A system configuration relying on a single unreliable communications route is further up the causal chain.
sokols 35 minutes ago [-]
Well somebody has to sign off the solution, in this case the hotel.
01HNNWZ0MV43FF 37 minutes ago [-]
We can assign fault, or we can ask which levers we can pull to make it better:

1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.

2. Could Microsoft have not had an outage? Hard to fix.

3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.

croes 16 minutes ago [-]
A light has to be seen, a siren has be heard. I bet email has a better chance of alarming the right person.

And you have to check those lights and sirens that they still work.

Everybody knew email was down.

The real problem is nobody thought of the consequences regarding the alarm system

stackskipton 48 seconds ago [-]
Generally, for sensitive safety systems like this, Light and Siren are visible to those using the facility. So if problem developed, all patrons would be going "Umm, you know your sauna has flashing red light and siren going off?"

Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.

macNchz 51 minutes ago [-]
Seems like if the system can send an email it should be able to, like, illuminate a red light somewhere or something, if what it’s doing can create an actual safety risk.
freitasm 47 minutes ago [-]
Came here to say this. A push notification to an app or Slack/Discord. Or even a light.

I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.

croes 23 minutes ago [-]
I doubt they have Slack or Discord and I bet there is nobody who constantly watches a light.

You know what they all have? An email account

xyst 33 minutes ago [-]
hindsight is 20/20.

I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.

Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.

In short, all parties suck here (MS and hotel)

Espressosaurus 23 minutes ago [-]
Hindsight is 20/20? Bullshit. Safety critical systems should not rely on transport layers that are guaranteed to fail. Email especially! Physical alarms are the way here. The email should be for convenience and to inform higher-ups.
reaperducer 17 minutes ago [-]
Safety critical systems should not rely on transport layers that are guaranteed to fail.

Go invent a transport layer that will never fail, and make yourself rich.

Because right now, there is no such thing.

Rygian 2 minutes ago [-]
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer for a life-or-death signal.

Email is not.

Espressosaurus 14 minutes ago [-]
Everything can fail even down to individual capacitors on the board but that doesn’t make the failure probabilities or failure mode equivalent to something like email that is best effort only and can be actively blocked by things like spam filters that aren’t even under your control
discordance 56 minutes ago [-]
I know it’s fun to blame Microsoft for things, but that is a poor and dangerous design.

Anything that could result in that sort of harm should have a more resilient and redundant architecture.

k310 1 hours ago [-]
https://www1-wdr-de.translate.goog/nrw/rheinland/rhein-erft-...

> 15 guests injured at Phantasialand

> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.

jamal-kumar 3 minutes ago [-]
excuse my language but holy shit thats fucking stupid. email? for chlorine gas alerts?
dekhn 50 minutes ago [-]
Perhaps as a proximal cause, but not a root cause.
mentalgear 55 minutes ago [-]
Core services must be as minimal and stable as possible, with safe fallbacks and never be part of a huge internet smart device big tech clusterfuck.
zkmon 18 minutes ago [-]
> This was because the automatic dosing system had pumped too much chlorine into the pool.

What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.

userbinator 39 minutes ago [-]
Surprising this happened in Germany of all places; don't they have a huge emphasis on Linux and OSS?
justincormack 31 minutes ago [-]
No, there are many Linux users there, and some small high profile government cases of trying to move of office, but like everywhere else it is hugely dominated by Windows, especially this type of smaller business area, hotels, OT systems etc.
johnnyApplePRNG 43 minutes ago [-]
Hot take.
dlcarrier 33 minutes ago [-]
Who needs a horn or a flashing light, when you can just send an email?
lwansbrough 40 minutes ago [-]
I suppose if you really think about it, butterflies are responsible for everything.
londons_explore 1 hours ago [-]
Employee finds pool water has dangerous amounts of chlorine, yet instead of locking the door to the pool they just send an email into the void?
59 minutes ago [-]
ocdtrekkie 1 hours ago [-]
This is an automated system almost certainly. It monitors the chemistry and sends alerts if things fall out of range.
dgoldstein0 56 minutes ago [-]
Whoever made email safety critical was not thinking... What a dumb design decision
throwaway85825 34 minutes ago [-]
OK you design a new out of band notification system that somehow has better reliability than email but is also used frequently enough that people will notice when it stops working.
CoastalCoder 26 minutes ago [-]
How about an audible alarm and a flashing light at the dangerous location?

This seems like a solved problem, e.g. for smoke detectors.

simoncion 29 minutes ago [-]
Phones are good. So are loud, annoying alarms as well as labeled, illuminated alert lights in the potentially-hazardous area.

The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.

Animats 1 hours ago [-]
Apparently the alarm system sent its alarms via email.

"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."

We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.

leni536 27 minutes ago [-]
I assume the backup pigeons didn't arrive in time either.
ErroneousBosh 25 minutes ago [-]
BRB, off to read the Therac-25 paper for the third time this month.
lifestyleguru 27 minutes ago [-]
lol Germans and the internet. Of course it's the internet's fault, not theirs.
lazzurs 45 minutes ago [-]
For anyone that doesn't read the link they were hurt but seriously.

This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 20:25:33 GMT+0000 (Coordinated Universal Time) with Vercel.