Nut upsuser - Apr 2022 - Edge case in our NUT deployment, asking for guidance

Hi,

I'm working on deploying NUT for our main server room, and I've put 
myself in a situation where there could be an edge case that I don't 
really like, and so I'm asking for a bit of guidance.

Most of the deploymentis fairly standard according to the documentation 
(big thanks for its exhaustiveness, by the way), the main change is my 
shutdown script. It checks if we have power on both PSUs using IPMI, and 
if this is the case, it doesn't shut down (in any other case it does : 
ipmi doesn't work, only one PSU is receiving power, only one PSU exists 
on the machine, etc). We did this because about 90% of our machines have 
dual redundant PSUs, with one on the UPS, the other on mains buton a 
separate circuit. So we could have a situation where the UPS loses 
power, but we still have some on a secondary circuit.

We choose to accept the fact that if the secondary circuit loses power 
after our NUT server sent a force shutdown sequence, we may have a bad 
shutdown at this point.

What could happen in this situation, is that a machine that is a 
nut-server (A) still has the FSD flag running (because it didn't shut 
itself down) even after power comes back and some machines restart. In 
this case, the upsmon on the freshly started machines will see the flag 
and then shut themselves down again.

Our workaround currently would be to be aware of this and restart 
nut-server and then nut-monitor on the machine A before starting back 
any of its clients that is currently down.

Is there any idea of a better way to handle this edge case ? Or a better 
way to articulate this ? Maybe a way to automatically clear the FSD flag ?

Thanks for the help
Arthur

-- 
Arthur Desplanches
Sysadmin @ BUF Compagnie (buf.com)

As far as I know the FSD flag by design can only be raised; many phrases
refer to it as "latching" - much for the same reasons as you outlined:
people usually want the datacenter in a predictable hands-off state. If
something begins to shut down due to critical power state of the UPS,
everything should power-cycle and come up together and in order. So the
only way to clear FSD is to restart the daemons raising it.

Note some UPSes and their smart drivers would treat as critical any
situation where battery charge is under a certain threshold - even if
online and charging at the moment, since the UPS is too depleted for a safe
shutdown if power is lost again.

I wonder if you can fiddle with ipmi-psu driver for your case. NUT has a
way to treat blade chassis as an ePDU for the blades. Maybe you can get
upsmon to monitor an UPS and the other PSU on redundant-PSU systems.

Also see if some smarter scripting with upssched (as the handler of signals
from upsmon for complex situations) can help...

Hope this helps,
Jim Klimov

On Fri, Apr 15, 2022, 14:17 Arthur Desplanches <adesplanches at buf.com>
wrote:
> Hi,
>
> I'm working on deploying NUT for our main server room, and I've put
> myself in a situation where there could be an edge case that I don't
> really like, and so I'm asking for a bit of guidance.
>
> Most of the deploymentis fairly standard according to the documentation
> (big thanks for its exhaustiveness, by the way), the main change is my
> shutdown script. It checks if we have power on both PSUs using IPMI, and
> if this is the case, it doesn't shut down (in any other case it does :
> ipmi doesn't work, only one PSU is receiving power, only one PSU exists
> on the machine, etc). We did this because about 90% of our machines have
> dual redundant PSUs, with one on the UPS, the other on mains buton a
> separate circuit. So we could have a situation where the UPS loses
> power, but we still have some on a secondary circuit.
>
> We choose to accept the fact that if the secondary circuit loses power
> after our NUT server sent a force shutdown sequence, we may have a bad
> shutdown at this point.
>
> What could happen in this situation, is that a machine that is a
> nut-server (A) still has the FSD flag running (because it didn't shut
> itself down) even after power comes back and some machines restart. In
> this case, the upsmon on the freshly started machines will see the flag
> and then shut themselves down again.
>
> Our workaround currently would be to be aware of this and restart
> nut-server and then nut-monitor on the machine A before starting back
> any of its clients that is currently down.
>
> Is there any idea of a better way to handle this edge case ? Or a better
> way to articulate this ? Maybe a way to automatically clear the FSD flag ?
>
> Thanks for the help
> Arthur
>
> --
> Arthur Desplanches
> Sysadmin @ BUF Compagnie (buf.com)
>
>
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20220417/26fb1aed/attachment.htm>