Nut upsuser - Nov 2020 - Possible to trigger commands based on slave disconnection?

Here's my situation:

APC Smart-UPS 750 powering a rack of equipment. Rack includes a NAS
(Debian-based) which is connected to the USB port of the UPS and is
happily running NUT. Rack also includes a router, switch, and various
other devices, including some Raspberry Pi computers which use the NAS
as their storage (via NFS) and are powered via PoE (from the switch).

When NUT tells the UPS to shut down, that of course will remove power
from everything in the rack. When mains power returns, the UPS will
apply power to everything in the rack. However, I *don't* want power
applied to the RPis, because the NAS takes quite some time (60-90
seconds) to fully boot, and they won't come up properly if they try
too early.

In addition, there is another RPi in another location, not connected
to this UPS, but which also uses the NAS as its file storage and thus
must be shut down if the NAS goes down.

At this point I have NUT running on all the systems, and they all
cleanly shut down when the UPS is heading into 'low battery' mode.
What I would like to additionally do is send commands to my network
switches to 'turn off the outlets' for the RPis (disable PoE on the
ports) as each one disconnects from the NUT master. I can do this over
SSH quite easily, if I can figure out a way to trigger the SSH command
to be executed.

I can also add commands to enable these ports in the NAS startup
sequence, at a point late enough for it to be safe.

So, the question is: is it possible on the NUT master to trigger a
command to be run when a slave disconnects? If so, I'd check the
"FSD"
flag at that moment, and if the FSD flag is set then that means the
power is being shut down, so I should disable the PoE port for that
slave.

I'll also need some way to turn the port back on if NUT entered FSD
mode but then didn't actually shut down (due to mains power returning
before the shutdown sequence was completed), so that I don't leave the
RPis shut down.

On November 22, 2020 6:56:28 PM UTC, "Kevin P. Fleming" <kevin at
km6g.us> wrote:
>Here's my situation:
>
>APC Smart-UPS 750 powering a rack of equipment. Rack includes a NAS
>(Debian-based) which is connected to the USB port of the UPS and is
>happily running NUT. Rack also includes a router, switch, and various
>other devices, including some Raspberry Pi computers which use the NAS
>as their storage (via NFS) and are powered via PoE (from the switch).
>
>When NUT tells the UPS to shut down, that of course will remove power
>from everything in the rack. When mains power returns, the UPS will
>apply power to everything in the rack. However, I *don't* want power
>applied to the RPis, because the NAS takes quite some time (60-90
>seconds) to fully boot, and they won't come up properly if they try
>too early.
>
>In addition, there is another RPi in another location, not connected
>to this UPS, but which also uses the NAS as its file storage and thus
>must be shut down if the NAS goes down.
>
>At this point I have NUT running on all the systems, and they all
>cleanly shut down when the UPS is heading into 'low battery' mode.
>What I would like to additionally do is send commands to my network
>switches to 'turn off the outlets' for the RPis (disable PoE on the
>ports) as each one disconnects from the NUT master. I can do this over
>SSH quite easily, if I can figure out a way to trigger the SSH command
>to be executed.
>
>I can also add commands to enable these ports in the NAS startup
>sequence, at a point late enough for it to be safe.
>
>So, the question is: is it possible on the NUT master to trigger a
>command to be run when a slave disconnects? If so, I'd check the
"FSD"
>flag at that moment, and if the FSD flag is set then that means the
>power is being shut down, so I should disable the PoE port for that
>slave.
>
>I'll also need some way to turn the port back on if NUT entered FSD
>mode but then didn't actually shut down (due to mains power returning
>before the shutdown sequence was completed), so that I don't leave the
>RPis shut down.
>
>_______________________________________________
>Nut-upsuser mailing list
>Nut-upsuser at alioth-lists.debian.net
>https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>From a NUT perspective, I think it could be interesting to add drivers that
abstract a PoE switch as an ePDU (maybe separate facades for SSH or serial-OOB
or SNMP with write access, but same core).
Possibly ports' power and comms can be turned off/on separately at least on
some vendors, which may need separate commands on ePDU shim side. Maybe
you'd have to save switch config for ports to stay down after reboot.
Otherwise it may make sense to put the switch onto a separately manageable
outlet group of the UPS so you can turn that one on sone time after total boot
by server command or after a delay counted by UPS itself.

NFS is quite a sturdy protocol, at least was in original Sun implementation; we
had it running cross-continent over VPN with systems rebooting sometimes (server
should persist state somehow though). On the remote RPi's you may want to
play with client mount options like 'nointr' and/or 'hard'
effectively so client processes would freeze on I/O requests until server
becomes functional again. A common (and maybe default now) setup may be an
opposite of that, to allow for timeouts and application-driven fault handling.

I don't think upsd currently has a way to call detailed shutdown logic,
other than what you specify in SHUTDOWN_CMD - which may be a script, that does
the complex logic of tracking accessibility and shutdowns for RPi's, and
eventually (all clients down, timeout, UPS charge too low, ...) tells the NFS
server and UPS to go down.

Given Debian and systemd, you might define a template to make unit instances
(like rpi-power at 13.service) that would power on a port named by that instance
when started, power off when stopped, and as Pre*Exec methods might check if the
client is alive. Maybe some systemd versions have a more explicit health
monitoring framework. That is easy to pluck into dependency chains (start after
NUT and NFS, stop before them).

Typical stance for FSD is that it is a "forced shut down" (and often
forced externally as a flag raised by UPS by its policy) - once started, you
should not abort it leaving systems in undefined midway state. So whatever
happens to power, you go down and reboot cleanly. Frameworks like systemd or SMF
do take some unpredictability out, but on a distributed rack you'd still
have to design very carefully to abort a shutdown (change of runlevel if it is
not too late, and maybe make upsd - or even a depleted UPS - believe the state
of FSD is no longer relevant so it does not trigger another shutdown) and start
all stuff in order.

Hope this helps,
Jim Klimov

--
Typos courtesy of K-9 Mail on my Android