Nut upsuser - Nov 2020 - Possible to trigger commands based on slave disconnection?

On November 22, 2020 6:56:28 PM UTC, "Kevin P. Fleming" <kevin at
km6g.us> wrote:
>Here's my situation:
>
>APC Smart-UPS 750 powering a rack of equipment. Rack includes a NAS
>(Debian-based) which is connected to the USB port of the UPS and is
>happily running NUT. Rack also includes a router, switch, and various
>other devices, including some Raspberry Pi computers which use the NAS
>as their storage (via NFS) and are powered via PoE (from the switch).
>
>When NUT tells the UPS to shut down, that of course will remove power
>from everything in the rack. When mains power returns, the UPS will
>apply power to everything in the rack. However, I *don't* want power
>applied to the RPis, because the NAS takes quite some time (60-90
>seconds) to fully boot, and they won't come up properly if they try
>too early.
>
>In addition, there is another RPi in another location, not connected
>to this UPS, but which also uses the NAS as its file storage and thus
>must be shut down if the NAS goes down.
>
>At this point I have NUT running on all the systems, and they all
>cleanly shut down when the UPS is heading into 'low battery' mode.
>What I would like to additionally do is send commands to my network
>switches to 'turn off the outlets' for the RPis (disable PoE on the
>ports) as each one disconnects from the NUT master. I can do this over
>SSH quite easily, if I can figure out a way to trigger the SSH command
>to be executed.
>
>I can also add commands to enable these ports in the NAS startup
>sequence, at a point late enough for it to be safe.
>
>So, the question is: is it possible on the NUT master to trigger a
>command to be run when a slave disconnects? If so, I'd check the
"FSD"
>flag at that moment, and if the FSD flag is set then that means the
>power is being shut down, so I should disable the PoE port for that
>slave.
>
>I'll also need some way to turn the port back on if NUT entered FSD
>mode but then didn't actually shut down (due to mains power returning
>before the shutdown sequence was completed), so that I don't leave the
>RPis shut down.
>
>_______________________________________________
>Nut-upsuser mailing list
>Nut-upsuser at alioth-lists.debian.net
>https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>From a NUT perspective, I think it could be interesting to add drivers that
abstract a PoE switch as an ePDU (maybe separate facades for SSH or serial-OOB
or SNMP with write access, but same core).
Possibly ports' power and comms can be turned off/on separately at least on
some vendors, which may need separate commands on ePDU shim side. Maybe
you'd have to save switch config for ports to stay down after reboot.
Otherwise it may make sense to put the switch onto a separately manageable
outlet group of the UPS so you can turn that one on sone time after total boot
by server command or after a delay counted by UPS itself.

NFS is quite a sturdy protocol, at least was in original Sun implementation; we
had it running cross-continent over VPN with systems rebooting sometimes (server
should persist state somehow though). On the remote RPi's you may want to
play with client mount options like 'nointr' and/or 'hard'
effectively so client processes would freeze on I/O requests until server
becomes functional again. A common (and maybe default now) setup may be an
opposite of that, to allow for timeouts and application-driven fault handling.

I don't think upsd currently has a way to call detailed shutdown logic,
other than what you specify in SHUTDOWN_CMD - which may be a script, that does
the complex logic of tracking accessibility and shutdowns for RPi's, and
eventually (all clients down, timeout, UPS charge too low, ...) tells the NFS
server and UPS to go down.

Given Debian and systemd, you might define a template to make unit instances
(like rpi-power at 13.service) that would power on a port named by that instance
when started, power off when stopped, and as Pre*Exec methods might check if the
client is alive. Maybe some systemd versions have a more explicit health
monitoring framework. That is easy to pluck into dependency chains (start after
NUT and NFS, stop before them).

Typical stance for FSD is that it is a "forced shut down" (and often
forced externally as a flag raised by UPS by its policy) - once started, you
should not abort it leaving systems in undefined midway state. So whatever
happens to power, you go down and reboot cleanly. Frameworks like systemd or SMF
do take some unpredictability out, but on a distributed rack you'd still
have to design very carefully to abort a shutdown (change of runlevel if it is
not too late, and maybe make upsd - or even a depleted UPS - believe the state
of FSD is no longer relevant so it does not trigger another shutdown) and start
all stuff in order.

Hope this helps,
Jim Klimov

--
Typos courtesy of K-9 Mail on my Android

Just a left field idea, is there any way to halt the load process on the
Rpi until it can see the NAS is up and running? Then proceed once it can
connect.

The only pi I have messed with is the ones you eat and the button on a
calculator.

On Sun, Nov 22, 2020, 5:05 PM Jim Klimov <jimklimov at cos.ru> wrote:
> On November 22, 2020 6:56:28 PM UTC, "Kevin P. Fleming" <kevin
at km6g.us>
> wrote:
> >Here's my situation:
> >
> >APC Smart-UPS 750 powering a rack of equipment. Rack includes a NAS
> >(Debian-based) which is connected to the USB port of the UPS and is
> >happily running NUT. Rack also includes a router, switch, and various
> >other devices, including some Raspberry Pi computers which use the NAS
> >as their storage (via NFS) and are powered via PoE (from the switch).
> >
> >When NUT tells the UPS to shut down, that of course will remove power
> >from everything in the rack. When mains power returns, the UPS will
> >apply power to everything in the rack. However, I *don't* want
power
> >applied to the RPis, because the NAS takes quite some time (60-90
> >seconds) to fully boot, and they won't come up properly if they try
> >too early.
> >
> >In addition, there is another RPi in another location, not connected
> >to this UPS, but which also uses the NAS as its file storage and thus
> >must be shut down if the NAS goes down.
> >
> >At this point I have NUT running on all the systems, and they all
> >cleanly shut down when the UPS is heading into 'low battery'
mode.
> >What I would like to additionally do is send commands to my network
> >switches to 'turn off the outlets' for the RPis (disable PoE on
the
> >ports) as each one disconnects from the NUT master. I can do this over
> >SSH quite easily, if I can figure out a way to trigger the SSH command
> >to be executed.
> >
> >I can also add commands to enable these ports in the NAS startup
> >sequence, at a point late enough for it to be safe.
> >
> >So, the question is: is it possible on the NUT master to trigger a
> >command to be run when a slave disconnects? If so, I'd check the
"FSD"
> >flag at that moment, and if the FSD flag is set then that means the
> >power is being shut down, so I should disable the PoE port for that
> >slave.
> >
> >I'll also need some way to turn the port back on if NUT entered FSD
> >mode but then didn't actually shut down (due to mains power
returning
> >before the shutdown sequence was completed), so that I don't leave
the
> >RPis shut down.
> >
> >_______________________________________________
> >Nut-upsuser mailing list
> >Nut-upsuser at alioth-lists.debian.net
> >https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>
> From a NUT perspective, I think it could be interesting to add drivers
> that abstract a PoE switch as an ePDU (maybe separate facades for SSH or
> serial-OOB or SNMP with write access, but same core).
>
> Possibly ports' power and comms can be turned off/on separately at
least
> on some vendors, which may need separate commands on ePDU shim side. Maybe
> you'd have to save switch config for ports to stay down after reboot.
> Otherwise it may make sense to put the switch onto a separately manageable
> outlet group of the UPS so you can turn that one on sone time after total
> boot by server command or after a delay counted by UPS itself.
>
> NFS is quite a sturdy protocol, at least was in original Sun
> implementation; we had it running cross-continent over VPN with systems
> rebooting sometimes (server should persist state somehow though). On the
> remote RPi's you may want to play with client mount options like
'nointr'
> and/or 'hard' effectively so client processes would freeze on I/O
requests
> until server becomes functional again. A common (and maybe default now)
> setup may be an opposite of that, to allow for timeouts and
> application-driven fault handling.
>
> I don't think upsd currently has a way to call detailed shutdown logic,
> other than what you specify in SHUTDOWN_CMD - which may be a script, that
> does the complex logic of tracking accessibility and shutdowns for
RPi's,
> and eventually (all clients down, timeout, UPS charge too low, ...) tells
> the NFS server and UPS to go down.
>
> Given Debian and systemd, you might define a template to make unit
> instances (like rpi-power at 13.service) that would power on a port named
by
> that instance when started, power off when stopped, and as Pre*Exec methods
> might check if the client is alive. Maybe some systemd versions have a more
> explicit health monitoring framework. That is easy to pluck into dependency
> chains (start after NUT and NFS, stop before them).
>
> Typical stance for FSD is that it is a "forced shut down" (and
often
> forced externally as a flag raised by UPS by its policy) - once started,
> you should not abort it leaving systems in undefined midway state. So
> whatever happens to power, you go down and reboot cleanly. Frameworks like
> systemd or SMF do take some unpredictability out, but on a distributed rack
> you'd still have to design very carefully to abort a shutdown (change
of
> runlevel if it is not too late, and maybe make upsd - or even a depleted
> UPS - believe the state of FSD is no longer relevant so it does not trigger
> another shutdown) and start all stuff in order.
>
> Hope this helps,
> Jim Klimov
>
> --
> Typos courtesy of K-9 Mail on my Android
>
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20201122/7aef6bad/attachment.html>

On Sun, Nov 22, 2020 at 4:54 PM Jim Klimov <jimklimov at cos.ru>
wrote:
> Possibly ports' power and comms can be turned off/on separately at
least on some vendors, which may need separate commands on ePDU shim side. Maybe
you'd have to save switch config for ports to stay down after reboot.
Otherwise it may make sense to put the switch onto a separately manageable
outlet group of the UPS so you can turn that one on sone time after total boot
by server command or after a delay counted by UPS itself.
Ahh... this is a good point. My switch won't store the port state
unless it is 'written to the boot config', so when the switch itself
loses power the config changes would be lost. The config can also be
persisted over SSH, but if multiple systems tried to do that at the
same time the results would be quite unpredictable.
> I don't think upsd currently has a way to call detailed shutdown logic,
other than what you specify in SHUTDOWN_CMD - which may be a script, that does
the complex logic of tracking accessibility and shutdowns for RPi's, and
eventually (all clients down, timeout, UPS charge too low, ...) tells the NFS
server and UPS to go down.
On the shutdown side, it turns out this would be pretty simple: on the
secondary systems, I'd use "shutdown -p +0" to force a poweroff,
and
then add a custom script in the system 'poweroff' service to actually
cut off the power from the switch. This doesn't solve the persistence
issue though, so I have some more thinking to do on that front.