Nut upsuser - Oct 2023 - FSD sequence: Waiting for bigger and slower clients before cutting power

Hi, this does sound like a useful idea - although for the principle of
least surprise and for variation in deployments, I'd rather have it as a
(non-default state of a) configuration toggle that can be set via
`upsmon.conf`: whether this particular client exits after processing FSD or
not. The onus for the rest would be on general systems integration - e.g.
ensure that init scripts `K*`ill the long-running services before they go
after upsmon and upsd, or add a drop-in systemd config snippet for
nut-monitor to not-conflict with "shutdown.target" (and half a dozen
of its
equivalents for halt/reboot/poweroff/...), and possibly to break the
shutdown-dependency between nut-monitor/nut-server/nut-driver units.

On a related note - there was lately work to allow daemonized drivers to
kill power of the UPS (may be useful especially for devices with long
protocol init times), with a safety switch to flip about this and actually
allow the driver to issue killpower commands. So stopping driver daemons
might eventually be not needed - but I'm not sure any OS integrations took
note of this possibility yet. It was not officially released so far, just
is in master branch.

Note however that typically FSD happens when the power is critical.
Definitions of that vary, as well as ability or not to set certain
thresholds for when the device would emit (and a driver would relay) the
low-battery condition. You might not physically have those 2 minutes worth
of remaining battery charge to shut down the VMs or other long-stopping
services (e.g. app servers to flush in-flight operations, and only later
their databases) - more so with the probable storage I/O and power-draw
burst to flush out databases or hibernate those VMs.

In this case fiddling with upssched or setting up dummy-ups relays with an
override for defining earlier trigger of critical state (usually by battery
charge or time remaining) may fare better: your NUT primary server would
seem to serve several UPSes (the "real" device and a few dummies with
different "criticality" levels), and various secondary hosts would
MONITOR
the suitable dummy to begin their shutdown earlier into the outage. This
approach may also be useful for Dan's post :)

Jim

On Fri, Oct 27, 2023 at 4:55?PM Magnus Holmgren <
magnus.holmgren at milientsoftware.com> wrote:
> Hi, and thanks for this great piece of free software! I've been meaning
to
> sort this out for some time, but we don't get power outages that often,
> fortunately...
>
> So, correct me if I'm wrong, but from the documentation at https://
> networkupstools.org/docs/user-manual.chunked/
> Configuration_notes.html#UPS_shutdown, and also reading upsmon.c, when a
> UPS
> goes OB LB (assuming we have a single UPS connected to a primary and
> supplying
> power to the primary and some number of secondaries), the primary notifies
> the
> secondaries, the secondaries wait for FINALDELAY and then execute
> SHUTDOWNCMD
> immediately followed by exiting, thereby disconnecting from the primary,
> and
> the primary, after seeing all secondaries disconnect, proceed with its
> shutdown (only waiting for FINALDELAY), which ends with telling the UPS to
> cut
> the power (without delay too, right?).
>
> Again, correct me if I'm wrong, Is it only I who find this a bit
flawed? I
> would like for the secondaries to stay connected until they shut down. We
> have
> a server with a bunch of virtual machines on, and they can take a couple
> of
> minutes to shut down. Otherwise the primary can easily cut the power
> prematurely. Avoiding this, it seems, could pretty easily be accomplished
> by
> having upsmon wait, perhaps in a separate loop, for the INT/TERM/QUIT
> signal
> (it would still be necessary to configure the service manager such that
> upsmon
> is terminated as late as possible). The primary could start shutting down
> its
> services in the meantime, but upsmon would hold the poweroff until the
> secondaries have disconnected (or HOSTSYNC expires).
>
> Surely this would be better than cranking up FINALDELAY on the primary and
> always waiting for a fixed period of time, as suggested in
>
https://alioth-lists.debian.net/pipermail/nut-upsuser/2012-April/007550.html?
> I guess I could
> try writing a SHUTDOWNCMD script that doesn't exit until most other
> services
> have also done so, taking care not to create a deadlock situation.
>
> Another option would be to use upssched to shut down the "big
rig"
> earlier. It
> just seems unsatisfying to me that upssched is entirely time-based. It
> would
> be nice if it were easier to trigger off battery.charge or battery.runtime
> going below arbitrary values instead of just the on battery and low
> battery
> statuses.
>
> How do others solve this?
>
> --
> Magnus Holmgren
> ./?\_/?\. Milient
> (also holmgren at debian.org)
>
>
>
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20231027/9cc04311/attachment.htm>

Check it out now,
my NUT project braza!..

https://github.com/networkupstools/nut/pull/2133
https://github.com/networkupstools/nut/wiki/Building-NUT-for-in%E2%80%90place-upgrades-or-non%E2%80%90disruptive-tests

Jim


On Fri, Oct 27, 2023 at 8:07?PM Jim Klimov <jimklimov+nut at gmail.com>
wrote:
> Hi, this does sound like a useful idea - although for the principle of
> least surprise and for variation in deployments, I'd rather have it as
a
> (non-default state of a) configuration toggle that can be set via
> `upsmon.conf`: whether this particular client exits after processing FSD or
> not. The onus for the rest would be on general systems integration - e.g.
> ensure that init scripts `K*`ill the long-running services before they go
> after upsmon and upsd, or add a drop-in systemd config snippet for
> nut-monitor to not-conflict with "shutdown.target" (and half a
dozen of its
> equivalents for halt/reboot/poweroff/...), and possibly to break the
> shutdown-dependency between nut-monitor/nut-server/nut-driver units.
>
> On a related note - there was lately work to allow daemonized drivers to
> kill power of the UPS (may be useful especially for devices with long
> protocol init times), with a safety switch to flip about this and actually
> allow the driver to issue killpower commands. So stopping driver daemons
> might eventually be not needed - but I'm not sure any OS integrations
took
> note of this possibility yet. It was not officially released so far, just
> is in master branch.
>
> Note however that typically FSD happens when the power is critical.
> Definitions of that vary, as well as ability or not to set certain
> thresholds for when the device would emit (and a driver would relay) the
> low-battery condition. You might not physically have those 2 minutes worth
> of remaining battery charge to shut down the VMs or other long-stopping
> services (e.g. app servers to flush in-flight operations, and only later
> their databases) - more so with the probable storage I/O and power-draw
> burst to flush out databases or hibernate those VMs.
>
> In this case fiddling with upssched or setting up dummy-ups relays with an
> override for defining earlier trigger of critical state (usually by battery
> charge or time remaining) may fare better: your NUT primary server would
> seem to serve several UPSes (the "real" device and a few dummies
with
> different "criticality" levels), and various secondary hosts
would MONITOR
> the suitable dummy to begin their shutdown earlier into the outage. This
> approach may also be useful for Dan's post :)
>
> Jim
>
> On Fri, Oct 27, 2023 at 4:55?PM Magnus Holmgren <
> magnus.holmgren at milientsoftware.com> wrote:
>
>> Hi, and thanks for this great piece of free software! I've been
meaning
>> to
>> sort this out for some time, but we don't get power outages that
often,
>> fortunately...
>>
>> So, correct me if I'm wrong, but from the documentation at https://
>> networkupstools.org/docs/user-manual.chunked/
>> Configuration_notes.html#UPS_shutdown, and also reading upsmon.c, when
a
>> UPS
>> goes OB LB (assuming we have a single UPS connected to a primary and
>> supplying
>> power to the primary and some number of secondaries), the primary
>> notifies the
>> secondaries, the secondaries wait for FINALDELAY and then execute
>> SHUTDOWNCMD
>> immediately followed by exiting, thereby disconnecting from the
primary,
>> and
>> the primary, after seeing all secondaries disconnect, proceed with its
>> shutdown (only waiting for FINALDELAY), which ends with telling the UPS
>> to cut
>> the power (without delay too, right?).
>>
>> Again, correct me if I'm wrong, Is it only I who find this a bit
flawed?
>> I
>> would like for the secondaries to stay connected until they shut down.
We
>> have
>> a server with a bunch of virtual machines on, and they can take a
couple
>> of
>> minutes to shut down. Otherwise the primary can easily cut the power
>> prematurely. Avoiding this, it seems, could pretty easily be
accomplished
>> by
>> having upsmon wait, perhaps in a separate loop, for the INT/TERM/QUIT
>> signal
>> (it would still be necessary to configure the service manager such that
>> upsmon
>> is terminated as late as possible). The primary could start shutting
down
>> its
>> services in the meantime, but upsmon would hold the poweroff until the
>> secondaries have disconnected (or HOSTSYNC expires).
>>
>> Surely this would be better than cranking up FINALDELAY on the primary
>> and
>> always waiting for a fixed period of time, as suggested in
>>
https://alioth-lists.debian.net/pipermail/nut-upsuser/2012-April/007550.html?
>> I guess I could
>> try writing a SHUTDOWNCMD script that doesn't exit until most other
>> services
>> have also done so, taking care not to create a deadlock situation.
>>
>> Another option would be to use upssched to shut down the "big
rig"
>> earlier. It
>> just seems unsatisfying to me that upssched is entirely time-based. It
>> would
>> be nice if it were easier to trigger off battery.charge or
>> battery.runtime
>> going below arbitrary values instead of just the on battery and low
>> battery
>> statuses.
>>
>> How do others solve this?
>>
>> --
>> Magnus Holmgren
>> ./?\_/?\. Milient
>> (also holmgren at debian.org)
>>
>>
>>
>> _______________________________________________
>> Nut-upsuser mailing list
>> Nut-upsuser at alioth-lists.debian.net
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20231027/3755773f/attachment.htm>

fredag 27 oktober 2023 20:07:58 CET skrev  Jim Klimov:
> Hi, this does sound like a useful idea - although for the principle of
> least surprise and for variation in deployments, I'd rather have it as
a
> (non-default state of a) configuration toggle that can be set via
> `upsmon.conf`: whether this particular client exits after processing FSD or
> not. The onus for the rest would be on general systems integration - e.g.
> ensure that init scripts `K*`ill the long-running services before they go
> after upsmon and upsd, or add a drop-in systemd config snippet for
> nut-monitor to not-conflict with "shutdown.target" (and half a
dozen of its
> equivalents for halt/reboot/poweroff/...), and possibly to break the
> shutdown-dependency between nut-monitor/nut-server/nut-driver units.
Yes, I figure this is up to the distributions to get right.
> On a related note - there was lately work to allow daemonized drivers to
> kill power of the UPS (may be useful especially for devices with long
> protocol init times), with a safety switch to flip about this and actually
> allow the driver to issue killpower commands. So stopping driver daemons
> might eventually be not needed - but I'm not sure any OS integrations
took
> note of this possibility yet. It was not officially released so far, just
> is in master branch.
I don't understand what you're saying here. Aren't the drivers
daemonized and
isn't it the drivers that kill power already?
> Note however that typically FSD happens when the power is critical.
> Definitions of that vary, as well as ability or not to set certain
> thresholds for when the device would emit (and a driver would relay) the
> low-battery condition. You might not physically have those 2 minutes worth
> of remaining battery charge to shut down the VMs or other long-stopping
> services (e.g. app servers to flush in-flight operations, and only later
> their databases) - more so with the probable storage I/O and power-draw
> burst to flush out databases or hibernate those VMs.
Obviously I'll have to set the LB threshold to give sufficient margin, but i
figure it'd be better if the primary could shut down as soon as it's
safe to
do so rather than waiting for a fixed amount of time. Though I'd still need
to
ensure that the battery can last for as long as the primary will wait.
> In this case fiddling with upssched or setting up dummy-ups relays with an
> override for defining earlier trigger of critical state (usually by battery
> charge or time remaining) may fare better: your NUT primary server would
> seem to serve several UPSes (the "real" device and a few dummies
with
> different "criticality" levels), and various secondary hosts
would MONITOR
> the suitable dummy to begin their shutdown earlier into the outage. This
> approach may also be useful for Dan's post :)
Ah, that seems like a workable way of shutting down different machines based 
on different battery levels. One caveat though: The manual makes a significant 
point of the idea that when an FSD event occurs, everything should be shut 
down and the power cut so that the UPS later will power everything back up 
again, whereas if some consumers are shut down early, they won't
automatically
come back up (unless you arrange for WoL packets to be sent out, or 
something).
> Jim
> 
> On Fri, Oct 27, 2023 at 4:55?PM Magnus Holmgren <
> 
> magnus.holmgren at milientsoftware.com> wrote:
> > Hi, and thanks for this great piece of free software! I've been
meaning to
> > sort this out for some time, but we don't get power outages that
often,
> > fortunately...
> > 
> > So, correct me if I'm wrong, but from the documentation at
https://
> > networkupstools.org/docs/user-manual.chunked/
> > Configuration_notes.html#UPS_shutdown, and also reading upsmon.c, when
a
> > UPS
> > goes OB LB (assuming we have a single UPS connected to a primary and
> > supplying
> > power to the primary and some number of secondaries), the primary
notifies
> > the
> > secondaries, the secondaries wait for FINALDELAY and then execute
> > SHUTDOWNCMD
> > immediately followed by exiting, thereby disconnecting from the
primary,
> > and
> > the primary, after seeing all secondaries disconnect, proceed with its
> > shutdown (only waiting for FINALDELAY), which ends with telling the
UPS to
> > cut
> > the power (without delay too, right?).
> > 
> > Again, correct me if I'm wrong, Is it only I who find this a bit
flawed? I
> > would like for the secondaries to stay connected until they shut down.
We
> > have
> > a server with a bunch of virtual machines on, and they can take a
couple
> > of
> > minutes to shut down. Otherwise the primary can easily cut the power
> > prematurely. Avoiding this, it seems, could pretty easily be
accomplished
> > by
> > having upsmon wait, perhaps in a separate loop, for the INT/TERM/QUIT
> > signal
> > (it would still be necessary to configure the service manager such
that
> > upsmon
> > is terminated as late as possible). The primary could start shutting
down
> > its
> > services in the meantime, but upsmon would hold the poweroff until the
> > secondaries have disconnected (or HOSTSYNC expires).
> > 
> > Surely this would be better than cranking up FINALDELAY on the primary
and
> > always waiting for a fixed period of time, as suggested in
> >
https://alioth-lists.debian.net/pipermail/nut-upsuser/2012-April/007550.ht
> > ml? I guess I could
> > try writing a SHUTDOWNCMD script that doesn't exit until most
other
> > services
> > have also done so, taking care not to create a deadlock situation.
> > 
> > Another option would be to use upssched to shut down the "big
rig"
> > earlier. It
> > just seems unsatisfying to me that upssched is entirely time-based. It
> > would
> > be nice if it were easier to trigger off battery.charge or
battery.runtime
> > going below arbitrary values instead of just the on battery and low
> > battery
> > statuses.
> > 
> > How do others solve this?
-- 
Magnus Holmgren
./?\_/?\. Milient