Nut upsuser - Aug 2020 - Synology NAS is shutting down Ubuntu servers after very brief power outage

I'm hoping that someone can shed some light on this… I have a Synology NAS
(DS416) that has a feature where you can enable a “Network UPS Server” which is
a NUT server.  I have been trying to get the Synology to shut down three Ubuntu
20.04 servers that I have. While it does work when I test it out manually,
sometimes when I am away and the power goes out briefly, the servers shut down
when the power has been out for like five seconds (or less sometimes).  When I
come home and turn on the servers, they boot up but then immediately shut down
again.  This happens until I restart the Synology; then they will boot up and
stay up.  I‘d like them to stay up a little longer than that.  Ideally, I’d like
them to stay up until battery is low, then shut down, then all come back on when
power is restored.

This is what I have done so far:  I have enabled "Enable Network UPS
server" on the Synology and have installed NUT on each of my servers
running Ubuntu 20.04.  I have added the appropriate IPs to the Permitted
DiskStation Devices” list.  I have also tried setting it on the Synology to shut
down when battery is low and after a specified amount of time (20
minutes).  Either way, the servers will shut down after like 5 seconds or
less.  I have edited upsmon.conf and added my MONITOR line and setup systemctl
so that the nut-client service starts automatically at boot.  I have no made any
other changes to the file; the rest is still set to defaults.

So I'm not sure where exactly the problem is; if it's the Synology or
NUT on Ubuntu.  Strange thing is, when I manually test by shutting off the power
briefly (or for a few seconds, or a few minutes - I've tried everything;)
every time I do a test, everything works perfect and they will shut down when
they are supposed to.  Seems to only happen when there is a passing storm that
knocks the power out for a few seconds.

Also, this is what I found in the syslog on one of the machines:
Jul 31 18:33:29 plex upsmon[970]: UPS ups at 192.168.1.70 on battery
Jul 31 18:33:34 plex upsmon[970]: UPS ups at 192.168.1.70 on line power
Jul 31 18:34:04 plex upsmon[970]: Executing automatic power-fail shutdown
Jul 31 18:34:04 plex upsmon[970]: Auto logout and shutdown proceeding
Jul 31 18:34:09 plex systemd[1]: nut-monitor.service: Succeeded.

If I’m not mistaken, it is shutting down after power came back on line….?

Any help would be greatly appreciated!

Details:  Nut version 2.7.4-11ubuntu4, Ubuntu 20.04, NUT installed from package,
Synology DS416, UPS is APC Back-UPS NS 650M1

--
Todd Benivegna // todd at benivegna.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200801/35a49ee6/attachment.html>

On Sat, 1 Aug 2020, Todd Benivegna wrote:
> Also, this is what I found in the syslog on one of the machines:
> Jul 31 18:33:29 plex upsmon[970]: UPS ups at 192.168.1.70 on battery
> Jul 31 18:33:34 plex upsmon[970]: UPS ups at 192.168.1.70 on line power
> Jul 31 18:34:04 plex upsmon[970]: Executing automatic power-fail shutdown
> Jul 31 18:34:04 plex upsmon[970]: Auto logout and shutdown proceeding
> Jul 31 18:34:09 plex systemd[1]: nut-monitor.service: Succeeded.
I suspect that getting to the bottom of this will need careful tracing of all 
the status changes reported by the upsd instance running in the NAS.  It would 
be helpful if your upsmon.conf had NOTIFYMSG entries for all possible events, 
with SYSLOG options set in the corresponding NOTIFYFLAG entries.

upsmon.conf doesn't support reporting battery charge in the notification 
messages, but you can specify multiple shell commands in the SHUTDOWNCMD entry. 
E.g. `upsc ups at 192.168.1.70 battery.charge` and `upsc ups at 192.168.1.70 
ups.status`.  This will show what the situation was at that moment.

The question that always gets asked: how old are the batteries?

Roger

Roger,

Thank you for the info. I will look into that and set that up ASAP.

Regarding the battery age, I’m not too it’d but I’m guessing approx 3-4 years.
You think it’d be worth replacing, in light of this issue? Run time still seems
Ok ~45 mins, but maybe the load when it switches over to battery may be too
much?

--
Todd Benivegna // todd at benivegna.com
On Aug 2, 2020, 9:38 AM -0400, Roger Price <roger at rogerprice.org>,
wrote:
> On Sat, 1 Aug 2020, Todd Benivegna wrote:
>
> > Also, this is what I found in the syslog on one of the machines:
> > Jul 31 18:33:29 plex upsmon[970]: UPS ups at 192.168.1.70 on battery
> > Jul 31 18:33:34 plex upsmon[970]: UPS ups at 192.168.1.70 on line
power
> > Jul 31 18:34:04 plex upsmon[970]: Executing automatic power-fail
shutdown
> > Jul 31 18:34:04 plex upsmon[970]: Auto logout and shutdown proceeding
> > Jul 31 18:34:09 plex systemd[1]: nut-monitor.service: Succeeded.
>
> I suspect that getting to the bottom of this will need careful tracing of
all
> the status changes reported by the upsd instance running in the NAS. It
would
> be helpful if your upsmon.conf had NOTIFYMSG entries for all possible
events,
> with SYSLOG options set in the corresponding NOTIFYFLAG entries.
>
> upsmon.conf doesn't support reporting battery charge in the
notification
> messages, but you can specify multiple shell commands in the SHUTDOWNCMD
entry.
> E.g. `upsc ups at 192.168.1.70 battery.charge` and `upsc ups at
192.168.1.70
> ups.status`. This will show what the situation was at that moment.
>
> The question that always gets asked: how old are the batteries?
>
> Roger
>
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200802/02c1d563/attachment.html>

On 8/1/20 11:25 PM, Todd Benivegna wrote:
> I'm hoping that someone can shed some light on this… I have a Synology 
> NAS (DS416) that has a feature where you can enable a “Network UPS 
> Server” which is a NUT server.  I have been trying to get the Synology 
> to shut down three Ubuntu 20.04 servers that I have. While it does 
> work when I test it out manually, sometimes when I am away and the 
> power goes out briefly, the servers shut down when the power has been 
> out for like five seconds (or less sometimes).  When I come home and 
> turn on the servers, they boot up but then immediately shut down 
> again.  This happens until I restart the Synology; then they will boot 
> up and stay up.  I‘d like them to stay up a little longer than 
> that.  Ideally, I’d like them to stay up until battery is low, then 
> shut down, then all come back on when power is restored.
>
> This is what I have done so far:  I have enabled "Enable Network UPS 
> server" on the Synology and have installed NUT on each of my servers 
> running Ubuntu 20.04.  I have added the appropriate IPs to the 
> Permitted DiskStation Devices” list.  I have also tried setting it on 
> the Synology to shut down when battery is low and after a specified 
> amount of time (20 minutes).  Either way, the servers will shut down 
> after like 5 seconds or less.  I have edited upsmon.conf and added my 
> MONITOR line and setup systemctl so that the nut-client service starts 
> automatically at boot.  I have no made any other changes to the file; 
> the rest is still set to defaults.
>
> So I'm not sure where exactly the problem is; if it's the Synology
or
> NUT on Ubuntu.  Strange thing is, when I manually test by shutting off 
> the power briefly (or for a few seconds, or a few minutes - I've tried 
> everything;) every time I do a test, everything works perfect and they 
> will shut down when they are supposed to.  Seems to only happen when 
> there is a passing storm that knocks the power out for a few seconds.
>
> Also, this is what I found in the syslog on one of the machines:
> Jul 31 18:33:29 plex upsmon[970]: UPS ups at 192.168.1.70 on battery
> Jul 31 18:33:34 plex upsmon[970]: UPS ups at 192.168.1.70 on line power
> Jul 31 18:34:04 plex upsmon[970]: Executing automatic power-fail shutdown
> Jul 31 18:34:04 plex upsmon[970]: Auto logout and shutdown proceeding
> Jul 31 18:34:09 plex systemd[1]: nut-monitor.service: Succeeded.
>
> If I’m not mistaken, it is shutting down after power came back on line….?
My first suspect is the Synology version of nut. More specifically, I 
suspect that nut triggers a shutdown immediately after the switch to "on 
battery" state and only cancels it after a restart.

What I would do is to use wireshark to sniff the communication between 
the Synology and one of the Ubuntu machines and log what exactly ( and 
when ) is sent by the nut server to the clients. I'd also couple that 
with a bit of logging on the nut side 
<https://networkupstools.org/docs/man/upslog.html> as well ( for 60 secs 
or so, starting shortly before the switch to on-battery, with a very 
short interval between polls )

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200802/5cc7bd5e/attachment.html>

On Sun, 2 Aug 2020, Manuel Wolfshant wrote:
> My first suspect is the Synology version of nut.
How much of NUT is installed in the NAS?  All of upsd + upsmon + upssched + 
upssched-cmd?  Are the configuratiion files accessible and modifiable?

Is is possible to run BASH scripts on the NAS?  If so it would be interesting to
try to run the script at http://rogerprice.org/NUT/nut-report which reports on a
NUT configuration.

Roger

How would I capture with Wireshark when it seemingly happens at
random?  Whenever I test, everything appears to work normally.  I wish I could
somehow replicate it.  Maybe I need to test manually and see if it’ll happen in
front of me; I’ve never actually seen it happen… always happens when I’m away.

--
Todd Benivegna // todd at benivegna.com
On Aug 2, 2020, 12:12 PM -0400, Manuel Wolfshant <wolfy at
nobugconsulting.ro>, wrote:
> On 8/1/20 11:25 PM, Todd Benivegna wrote:
> > I'm hoping that someone can shed some light on this… I have a
Synology NAS (DS416) that has a feature where you can enable a “Network UPS
Server” which is a NUT server.  I have been trying to get the Synology to shut
down three Ubuntu 20.04 servers that I have. While it does work when I test it
out manually, sometimes when I am away and the power goes out briefly, the
servers shut down when the power has been out for like five seconds (or less
sometimes).  When I come home and turn on the servers, they boot up but then
immediately shut down again.  This happens until I restart the Synology; then
they will boot up and stay up.  I‘d like them to stay up a little longer than
that.  Ideally, I’d like them to stay up until battery is low, then shut down,
then all come back on when power is restored.
> >
> > This is what I have done so far:  I have enabled "Enable Network
UPS server" on the Synology and have installed NUT on each of my servers
running Ubuntu 20.04.  I have added the appropriate IPs to the Permitted
DiskStation Devices” list.  I have also tried setting it on the Synology to shut
down when battery is low and after a specified amount of time (20
minutes).  Either way, the servers will shut down after like 5 seconds or
less.  I have edited upsmon.conf and added my MONITOR line and setup systemctl
so that the nut-client service starts automatically at boot.  I have no made any
other changes to the file; the rest is still set to defaults.
> >
> > So I'm not sure where exactly the problem is; if it's the
Synology or NUT on Ubuntu.  Strange thing is, when I manually test by shutting
off the power briefly (or for a few seconds, or a few minutes - I've tried
everything;) every time I do a test, everything works perfect and they will shut
down when they are supposed to.  Seems to only happen when there is a passing
storm that knocks the power out for a few seconds.
> >
> > Also, this is what I found in the syslog on one of the machines:
> > Jul 31 18:33:29 plex upsmon[970]: UPS ups at 192.168.1.70 on battery
> > Jul 31 18:33:34 plex upsmon[970]: UPS ups at 192.168.1.70 on line
power
> > Jul 31 18:34:04 plex upsmon[970]: Executing automatic power-fail
shutdown
> > Jul 31 18:34:04 plex upsmon[970]: Auto logout and shutdown proceeding
> > Jul 31 18:34:09 plex systemd[1]: nut-monitor.service: Succeeded.
> >
> > If I’m not mistaken, it is shutting down after power came back on
line….?
> My first suspect is the Synology version of nut. More specifically, I
suspect that nut triggers a shutdown immediately after the switch to "on
battery" state and only cancels it after a restart.
> What I would do is to use wireshark to sniff the communication between the
Synology and one of the Ubuntu machines and log what exactly ( and when ) is
sent by the nut server to the clients. I'd also couple that with a bit of
logging on the nut side as well ( for 60 secs or so, starting shortly before the
switch to on-battery, with a very short interval between polls )
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200802/10529a42/attachment-0001.html>

Roger,

How would you write the SHUTDOWNCMD line with multiple commands?  I’ve been
looking at the manual and see that you have to escape the internal “ but am
still a little confused on how to do this exactly.

--
Todd Benivegna // todd at benivegna.com
On Aug 2, 2020, 9:38 AM -0400, Roger Price <roger at rogerprice.org>,
wrote:
>
> upsmon.conf doesn't support reporting battery charge in the
notification
> messages, but you can specify multiple shell commands in the SHUTDOWNCMD
entry.
> E.g. `upsc ups at 192.168.1.70 battery.charge` and `upsc ups at
192.168.1.70
> ups.status`. This will show what the situation was at that moment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200802/92f586d5/attachment.html>