Nut upsuser - Aug 2020 - Synology NAS is shutting down Ubuntu servers after very brief power outage (fwd)

On Fri, 7 Aug 2020, Todd Benivegna wrote:
> APC Back-UPS NS 650M1 UPS ---USB---> Synology NAS (DS416 - Master?) 
> ---Ethernet---> Netgear Managed Switch w/ uplink to router
<---Ethernet---
> Servers (Ubuntu 20.04 - Plex, Pulsar, Proton - All three set as slaves)
I'm guessing that the UPS supplies only the NAS, not the 3 Ubuntu machines. 
Do
they have their own UPS's?
> I have all three servers set as slaves, so is the Synology considered the 
> master?  Or do I need to set one of the servers as the Master?  I've
been
> under the impression that the Synology is the master, but have been unable
to
> confirm this.
I've been looking at the Synology documentation and their NUT setup is not
at
all clear.  Some of their site is nonsense.  I gather from other sites that the 
NAS is indeed the master and that upsmon runs in the NAS. This makes sense if 
the UPS is for the NAS and nothing else.  It also simplifies shutdown if NAS 
users mount NFS supplied directories in the NAS.
> So what I have done so far is enable the "Network UPS Server" on
the Synology,
> entered the three IPs of the servers in there, set it to shutdown when
battery
> is low and enabled "Shutdown UPS when the system enters safe
mode".  I then
> installed NUT on all three servers. In nut.conf I changed MODE to 
> "MODE=netclient".  I then added my MONITOR line in upsmon.conf.
on all three. 
> Looks something like this:  
> 
> MONITOR ups at 192.168.1.70 1 monuser secret slave
> 
> My SHUTDOWNCMD looks like this:
> SHUTDOWNCMD "getUPSstatus ups at 192.168.1.70 ; logger -t upsmon.conf
\"UPS status is $UPSstatus\" ; /sbin/shutdown -h +0"  
>From your previous reports it looks as if getUPSstatus does not work in a 
SHUTDOWNCMD declaration since the shell variable it creates gets lost. It 
probably better to declare something like

  SHUTDOWNCMD "logger -t upsmon.conf \"UPS status [$( upsc ups at
192.168.1.70 ups.status )]:$( upsc ups at 192.168.1.70 )\" ; /sbin/shutdown
-h +0"  
> I think I've definitely made a mistake though, in that I have not
set RUN_AS_USER in upsmon.conf
The default user is usually set when NUT is built for a specific Linux 
distribution.  I don't know what user Ubuntu have chosen, but I will guess
that
they have followed Debian and use "nut".  I suggest you do not change
this.
> and set up the appropriate permissions.
Again, I assume Ubuntu build NUT with the correct file permissions for their 
default user.

Roger

On Sat, 8 Aug 2020, Roger Price wrote:

Better:

  SHUTDOWNCMD "logger -t upsmon.conf \"UPS status [$( upsc ups at
192.168.1.70 ups.status )]:$( upsc ups at 192.168.1.70 battery.charge )\" ;
/sbin/shutdown -h +0"

I forgot the battery.charge.  Roger

Roger,

Ok, so how does this look...

Updated upsmon.conf:  https://hastebin.com/jisinaquso.pl
> I'm guessing that the UPS supplies only the NAS, not the 3 Ubuntu
machines. Do
> they have their own UPS's?
No, the Synology and the three servers are all on the one UPS (also my switch
and spare monitor). All these are super low power devices (two Intel NUCs and a
Raspberry Pi) so at idle the draw like 50-75w and at max load it’s like 100-150w
tops.  UPS is rated for 300w.
> Better:
>
>  SHUTDOWNCMD "logger -t upsmon.conf \"UPS status [$( upsc ups at
192.168.1.70 ups.status )]:$( upsc ups at 192.168.1.70 battery.charge )\" ;
/sbin/shutdown -h +0"
>
> I forgot the battery.charge. Roger
I changed it to this, thanks.

So does everything look good with my config files?  Any thoughts on what may be
going on here?  I really wish I could reproduce this, I think that’d make this a
lot easier to troubleshoot.

Thanks,

Todd

--
Todd Benivegna // todd at benivegna.com
On Aug 8, 2020, 5:12 AM -0400, Roger Price <roger at rogerprice.org>,
wrote:
> On Fri, 7 Aug 2020, Todd Benivegna wrote:
>
> > APC Back-UPS NS 650M1 UPS ---USB---> Synology NAS (DS416 - Master?)
> > ---Ethernet---> Netgear Managed Switch w/ uplink to router
<---Ethernet---
> > Servers (Ubuntu 20.04 - Plex, Pulsar, Proton - All three set as
slaves)
>
> I'm guessing that the UPS supplies only the NAS, not the 3 Ubuntu
machines. Do
> they have their own UPS's?
>
> > I have all three servers set as slaves, so is the Synology
considered the
> > master?  Or do I need to set one of the servers as the Master? 
I've been
> > under the impression that the Synology is the master, but have been
unable to
> > confirm this.
>
> I've been looking at the Synology documentation and their NUT setup is
not at
> all clear. Some of their site is nonsense. I gather from other sites that
the
> NAS is indeed the master and that upsmon runs in the NAS. This makes sense
if
> the UPS is for the NAS and nothing else. It also simplifies shutdown if NAS
> users mount NFS supplied directories in the NAS.
>
> > So what I have done so far is enable the "Network UPS
Server" on the Synology,
> > entered the three IPs of the servers in there, set it to shutdown when
battery
> > is low and enabled "Shutdown UPS when the system enters safe
mode".  I then
> > installed NUT on all three servers. In nut.conf I changed MODE to
> > "MODE=netclient".  I then added my MONITOR line in
upsmon.conf. on all three.
> > Looks something like this:
> >
> > MONITOR ups at 192.168.1.70 1 monuser secret slave
> >
> > My SHUTDOWNCMD looks like this:
> > SHUTDOWNCMD "getUPSstatus ups at 192.168.1.70 ; logger -t
upsmon.conf \"UPS status is $UPSstatus\" ; /sbin/shutdown -h +0"
>
> From your previous reports it looks as if getUPSstatus does not work in a
> SHUTDOWNCMD declaration since the shell variable it creates gets lost. It
> probably better to declare something like
>
> SHUTDOWNCMD "logger -t upsmon.conf \"UPS status [$( upsc ups at
192.168.1.70 ups.status )]:$( upsc ups at 192.168.1.70 )\" ; /sbin/shutdown
-h +0"
>
> > I think I've definitely made a mistake though, in that I have not
set RUN_AS_USER in upsmon.conf
>
> The default user is usually set when NUT is built for a specific Linux
> distribution. I don't know what user Ubuntu have chosen, but I will
guess that
> they have followed Debian and use "nut". I suggest you do not
change this.
>
> > and set up the appropriate permissions.
>
> Again, I assume Ubuntu build NUT with the correct file permissions for
their
> default user.
>
> Roger
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200808/d40ac623/attachment.html>

Roger,

I just ran a manual test, killing power and see what happens.  I set the
Synology “Time before DiskStation goes into Safe mode” to 5 minutes so I didn’t
have to wait like an hour until it powered down.  Here is the log:

https://hastebin.com/ovuwilufeb.sql

Everything appeared to be normal; the servers powered off and the Synology went
into safe mode.  Power was then cut to the Synology and my UPS turned off.  I
waited a couple minutes, restored power and the UPS came on, the Synology and
all three servers powered on too.  Everything appears to be perfect.  I just
don’t understand why in a real power loss situation, that this same thing does
not apparently happen.  Why would they shut down if power was only out for a
second?

Could it be something like the Synology drives were in hibernation and the
Synology wasn’t responding (was probably coming out of hibernation - it takes
maybe 5-10 seconds) and NUT thought the server was dead and shut everything
down?  On that note, I did have the Synology set to hibernate/spin down the
disks after 1 hour, but just disabled that just in case; will be on all the time
now.  You think that was possibly the problem?

--
Todd Benivegna // todd at benivegna.com
On Aug 8, 2020, 2:12 PM -0400, Todd Benivegna <todd at benivegna.com>,
wrote:
> Roger,
>
> Ok, so how does this look...
>
> Updated upsmon.conf:  https://hastebin.com/jisinaquso.pl
> > I'm guessing that the UPS supplies only the NAS, not the 3 Ubuntu
machines. Do
> > they have their own UPS's?
> No, the Synology and the three servers are all on the one UPS (also my
switch and spare monitor). All these are super low power devices (two Intel NUCs
and a Raspberry Pi) so at idle the draw like 50-75w and at max load it’s like
100-150w tops.  UPS is rated for 300w.
> > Better:
> >
> >  SHUTDOWNCMD "logger -t upsmon.conf \"UPS status [$(
upsc ups at 192.168.1.70 ups.status )]:$( upsc ups at
192.168.1.70 battery.charge )\" ; /sbin/shutdown -h +0"
> >
> > I forgot the battery.charge. Roger
> I changed it to this, thanks.
>
> So does everything look good with my config files?  Any thoughts on what
may be going on here?  I really wish I could reproduce this, I think that’d make
this a lot easier to troubleshoot.
>
> Thanks,
>
> Todd
>
> --
> Todd Benivegna // todd at benivegna.com
> On Aug 8, 2020, 5:12 AM -0400, Roger Price <roger at rogerprice.org>,
wrote:
> > On Fri, 7 Aug 2020, Todd Benivegna wrote:
> >
> > > APC Back-UPS NS 650M1 UPS ---USB---> Synology NAS (DS416 -
Master?)
> > > ---Ethernet---> Netgear Managed Switch w/ uplink to router
<---Ethernet---
> > > Servers (Ubuntu 20.04 - Plex, Pulsar, Proton - All three set as
slaves)
> >
> > I'm guessing that the UPS supplies only the NAS, not the 3 Ubuntu
machines. Do
> > they have their own UPS's?
> >
> > > I have all three servers set as slaves, so is the Synology
considered the
> > > master?  Or do I need to set one of the servers as the Master? 
I've been
> > > under the impression that the Synology is the master, but have
been unable to
> > > confirm this.
> >
> > I've been looking at the Synology documentation and their NUT
setup is not at
> > all clear. Some of their site is nonsense. I gather from other sites
that the
> > NAS is indeed the master and that upsmon runs in the NAS. This makes
sense if
> > the UPS is for the NAS and nothing else. It also simplifies shutdown
if NAS
> > users mount NFS supplied directories in the NAS.
> >
> > > So what I have done so far is enable the "Network UPS
Server" on the Synology,
> > > entered the three IPs of the servers in there, set it to shutdown
when battery
> > > is low and enabled "Shutdown UPS when the system enters safe
mode".  I then
> > > installed NUT on all three servers. In nut.conf I changed MODE to
> > > "MODE=netclient".  I then added my MONITOR line in
upsmon.conf. on all three.
> > > Looks something like this:
> > >
> > > MONITOR ups at 192.168.1.70 1 monuser secret slave
> > >
> > > My SHUTDOWNCMD looks like this:
> > > SHUTDOWNCMD "getUPSstatus ups at 192.168.1.70 ; logger -t
upsmon.conf \"UPS status is $UPSstatus\" ; /sbin/shutdown -h +0"
> >
> > From your previous reports it looks as if getUPSstatus does not work
in a
> > SHUTDOWNCMD declaration since the shell variable it creates gets lost.
It
> > probably better to declare something like
> >
> > SHUTDOWNCMD "logger -t upsmon.conf \"UPS status [$( upsc ups
at 192.168.1.70 ups.status )]:$( upsc ups at 192.168.1.70 )\" ;
/sbin/shutdown -h +0"
> >
> > > I think I've definitely made a mistake though, in that I have
not set RUN_AS_USER in upsmon.conf
> >
> > The default user is usually set when NUT is built for a specific Linux
> > distribution. I don't know what user Ubuntu have chosen, but I
will guess that
> > they have followed Debian and use "nut". I suggest you do
not change this.
> >
> > > and set up the appropriate permissions.
> >
> > Again, I assume Ubuntu build NUT with the correct file permissions for
their
> > default user.
> >
> > Roger
> > _______________________________________________
> > Nut-upsuser mailing list
> > Nut-upsuser at alioth-lists.debian.net
> > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200808/61af6e24/attachment-0001.html>

On Sat, 8 Aug 2020, Todd Benivegna wrote:
> Updated upsmon.conf:  https://hastebin.com/jisinaquso.pl
You should remove line 1 : RUN_AS_USER nut

Lines 12-21: it would be possible to display more information, such as the UPS 
status, but this requires pointing NOTIFYCMD to upssched and then using 
a upssched-cmd script to display details of each event.
> No, the Synology and the three servers are all on the one UPS (also my
switch
> and spare monitor). All these are super low power devices (two Intel NUCs
and
> a Raspberry Pi) so at idle the draw like 50-75w and at max load it’s like 
> 100-150w tops.  UPS is rated for 300w.
In that case, nothing stops you from using one of the "p" Ubuntu
machines as the
master with the Synology NAS as a slave. See 
https://community.synology.com/enu/forum/1/post/119995

Roger