Nut upsuser - Aug 2020 - Synology NAS is shutting down Ubuntu servers after very brief power outage (fwd)

upsmon.conf on server:  https://pastebin.com/z4CrUTxb
nut.conf on server:  https://pastebin.com/540ShZH7
Permissions for /etc/nut:  https://hastebin.com/qecolodapi.diff

On the Synology (I didn’t edit any of these files):
ups.conf:  https://hastebin.com/dedereqizi.shell
upsd.conf:  https://hastebin.com/pupeseweda.css
upsd.users:  https://hastebin.com/ocenamecex.cs

I don’t think I am able to run that script.  If you can, I wouldn’t know how.

Please let me know what you think.

Thanks for all the help!

Todd

--
Todd Benivegna // todd at benivegna.com
On Aug 7, 2020, 12:14 PM -0400, Todd Benivegna <todd at benivegna.com>,
wrote:
> Hi Roger,
>
> I am not home, but when I do get home I will check out everything
you've mentioned.  I do have some time where I can give you a breakdown of
my topology though.  I have a feeling all of this is probably due to a
configuration error somewhere on my part.   Here's what I have done so far.
>
> APC Back-UPS NS 650M1 UPS ---USB---> Synology NAS (DS416 - Master?)
---Ethernet---> Netgear Managed Switch w/ uplink to router <---Ethernet---
Servers (Ubuntu 20.04 - Plex, Pulsar, Proton - All three set as slaves)
>
> I have all three servers set as slaves, so is the Synology considered the
master?  Or do I need to set one of the servers as the Master?  I've been
under the impression that the Synology is the master, but have been unable to
confirm this.
>
> So what I have done so far is enable the "Network UPS Server" on
the Synology, entered the three IPs of the servers in there, set it to shutdown
when battery is low and enabled "Shutdown UPS when the system enters safe
mode".  I then installed NUT on all three servers. In nut.conf I changed
MODE to "MODE=netclient".  I then added my MONITOR line in
upsmon.conf. on all three.  Looks something like this:
>
> MONITOR ups at 192.168.1.70 1 monuser secret slave
>
> My SHUTDOWNCMD looks like this:
>
> SHUTDOWNCMD "getUPSstatus ups at 192.168.1.70 ; logger -t upsmon.conf
\"UPS status is $UPSstatus\" ; /sbin/shutdown -h +0"
>
> Would that not work?  I believe that's what you told me to put there.
>
> I think I've definitely made a mistake though, in that I have not
set RUN_AS_USER in upsmon.conf and set up the appropriate permissions.  I read
somewhere that it would then run as user NOBODY and not be able read
upsmon.conf.  Is that true?  Could that be the whole problem?
>
> I'm thinking I need to put in..
>
> RUN_AS_USER nut
>
> in upsmon.conf and then do:
>
> $ sudo chown -R root:nut /etc/nut
> $ sudo chmod 0770 /etc/nut
> $ sudo chmod 0640 /etc/nut/*
>
> Does that last part look correct to you?  I think what I will do is put the
contents of the files in pastebin and send the links to you.  May be easier that
way.
>
> Thanks,
>
> Todd
>
> > On Fri, Aug 7, 2020 at 5:47 AM Roger Price <roger at
rogerprice.org> wrote:
> > > On Thu, 6 Aug 2020, Todd Benivegna wrote:
> > >
> > > > ... I grep’d the syslog and here’s the results:
> > >
> > > Could you also grep for upsd and upsmon in the NAS log?  Is this
possible?
> > >
> > > >       proton at proton:~$ sudo grep upsmon /var/log/syslog
> > > >       Aug  6 19:19:09 proton upsmon[1552]: UPS ups at
192.168.1.70 on battery
> > > >       Aug  6 19:19:14 proton upsmon[1552]: UPS ups at
192.168.1.70 on line power
> > > >       Aug  6 19:19:44 proton upsmon[1552]: UPS ups at
192.168.1.70: forced shutdown in progress
> > > >       Aug  6 19:19:44 proton upsmon[1552]: Executing
automatic power-fail shutdown
> > >
> > > So proton is a slave, and is being made to shut down.
> > >
> > > >       Aug  6 19:19:45 proton upsmon[1552]: Auto logout and
shutdown proceeding
> > > >       Aug  6 19:19:50 proton upsmon.conf: UPS status is
> > >
> > > The variable USPstatus set by getUPSstatus was not in the
environment for the
> > > echo command.  To get this to work, you'll have to put the
upsc commands
> > > directly in SHUTDOWNCMD, or create a short script and call the
script in
> > > SHUTDOWNCMD.
> > >
> > > >       Aug  6 19:19:59 proton upsmon[1545]: UPS: ups at
192.168.1.70 (slave) (power value 1)
> > >
> > > Could you tell us your topology?  Which system is master and
which is slave?
> > > Do you have multiple slaves?  The decision to shutdown is taken
by the master,
> > > not the slave.
> > >
> > > Is it possible to run script http://rogerprice.org/NUT/nut-report
on the NAS?
> > > Could you run the script on one of the slaves and post the result
here?
> > >
> > > Roger_______________________________________________
> > > Nut-upsuser mailing list
> > > Nut-upsuser at alioth-lists.debian.net
> > >
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>
>
> --
> Todd Benivegna // todd at benivegna.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200807/664644b8/attachment-0001.html>

On Fri, 7 Aug 2020, Todd Benivegna wrote:

Hello Todd, sorry for the delay replying, I was away for a while. It's ok to
post configuration files in this list if blank lines and comments are removed.
> On the Synology (I didn’t edit any of these files):
> ups.conf:  https://hastebin.com/dedereqizi.shell
pollinterval = 5
[ups]
 	driver = usbhid-ups
 	port = auto
> upsd.conf:  https://hastebin.com/pupeseweda.css
LISTEN 192.168.1.70
LISTEN 2601:cf:8200:43d0:211:32ff:fe63:60d7
LISTEN fe80::211:32ff:fe63:60d7
LISTEN 127.0.0.1
LISTEN ::1
> upsd.users:  https://hastebin.com/ocenamecex.cs
[monuser]
 		password = secret
 		upsmon master

Since it's NUT in the NAS which is deciding to order the shutdown, it would
be
useful to see upsmon.conf, upssched.conf and upssched-cmd (or whatever Synology 
use if anything) from the NAS to see what criteria they use.

Roger

On Fri, 7 Aug 2020, Todd Benivegna wrote:
> I don’t think I am able to run that script.  If you can, I wouldn’t know
how.
You have to activate SSH on the NAS, log in to the NAS as root (admin) and 
create a temporary directory.  Based on what I read on a french language site, 
the commands are:

  mkdir /volume1/tmp
  cd /volume1/tmp

You can then install the script and run it:

  wget http://rogerprice.org/NUT/nut-report
  ./nut-report

Roger

> Since it's NUT in the NAS which is deciding to order the shutdown, it
would be
> useful to see upsmon.conf, upssched.conf and upssched-cmd (or whatever
Synology
> use if anything) from the NAS to see what criteria they use.
upsmon.conf (on Synology):
RUN_AS_USER root
MONITOR ups at localhost 1 monuser secret master
MINSUPPLIES 1
SHUTDOWNCMD ""
NOTIFYCMD /usr/sbin/upssched
POLLFREQ 5
POLLFREQALERT 5
HOSTSYNC 15
DEADTIME 15
POWERDOWNFLAG /etc/killpower
NOTIFYFLAG ONLINE EXEC
NOTIFYFLAG ONBATT EXEC
NOTIFYFLAG LOWBATT EXEC
NOTIFYFLAG NOCOMM EXEC
NOTIFYFLAG COMMBAD IGNORE
NOTIFYFLAG COMMOK IGNORE
NOTIFYFLAG SHUTDOWN IGNORE
NOTIFYFLAG FSD EXEC
NOTIFYFLAG NOPARENT SYSLOG
RBWARNTIME 43200
NOCOMMWARNTIME 300
FINALDELAY 5

upssched.conf (on Synology):
CMDSCRIPT /usr/syno/bin/synoups
PIPEFN /var/run/upssched.pipe
LOCKFN /var/run/upssched.lock
AT ONLINE * EXECUTE online
AT ONLINE * CANCEL-TIMER fsd
AT LOWBATT * EXECUTE lowbatt
AT NOCOMM * EXECUTE nocomm
AT FSD * EXECUTE fsd
AT ONBATT * EXECUTE onbatt

upssched-cmd (on Synology):
I don’t see this file.  Here are all the files in the ups folder:
nutscan-usb.h  ups.conf  upsd.conf  upsd.users  upsmon.conf  upssched.conf
> What is "Safe mode"?  Is it complete power down?, or some sort of
hibernation?
> If it's not a complete power down, how is the hibernation powered?
	"When the Synology NAS enters Safe Mode, it stops all services and
unmounts volumes in order to prevent data loss and shut down (halt for EDS14)
safely when the UPS device runs out of power. By default, the system enters Safe
Mode when the UPS device starts running low on power. You can also specify the
amount of time before the Synology NAS enters Safe Mode when power failure
occurs. However, if the UPS device reaches low battery before the specified
time, the system enters Safe Mode immediately.

	In situations where the Synology NAS shuts down during Safe Mode, it will
automatically turn on when power is restored if you have enabled the Restart
automatically after a power failure option (located at Control Panel >
Hardware & Power > General)."

I gather this puts it in a state where it is ready for imminent power loss and
won’t damage or lose any data.  I have enabled “Shutdown UPS when the system
enter Safe Mode” and I have also enabled “Restart automatically after a power
failure” as noted above.  I have always had these two options enabled.

Yesterday I recently disabled HDD Advanced Hibernation option as I thought that
may interfere with NUT on my servers being able to communicate with the NUT
server on the NAS, hence the immediate shutdown.  Now they will spin-up on their
own, but it takes 5-10 seconds.  My thought was if they can’t communicate
initially, they assume the server is dead and shut down.  Would that make sense
at all?

HDD Hibernation essentially spins down the hard drives when there has been no
activity.  I had it set at one hour; I have since disabled that setting.  I have
not ever set the Synology itself to go to sleep or to auto-shutdown.

From the Synology support site:
	You can specify the period of time hard disks remain idle before entering HDD
hibernation mode. During HDD hibernation, hard disks stop spinning, reducing
power consumption and extending the lifespan of hard disks. You can specify
different periods of time for internal hard disks and external eSATA/USB hard
disks.

	Phase 1: HDDs power down after a period of inactivity (idle time). You may go
to DSM > Control Panel > Hardware & Power > HDD Hibernation to
configure the length of inactivity for the HDDs to enter hibernation.

	Phase 2: Advanced Hibernation/Deep Sleep will be implemented for the HDDs to
further reduce power consumption. You may go to DSM > Control Panel >
Hardware & Power > HDD Hibernation to enable advanced HDD hibernation.
> You should remove line 1 : RUN_AS_USER nut
Ok, I will comment that back out.  That won’t interfere with anything?  What
user would it run as?  Will it have access to upsmon.conf?


Thanks,

Todd

--
Todd Benivegna // todd at benivegna.com
On Aug 9, 2020, 3:07 AM -0400, Roger Price <roger at rogerprice.org>,
wrote:
> On Fri, 7 Aug 2020, Todd Benivegna wrote:
>
> Hello Todd, sorry for the delay replying, I was away for a while. It's
ok to
> post configuration files in this list if blank lines and comments are
removed.
>
> > On the Synology (I didn’t edit any of these files):
> > ups.conf:  https://hastebin.com/dedereqizi.shell
>
> pollinterval = 5
> [ups]
> driver = usbhid-ups
> port = auto
>
> > upsd.conf:  https://hastebin.com/pupeseweda.css
>
> LISTEN 192.168.1.70
> LISTEN 2601:cf:8200:43d0:211:32ff:fe63:60d7
> LISTEN fe80::211:32ff:fe63:60d7
> LISTEN 127.0.0.1
> LISTEN ::1
>
> > upsd.users:  https://hastebin.com/ocenamecex.cs
>
> [monuser]
> password = secret
> upsmon master
>
> Since it's NUT in the NAS which is deciding to order the shutdown, it
would be
> useful to see upsmon.conf, upssched.conf and upssched-cmd (or whatever
Synology
> use if anything) from the NAS to see what criteria they use.
>
> Roger
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200809/07ce9cfc/attachment.html>