Nut upsuser - Aug 2020 - Synology NAS is shutting down Ubuntu servers after very brief power outage (fwd)

On Mon, 10 Aug 2020, Todd Benivegna wrote:
> synoups: https://hastebin.com/xexafofiha.bash
Wow! What a mess.  It looks as if Synology wanted to write their own
"NUT", but
decided it would be easier to put their ideas in a script when they saw they 
could use upssched.conf to call it.  NUT intends such a script for timer 
management.  Synology use it for general system management.

  1. Could you show us the file /usr/syno/bin/synoupscommon ?

  2. Could you log into the NAS and execute ls -alF /bin/sh ?

Roger

Hi Roger,

Ok, very interesting…!  Here is what you are looking for:

1.  https://hastebin.com/evadipomew.bash
2. lrwxrwxrwx 1 root root 4 May 27 05:26 /bin/sh -> bash*

Hope that helps!

Todd

--
Todd Benivegna // todd at benivegna.com
On Aug 11, 2020, 1:46 AM -0400, Roger Price <roger at rogerprice.org>,
wrote:
> On Mon, 10 Aug 2020, Todd Benivegna wrote:
>
> > synoups: https://hastebin.com/xexafofiha.bash
>
> Wow! What a mess. It looks as if Synology wanted to write their own
"NUT", but
> decided it would be easier to put their ideas in a script when they saw
they
> could use upssched.conf to call it. NUT intends such a script for timer
> management. Synology use it for general system management.
>
> 1. Could you show us the file /usr/syno/bin/synoupscommon ?
>
> 2. Could you log into the NAS and execute ls -alF /bin/sh ?
>
> Roger
>
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200811/301106eb/attachment.html>

On Tue, Aug 11, 2020 at 12:46 AM Roger Price <roger at rogerprice.org>
wrote:
> On Mon, 10 Aug 2020, Todd Benivegna wrote:
>
> > synoups: https://hastebin.com/xexafofiha.bash
>
> Wow! What a mess.  It looks as if Synology wanted to write their own
> "NUT", but
> decided it would be easier to put their ideas in a script when they saw
> they
> could use upssched.conf to call it.  NUT intends such a script for timer
> management.  Synology use it for general system management.
>
Roger's comment confirms my suspicion of NUT as provided by Synology. They
make a great NAS product, but then they bolt on all manner of other things.
In my opinion, best to leave the NAS as an appliance configured and managed
by their GUI tools, and let it just be a NUT client rather than trying to
configure it to be the NUT server. I use and find the RaspberryPi's to be
very capable NUT servers with the rest of my systems (including my Synology
NAS) as NUT clients. Much simpler to manage that way as you have complete
control over a fairly current NUT as provided by Raspbian (a Debian
derivative). The only kink I've run into is that the Synology NAS as a NUT
client provides no means of changing the NUT credentials, so you have to
use default credentials for NUT (another reason to make sure NUT is on a
protected network).

--Larry

-- 
Larry Fahnoe, Fahnoe Technology Consulting, fahnoe at FahnoeTech.com
           Minneapolis, Minnesota       www.FahnoeTech.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200811/f8bec641/attachment.html>

Hi everyone,

Well, we lost power here again.  I was not at home, but I guess a dump truck
smashed into some power poles and took out half the city; actually tripped most
of the breakers in the panel.  Anyways, the power was out for just a few seconds
and of course my servers shutdown on me after power being out for just a second
or two.  Here is my syslog from two different machines.  I included everything
just in case.

“Proton":  https://hastebin.com/uluqaqetuc.bash
"Plex”:  https://hastebin.com/apudatonun.sql

Let me know what you think.  I just tested everything over the weekend and it
all worked perfectly; cut power and machines stayed up until it was time to shut
them down (I temporarily configured them to shut down after 5 minutes so I
didn’t have to wait for the entire battery to drain).  I seriously at am a total
loss.  I guess HDD Hibernation was not the culprit either since I’m still
getting the same results.   Ugh.  Thanks everyone for all your help so far!!!!!

Regards,

Todd

--
Todd Benivegna // todd at benivegna.com
On Aug 11, 2020, 7:32 AM -0400, Larry Fahnoe <fahnoe at fahnoetech.com>,
wrote:
> On Tue, Aug 11, 2020 at 12:46 AM Roger Price <roger at
rogerprice.org> wrote:
> > > On Mon, 10 Aug 2020, Todd Benivegna wrote:
> > >
> > > > synoups: https://hastebin.com/xexafofiha.bash
> > >
> > > Wow! What a mess.  It looks as if Synology wanted to write their
own "NUT", but
> > > decided it would be easier to put their ideas in a script when
they saw they
> > > could use upssched.conf to call it.  NUT intends such a script
for timer
> > > management.  Synology use it for general system management.
>
> Roger's comment confirms my suspicion of NUT as provided by Synology.
They make a great NAS product, but then they bolt on all manner of other things.
In my opinion, best to leave the NAS as an appliance configured and managed by
their GUI tools, and let it just be a NUT client rather than trying to configure
it to be the NUT server. I use and find the RaspberryPi's to be very capable
NUT servers with the rest of my systems (including my Synology NAS) as NUT
clients. Much simpler to manage that way as you have complete control over a
fairly current NUT as provided by Raspbian (a Debian derivative). The only kink
I've run into is that the Synology NAS as a NUT client provides no means of
changing the NUT credentials, so you have to use default credentials for NUT
(another reason to make sure NUT is on a protected network).
>
> --Larry
>
> --
> Larry Fahnoe, Fahnoe Technology Consulting, fahnoe at FahnoeTech.com
>            Minneapolis, Minnesota       www.FahnoeTech.com
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20200811/d5728801/attachment.html>

Synology, in their NAS products, include NUT for UPS based power management. But
although Synology use upsd to talk to the UPS unit, they have their own power 
management software called "Safe Mode".  This reduces upsmon and
upssched to
pure passthrough with the status changes handled by Synology scripts 
synoupscommon + synoups.

  What NUT does
  -------------

NUT, in upsmon, provides a basic UPS management which I will call
"Emergency
Shutdown" based on status [LB], and in upssched NUT provides a more complex
time-based UPS management which I call "Managed Shutdown", and which
is more
suitable for installations with frequent wall power failures.  Managed Shutdown 
becomes Emergency Shutdown when the battery is exhausted.  The software design 
is simple with rapid handling by upsmon of the received status changes following
the declarations in the configuration file..  Timers run independently and 
generate further status changes for upssched, again following the declarations 
in the configuration file.

  What Synology does
  ------------------

Synology do neither of NUT's Emergency or Managed shutdowns.  Even the basic
Emergency Shutdown mechanism of NUT's upsmon is absent.  See SHUTDOWNCMD
"" in
their upsmon.conf.  They have invented their own shutdown plan which they call 
"Safe Mode".  It's an emotive term which is good in sales
literature but
something of a red flag in a disaster recovery audit.

The software design differs from NUT.  While handling a status change, Synology 
will loop waiting for confirmation via upsc of the UPS status.  This can lead to
complex situations.

The following descriptions will refer to Synology's SZF flags.  These are
files,
often in /tmp, which are either defined or not defined.  Their contents are 
comments on what is happening.  Synology's function UPSStatusGet returns the
upsc report for ups.status.

Synology's Safe Mode accepts 5 status changes: online, onbatt, lowbatt, fsd
and
shutdownups. I summarise each one:

  Status change: online
  ---------------------

Function UPSRestart loops waiting up to 3 minutes for flag SZF_SAFEMODE_DONE aka
/tmp/safeModeDone to be defined before calling telinit 6 for a system re-boot. 
Note that the timer is in a local loop. If power fails during this time, it 
looks as if the status change is ignored.  Good timer management would requires 
the timer to be run externally producing a new call of Synology's software.

  Status change: onbatt
  ---------------------

Sets flag SZF_ONBATT aka /tmp/ups.onbatt

  Status changes: lowbatt, fsd
  ----------------------------

Calls function UPSSafeMode which checks the UPS status and exits if [OL].  This 
code will fail if the UPS status is [OL CHRG], and although the UPS is online, 
the NAS will shutdown.   Here is the relevant snippet:

 	St=`UPSStatusGet`
 	if [ "$St" = "OL" ]; then
 		SYSLOG "WARNING: UPS is On-Line"
 		return
 	fi
 	echo "$1" > $SZF_SAFEMODE

 	/usr/syno/bin/synologset1 sys warn 0x11300011
 	SYSLOG "[UPS] Server is going to Safe Mode. It will stop all services and
umount all volumes."


  Status change: shutdownups
  --------------------------

Calls function UPSShutdown. which checks the UPS status and reboots if [OL]. 
This code will fail if the UPS status is [OL CHRG], and although the UPS is 
online, the NAS will shutdown.  Here is the relevant snippet:

 	OL=0
 	LB=0
 	while [ $OL -ne 2 -a $LB -ne 2 ]; do
 		sleep 10
 		St=`UPSStatusGet`

The following test will fail if St="OL CHRG".

 		if [ "$St" = "OL" ]; then
 			OL=`expr $OL + 1`
 			LB=0
 		else
 			LB=`expr $LB + 1`
 			OL=0
 		fi
 		echo "OL=$OL LB=$LB" >> $SZF_SAFEMODE
 	done
 	if [ $OL -eq 2 ]; then
 		synologset1 sys warn 0x11300012
 		echo "UPS back to On-Line and reboot." >> $SZF_SAFEMODE
 		$SYNOBOOTBIN --unset-safe-shutdown
 		telinit 6

If St="OL CHRG", the following code is executed and the NAS shuts down
even
though the UPS is online.

 	elif [ $LB -eq 2 ]; then
 		touch /var/.NormalShutdown
 		if [ $UPSSafeShutdown -eq 0 ]; then
 			echo "Waiting UPS exhausted." >> $SZF_SAFEMODE
 			SYSLOG "Waiting UPS exhausted."
 		else
 			echo "UPS safe shutdown." >> $SZF_SAFEMODE
 			SYSLOG "UPS safe shutdown."


  Summary
  -------

Synology's Safe Mode is a complete replacement for NUT's upsmon and
upssched and
looks to be buggy with at least some UPS units.  The Safe Mode seems to be an 
attempt to do a Managed Shutdown, but keeps the NAS running which will deplete 
the UPS.  The overall design is weak.  The logic is code-based rather than 
configuration file based, and is unduly complex.

If you have problems with having the NAS as master, make it a slave, and run the
NUT configuration of your choice in your PC/workstation.

Roger