Nut upsdev - Feb 2014 - Logic problem in NUT with upscode2 driver

Hi All,

   I have what is probably a 12 year old (or older) Compaq R3000 UPS
plugged into a FreeBSD 8 system running NUT version 2.6.5

   This UPS contains a total of 16 batteries in 2 banks of 8,
each battery is 6 volts and the bank produces 50 volts.

   The UPS monitors battery voltage and the battery charger in
the UPS works like this.  For about 2 days it will apply a
charge/float/topping voltage of 55.2 volts to the batteries.
During that time the battery voltage measures 55.2 volts
both with a voltmeter on the batteries and as reported by
upsc, and the "calculated state of charge" of the UPS is 92.7%

(keep in mind that UPSes that are driven by upscode2 are unable
to report battery charge, so per the man page the upscode2 driver 
synthesizes that value based on nominal battery min/max and current 
battery voltage)

   Anyway, then the UPS shuts off the battery charger.  The battery
voltage then starts dropping gradually until the battery voltage hits a 
bit above 48v then the charger turns back on.  (this takes about 2 days) 
  For example, right now the battery voltage of each bank (as shown by 
upsc and confirmed by a voltmeter) is 50.5 volts yet the calculated 
state of charge is 64% even though the UPS is reporting
a nominal battery voltage of 48v and the batteries are 2 volts ABOVE
that.

   The problem I am having appears to be the logic in the upscode2
driver.  For starters, the calculated state of charge is quite wrong.
48v on a bank of 8  6v batteries is normal, it IS fully charged, it
is NOT 64% charged.  50.5 volts on a 48v bank of batteries should be
almost 100% charged, not 64%.

   Worse, however, is if there's a power failure right near the end
of the 2-days-off cycle.  That happened to me last week - it was a
short duration 15 second loss - and the upscode2 driver decided it 
needed to issue a forced shutdown.

   Very likely this was because upscode2 had decided the batteries
were dangerously low discharged.  But they were NOT discharged and
easily kept the servers up and online.

   The UPS reports a battery max voltage of 56.4 and minimum voltage of
40 and nominal voltage of 48  (this is obviously for a given bank)

   Now I do not know if OTHER upses that require upscode2 have battery
chargers that sense battery voltage like this, and that shut off. If
I disconnect the battery pack for a second and plug it back in, the
battery charger sees the loss of battery voltage and turns back on,
so it's obvious the charger is sensing battery voltage.

   I am guessing the upscode2 driver is assuming that the battery charger
would be switched on all of the time.  55.2v is the correct float charge
voltage for a bank of 8 6v SLA batteries, although the driver is 
calculating a state of charge of 92.7% when the charger is switched on 
and the battery bank is at 55.2v.

   I believe the upscode2 driver should be calculating a battery state
of charge of 97% with a max battery voltage of 56.4 and a measured
voltage of 55.2.  It should not be reporting 92%

   And, I believe the upscode2 driver should NOT be calculating that 
state of charge below 97% until the measured voltage drops below the 
nominal voltage.

   Below the nominal voltage - 48v - is where the calculated state of
charge should be dropping down below 90%

Can anyone tell me how to correct the logic in upscode2?  I can't have
the NUT software shutting servers down when the UPS has plenty of
runtime and capacity.

Thanks!!!
Ted

Hi All again,

OK I found the section of code in upscode2.c I think is applicable - my 
UPS does not supply battery charge current, meaning I think that it
operates using this logic:

}
         /* Old method, assumes battery high/low-voltages provided by 
UPS are
          * applicable to battery charge, but they usually aren't */
         else if (batt_volt_low > 0 && batt_volt_high > 0
&& batt_volt >
0) {
                 if (batt_volt > batt_volt_high) {
                         chg=100;
                 }
                 else if (batt_volt < batt_volt_low) {
                         chg=0;
                 }
                 else {
                         chg = (batt_volt - batt_volt_low) /
                                 (batt_volt_high - batt_volt_low);
                         chg*=100;
                 }
         }
         else {
                 return -1;
         }

I believe the bug here is:

if (batt_volt > batt_volt_high) {
                         chg=100;

this should be:

if (batt_volt > batt_volt_nom) {
                         chg=100;

The problem (with my UPS) is that battery voltage is never going to 
exceed the battery voltage maximum, (that would be a bad overcharge) 
thus that decision is never taken
and we are always dropping through to the next formula which is 
senseless when calculating state of charge of batteries that are ABOVE 
their nominal value.   By definition a lead acid cell above 2 volts is
fully charged.

I propose that anytime the batteries are above nominal, (48 volts), they 
should be considered at 100% of charge.

I further propose that in the formula

else {
                         chg = (batt_volt - batt_volt_low) /
                                 (batt_volt_high - batt_volt_low);

we change this to:


else {
                         chg = (batt_volt_low / batt_volt);

Thus, a 48v battery pack that has dropped below nominal value of 48 
volts to 46v would be considered 86% charged.

Suggestions?


Ted

PS:  Here's a patch

--- upscode2.c.original 2012-07-31 10:38:59.000000000 -0700
+++ upscode2.c  2014-02-19 10:54:27.000000000 -0800
@@ -1391,15 +1391,15 @@
         /* Old method, assumes battery high/low-voltages provided by 
UPS are
          * applicable to battery charge, but they usually aren't */
         else if (batt_volt_low > 0 && batt_volt_high > 0
&& batt_volt >
0) {
-               if (batt_volt > batt_volt_high) {
+               if (batt_volt > batt_volt_nom) {
                         chg=100;
                 }
                 else if (batt_volt < batt_volt_low) {
                         chg=0;
                 }
                 else {
-                       chg = (batt_volt - batt_volt_low) /
-                               (batt_volt_high - batt_volt_low);
+                       chg = ( batt_volt_low /
+                               batt_volt);
                         chg*=100;
                 }
         }



On 2/19/2014 9:50 AM, Ted Mittelstaedt wrote:
> Hi All,
>
> I have what is probably a 12 year old (or older) Compaq R3000 UPS
> plugged into a FreeBSD 8 system running NUT version 2.6.5
>
> This UPS contains a total of 16 batteries in 2 banks of 8,
> each battery is 6 volts and the bank produces 50 volts.
>
> The UPS monitors battery voltage and the battery charger in
> the UPS works like this. For about 2 days it will apply a
> charge/float/topping voltage of 55.2 volts to the batteries.
> During that time the battery voltage measures 55.2 volts
> both with a voltmeter on the batteries and as reported by
> upsc, and the "calculated state of charge" of the UPS is 92.7%
>
> (keep in mind that UPSes that are driven by upscode2 are unable
> to report battery charge, so per the man page the upscode2 driver
> synthesizes that value based on nominal battery min/max and current
> battery voltage)
>
> Anyway, then the UPS shuts off the battery charger. The battery
> voltage then starts dropping gradually until the battery voltage hits a
> bit above 48v then the charger turns back on. (this takes about 2 days)
> For example, right now the battery voltage of each bank (as shown by
> upsc and confirmed by a voltmeter) is 50.5 volts yet the calculated
> state of charge is 64% even though the UPS is reporting
> a nominal battery voltage of 48v and the batteries are 2 volts ABOVE
> that.
>
> The problem I am having appears to be the logic in the upscode2
> driver. For starters, the calculated state of charge is quite wrong.
> 48v on a bank of 8 6v batteries is normal, it IS fully charged, it
> is NOT 64% charged. 50.5 volts on a 48v bank of batteries should be
> almost 100% charged, not 64%.
>
> Worse, however, is if there's a power failure right near the end
> of the 2-days-off cycle. That happened to me last week - it was a
> short duration 15 second loss - and the upscode2 driver decided it
> needed to issue a forced shutdown.
>
> Very likely this was because upscode2 had decided the batteries
> were dangerously low discharged. But they were NOT discharged and
> easily kept the servers up and online.
>
> The UPS reports a battery max voltage of 56.4 and minimum voltage of
> 40 and nominal voltage of 48 (this is obviously for a given bank)
>
> Now I do not know if OTHER upses that require upscode2 have battery
> chargers that sense battery voltage like this, and that shut off. If
> I disconnect the battery pack for a second and plug it back in, the
> battery charger sees the loss of battery voltage and turns back on,
> so it's obvious the charger is sensing battery voltage.
>
> I am guessing the upscode2 driver is assuming that the battery charger
> would be switched on all of the time. 55.2v is the correct float charge
> voltage for a bank of 8 6v SLA batteries, although the driver is
> calculating a state of charge of 92.7% when the charger is switched on
> and the battery bank is at 55.2v.
>
> I believe the upscode2 driver should be calculating a battery state
> of charge of 97% with a max battery voltage of 56.4 and a measured
> voltage of 55.2. It should not be reporting 92%
>
> And, I believe the upscode2 driver should NOT be calculating that state
> of charge below 97% until the measured voltage drops below the nominal
> voltage.
>
> Below the nominal voltage - 48v - is where the calculated state of
> charge should be dropping down below 90%
>
> Can anyone tell me how to correct the logic in upscode2? I can't have
> the NUT software shutting servers down when the UPS has plenty of
> runtime and capacity.
>
> Thanks!!!
> Ted
>
> _______________________________________________
> Nut-upsdev mailing list
> Nut-upsdev at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/nut-upsdev

On Feb 19, 2014, at 12:50 PM, Ted Mittelstaedt wrote:
>  Worse, however, is if there's a power failure right near the end
> of the 2-days-off cycle.  That happened to me last week - it was a
> short duration 15 second loss - and the upscode2 driver decided it needed
to issue a forced shutdown.
> 
>  Very likely this was because upscode2 had decided the batteries
> were dangerously low discharged.  But they were NOT discharged and
> easily kept the servers up and online.
As far as I can tell, the upscode2 driver does not use the battery voltage to
determine when to shut down - it uses one of the UPS status bits from the STAT
or STMF responses.

We have a few other drivers that calculate a cosmetic state-of-charge, and off
the top of my head, the drivers have parameters to adjust for the variations in
voltages sensed by the UPS. That's certainly a possible improvement for this
driver, but I don't think it is going to fix the internal UPS determination
of whether the battery is low.

-- 
Charles Lepple
clepple at gmail

On 2/20/2014 6:55 AM, Charles Lepple wrote:
> On Feb 19, 2014, at 12:50 PM, Ted Mittelstaedt wrote:
>
>> Worse, however, is if there's a power failure right near the end of
>> the 2-days-off cycle.  That happened to me last week - it was a
>> short duration 15 second loss - and the upscode2 driver decided it
>> needed to issue a forced shutdown.
>>
>> Very likely this was because upscode2 had decided the batteries
>> were dangerously low discharged.  But they were NOT discharged and
>> easily kept the servers up and online.
>
> As far as I can tell, the upscode2 driver does not use the battery
> voltage to determine when to shut down - it uses one of the UPS
> status bits from the STAT or STMF responses.
>
Really!  I was afraid of that.  With this blip, this is what
upslog was showing:

20140212 032546 65.9 119.30 60.9 [OL] NA NA
20140212 032626 62.8 119.10 55.5 [FSD OL] NA NA
20140212 043722 NA NA NA [WAIT] NA NA
20140212 044222 67.7 118.90 54.4 [OL] NA NA
20140212 044722 65.9 119.20 52.4 [OL] NA NA

The blip obviously took place between 3:25:46 and
3:26:26, I was lucky that upslog caught it then.  You
can see the calculated state of charge going to 62.8% so
the battery voltage probably hit 48 volts at that time -
and I'm assuming the UPS considered the batteries in imminent
danger of failing - I guess - even though they weren't.

I did a battery replacement on this UPS about 8 months ago,
I wonder if the batteries I used have a slightly lower run
voltage than what the UPS was calibrated for?

What I don't understand exactly is why FSD and OL were _both_
showing in the status?

That's forced shutdown, I believe and online at the same time?!?
> We have a few other drivers that calculate a cosmetic
> state-of-charge, and off the top of my head, the drivers have
> parameters to adjust for the variations in voltages sensed by the
> UPS. That's certainly a possible improvement for this driver, but I
> don't think it is going to fix the internal UPS determination of
> whether the battery is low.
>
Well, do you think my patch logic makes sense?  From the looks of
it the upscode2 protocol is obsolete and was not used by many
UPSes (and most likely most of those are being retired) and
more modern UPSes have their own logic to calculate battery state
of charge that is much better than having the driver do it.  But
this may help someone else who is dealing with one of these.

is there anyway to delay how long NUT reacts to an FSD from the
driver?  Since with a little blip like this lasting less than a
minute, my preference is to NOT shut any of the servers down but
to let it ride.

I also suggest the following patch to upslog.c  (little line wrapping there)


--- upslog.c.orig       2012-07-31 10:38:58.000000000 -0700
+++ upslog.c    2014-02-20 09:23:14.000000000 -0800
@@ -50,6 +50,7 @@
         static  flist_t *fhead = NULL;

  #define DEFAULT_LOGFORMAT "%TIME @Y at m@d @H at M@S% %VAR
battery.charge% " \
+               "%VAR battery.voltage% %VAR output.current% " \
                 "%VAR input.voltage% %VAR ups.load% [%VAR ups.status%]
" \
                 "%VAR ups.temperature% %VAR input.frequency%"


Mainly because the only real indicator of true battery health with this
UPS is the voltage off the battery bank.

Ted