>> last night we had what looked like a power spike and 3 machines plugged >> into 2 Eaton UPS went down instantly. I heard the click the UPS makes when >> it shuts the load on the ports and then shuts itself down (ie same thing if >> I simulate with upsmon -fsd). The machines instantly came back, there was no >> actual downtime, and the machine next to them which is not on the UPS never >> went down. >> >> This to me suggests some kind of overvoltage protection or something where >> the UPS decided to shut down the load, but I'm quite puzzled by it as I'd >> expect the UPS to actually shield the machines from it and certainly not >> turn something off brutally like that..I'm going to CC Arnaud's work address just in case they have any additional information at Eaton, but for a variety of reasons, NUT is not very strong when it comes to post-mortem analysis of events. If the system running the NUT driver did not have a chance to log the event before power went down, it is difficult to ascertain what happened. Is there an "upsc" dump for these units in one of the other email/GitHub threads? (Feel free to redact all or part of the serial number.) I don't know of any particular combination of settings which might cause an immediate power-down for over-voltage, but it can't hurt to check. Arno: do these models keep an event log in EEPROM? Didn't see anything in the MGE OPS protocol library on the NUT website, but I haven't had a chance to poke around the Eaton website. -- - Charles Lepple
Hello Charles, I've attached a dump below. There were no long entries on any of the hosts, neither the one running standalone, nor the master/slave combo, which is not too surprising, it was really fast: I saw the light flickering in the room, almost like flash, and all those 3 systems went down. I first heard the "click" of the ports being disabled, heard the systems shut down, and then the louder click of the UPS turning itself off. Probably less than 3secs in total. battery.charge: 100 battery.charge.low: 20 battery.runtime: 754 battery.type: PbAc device.mfr: EATON device.model: Ellipse PRO 1500 device.type: ups driver.name: usbhid-ups driver.parameter.pollfreq: 30 driver.parameter.pollinterval: 2 driver.parameter.port: auto driver.version: 2.7.2 driver.version.data: MGE HID 1.33 driver.version.internal: 0.38 input.frequency: 60.0 input.transfer.high: 138 input.transfer.low: 93 input.voltage: 115.0 input.voltage.extended: no outlet.1.desc: PowerShare Outlet 1 outlet.1.id: 2 outlet.1.status: on outlet.1.switchable: no outlet.2.desc: PowerShare Outlet 2 outlet.2.id: 3 outlet.2.status: on outlet.2.switchable: no outlet.desc: Main Outlet outlet.id: 1 outlet.switchable: no output.frequency: 60.0 output.frequency.nominal: 60 output.voltage: 114.0 output.voltage.nominal: 115 ups.beeper.status: enabled ups.delay.shutdown: 20 ups.delay.start: 30 ups.firmware: 01.08.0016 ups.load: 34 ups.mfr: EATON ups.model: Ellipse PRO 1500 ups.power: 316 ups.power.nominal: 1500 ups.productid: ffff ups.realpower: 316 ups.status: OL CHRG ups.timer.shutdown: 0 ups.timer.start: 0 ups.vendorid: 0463 This is on ubuntu xenial. The servers have dual power supplies, total 4 ports, plugged into the battery backups + surge protection ports on the right hand side. Eco mode is disabled. thank you for your input, Spike On Fri, Apr 7, 2017 at 2:52 PM Charles Lepple <clepple at gmail.com> wrote:> >> last night we had what looked like a power spike and 3 machines plugged > >> into 2 Eaton UPS went down instantly. I heard the click the UPS makes > when > >> it shuts the load on the ports and then shuts itself down (ie same > thing if > >> I simulate with upsmon -fsd). The machines instantly came back, there > was no > >> actual downtime, and the machine next to them which is not on the UPS > never > >> went down. > >> > >> This to me suggests some kind of overvoltage protection or something > where > >> the UPS decided to shut down the load, but I'm quite puzzled by it as > I'd > >> expect the UPS to actually shield the machines from it and certainly not > >> turn something off brutally like that.. > > I'm going to CC Arnaud's work address just in case they have any > additional information at Eaton, but for a variety of reasons, NUT is > not very strong when it comes to post-mortem analysis of events. If > the system running the NUT driver did not have a chance to log the > event before power went down, it is difficult to ascertain what > happened. > > Is there an "upsc" dump for these units in one of the other > email/GitHub threads? (Feel free to redact all or part of the serial > number.) I don't know of any particular combination of settings which > might cause an immediate power-down for over-voltage, but it can't > hurt to check. > > Arno: do these models keep an event log in EEPROM? Didn't see anything > in the MGE OPS protocol library on the NUT website, but I haven't had > a chance to poke around the Eaton website. > > -- > - Charles Lepple >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.alioth.debian.org/pipermail/nut-upsuser/attachments/20170407/8cbda057/attachment-0001.html>
fwiw, in case somebody in the future stumbles on this thread, I finally got around to talk to a tech at Eaton and look again at our systems and it may have been a ground fault. What I did not know and it's not in the manuals, but which makes sense, is that the only condition under which those UPSs would shut themselves down like that is if they detected a ground fault/power on the neutral. Furthermore, I also confirmed that the unit does not store any kind of logs, so if any event was logged that would have been on the PC through the monitoring sw (nut didn't, but I suspect there was no event generated). best, Spike On Fri, Apr 7, 2017 at 4:28 PM Spike <spike at drba.org> wrote:> Hello Charles, > > I've attached a dump below. There were no long entries on any of the > hosts, neither the one running standalone, nor the master/slave combo, > which is not too surprising, it was really fast: I saw the light flickering > in the room, almost like flash, and all those 3 systems went down. I first > heard the "click" of the ports being disabled, heard the systems shut down, > and then the louder click of the UPS turning itself off. Probably less than > 3secs in total. > > battery.charge: 100 > battery.charge.low: 20 > battery.runtime: 754 > battery.type: PbAc > device.mfr: EATON > device.model: Ellipse PRO 1500 > device.type: ups > driver.name: usbhid-ups > driver.parameter.pollfreq: 30 > driver.parameter.pollinterval: 2 > driver.parameter.port: auto > driver.version: 2.7.2 > driver.version.data: MGE HID 1.33 > driver.version.internal: 0.38 > input.frequency: 60.0 > input.transfer.high: 138 > input.transfer.low: 93 > input.voltage: 115.0 > input.voltage.extended: no > outlet.1.desc: PowerShare Outlet 1 > outlet.1.id: 2 > outlet.1.status: on > outlet.1.switchable: no > outlet.2.desc: PowerShare Outlet 2 > outlet.2.id: 3 > outlet.2.status: on > outlet.2.switchable: no > outlet.desc: Main Outlet > outlet.id: 1 > outlet.switchable: no > output.frequency: 60.0 > output.frequency.nominal: 60 > output.voltage: 114.0 > output.voltage.nominal: 115 > ups.beeper.status: enabled > ups.delay.shutdown: 20 > ups.delay.start: 30 > ups.firmware: 01.08.0016 > ups.load: 34 > ups.mfr: EATON > ups.model: Ellipse PRO 1500 > ups.power: 316 > ups.power.nominal: 1500 > ups.productid: ffff > ups.realpower: 316 > ups.status: OL CHRG > ups.timer.shutdown: 0 > ups.timer.start: 0 > ups.vendorid: 0463 > > This is on ubuntu xenial. The servers have dual power supplies, total 4 > ports, plugged into the battery backups + surge protection ports on the > right hand side. Eco mode is disabled. > > thank you for your input, > > Spike > > On Fri, Apr 7, 2017 at 2:52 PM Charles Lepple <clepple at gmail.com> wrote: > >> >> last night we had what looked like a power spike and 3 machines plugged >> >> into 2 Eaton UPS went down instantly. I heard the click the UPS makes >> when >> >> it shuts the load on the ports and then shuts itself down (ie same >> thing if >> >> I simulate with upsmon -fsd). The machines instantly came back, there >> was no >> >> actual downtime, and the machine next to them which is not on the UPS >> never >> >> went down. >> >> >> >> This to me suggests some kind of overvoltage protection or something >> where >> >> the UPS decided to shut down the load, but I'm quite puzzled by it as >> I'd >> >> expect the UPS to actually shield the machines from it and certainly >> not >> >> turn something off brutally like that.. >> >> I'm going to CC Arnaud's work address just in case they have any >> additional information at Eaton, but for a variety of reasons, NUT is >> not very strong when it comes to post-mortem analysis of events. If >> the system running the NUT driver did not have a chance to log the >> event before power went down, it is difficult to ascertain what >> happened. >> >> Is there an "upsc" dump for these units in one of the other >> email/GitHub threads? (Feel free to redact all or part of the serial >> number.) I don't know of any particular combination of settings which >> might cause an immediate power-down for over-voltage, but it can't >> hurt to check. >> >> Arno: do these models keep an event log in EEPROM? Didn't see anything >> in the MGE OPS protocol library on the NUT website, but I haven't had >> a chance to poke around the Eaton website. >> >> -- >> - Charles Lepple >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.alioth.debian.org/pipermail/nut-upsuser/attachments/20170501/6e7cc9af/attachment.html>