David Zomaya
2019-Jun-19 20:16 UTC
[Nut-upsdev] [EXTERNAL] Re: Fixing Drops With SMART1500LCDXL & USB-HID Driver
Thanks for the inputs. I’ll get CentOS 7.6 installed on a physical machine this week and follow up. Few responses and notes below in the interim. ”The log format looks okay, but I may not have been clear that I was looking for the log at the same time as when the kernel errors occur. So without the udev rules would be good.” Ah, sorry. Attached are these two files: 1- norules.log.gz -Neither 62-nut-usbups.rules or 42-usb-hid-pm.rules at /etc/udev/rules.d/ or /usr/lib/udev/rules.d/. This really just left me with “permission denied”. I assume this is because 62-nut-usbups.rules is needed and #2 is the relevant file. 2- norule42.log.gz- 62-nut-usbups.rules at /lib/udev/rules.d/ and no “pollinterval = 1” value set in ups.conf. This yielded better results. I also included the output from /var/log/messages, lsusb –v, uname –r, and cat /etc/os-release. In the /var/log/messages output, you can see a few of the drops we are trying to prevent. “I think you mentioned the CentOS version - which kernel version does that run? ("uname -r" is probably sufficient)” 3.10.0-957.el7.x86_64 “I moved my SMART1500LCDT off of the always-on system, so I don't have continuous data for it. I don't think it worked on Raspbian stretch the last time I tried. Others have rigged up scripts to reset a USB hub to simulate re-plugging the UPS USB connection.” Ok, if issues come up going forward and I can help, feel free to reach out (same for any new Tripp Lite issues). “I think that a lot of the content of the protocol docs would end up in the logic of the source code. Trying to write open-source code based on a document that can't be shared seems risky to me.” That’s fair. I’d like to see us do something to help though. Let me see where this goes internally and follow up. “I think a range of IDs (even with a few exceptions for the older units) would be sufficient. The ideal scenario is that the range is somewhat future-proof, so a version of NUT from this year can properly identify next year's UPS. If not, we still have manual ways for users to add their idProduct to ups.conf and the udev files, but as you can imagine, that is frustrating for a new user.” Almost everything we make now is PDC compliant, so we should at least be able to do this. We’re looking into something easy to identify in the output of lsusb –v though since that could help with legacy support. Note: Interestingly, I did some tinkering with openSUSE 15.1 Leap on a physical box and saw the same drops there, but was able to get them to stop after installing NUT and just doing the “standard” configuration. However, I ran into some connection refused messages, e.g.: user at linux-nxmm:/dev/bus> sudo upsc -l Error: Connection failure: Connection refused When trying to pull data from the UPS, so I was clearly doing something wrong. But, the UPS stopped dropping (i.e. retained Device # in lsusb and no messages in /var/log/messages) . I did NOT have to make the changes to “42-hd-audio-pm.rules” to make the drops stop in this case. Given the different kernel (4.12.14-lp151.27-default) and newer version of (NUT 2.7.4) this may not be apples to apples though. I hope that I will be able to provide better data once I get to test further later this week. Topically, this has me thinking at the least I need to confirm the reproducibility of the fix I mentioned on a physical box. I will follow up by close of business Friday. Thank you, David Zomaya Tripp Lite 1111 W. 35th Street | Chicago, IL 60609 USA -----Original Message----- From: Charles Lepple <clepple at gmail.com> Sent: Tuesday, June 18, 2019 10:00 PM To: David Zomaya <David_Zomaya at tripplite.com> Cc: nut-upsdev at alioth-lists.debian.net; Jonathan Manzanilla <Jonathan_Manzanilla at tripplite.com>; Eric Cobb <Eric_Cobb at tripplite.com> Subject: Re: [EXTERNAL] Re: [Nut-upsdev] Fixing Drops With SMART1500LCDXL & USB-HID Driver On Jun 18, 2019, at 1:47 PM, David Zomaya wrote:>> Charles and Wolfy,>> Thanks for the replies.>> I added some responses below.> I think I got the driver debugging right, but let me know if it is off.>> I should note:> The CentOS 7.6 machine I have been testing with is a virtual machine (running on VMware ESXi 7.6). At least 1 customer and I have seen the same issue on VMs and physical boxes, so I don’t think that matters, but if it does let me know.>>> “This is a little off-topic, but I would like to point out that not including a machine-readable serial number in the USB device descriptor makes it difficult for people to reliably use two or more UPSes on a single *nix system. Due to some complications with the way that USB devices are opened in libusb, there is no easy way to "open the next unused USB device", so we recommend that people match against the serial number.”>>> Noted. I’ll kick this around internally and follow up. I know other units we make report the serial number. Maybe it’s a 2012 thing.I think we saw a similar problem with APC UPSes not having string descriptors when attached to a VM, so that is worth checking on a physical box. If so, I apologize for jumping to the conclusion that the UPS was not providing the serial number.>> “It would be interesting to see the debug log from usbhid-ups as well. It would give a little more context to the kernel errors. I haven't used a physical CentOS or RedHat system in a while, so I am not sure of the specifics needed to just stop the usbhid-ups driver, but then you can restart it with a few "-D" flags (3 should be sufficient for this kind of problem) and "-a TrippLiteUPS" to match this configuration. Please compress any log files (gzip preferred; zip works).”>> Attached. This is with the settings and udev rules mentioned in the earlier. I can remove and redo if useful.The log format looks okay, but I may not have been clear that I was looking for the log at the same time as when the kernel errors occur. So without the udev rules would be good.>> “pollinterval defaults to 2, and to be honest, for most other UPSes, we suggest that people raise the value (since many UPSes do not update their filtered values more frequently than that anyway).”>> Good to know on the default “2”. Reading upsmon.conf, I thought it was related to “POLLFREQ” which defaults to 30.We've reworked some of the documentation for the next release, but the short answer is that USB HID drivers poll the essential status bits every `pollinterval` seconds ("Quick update..." in the log), and then grab the rest every `pollfreq` seconds ("Full update").>> “Do you know how frequently the Windows software polls the UPS?”>> For our PowerAlert Local software, generally we do half a second for USB (I’d need to check if that holds for all protocols, but I cannot think of any exceptions off the top of my head). In this particular case we’ve just been testing with baked-in Windows Power Options as a reference point here, so I’ll need to look into how frequently Windows polls.Thanks, that is useful to know.> “Should this be applied to other models as well, or just protocol 2012?”>> I’ll dig into that and confirm. I’m not entirely convinced it is a protocol issue, but it very well could be.(By "protocol" I am just referring to the idProduct field, rather than the protocol itself, since that is what the driver would eventually use to enable any special cases.)>> “The 62-nut-usbups.rules file looks pretty standard. Do you know if the changes to 42-usb-hd-pm.rules are needed? It seems like none of the USB devices would have the right permissions if 62-nut-usbups.rules isn't sufficient (though this happened in Debian once).”>> My means of testing wasn’t the most rigorous, but I did try to use variable isolation with these changes and some other changes. I could not make the drops stop without having all 3 of these changes present. I believe a web search lead me to this udev rule so I’ll dig up the link for context.This is starting to make sense, though. The link would be helpful, but no worries if you can't find it. I think you mentioned the CentOS version - which kernel version does that run? ("uname -r" is probably sufficient) I will try to read up on the USB power management settings that the udev file is changing.> “Note that NUT 2.7.4 has been out for some time now.”>> Wolfy nailed this one, I just installed whatever the repository gave me. Should I test with 2.7.4?Might not be necessary.>> “I thought we did, but maybe I am confusing it with protocol 3016 devices. We actually added a lot of the protocol 2012 devices to the hardware compatibility list based on the test results that Eric provided, so I assume they worked then (about six years ago).>>>> The protocol 3016 devices (in particular, the SMART1500LCDT and>> OMNI1500LCDT) sometimes don't even stay on USB long enough to read a>> USB descriptor, and this does seem correlated with newer>> motherboards. Example:>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netwo>> rkupstools_nut_issues_577&d=DwIFaQ&c=f9s1WCuF-N6cmD_YaZ7gBg&r=lhr3k4a>> u5dVQgHY_iS-v_t9g8PHVkn8Px_wyaupZGfQ&m=6vPDH1sqJhcwwVRECtQO7H6Tzh-L4I>> Yo9Ryzpa4A9IE&s=oV3u25jYPtbrcgAWLEAQujFDFzyrf1zXc7KsMK_XDaQ&e >>>> From where I stand, there really shouldn't be anything that a user-space program (like a NUT driver) can do that should be able to cause a USB device to disconnect during normal polling. (Aside from firmware updates, which we don't attempt.) That said, I recognize that USB Phy layer problems can be hard to diagnose, and power management can compound the issue.”>>> Interesting threads. Are you still seeing issues with LCDTs or have they subsided?I moved my SMART1500LCDT off of the always-on system, so I don't have continuous data for it. I don't think it worked on Raspbian stretch the last time I tried. Others have rigged up scripts to reset a USB hub to simulate re-plugging the UPS USB connection.>> “Some users prefer not to post the entire serial numbers from their UPS when reporting issues. Is there a convention for the serial number digits such that we can ask for just the first few digits, and get an idea as to whether the problem is limited to a given hardware or firmware revision? There seems to be a firmware revision buried in the HID descriptor for some models, but I don't know how to interpret it, and some of these connection problems present themselves before the UPS can return that HID report.”>> Our serial numbers break down like this:> https://www.tripplite.com/support/identify-products> If they give you the first 13 characters of the serial number, you’d know the SKU and the datecode without having the full serial number. Firmware isn’t inherently baked into that though. i.e. you could have the same firmware on different SKUs.> Does this help at all?Very useful, thanks!> “Although these models are not as common, we still hear from users with non-HID-PDC-based USB devices (and some serial UPSes as well). Publicly-available protocol documents would help us write better drivers for those devices. If not, a better way to identify models with proprietary protocols would be useful.”>> Does publicly available protocol = needs to be accessible to anyone Or> We provide you with protocol docs if you agree not to share?> I can look into the latter. I feel like we should be able to help here.I think that a lot of the content of the protocol docs would end up in the logic of the source code. Trying to write open-source code based on a document that can't be shared seems risky to me.>> “Another thing is considering how users get started with NUT. Sometimes a user inherits an UPS on a given system, and they want to set up NUT to monitor it. Ideally, we'd like to have a way for them to quickly triage whether a particular UPS model will work, and before they have NUT installed, they will likely have "lsusb" or similar tools to enumerate devices. Other times, a user is replacing another UPS, and they want to know which models are supported by NUT before purchasing one. In both of those cases, more information about how USB IDs map to models can help smooth out those processes. At the moment, we manually add each protocol number to the usbhid-ups driver when a user tries an UPS that isn't listed already. If there were a convention that all USB idDevice values in a certain range were going to be HID PDC compliant, we could change the default from opt-in to enabled-by-default (but we wouldn't want the UPS driver to try to control a USB hub).”>>> Let me kick this around. If we were able to say: USB UPSes with IDs from “09ae:XXXX” to 09ae:YYYY” are PDC compliant, would that be enough?> Topically the problem I can think of is determining if this holds for older units (I’ll dig into this).I think a range of IDs (even with a few exceptions for the older units) would be sufficient. The ideal scenario is that the range is somewhat future-proof, so a version of NUT from this year can properly identify next year's UPS. If not, we still have manual ways for users to add their idProduct to ups.conf and the udev files, but as you can imagine, that is frustrating for a new user. ________________________________ This message is for the addressee's use only. It may contain confidential information. If you receive this message in error, please delete it and notify the sender. Tripp Lite disclaims all warranties and liabilities, and assumes no responsibility for viruses which may infect an email sent to you from Tripp Lite and which damage your electronic systems or information. It is your responsibility to maintain virus detection systems to prevent damage to your electronic systems and information. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://alioth-lists.debian.net/pipermail/nut-upsdev/attachments/20190619/f78da1d8/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: norules.log.gz Type: application/x-gzip Size: 348 bytes Desc: norules.log.gz URL: <http://alioth-lists.debian.net/pipermail/nut-upsdev/attachments/20190619/f78da1d8/attachment-0002.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: norule42.log.gz Type: application/x-gzip Size: 16001 bytes Desc: norule42.log.gz URL: <http://alioth-lists.debian.net/pipermail/nut-upsdev/attachments/20190619/f78da1d8/attachment-0003.bin>
Charles Lepple
2019-Jun-20 12:46 UTC
[Nut-upsdev] [EXTERNAL] Fixing Drops With SMART1500LCDXL & USB-HID Driver
On Jun 19, 2019, at 4:16 PM, David Zomaya wrote:> > Interestingly, I did some tinkering with openSUSE 15.1 Leap on a physical box and saw the same drops there, but was able to get them to stop after installing NUT and just doing the “standard” configuration. However, I ran into some connection refused messages, e.g.: > > user at linux-nxmm:/dev/bus> sudo upsc -l > Error: Connection failure: Connection refused >I keep meaning to put together a diagram to help with this, but in the mean time: UPS <-- driver <-- upsd <-- clients (upsc, upsmon, NUT Monitor GUI, etc.) "upsc -l" is making a TCP connection to upsd localhost:3493 (no sudo needed here), so a "Connection refused" is likely due to upsd not running. (If it were over the network, I'd recommend first logging into the master system and running "upsc" there, then check for the LISTEN address in upsd.conf, and any firewalls in between the systems running upsd and upsc.) You should be able to get it working for testing with a quick "sudo upsd", though it is probably some sort of startup script issue. I do not know exactly how openSUSE's packages interact with their init system- maybe others on the list can help. (A quick search of nut-upsuser indicates that 42.3 is using systemd, so if 15.1 does as well, maybe "systemctl | grep ^nut" will show a unit that just needs to be restarted.)
Charles Lepple
2019-Jun-20 13:20 UTC
[Nut-upsdev] [EXTERNAL] Fixing Drops With SMART1500LCDXL & USB-HID Driver
On Jun 19, 2019, at 4:16 PM, David Zomaya wrote:> >> “I think a range of IDs (even with a few exceptions for the older units) would be sufficient. The ideal scenario is that the range is somewhat future-proof, so a version of NUT from this year can properly identify next year's UPS. If not, we still have manual ways for users to add their idProduct to ups.conf and the udev files, but as you can imagine, that is frustrating for a new user.” > > Almost everything we make now is PDC compliant, so we should at least be able to do this. We’re looking into something easy to identify in the output of lsusb –v though since that could help with legacy support. >As you probably saw with lsusb, some of the output is not shown when not running as root (it does not look up descriptors, for instance, including string descriptor). One value that should always be available is "wDescriptorLength". PDC-compliant UPSes need several hundred bytes in the HID descriptor, whereas the proprietary protocols are just describing an 8-byte buffer (or similar). (I'm still noodling over the 42-usb-hd-pm.rules file - not sure why the stop/start cycle is necessary, because the logs indicated that the driver is reconnecting.) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://alioth-lists.debian.net/pipermail/nut-upsdev/attachments/20190620/45eb0626/attachment.html>
David Zomaya
2019-Jun-21 18:54 UTC
[Nut-upsdev] [EXTERNAL] Fixing Drops With SMART1500LCDXL & USB-HID Driver
Ok, so I have made some progress that seems to suggest the initial “fix” I stumbled upon isn’t required. I haven’t yet worked backwards to figure out reasons why our customers have had similar complaints or solve the problem on the CentOS VM yet, but that is on the todo list. I am hoping it ends up just being not enabling the service on startup, but I can’t say that explains away all the symptoms just yet. For now, below is where I am at after testing with CentOS 7.6 and the same SMART1500LCDXL (SM886B) UPS on an HP Compaq Pro 6305. · No special udev rules seemed to be needed. · I did not need to add “pollinterval = 1” to the ups.conf. · It seemed that starting upsd and the driver would fix the drops, but they would return after reboots of the operating system. · Enabling NUT to start with: $ sudo systemctl enable nut-server.service $ sudo systemctl enable nut-monitor.service Seemed to make the drops go away completely. These leaves me with the questions of “why are we dropping in the first place?” (obviously not a NUT problem and maybe not as big a deal now that it seems to work) and “why has it been impacting users ability to use this model and a similar model?”. I am hoping to better understand this as I go back and dig into the other issues. Any inputs are appreciated. Below are the details from the system, UPS, and version of NUT. Kernel: 3.10.0-957.el7.x86_64 CentOS info: NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" Network UPS Tools (installed from EPEL repo using “yum install nut”) version: Network UPS Tools upsd 2.7.2 Messages in /var/log/messages when UPS was dropping (note that you were correct about the serial number reporting “0”): Jun 21 11:34:49 localhost kernel: usb 8-2: USB disconnect, device number 103 Jun 21 11:34:49 localhost kernel: usb 8-2: new low-speed USB device number 104 using xhci_hcd Jun 21 11:34:49 localhost kernel: usb 8-2: New USB device found, idVendor=09ae, idProduct=2012 Jun 21 11:34:49 localhost kernel: usb 8-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Jun 21 11:34:49 localhost kernel: usb 8-2: Product: Tripp Lite UPS Jun 21 11:34:49 localhost kernel: usb 8-2: Manufacturer: Tripp Lite Jun 21 11:34:50 localhost kernel: hid-generic 0003:09AE:2012.1027: hiddev0,hidraw0: USB HID v1.10 Device [Tripp Lite Tripp Lite UPS ] on usb-0000:00:10.1-2/input0 Jun 21 11:35:04 localhost kernel: usb 8-2: USB disconnect, device number 104 Jun 21 11:35:04 localhost kernel: usb 8-2: new low-speed USB device number 105 using xhci_hcd Jun 21 11:35:05 localhost kernel: usb 8-2: New USB device found, idVendor=09ae, idProduct=2012 Jun 21 11:35:05 localhost kernel: usb 8-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Jun 21 11:35:05 localhost kernel: usb 8-2: Product: Tripp Lite UPS Jun 21 11:35:05 localhost kernel: usb 8-2: Manufacturer: Tripp Lite Jun 21 11:35:05 localhost kernel: hid-generic 0003:09AE:2012.1028: hiddev0,hidraw0: USB HID v1.10 Device [Tripp Lite Tripp Lite UPS ] on usb-0000:00:10.1-2/input0 Jun 21 11:35:19 localhost kernel: usb 8-2: USB disconnect, device number 105 Jun 21 11:35:19 localhost kernel: usb 8-2: new low-speed USB device number 106 using xhci_hcd Jun 21 11:35:20 localhost kernel: usb 8-2: New USB device found, idVendor=09ae, idProduct=2012 Jun 21 11:35:20 localhost kernel: usb 8-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Jun 21 11:35:20 localhost kernel: usb 8-2: Product: Tripp Lite UPS Jun 21 11:35:20 localhost kernel: usb 8-2: Manufacturer: Tripp Lite Jun 21 11:35:20 localhost kernel: hid-generic 0003:09AE:2012.1029: hiddev0,hidraw0: USB HID v1.10 Device [Tripp Lite Tripp Lite UPS ] on usb-0000:00:10.1-2/input0 /etc/ups/ups.conf [TrippLiteUPS] driver = usbhid-ups port = auto desc = "SMART1500LCD" sudo lsusb –v output for UPS Bus 008 Device 098: ID 09ae:2012 Tripp Lite Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 1.10 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 8 idVendor 0x09ae Tripp Lite idProduct 0x2012 bcdDevice 0.09 iManufacturer 1 Tripp Lite iProduct 2 Tripp Lite UPS iSerial 0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 34 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 3 Human Interface Device bInterfaceSubClass 0 No Subclass bInterfaceProtocol 0 None iInterface 0 HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 1.10 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 662 Report Descriptors: ** UNAVAILABLE ** Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0008 1x 8 bytes bInterval 40 Device Status: 0x0001 Self Powered **************************** On a related note: Charles, This “I keep meaning to put together a diagram to help with this, but in the mean time: UPS <-- driver <-- upsd <-- clients (upsc, upsmon, NUT Monitor GUI, etc.) "upsc -l" is making a TCP connection to upsd localhost:3493 (no sudo needed here), so a "Connection refused" is likely due to upsd not running. (If it were over the network, I'd recommend first logging into the master system and running "upsc" there, then check for the LISTEN address in upsd.conf, and any firewalls in between the systems running upsd and upsc.) You should be able to get it working for testing with a quick "sudo upsd", though it is probably some sort of startup script issue. I do not know exactly how openSUSE's packages interact with their init system- maybe others on the list can help. (A quick search of nut-upsuser indicates that 42.3 is using systemd, so if 15.1 does as well, maybe "systemctl | grep ^nut" will show a unit that just needs to be restarted.)” Was very helpful in conceptualizing and working through the setup. Thank you. **************************** Thank you, David Zomaya Tripp Lite -----Original Message----- From: Charles Lepple <clepple at gmail.com> Sent: Thursday, June 20, 2019 7:47 AM To: David Zomaya <David_Zomaya at tripplite.com> Cc: nut-upsdev at alioth-lists.debian.net; Jonathan Manzanilla <Jonathan_Manzanilla at tripplite.com>; Eric Cobb <Eric_Cobb at tripplite.com> Subject: Re: [EXTERNAL] [Nut-upsdev] Fixing Drops With SMART1500LCDXL & USB-HID Driver On Jun 19, 2019, at 4:16 PM, David Zomaya wrote:>> Interestingly, I did some tinkering with openSUSE 15.1 Leap on a physical box and saw the same drops there, but was able to get them to stop after installing NUT and just doing the “standard” configuration. However, I ran into some connection refused messages, e.g.:>> user at linux-nxmm:/dev/bus> sudo upsc -l> Error: Connection failure: Connection refused>I keep meaning to put together a diagram to help with this, but in the mean time: UPS <-- driver <-- upsd <-- clients (upsc, upsmon, NUT Monitor GUI, etc.) "upsc -l" is making a TCP connection to upsd localhost:3493 (no sudo needed here), so a "Connection refused" is likely due to upsd not running. (If it were over the network, I'd recommend first logging into the master system and running "upsc" there, then check for the LISTEN address in upsd.conf, and any firewalls in between the systems running upsd and upsc.) You should be able to get it working for testing with a quick "sudo upsd", though it is probably some sort of startup script issue. I do not know exactly how openSUSE's packages interact with their init system- maybe others on the list can help. (A quick search of nut-upsuser indicates that 42.3 is using systemd, so if 15.1 does as well, maybe "systemctl | grep ^nut" will show a unit that just needs to be restarted.) ________________________________ This message is for the addressee's use only. It may contain confidential information. If you receive this message in error, please delete it and notify the sender. Tripp Lite disclaims all warranties and liabilities, and assumes no responsibility for viruses which may infect an email sent to you from Tripp Lite and which damage your electronic systems or information. It is your responsibility to maintain virus detection systems to prevent damage to your electronic systems and information. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://alioth-lists.debian.net/pipermail/nut-upsdev/attachments/20190621/352cbf9d/attachment-0001.html>
Reasonably Related Threads
- [EXTERNAL] Re: Fixing Drops With SMART1500LCDXL & USB-HID Driver
- Fixing Drops With SMART1500LCDXL & USB-HID Driver
- Fixing Drops With SMART1500LCDXL & USB-HID Driver
- [EXTERNAL] Re: Fixing Drops With SMART1500LCDXL & USB-HID Driver
- [EXTERNAL] Re: Fixing Drops With SMART1500LCDXL & USB-HID Driver