A server just all of a sudden dropped from the network.
uptime was 26days.
This got my ZFS server hanging:
Aug 1 23:39:58 zfs kernel: em0: Watchdog timeout -- resetting
Aug 1 23:39:58 zfs kernel: em0: Queue(0) tdh = 942, hw tdt = 977
Aug 1 23:39:58 zfs kernel: em0: TX(0) desc avail = 985,Next TX to Clean
= 938
Aug 1 23:43:24 zfs kernel: em0: Watchdog timeout -- resetting
Aug 1 23:43:24 zfs kernel: em0: Queue(0) tdh = 147, hw tdt = 163
Aug 1 23:43:24 zfs kernel: em0: TX(0) desc avail = 1006,Next TX to
Clean = 145
ifconfig down/up did not fix anything, un/plugging the ethernet did not
do anything either. rebooting did fix it.
Serious maintenance jobs only starts after 0:00.
--WjW
uname -a:
FreeBSD zfs.digiware.nl 8.2-STABLE FreeBSD 8.2-STABLE #10: Wed Jul 6
21:57:36 CEST 2011
root@zfs.digiware.nl:/home/obj/usr/src/src8/src/sys/ZFS amd64
pciconf -lv:
hostb0@pci0:0:0:0: class=0x060000 card=0xd98015d9 chip=0x29e08086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X38/X48 (Bearlake) Processor to I/O Controller'
class = bridge
subclass = HOST-PCI
em0@pci0:0:25:0: class=0x020000 card=0x10bd15d9 chip=0x10bd8086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)'
class = network
subclass = ethernet
uhci0@pci0:0:26:0: class=0x0c0300 card=0xd98015d9 chip=0x29378086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB Universal Host
Controller'
class = serial bus
subclass = USB
uhci1@pci0:0:26:1: class=0x0c0300 card=0xd98015d9 chip=0x29388086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB Universal Host
Controller'
class = serial bus
subclass = USB
uhci2@pci0:0:26:2: class=0x0c0300 card=0xd98015d9 chip=0x29398086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB Universal Host
Controller'
class = serial bus
subclass = USB
ehci0@pci0:0:26:7: class=0x0c0320 card=0xd98015d9 chip=0x293c8086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB2 Enhanced Host
Controller'
class = serial bus
subclass = USB
none0@pci0:0:27:0: class=0x040300 card=0xd98015d9 chip=0x293e8086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) HD Audio Controller'
class = multimedia
subclass = HDA
pcib1@pci0:0:28:0: class=0x060400 card=0xd98015d9 chip=0x29408086
rev=0x02 hdr=0x01
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) PCIe Root Port 1'
class = bridge
subclass = PCI-PCI
uhci3@pci0:0:29:0: class=0x0c0300 card=0xd98015d9 chip=0x29348086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB Universal Host
Controller'
class = serial bus
subclass = USB
uhci4@pci0:0:29:1: class=0x0c0300 card=0xd98015d9 chip=0x29358086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB Universal Host
Controller'
class = serial bus
subclass = USB
uhci5@pci0:0:29:2: class=0x0c0300 card=0xd98015d9 chip=0x29368086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB Universal Host
Controller'
class = serial bus
subclass = USB
ehci1@pci0:0:29:7: class=0x0c0320 card=0xd98015d9 chip=0x293a8086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) USB2 Enhanced Host
Controller'
class = serial bus
subclass = USB
pcib4@pci0:0:30:0: class=0x060401 card=0xd98015d9 chip=0x244e8086
rev=0x92 hdr=0x01
vendor = 'Intel Corporation'
device = '82801 Family (ICH2/3/4/5/6/7/8/9,63xxESB) Hub
Interface to PCI Bridge'
class = bridge
subclass = PCI-PCI
isab0@pci0:0:31:0: class=0x060100 card=0xd98015d9 chip=0x29168086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IR (ICH9R) LPC Interface Controller'
class = bridge
subclass = PCI-ISA
atapci1@pci0:0:31:2: class=0x010601 card=0xd98015d9 chip=0x29228086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) 6 port SATA AHCI
Controller'
class = mass storage
subclass = SATA
none1@pci0:0:31:3: class=0x0c0500 card=0xd98015d9 chip=0x29308086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel(R) ICH9 Family SMBus Controller working fine
with http://download.cnet.com/Chipset-Driver-Inte (8086)'
class = serial bus
subclass = SMBus
none2@pci0:0:31:6: class=0x118000 card=0x000015d9 chip=0x29328086
rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = '82801IB/IR/IH (ICH9 Family) Thermal Subsystem'
class = dasp
pcib2@pci0:5:0:0: class=0x060400 card=0x00000000 chip=0x032c8086
rev=0x09 hdr=0x01
vendor = 'Intel Corporation'
device = 'PCI Express-to-PCI Express Bridge (6702PXH)'
class = bridge
subclass = PCI-PCI
ioapic0@pci0:5:0:1: class=0x080020 card=0xd98015d9 chip=0x03268086
rev=0x09 hdr=0x00
vendor = 'Intel Corporation'
device = '6700/6702PXH I/OxAPIC Interrupt Controller A'
class = base peripheral
subclass = interrupt controller
pcib3@pci0:6:7:0: class=0x060400 card=0x00000000 chip=0x03358086
rev=0x0a hdr=0x01
vendor = 'Intel Corporation'
device = '80331 [Lindsay] I/O processor PCI-X bridge'
class = bridge
subclass = PCI-PCI
arcmsr0@pci0:7:14:0: class=0x010400 card=0x112017d3 chip=0x112017d3
rev=0x00 hdr=0x00
vendor = 'Areca Technology Corporation'
device = 'ARC-1120 8-Port PCI-X to SATA RAID Controller'
class = mass storage
subclass = RAID
vgapci0@pci0:17:1:0: class=0x030000 card=0x47501002 chip=0x47501002
rev=0x5c hdr=0x00
vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.'
device = 'ATI 3D Rage Pro 215GP (ATI 3D Rage Pro 215GP)'
class = display
subclass = VGA
atapci0@pci0:17:4:0: class=0x010185 card=0x82131283 chip=0x82131283
rev=0x00 hdr=0x00
vendor = 'Integrated Technology Express (ITE) Inc'
device = 'IDE Controller (IT8213F)'
class = mass storage
subclass = ATA
On Tue, Aug 02, 2011 at 12:27:57AM +0200, Willem Jan Withagen wrote:> A server just all of a sudden dropped from the network. > uptime was 26days. > > This got my ZFS server hanging: > > Aug 1 23:39:58 zfs kernel: em0: Watchdog timeout -- resetting > Aug 1 23:39:58 zfs kernel: em0: Queue(0) tdh = 942, hw tdt = 977 > Aug 1 23:39:58 zfs kernel: em0: TX(0) desc avail = 985,Next TX to Clean > = 938 > Aug 1 23:43:24 zfs kernel: em0: Watchdog timeout -- resetting > Aug 1 23:43:24 zfs kernel: em0: Queue(0) tdh = 147, hw tdt = 163 > Aug 1 23:43:24 zfs kernel: em0: TX(0) desc avail = 1006,Next TX to > Clean = 145 > > ifconfig down/up did not fix anything, un/plugging the ethernet did not > do anything either. rebooting did fix it. > > Serious maintenance jobs only starts after 0:00. > > --WjW > > uname -a: > FreeBSD zfs.digiware.nl 8.2-STABLE FreeBSD 8.2-STABLE #10: Wed Jul 6 > 21:57:36 CEST 2011 > root@zfs.digiware.nl:/home/obj/usr/src/src8/src/sys/ZFS amd64Please provide "dmesg" output pertaining to the NIC (dmesg | grep em0 would be sufficient).> pciconf -lv:Please re-run this with -lvcb and include only the Intel NIC (em0). Also please provide output from command "sysctl dev.em.0". Thanks. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
Do you happen to run nfs on the server? I had weird problems with igb-timeouts when many nfs-reads occured and a down and up on the interface would restore the network connection for a while. I had vmware-servers on a nfs-share and either when booting or installing programs from windows Regards Claus Sendt fra min iPhone Den 02/08/2011 kl. 00.27 skrev Willem Jan Withagen <wjw@digiware.nl>:> A server just all of a sudden dropped from the network. > uptime was 26days. > > This got my ZFS server hanging: > > Aug 1 23:39:58 zfs kernel: em0: Watchdog timeout -- resetting > Aug 1 23:39:58 zfs kernel: em0: Queue(0) tdh = 942, hw tdt = 977 > Aug 1 23:39:58 zfs kernel: em0: TX(0) desc avail = 985,Next TX to Clean > = 938 > Aug 1 23:43:24 zfs kernel: em0: Watchdog timeout -- resetting > Aug 1 23:43:24 zfs kernel: em0: Queue(0) tdh = 147, hw tdt = 163 > Aug 1 23:43:24 zfs kernel: em0: TX(0) desc avail = 1006,Next TX to > Clean = 145 > > ifconfig down/up did not fix anything, un/plugging the ethernet did not > do anything either. rebooting did fix it. > > Serious maintenance jobs only starts after 0:00. > > --WjW > > uname -a: > FreeBSD zfs.digiware.nl 8.2-STABLE FreeBSD 8.2-STABLE #10: Wed Jul 6 > 21:57:36 CEST 2011 > root@zfs.digiware.nl:/home/obj/usr/src/src8/src/sys/ZFS amd64 > > pciconf -lv: > hostb0@pci0:0:0:0: class=0x060000 card=0xd98015d9 chip=0x29e08086 > rev=0x01 hdr=0x00 > vendor = 'Intel Corporation' > device = 'X38/X48 (Bearlake) Processor to I/O Controller' > class = bridge > subclass = HOST-PCI > em0@pci0:0:25:0: class=0x020000 card=0x10bd15d9 chip=0x10bd8086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)' > class = network > subclass = ethernet > uhci0@pci0:0:26:0: class=0x0c0300 card=0xd98015d9 chip=0x29378086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB Universal Host Controller' > class = serial bus > subclass = USB > uhci1@pci0:0:26:1: class=0x0c0300 card=0xd98015d9 chip=0x29388086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB Universal Host Controller' > class = serial bus > subclass = USB > uhci2@pci0:0:26:2: class=0x0c0300 card=0xd98015d9 chip=0x29398086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB Universal Host Controller' > class = serial bus > subclass = USB > ehci0@pci0:0:26:7: class=0x0c0320 card=0xd98015d9 chip=0x293c8086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB2 Enhanced Host Controller' > class = serial bus > subclass = USB > none0@pci0:0:27:0: class=0x040300 card=0xd98015d9 chip=0x293e8086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) HD Audio Controller' > class = multimedia > subclass = HDA > pcib1@pci0:0:28:0: class=0x060400 card=0xd98015d9 chip=0x29408086 > rev=0x02 hdr=0x01 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) PCIe Root Port 1' > class = bridge > subclass = PCI-PCI > uhci3@pci0:0:29:0: class=0x0c0300 card=0xd98015d9 chip=0x29348086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB Universal Host Controller' > class = serial bus > subclass = USB > uhci4@pci0:0:29:1: class=0x0c0300 card=0xd98015d9 chip=0x29358086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB Universal Host Controller' > class = serial bus > subclass = USB > uhci5@pci0:0:29:2: class=0x0c0300 card=0xd98015d9 chip=0x29368086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB Universal Host Controller' > class = serial bus > subclass = USB > ehci1@pci0:0:29:7: class=0x0c0320 card=0xd98015d9 chip=0x293a8086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) USB2 Enhanced Host Controller' > class = serial bus > subclass = USB > pcib4@pci0:0:30:0: class=0x060401 card=0xd98015d9 chip=0x244e8086 > rev=0x92 hdr=0x01 > vendor = 'Intel Corporation' > device = '82801 Family (ICH2/3/4/5/6/7/8/9,63xxESB) Hub > Interface to PCI Bridge' > class = bridge > subclass = PCI-PCI > isab0@pci0:0:31:0: class=0x060100 card=0xd98015d9 chip=0x29168086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IR (ICH9R) LPC Interface Controller' > class = bridge > subclass = PCI-ISA > atapci1@pci0:0:31:2: class=0x010601 card=0xd98015d9 chip=0x29228086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) 6 port SATA AHCI Controller' > class = mass storage > subclass = SATA > none1@pci0:0:31:3: class=0x0c0500 card=0xd98015d9 chip=0x29308086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel(R) ICH9 Family SMBus Controller working fine > with http://download.cnet.com/Chipset-Driver-Inte (8086)' > class = serial bus > subclass = SMBus > none2@pci0:0:31:6: class=0x118000 card=0x000015d9 chip=0x29328086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801IB/IR/IH (ICH9 Family) Thermal Subsystem' > class = dasp > pcib2@pci0:5:0:0: class=0x060400 card=0x00000000 chip=0x032c8086 > rev=0x09 hdr=0x01 > vendor = 'Intel Corporation' > device = 'PCI Express-to-PCI Express Bridge (6702PXH)' > class = bridge > subclass = PCI-PCI > ioapic0@pci0:5:0:1: class=0x080020 card=0xd98015d9 chip=0x03268086 > rev=0x09 hdr=0x00 > vendor = 'Intel Corporation' > device = '6700/6702PXH I/OxAPIC Interrupt Controller A' > class = base peripheral > subclass = interrupt controller > pcib3@pci0:6:7:0: class=0x060400 card=0x00000000 chip=0x03358086 > rev=0x0a hdr=0x01 > vendor = 'Intel Corporation' > device = '80331 [Lindsay] I/O processor PCI-X bridge' > class = bridge > subclass = PCI-PCI > arcmsr0@pci0:7:14:0: class=0x010400 card=0x112017d3 chip=0x112017d3 > rev=0x00 hdr=0x00 > vendor = 'Areca Technology Corporation' > device = 'ARC-1120 8-Port PCI-X to SATA RAID Controller' > class = mass storage > subclass = RAID > vgapci0@pci0:17:1:0: class=0x030000 card=0x47501002 chip=0x47501002 > rev=0x5c hdr=0x00 > vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' > device = 'ATI 3D Rage Pro 215GP (ATI 3D Rage Pro 215GP)' > class = display > subclass = VGA > atapci0@pci0:17:4:0: class=0x010185 card=0x82131283 chip=0x82131283 > rev=0x00 hdr=0x00 > vendor = 'Integrated Technology Express (ITE) Inc' > device = 'IDE Controller (IT8213F)' > class = mass storage > subclass = ATA > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
On 2011-08-02 0:49, Claus Guttesen wrote:> Do you happen to run nfs on the server? > > I had weird problems with igb-timeouts when many nfs-reads occured > and a down and up on the interface would restore the network > connection for a while. I had vmware-servers on a nfs-share and > either when booting or installing programs from windowsYup, this server runs: nfsd samba rsyncd --WjW