thr3ads.net - freebsd stable - mfi(4) IO performance regression, post 8.1 [Jun 2012]

If this information is useful, please help other people find it:
Share via:

Charles Owens

2012-Jun-15 05:19 UTC

mfi(4) IO performance regression, post 8.1

Hello FreeBSD folk,

We're seeing what appears to be a storage performance regression as we 
try to move from 8.1 (i386) to 8.3.   We looked at 8.2 also and it 
appears that the regression happened between 8.1 and 8.2.

Our system is an Intel S5520UR Server with 12 GB RAM, dual 4-core CPUs.  
Storage is a LSI MegaSAS 1078 controller (mfi) in a RAID-10 
configuration, using UFS + geom_journal for filesystem.

Postgresql performance, as seen via pgbench, dropped by approx 20%.  
This testing was done with our usual PAE-enabled kernels.  We then went 
back to GENERIC kernels and did comparisons using "bonnie", results 
below.  Following that is a kernel boot log.

Notably, we're seeing this regression only with our RAID mfi(4) based 
systems.  Notably, from looking at FreeBSD source changelogs it appears 
that the mfi(4) code has seen some changes since 8.1.

How can I investigate further?  Assistance with sorting this out would 
be greatly appreciated.

Thanks much,

Charles



Bonnie comparison

8.3 GENERIC
               -------Sequential Output-------- ---Sequential Input-- --Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
           100 93580 98.3 82136 26.6 113709 78.8 152081 98.8 3223876 100.0
233590.3 240.4

8.2 GENERIC
               -------Sequential Output-------- ---Sequential Input-- --Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
           100 95923 99.1 84042 56.7 110568 69.8 152088 100.6 4290802 82.8
239779.4 234.5

8.1 GENERIC
               -------Sequential Output-------- ---Sequential Input-- --Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
           100 140708 100.0 164261 44.4 208553 48.5 153472 100.0 3298756 100.0
270325.1 238.4



Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p8 #1: Fri Jan  6 12:13:34 EST 2012
    
cowens@newcastle.greatbaysoftware.com:/usr/obj/usr/relbuild/os/RELENG_8_1/sys/GENERIC
i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz (2394.28-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0x106a5  Family = 6  Model = 1a 
Stepping = 5
  
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  
Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
   AMD Features=0x28100000<NX,RDTSCP,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant
real memory  = 12884901888 (12288 MB)
avail memory = 2289147904 (2183 MB)
ACPI APIC Table:<INTEL  S5520UR>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
  cpu4 (AP): APIC ID:  4
  cpu5 (AP): APIC ID:  5
  cpu6 (AP): APIC ID:  6
  cpu7 (AP): APIC ID:  7
  cpu8 (AP): APIC ID: 16
  cpu9 (AP): APIC ID: 17
  cpu10 (AP): APIC ID: 18
  cpu11 (AP): APIC ID: 19
  cpu12 (AP): APIC ID: 20
  cpu13 (AP): APIC ID: 21
  cpu14 (AP): APIC ID: 22
  cpu15 (AP): APIC ID: 23
ioapic0<Version 2.0>  irqs 0-23 on motherboard
ioapic1<Version 2.0>  irqs 24-47 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd0 at kbdmux0
acpi0:<INTEL S5520UR>  on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0:<24-bit timer at 3.579545MHz>  port 0x408-0x40b on acpi0
cpu0:<ACPI CPU>  on acpi0
cpu1:<ACPI CPU>  on acpi0
cpu2:<ACPI CPU>  on acpi0
cpu3:<ACPI CPU>  on acpi0
cpu4:<ACPI CPU>  on acpi0
cpu5:<ACPI CPU>  on acpi0
cpu6:<ACPI CPU>  on acpi0
cpu7:<ACPI CPU>  on acpi0
cpu8:<ACPI CPU>  on acpi0
cpu9:<ACPI CPU>  on acpi0
cpu10:<ACPI CPU>  on acpi0
cpu11:<ACPI CPU>  on acpi0
cpu12:<ACPI CPU>  on acpi0
cpu13:<ACPI CPU>  on acpi0
cpu14:<ACPI CPU>  on acpi0
cpu15:<ACPI CPU>  on acpi0
acpi_hpet0:<High Precision Event Timer>  iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0:<ACPI Host-PCI bridge>  port 0xcf8-0xcff on acpi0
pci0:<ACPI PCI bus>  on pcib0
pcib1:<ACPI PCI-PCI bridge>  irq 28 at device 1.0 on pci0
pci1:<ACPI PCI bus>  on pcib1
igb0:<Intel(R) PRO/1000 Network Connection version - 1.9.5>  port
0x4020-0x403f mem 0xb1f20000-0xb1f3ffff,0xb1f44000-0xb1f47fff irq 40 at device
0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: Ethernet address: 00:15:17:f2:1b:a0
igb1:<Intel(R) PRO/1000 Network Connection version - 1.9.5>  port
0x4000-0x401f mem 0xb1f00000-0xb1f1ffff,0xb1f40000-0xb1f43fff irq 28 at device
0.1 on pci1
igb1: Using MSIX interrupts with 5 vectors
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: Ethernet address: 00:15:17:f2:1b:a1
pcib2:<ACPI PCI-PCI bridge>  irq 24 at device 3.0 on pci0
pci2:<ACPI PCI bus>  on pcib2
pcib3:<PCI-PCI bridge>  mem 0xb1e00000-0xb1e1ffff irq 24 at device 0.0 on
pci2
pci3:<PCI bus>  on pcib3
pcib4:<PCI-PCI bridge>  irq 24 at device 0.0 on pci3
pci4:<PCI bus>  on pcib4
igb2:<Intel(R) PRO/1000 Network Connection version - 1.9.5>  port
0x3020-0x303f mem
0xb1c60000-0xb1c7ffff,0xb1c40000-0xb1c5ffff,0xb1d04000-0xb1d07fff irq 34 at
device 0.0 on pci4
igb2: Using MSIX interrupts with 10 vectors
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: [ITHREAD]
igb2: Ethernet address: 00:0c:bd:00:9d:eb
igb3:<Intel(R) PRO/1000 Network Connection version - 1.9.5>  port
0x3000-0x301f mem
0xb1c20000-0xb1c3ffff,0xb1c00000-0xb1c1ffff,0xb1d00000-0xb1d03fff irq 24 at
device 0.1 on pci4
igb3: Using MSIX interrupts with 10 vectors
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: Ethernet address: 00:0c:bd:00:9d:ea
pcib5:<PCI-PCI bridge>  irq 34 at device 1.0 on pci3
pci7:<PCI bus>  on pcib5
igb4:<Intel(R) PRO/1000 Network Connection version - 1.9.5>  port
0x2020-0x203f mem
0xb1a60000-0xb1a7ffff,0xb1a40000-0xb1a5ffff,0xb1b04000-0xb1b07fff irq 35 at
device 0.0 on pci7
igb4: Using MSIX interrupts with 10 vectors
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: [ITHREAD]
igb4: Ethernet address: 00:0c:bd:00:9d:ed
igb5:<Intel(R) PRO/1000 Network Connection version - 1.9.5>  port
0x2000-0x201f mem
0xb1a20000-0xb1a3ffff,0xb1a00000-0xb1a1ffff,0xb1b00000-0xb1b03fff irq 34 at
device 0.1 on pci7
igb5: Using MSIX interrupts with 10 vectors
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: [ITHREAD]
igb5: Ethernet address: 00:0c:bd:00:9d:ec
pcib6:<PCI-PCI bridge>  irq 24 at device 4.0 on pci3
pci10:<PCI bus>  on pcib6
pcib7:<PCI-PCI bridge>  irq 34 at device 5.0 on pci3
pci11:<PCI bus>  on pcib7
pcib8:<ACPI PCI-PCI bridge>  irq 30 at device 7.0 on pci0
pci12:<ACPI PCI bus>  on pcib8
pcib9:<ACPI PCI-PCI bridge>  irq 32 at device 9.0 on pci0
pci13:<ACPI PCI bus>  on pcib9
pcib10:<ACPI PCI-PCI bridge>  irq 33 at device 10.0 on pci0
pci14:<ACPI PCI bus>  on pcib10
pci0:<base peripheral, interrupt controller>  at device 16.0 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 16.1 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 17.0 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 17.1 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 20.0 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 20.1 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 20.2 (no driver
attached)
pci0:<base peripheral, interrupt controller>  at device 20.3 (no driver
attached)
pci0:<base peripheral>  at device 22.0 (no driver attached)
pci0:<base peripheral>  at device 22.1 (no driver attached)
pci0:<base peripheral>  at device 22.2 (no driver attached)
pci0:<base peripheral>  at device 22.3 (no driver attached)
pci0:<base peripheral>  at device 22.4 (no driver attached)
pci0:<base peripheral>  at device 22.5 (no driver attached)
pci0:<base peripheral>  at device 22.6 (no driver attached)
pci0:<base peripheral>  at device 22.7 (no driver attached)
uhci0:<Intel 82801JI (ICH10) USB controller USB-D>  port 0x50c0-0x50df irq
19 at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x3000
usbus0:<Intel 82801JI (ICH10) USB controller USB-D>  on uhci0
uhci1:<Intel 82801JI (ICH10) USB controller USB-E>  port 0x50a0-0x50bf irq
19 at device 26.1 on pci0
uhci1: [ITHREAD]
usbus1:<Intel 82801JI (ICH10) USB controller USB-E>  on uhci1
uhci2:<Intel 82801JI (ICH10) USB controller USB-F>  port 0x5080-0x509f irq
19 at device 26.2 on pci0
uhci2: [ITHREAD]
usbus2:<Intel 82801JI (ICH10) USB controller USB-F>  on uhci2
ehci0:<Intel 82801JI (ICH10) USB 2.0 controller USB-B>  mem
0xb2021000-0xb20213ff irq 19 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3:<Intel 82801JI (ICH10) USB 2.0 controller USB-B>  on ehci0
pcib11:<ACPI PCI-PCI bridge>  irq 16 at device 28.0 on pci0
pci15:<ACPI PCI bus>  on pcib11
mfi0:<LSI MegaSAS 1078>  port 0x1000-0x10ff mem
0xb1900000-0xb193ffff,0xb1940000-0xb197ffff irq 16 at device 0.0 on pci15
mfi0: Megaraid SAS driver Ver 3.00
mfi0: 36759 (393046626s/0x0020/info) - Shutdown command received from host
mfi0: 36760 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID
0060/1000/3505/8086)
mfi0: 36761 (boot + 3s/0x0020/info) - Firmware version 1.40.232-1007
mfi0: 36762 (boot + 4s/0x0008/info) - Battery Present
mfi0: 36763 (boot + 5s/0x0020/info) - Board Revision
mfi0: 36764 (boot + 49s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 
reset (Type 03)
mfi0: 36765 (boot + 65s/0x0002/WARN) - PD 111(e0x00/s17) Path 50001017e13bdc00 
reset (Type 03)
mfi0: 36766 (boot + 66s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 
reset (Type 03)
mfi0: 36767 (boot + 82s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 
reset (Type 03)
mfi0: 36768 (boot + 83s/0x0002/WARN) - PD 111(e0x00/s17) Path 50001017e13bdc00 
reset (Type 03)
mfi0: 36769 (boot + 99s/0x0002/WARN) - PD 111(e0x00/s17) Path 50001017e13bdc00 
reset (Type 03)
mfi0: 36770 (boot + 100s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 
reset (Type 03)
mfi0: 36771 (boot + 104s/0x0002/info) - Inserted: PD 01(e0xff/s1)
mfi0: 36772 (boot + 104s/0x0002/info) - Inserted: PD 01(e0xff/s1) Info:
enclPd=ffff, scsiType=0, portMap=00, sasAddr=5000c5001ab0d441,0000000000000000
mfi0: 36773 (boot + 104s/0x0002/info) - Inserted: PD 02(e0xff/s2)
mfi0: 36774 (boot + 104s/0x0002/info) - Inserted: PD 02(e0xff/s2) Info:
enclPd=ffff, scsiType=0, portMap=01, sasAddr=5000c5001ab0d4ed,0000000000000000
mfi0: 36775 (boot + 104s/0x0002/info) - Inserted: PD 03(e0xff/s3)
mfi0: 36776 (boot + 104s/0x0002/info) - Inserted: PD 03(e0xff/s3) Info:
enclPd=ffff, scsiType=0, portMap=04, sasAddr=5000c5001ab0d585,0000000000000000
mfi0: 36777 (boot + 104s/0x0002/info) - Inserted: PD 04(e0xff/s4)
mfi0: 36778 (boot + 104s/0x0002/info) - Inserted: PD 04(e0xff/s4) Info:
enclPd=ffff, scsiType=0, portMap=02, sasAddr=5000c5001ab0d869,0000000000000000
mfi0: 36779 (boot + 104s/0x0002/info) - Inserted: PD 05(e0xff/s5)
mfi0: 36780 (boot + 104s/0x0002/info) - Inserted: PD 05(e0xff/s5) Info:
enclPd=ffff, scsiType=0, portMap=03, sasAddr=5000c5001ab0d655,0000000000000000
mfi0: 36781 (boot + 104s/0x0042/info) - Global Hot Spare created on PD
05(e0xff/s5) (global,rev)
mfi0: 36782 (393046751s/0x0020/info) - Time established as 06/15/12  3:39:11;
(113 seconds since power on)
mfi0: [ITHREAD]
pcib12:<ACPI PCI-PCI bridge>  irq 16 at device 28.4 on pci0
pci16:<ACPI PCI bus>  on pcib12
vgapci0:<VGA-compatible display>  mem
0xb0000000-0xb0ffffff,0xb1800000-0xb1803fff,0xb1000000-0xb17fffff irq 16 at
device 0.0 on pci16
pcib13:<ACPI PCI-PCI bridge>  irq 17 at device 28.5 on pci0
pci17:<ACPI PCI bus>  on pcib13
uhci3:<Intel 82801JI (ICH10) USB controller USB-A>  port 0x5060-0x507f irq
16 at device 29.0 on pci0
uhci3: [ITHREAD]
usbus4:<Intel 82801JI (ICH10) USB controller USB-A>  on uhci3
uhci4:<Intel 82801JI (ICH10) USB controller USB-B>  port 0x5040-0x505f irq
16 at device 29.1 on pci0
uhci4: [ITHREAD]
usbus5:<Intel 82801JI (ICH10) USB controller USB-B>  on uhci4
uhci5:<Intel 82801JI (ICH10) USB controller USB-C>  port 0x5020-0x503f irq
16 at device 29.2 on pci0
uhci5: [ITHREAD]
usbus6:<Intel 82801JI (ICH10) USB controller USB-C>  on uhci5
ehci1:<Intel 82801JI (ICH10) USB 2.0 controller USB-A>  mem
0xb2020000-0xb20203ff irq 16 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7:<Intel 82801JI (ICH10) USB 2.0 controller USB-A>  on ehci1
pcib14:<ACPI PCI-PCI bridge>  at device 30.0 on pci0
pci18:<ACPI PCI bus>  on pcib14
isab0:<PCI-ISA bridge>  at device 31.0 on pci0
isa0:<ISA bus>  on isab0
atapci0:<Intel ICH10 SATA300 controller>  port
0x5138-0x513f,0x514c-0x514f,0x5130-0x5137,0x5148-0x514b,0x5110-0x511f,0x5100-0x510f
irq 18 at device 31.2 on pci0
atapci0: [ITHREAD]
ata2:<ATA channel 0>  on atapci0
ata2: [ITHREAD]
ata3:<ATA channel 1>  on atapci0
ata3: [ITHREAD]
pci0:<serial bus, SMBus>  at device 31.3 (no driver attached)
atapci1:<Intel ICH10 SATA300 controller>  port
0x5128-0x512f,0x5144-0x5147,0x5120-0x5127,0x5140-0x5143,0x50f0-0x50ff,0x50e0-0x50ef
irq 21 at device 31.5 on pci0
atapci1: [ITHREAD]
ata4:<ATA channel 0>  on atapci1
ata4: [ITHREAD]
ata5:<ATA channel 1>  on atapci1
ata5: [ITHREAD]
acpi_button0:<Sleep Button>  on acpi0
atrtc0:<AT realtime clock>  port 0x70-0x71,0x74-0x77 irq 8 on acpi0
uart0:<16550 or compatible>  port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart0: console (9600,n,8,1)
uart1:<16550 or compatible>  port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
pmtimer0 on isa0
orm0:<ISA Option ROMs>  at iomem
0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcffff pnpid ORM0000 on
isa0
sc0:<System console>  at flags 0x100 on isa0
sc0: VGA<16 virtual consoles, flags=0x300>
vga0:<Generic ISA VGA>  at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata0: [ITHREAD]
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ata1: [ITHREAD]
ppc0: parallel port not found.
est0:<Enhanced SpeedStep Frequency Control>  on cpu0
p4tcc0:<CPU Frequency Thermal Control>  on cpu0
est1:<Enhanced SpeedStep Frequency Control>  on cpu1
p4tcc1:<CPU Frequency Thermal Control>  on cpu1
est2:<Enhanced SpeedStep Frequency Control>  on cpu2
p4tcc2:<CPU Frequency Thermal Control>  on cpu2
est3:<Enhanced SpeedStep Frequency Control>  on cpu3
p4tcc3:<CPU Frequency Thermal Control>  on cpu3
est4:<Enhanced SpeedStep Frequency Control>  on cpu4
p4tcc4:<CPU Frequency Thermal Control>  on cpu4
est5:<Enhanced SpeedStep Frequency Control>  on cpu5
p4tcc5:<CPU Frequency Thermal Control>  on cpu5
est6:<Enhanced SpeedStep Frequency Control>  on cpu6
p4tcc6:<CPU Frequency Thermal Control>  on cpu6
est7:<Enhanced SpeedStep Frequency Control>  on cpu7
p4tcc7:<CPU Frequency Thermal Control>  on cpu7
est8:<Enhanced SpeedStep Frequency Control>  on cpu8
p4tcc8:<CPU Frequency Thermal Control>  on cpu8
est9:<Enhanced SpeedStep Frequency Control>  on cpu9
p4tcc9:<CPU Frequency Thermal Control>  on cpu9
est10:<Enhanced SpeedStep Frequency Control>  on cpu10
p4tcc10:<CPU Frequency Thermal Control>  on cpu10
est11:<Enhanced SpeedStep Frequency Control>  on cpu11
p4tcc11:<CPU Frequency Thermal Control>  on cpu11
est12:<Enhanced SpeedStep Frequency Control>  on cpu12
p4tcc12:<CPU Frequency Thermal Control>  on cpu12
est13:<Enhanced SpeedStep Frequency Control>  on cpu13
p4tcc13:<CPU Frequency Thermal Control>  on cpu13
est14:<Enhanced SpeedStep Frequency Control>  on cpu14
p4tcc14:<CPU Frequency Thermal Control>  on cpu14
est15:<Enhanced SpeedStep Frequency Control>  on cpu15
p4tcc15:<CPU Frequency Thermal Control>  on cpu15
Timecounters tick every 1.000 msec
mfi0: 36783 (393046780s/0x0008/info) - Battery temperature is normal
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
mfid0:<MFI Logical Disk>  on mfi0
mfid0: 278472MB (570310656 sectors) RAID volume '' is optimal
lapic18: Forcing LINT1 to edge trigger
SMP: AP CPU #10 Launched!
lapic5: Forcing LINT1 to edge trigger
SMP: AP CPU #5 Launched!
lapic17: Forcing LINT1 to edge trigger
SMP: AP CPU #9 Launched!
lapic3: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
lapic6: Forcing LINT1 to edge trigger
SMP: AP CPU #6 Launched!
lapic23: Forcing LINT1 to edge trigger
SMP: AP CPU #15 Launched!
lapic22: Forcing LINT1 to edge trigger
SMP: AP CPU #14 Launched!
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic4: Forcing LINT1 to edge trigger
SMP: AP CPU #4 Launched!
lapic2: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
lapic16: Forcing LINT1 to edge trigger
SMP: AP CPU #8 Launched!
lapic21: Forcing LINT1 to edge trigger
SMP: AP CPU #13 Launched!
lapic19: Forcing LINT1 to edge trigger
SMP: AP CPU #11 Launched!
lapic20: Forcing LINT1 to edge trigger
SMP: AP CPU #12 Launched!
lapic7: Forcing LINT1 to edge trigger
SMP: AP CPU #7 Launched!
ugen2.1:<Intel>  at usbus2ugen5.1:<Intel>  at
usbus5ugen7.1:<Intel>  at usbus7ugen1.1:<Intel>  at
usbus1ugen4.1:<Intel>  at usbus4ugen6.1:<Intel>  at
usbus6ugen0.1:<Intel>  at usbus0ugen3.1:<Intel>  at usbus3
uhub0:
<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1>  on usbus2


uhub1:
<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1>  on usbus5

uhub2:
<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1>  on usbus0

uhub3:<Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1>  on usbus7
uhub4:<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1>  on usbus6
uhub5:<Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1>  on usbus3
uhub6:<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1>  on usbus1
uhub7:<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1>  on usbus4
GEOM_JOURNAL: Journal 3746672352: mfid0s1a contains data.
GEOM_JOURNAL: Journal 3746672352: mfid0s1a contains journal.
GEOM_JOURNAL: Journal mfid0s1a clean.
GEOM_JOURNALuhub0: : 2 ports with 2 removable, self poweredBIO_FLUSH not
supported by mfid0s1a.
uhub2: 2 ports with 2 removable, self powered

uhub4: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self poweredGEOM_JOURNAL
: Journal 137861468: mfid0s1d contains data.uhub7: 2 ports with 2 removable,
self powered
GEOM_JOURNAL
uhub6: : 2 ports with 2 removable, self poweredJournal 137861468: mfid0s1d
contains journal.

GEOM_JOURNAL: Journal mfid0s1d clean.
GEOM_JOURNAL: BIO_FLUSH not supported by mfid0s1d.
Root mount waiting for: usbus7 usbus3
Root mount waiting for: usbus7 usbus3
uhub5: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
Root mount waiting for: usbus7 usbus3
ugen3.2:<vendor 0x0424>  at usbus3
uhub8:<vendor 0x0424 product 0x2514, class 9/0, rev 2.00/0.00, addr 2>  on
usbus3
ugen7.2:<vendor 0x05e3>  at usbus7
umass0:<vendor 0x05e3 USB Storage, class 0/0, rev 2.00/0.09, addr 2>  on
usbus7
umass0:  SCSI over Bulk-Only; quirks = 0x0000
uhub8: 4 ports with 4 removable, self powered
umass0:0:0:-1: Attached to scbus0
Root mount waiting for: usbus7
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
(probe0:umass-sim0:0:0:0): CAM status: SCSI Status ErrorTrying to mount root
from ufs:/dev/ufs/root
(probe0:umass-sim0:0:0:0): SCSI status: Check Condition

(probe0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present -
tray open)
cd0 at umass-sim0 bus 0 scbus0 target 0 lun 0
cd0:<TEAC DV-28S-W 1.2A>  Removable CD-ROM SCSI-0 device
cd0: 40.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray
open
ugen5.2:<vendor 0x0557>  at usbus5ugen2.2:<American Megatrends Inc.>
at usbus2
uhub9:
<vendor 0x0557 product 0x8021, class 9/0, rev 1.10/1.00, addr 2>  on
usbus5
ukbd0:<Keyboard Interface>  on usbus2
kbd1 at ukbd0
ums0:<Mouse Interface>  on usbus2

ums0: 3 buttons and [Z] coordinates ID=0
uhub9: 4 ports with 4 removable, self powered
ugen5.3:<ATEN International Co. Ltd>  at usbus5
ukbd1:<ATEN International Co. Ltd GCS1808 V3.0.294, class 0/0, rev 1.10/1.00,
addr 3>  on usbus5
kbd2 at ukbd1
uhid0:<ATEN International Co. Ltd GCS1808 V3.0.294, class 0/0, rev 1.10/1.00,
addr 3>  on usbus5
ums1:<ATEN International Co. Ltd GCS1808 V3.0.294, class 0/0, rev 1.10/1.00,
addr 3>  on usbus5
ums1: 5 buttons and [XYZ] coordinates ID=0
ums2:<ATEN International Co. Ltd GCS1808 V3.0.294, class 0/0, rev 1.10/1.00,
addr 3>  on usbus5
ums2: 3 buttons and [Z] coordinates ID=0
eth0: link state changed to UP



-- 
Charles Owens
Great Bay Software, Inc.

Adrian Chadd

2012-Jun-15 05:55 UTC

head link

mfi(4) IO performance regression, post 8.1

Hm, can you try different subversion checkouts of the kernel tree
between 8.1 and 8.3, to pinpoint which commit(s) broke things?



ADrian

Charles Owens

2012-Jun-15 11:06 UTC

head link

mfi(4) IO performance regression, post 8.1

Yes, of course. So far I can say that the major shift appears to have occurred
between 8.1 and 8.2 .

Thanks,

Charles Owens 
Great Bay Software,  Inc.

Sent from my phone

----- Reply message -----
From: "Adrian Chadd" <adrian@freebsd.org>
To: "Charles Owens" <cowens@greatbaysoftware.com>
Cc: <stable@freebsd.org>
Subject: mfi(4) IO performance regression, post 8.1
Date: Fri, Jun 15, 2012 1:55 am


Hm, can you try different subversion checkouts of the kernel tree
between 8.1 and 8.3, to pinpoint which commit(s) broke things?



ADrian

Brian W.

2012-Jun-15 14:39 UTC

head link

mfi(4) IO performance regression, post 8.1

Curious why you are preferring i386 +PAE as opposed to amd64?
On Jun 15, 2012 4:09 AM, "Charles Owens"
<cowens@greatbaysoftware.com>
wrote:
> Yes, of course. So far I can say that the major shift appears to have
> occurred between 8.1 and 8.2 .
>
> Thanks,
>
> Charles Owens
> Great Bay Software,  Inc.
>
> Sent from my phone
>
> ----- Reply message -----
> From: "Adrian Chadd" <adrian@freebsd.org>
> To: "Charles Owens" <cowens@greatbaysoftware.com>
> Cc: <stable@freebsd.org>
> Subject: mfi(4) IO performance regression, post 8.1
> Date: Fri, Jun 15, 2012 1:55 am
>
>
> Hm, can you try different subversion checkouts of the kernel tree
> between 8.1 and 8.3, to pinpoint which commit(s) broke things?
>
>
>
> ADrian
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>

John Baldwin

2012-Jun-15 17:18 UTC

head link

mfi(4) IO performance regression, post 8.1

On Friday, June 15, 2012 12:28:59 am Charles Owens
wrote:> Hello FreeBSD folk,
> 
> We're seeing what appears to be a storage performance regression as we 
> try to move from 8.1 (i386) to 8.3.   We looked at 8.2 also and it 
> appears that the regression happened between 8.1 and 8.2.
> 
> Our system is an Intel S5520UR Server with 12 GB RAM, dual 4-core CPUs.  
> Storage is a LSI MegaSAS 1078 controller (mfi) in a RAID-10 
> configuration, using UFS + geom_journal for filesystem.
> 
> Postgresql performance, as seen via pgbench, dropped by approx 20%.  
> This testing was done with our usual PAE-enabled kernels.  We then went 
> back to GENERIC kernels and did comparisons using "bonnie",
results
> below.  Following that is a kernel boot log.
> 
> Notably, we're seeing this regression only with our RAID mfi(4) based 
> systems.  Notably, from looking at FreeBSD source changelogs it appears 
> that the mfi(4) code has seen some changes since 8.1.
Between 8.1 and 8.2 mfi has not had any significant changes.  The only changes
made to sys/dev/mfi were to add a new constant:
> svn diff svn+ssh://svn.freebsd.org/base/releng/8.1/sys/dev/mfi svn+ssh://svn.freebsd.org/base/releng/8.2/sys/dev/mfi
Index: mfireg.h
==================================================================--- mfireg.h  
(.../8.1/sys/dev/mfi)   (revision 237134)
+++ mfireg.h    (.../8.2/sys/dev/mfi)   (revision 237134)
@@ -975,7 +975,9 @@
        MFI_PD_STATE_OFFLINE = 0x10,
        MFI_PD_STATE_FAILED = 0x11,
        MFI_PD_STATE_REBUILD = 0x14,
-       MFI_PD_STATE_ONLINE = 0x18
+       MFI_PD_STATE_ONLINE = 0x18,
+       MFI_PD_STATE_COPYBACK = 0x20,
+       MFI_PD_STATE_SYSTEM = 0x40
 };
 
 union mfi_ld_ref {

The difference in write performance must be due to something else.  You 
mentioned you are using UFS + gjournal.  I think gjournal uses BIO_FLUSH, so I 
wonder if this is related:

------------------------------------------------------------------------
r212939 | gibbs | 2010-09-20 19:39:00 -0400 (Mon, 20 Sep 2010) | 61 lines

MFC 212160:

Correct bioq_disksort so that bioq_insert_tail() offers barrier semantic.
Add the BIO_ORDERED flag for struct bio and update bio clients to use it.

The barrier semantics of bioq_insert_tail() were broken in two ways:

 o In bioq_disksort(), an added bio could be inserted at the head of
   the queue, even when a barrier was present, if the sort key for
   the new entry was less than that of the last queued barrier bio.

 o The last_offset used to generate the sort key for newly queued bios
   did not stay at the position of the barrier until either the
   barrier was de-queued, or a new barrier (which updates last_offset)
   was queued.  When a barrier is in effect, we know that the disk
   will pass through the barrier position just before the
   "blocked bios" are released, so using the barrier's offset for
   last_offset is the optimal choice.

sys/geom/sched/subr_disk.c:
sys/kern/subr_disk.c:
        o Update last_offset in bioq_insert_tail().

        o Only update last_offset in bioq_remove() if the removed bio is
          at the head of the queue (typically due to a call via
          bioq_takefirst()) and no barrier is active.

        o In bioq_disksort(), if we have a barrier (insert_point is non-NULL),
          set prev to the barrier and cur to it's next element.  Now that
          last_offset is kept at the barrier position, this change isn't
          strictly necessary, but since we have to take a decision branch
          anyway, it does avoid one, no-op, loop iteration in the while
          loop that immediately follows.

        o In bioq_disksort(), bypass the normal sort for bios with the
          BIO_ORDERED attribute and instead insert them into the queue
          with bioq_insert_tail().  bioq_insert_tail() not only gives
          the desired command order during insertion, but also provides
          barrier semantics so that commands disksorted in the future
          cannot pass the just enqueued transaction.

sys/sys/bio.h:
        Add BIO_ORDERED as bit 4 of the bio_flags field in struct bio.

sys/cam/ata/ata_da.c:
sys/cam/scsi/scsi_da.c
        Use an ordered command for SCSI/ATA-NCQ commands issued in
        response to bios with the BIO_ORDERED flag set.

sys/cam/scsi/scsi_da.c
        Use an ordered tag when issuing a synchronize cache command.

        Wrap some lines to 80 columns.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
sys/geom/geom_io.c
        Mark bios with the BIO_FLUSH command as BIO_ORDERED.

Sponsored by:   Spectra Logic Corporation
------------------------------------------------------------------------

Can you try perhaps commenting out the 'bp->bio_flags |= BIO_ORDERED'
line
changed in geom_io.c in 8.2?  That would be effectively reverting this
portion of the diff:

Index: geom_io.c
==================================================================--- geom_io.c 
(.../8.1/sys/geom)      (revision 237134)
+++ geom_io.c   (.../8.2/sys/geom)      (revision 237134)
@@ -265,6 +265,7 @@
        g_trace(G_T_BIO, "bio_flush(%s)", cp->provider->name);
        bp = g_alloc_bio();
        bp->bio_cmd = BIO_FLUSH;
+       bp->bio_flags |= BIO_ORDERED;
        bp->bio_done = NULL;
        bp->bio_attribute = NULL;
        bp->bio_offset = cp->provider->mediasize;

-- 
John Baldwin

Charles Owens

2012-Jun-15 18:58 UTC

head link

mfi(4) IO performance regression, post 8.1

No reason other than history... will be changing over at some point.


Charles Owens
Great Bay Software, Inc.


On 6/15/12 10:38 AM, Brian W. wrote:>
> Curious why you are preferring i386 +PAE as opposed to amd64?
>
> On Jun 15, 2012 4:09 AM, "Charles Owens"
<cowens@greatbaysoftware.com
> <mailto:cowens@greatbaysoftware.com>> wrote:
>
>     Yes, of course. So far I can say that the major shift appears to
>     have occurred between 8.1 and 8.2 .
>
>     Thanks,
>
>     Charles Owens
>     Great Bay Software,  Inc.
>
>     Sent from my phone
>
>     ----- Reply message -----
>     From: "Adrian Chadd" <adrian@freebsd.org
<mailto:adrian@freebsd.org>>
>     To: "Charles Owens" <cowens@greatbaysoftware.com
>     <mailto:cowens@greatbaysoftware.com>>
>     Cc: <stable@freebsd.org <mailto:stable@freebsd.org>>
>     Subject: mfi(4) IO performance regression, post 8.1
>     Date: Fri, Jun 15, 2012 1:55 am
>
>
>     Hm, can you try different subversion checkouts of the kernel tree
>     between 8.1 and 8.3, to pinpoint which commit(s) broke things?
>
>
>
>     ADrian
>
>     _______________________________________________
>     freebsd-stable@freebsd.org <mailto:freebsd-stable@freebsd.org>
>     mailing list
>     http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>     To unsubscribe, send any mail to
>     "freebsd-stable-unsubscribe@freebsd.org
>     <mailto:freebsd-stable-unsubscribe@freebsd.org>"
>

Steve McCoy

2012-Jul-12 20:34 UTC

head link

mfi(4) IO performance regression, post 8.1

On 7/12/12 4:14 PM, Charles Owens wrote:> On Thursday, June 21, 2012 10:36:04 pm Charles Owens wrote:
>>
>> On 6/15/12 8:04 AM, John Baldwin wrote:
>> > On Friday, June 15, 2012 12:28:59 am Charles Owens wrote:
>> >> Hello FreeBSD folk,
>> >>
>> >> We're seeing what appears to be a storage performance
regression as we
>> >> try to move from 8.1 (i386) to 8.3.   We looked at 8.2 also
and it
>> >> appears that the regression happened between 8.1 and 8.2.
>> >>
>> >> Our system is an Intel S5520UR Server with 12 GB RAM, dual
4-core
>> CPUs.
>> >> Storage is a LSI MegaSAS 1078 controller (mfi) in a RAID-10
>> >> configuration, using UFS + geom_journal for filesystem.
>> >>
>> >> Postgresql performance, as seen via pgbench, dropped by approx
20%.
>> >> This testing was done with our usual PAE-enabled kernels.  We
then
>> went
>> >> back to GENERIC kernels and did comparisons using
"bonnie", results
>> >> below.  Following that is a kernel boot log.
>> >>
>> >> Notably, we're seeing this regression only with our RAID
mfi(4) based
>> >> systems.  Notably, from looking at FreeBSD source changelogs
it
>> appears
>> >> that the mfi(4) code has seen some changes since 8.1.
>> > Between 8.1 and 8.2 mfi has not had any significant changes.  The
>> only changes
>> > made to sys/dev/mfi were to add a new constant:
>> >
>> >> svn diff svn+ssh://svn.freebsd.org/base/releng/8.1/sys/dev/mfi
>> > svn+ssh://svn.freebsd.org/base/releng/8.2/sys/dev/mfi
>> > Index: mfireg.h
>> >
==================================================================>> >
--- mfireg.h    (.../8.1/sys/dev/mfi)   (revision 237134)
>> > +++ mfireg.h    (.../8.2/sys/dev/mfi)   (revision 237134)
>> > @@ -975,7 +975,9 @@
>> >          MFI_PD_STATE_OFFLINE = 0x10,
>> >          MFI_PD_STATE_FAILED = 0x11,
>> >          MFI_PD_STATE_REBUILD = 0x14,
>> > -       MFI_PD_STATE_ONLINE = 0x18
>> > +       MFI_PD_STATE_ONLINE = 0x18,
>> > +       MFI_PD_STATE_COPYBACK = 0x20,
>> > +       MFI_PD_STATE_SYSTEM = 0x40
>> >   };
>> >
>> >   union mfi_ld_ref {
>> >
>> > The difference in write performance must be due to something else.
You
>> > mentioned you are using UFS + gjournal.  I think gjournal uses
>> BIO_FLUSH, so I
>> > wonder if this is related:
>> >
>> >
>>
------------------------------------------------------------------------
>> > r212939 | gibbs | 2010-09-20 19:39:00 -0400 (Mon, 20 Sep 2010) |
61
>> lines
>> >
>> > MFC 212160:
>> >
>> > Correct bioq_disksort so that bioq_insert_tail() offers barrier
>> semantic.
>> > Add the BIO_ORDERED flag for struct bio and update bio clients to
>> use it.
>> >
>> > The barrier semantics of bioq_insert_tail() were broken in two
ways:
>> >
>> >   o In bioq_disksort(), an added bio could be inserted at the head
of
>> >     the queue, even when a barrier was present, if the sort key
for
>> >     the new entry was less than that of the last queued barrier
bio.
>> >
>> >   o The last_offset used to generate the sort key for newly queued
bios
>> >     did not stay at the position of the barrier until either the
>> >     barrier was de-queued, or a new barrier (which updates
last_offset)
>> >     was queued.  When a barrier is in effect, we know that the
disk
>> >     will pass through the barrier position just before the
>> >     "blocked bios" are released, so using the
barrier's offset for
>> >     last_offset is the optimal choice.
>> >
>> > sys/geom/sched/subr_disk.c:
>> > sys/kern/subr_disk.c:
>> >          o Update last_offset in bioq_insert_tail().
>> >
>> >          o Only update last_offset in bioq_remove() if the removed
>> bio is
>> >            at the head of the queue (typically due to a call via
>> >            bioq_takefirst()) and no barrier is active.
>> >
>> >          o In bioq_disksort(), if we have a barrier (insert_point
is
>> non-NULL),
>> >            set prev to the barrier and cur to it's next
element.
>> Now that
>> >            last_offset is kept at the barrier position, this
change
>> isn't
>> >            strictly necessary, but since we have to take a
decision
>> branch
>> >            anyway, it does avoid one, no-op, loop iteration in the
>> while
>> >            loop that immediately follows.
>> >
>> >          o In bioq_disksort(), bypass the normal sort for bios
with the
>> >            BIO_ORDERED attribute and instead insert them into the
queue
>> >            with bioq_insert_tail().  bioq_insert_tail() not only
gives
>> >            the desired command order during insertion, but also
>> provides
>> >            barrier semantics so that commands disksorted in the
future
>> >            cannot pass the just enqueued transaction.
>> >
>> > sys/sys/bio.h:
>> >          Add BIO_ORDERED as bit 4 of the bio_flags field in struct
bio.
>> >
>> > sys/cam/ata/ata_da.c:
>> > sys/cam/scsi/scsi_da.c
>> >          Use an ordered command for SCSI/ATA-NCQ commands issued
in
>> >          response to bios with the BIO_ORDERED flag set.
>> >
>> > sys/cam/scsi/scsi_da.c
>> >          Use an ordered tag when issuing a synchronize cache
command.
>> >
>> >          Wrap some lines to 80 columns.
>> >
>> > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
>> > sys/geom/geom_io.c
>> >          Mark bios with the BIO_FLUSH command as BIO_ORDERED.
>> >
>> > Sponsored by:   Spectra Logic Corporation
>> >
>>
------------------------------------------------------------------------
>> >
>> > Can you try perhaps commenting out the 'bp->bio_flags
|>> BIO_ORDERED' line
>> > changed in geom_io.c in 8.2?  That would be effectively reverting
this
>> > portion of the diff:
>> >
>> > Index: geom_io.c
>> >
==================================================================>> >
--- geom_io.c   (.../8.1/sys/geom)      (revision 237134)
>> > +++ geom_io.c   (.../8.2/sys/geom)      (revision 237134)
>> > @@ -265,6 +265,7 @@
>> >          g_trace(G_T_BIO, "bio_flush(%s)",
cp->provider->name);
>> >          bp = g_alloc_bio();
>> >          bp->bio_cmd = BIO_FLUSH;
>> > +       bp->bio_flags |= BIO_ORDERED;
>> >          bp->bio_done = NULL;
>> >          bp->bio_attribute = NULL;
>> >          bp->bio_offset = cp->provider->mediasize;
>> >
>>
>> John... thanks for the suggestion.  I've built and tested a kernel
with
>> this change made.  Result:  no change (same performance as with
>> 8.2-GENERIC). Any thoughts as to where to go next?
>
> Hmm.  That seemed the most plausible candidate when I looked at this.
>
> Do you use quotas (there is one change in UFS related to quotas)?
>
> There are 5 changes that involve sys/kern/vfs_bio.c in 8.2:
> 209459, 212229, 212562, 212583, and 213890.
>
> Can you possibly test out kernels from stable/8 at those revisions on an
> 8.1
> world and see if you can narrow it down futher?
>
> Barring that, can you do a binary search of kernels from stable/8
> between 8.1
> and 8.2 on an 8.1 world to see which commit caused the change in write
> performance?
>
Hi John, I'm working with Charles to narrow this down.

Looks like revision 212229 is the culprit, or at least around the same 
time to it, if this change isn't what slowed things down. The change to 
sys/kern/vfs_bio.c modifies some synchronization in dev_strategy():


--- stable/8/sys/kern/vfs_bio.c	2010/06/23 10:06:57	209459
+++ stable/8/sys/kern/vfs_bio.c	2010/09/05 14:27:55	212229
@@ -3195,6 +3195,7 @@
  {
  	struct cdevsw *csw;
  	struct bio *bip;
+	int ref;

  	if ((!bp->b_iocmd) || (bp->b_iocmd & (bp->b_iocmd - 1)))
  		panic("b_iocmd botch");
@@ -3216,7 +3217,7 @@
  	KASSERT(dev->si_refcount > 0,
  	    ("dev_strategy on un-referenced struct cdev *(%s)",
  	    devtoname(dev)));
-	csw = dev_refthread(dev);
+	csw = dev_refthread(dev, &ref);
  	if (csw == NULL) {
  		g_destroy_bio(bip);
  		bp->b_error = ENXIO;
@@ -3225,7 +3226,7 @@
  		return;
  	}
  	(*csw->d_strategy)(bip);
-	dev_relthread(dev);
+	dev_relthread(dev, ref);
  }

  /*


This is in line with the following changes to dev_refthread and 
dev_relthread, but it's not obvious to me how these alone would impact 
performance. Do you have any thoughts?


--- stable/8/sys/kern/kern_conf.c	2010/07/03 18:19:59	209668
+++ stable/8/sys/kern/kern_conf.c	2010/09/05 14:27:55	212229
@@ -177,12 +177,16 @@
  }

  struct cdevsw *
-dev_refthread(struct cdev *dev)
+dev_refthread(struct cdev *dev, int *ref)
  {
  	struct cdevsw *csw;
  	struct cdev_priv *cdp;

  	mtx_assert(&devmtx, MA_NOTOWNED);
+	if ((dev->si_flags & SI_ETERNAL) != 0) {
+		*ref = 0;
+		return (dev->si_devsw);
+	}
  	dev_lock();
  	csw = dev->si_devsw;
  	if (csw != NULL) {
@@ -193,36 +197,59 @@
  			csw = NULL;
  	}
  	dev_unlock();
+	*ref = 1;
  	return (csw);
  }

?

void	
-dev_relthread(struct cdev *dev)
+dev_relthread(struct cdev *dev, int ref)
  {

  	mtx_assert(&devmtx, MA_NOTOWNED);
+	if (!ref)
+		return;
  	dev_lock();
  	KASSERT(dev->si_threadcount > 0,
  	    ("%s threadcount is wrong", dev->si_name));

freebsd stable - Jun 2012 - mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1

mfi(4) IO performance regression, post 8.1