thr3ads.net - freebsd stable - 8-STABLE Slow Write Speeds on ESXI 4.0 [Aug 2010]

If this information is useful, please help other people find it:
Share via:

Joshua Boyd

2010-Aug-07 01:21 UTC

8-STABLE Slow Write Speeds on ESXI 4.0

Hello,

I'm experiencing slow write speeds on 8-STABLE running on an ESXI 4.0
server, despite whatever tunables I've thrown at it. Read speeds are slower
than they should be, but acceptable. Note, this is a thick provisioned disk,
not thin.

Speeds on Windows hosts are as expected for an MD3000 DAS, 250MB/s or so.

[root@git ~]# dd if=/dev/da0 of=/dev/null bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 3.304514 secs (158658118 bytes/sec

[root@git ~]# dd if=/dev/zero of=/var/testfile bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 52.083421 secs (10066313 bytes/sec)

Output of gstat during write:
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    6     82      0      0    0.0     82  10520   82.5  100.7| da0

[root@git ~]# vmstat da0 2
 procs      memory      page                   disk   faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0   in   sy   cs us sy
id
 0 0 0    664M  3684M    28   0   0   0    59   0   0    4   90  347  0  2
98
 0 0 0    664M  3667M     1   0   0   0    44   0 149  152  120 1340  0  3
97
 0 0 0    664M  3646M     0   0   0   0     0   0 163  166  123 1385  0  2
98
 0 0 0    664M  3626M     0   0   0   0     0   0 160  162  119 1393  0  4
96
 0 0 0    664M  3605M     1   0   0   0     1   0 164  165  156 1435  0  4
96

[root@git ~]# camcontrol tags da0
(pass0:mpt0:0:0:0): device openings: 127

[root@git ~]# camcontrol devlist
<VMware Virtual disk 1.0>          at scbus0 target 0 lun 0 (da0,pass0)

And now, for the rest of the information:

[root@git ~]# uname -ar
FreeBSD git.openfisma.org 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0: Mon Jun
28 17:46:33 EDT 2010     root@git.openfisma.org:/usr/obj/usr/src/sys/GENERIC
 amd64

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #0: Mon Jun 28 17:46:33 EDT 2010
    root@git.openfisma.org:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Quad-Core AMD Opteron(tm) Processor 2350 (1999.81-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f23  Family = 10  Model = 2 
Stepping 3
 
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x80802001<SSE3,CX16,POPCNT,<b31>>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x1e9<LAHF,ExtAPIC,ABM,SSE4A,MAS,Prefetch>
  TSC: P-state invariant
real memory  = 4294967296 (4096 MB)
avail memory = 4099297280 (3909 MB)
ACPI APIC Table: <PTLTD   APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 4 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem
0xd4000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port
0x2000-0x203f
mem 0xd8820000-0xd883ffff,0xd8800000-0xd880ffff irq 18 at device 0.0 on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:50:56:81:3c:c8
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem
0xd9c04000-0xd9c07fff,0xd9c10000-0xd9c1ffff irq 18 at device 0.0 on pci3
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.0.0
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
acpi_button0: <Sleep Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata1-master
UDMA33
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 40960MB (83886080 512 byte sectors: 255H 63S/T 5221C)
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/da0s1a
VMware memory control driver initialized

[root@git /var/log]# mptutil show adapter
mpt0 Adapter:
       Board Name: SAS3444
   Board Assembly:
        Chip Name: C1068E
    Chip Revision: UNUSED
      RAID Levels: none



-- 
Joshua Boyd
JBipNet

E-mail: boydjd@jbip.net

http://www.jbip.net

Ivan Voras

2010-Aug-07 12:25 UTC

head link

8-STABLE Slow Write Speeds on ESXI 4.0

On 7.8.2010 3:21, Joshua Boyd wrote:> Hello,
> 
> I'm experiencing slow write speeds on 8-STABLE running on an ESXI 4.0
> server, despite whatever tunables I've thrown at it. Read speeds are
slower
> than they should be, but acceptable. Note, this is a thick provisioned
disk,
> not thin.
> 
> Speeds on Windows hosts are as expected for an MD3000 DAS, 250MB/s or so.
> 
> [root@git ~]# dd if=/dev/da0 of=/dev/null bs=1M count=500
> 500+0 records in
> 500+0 records out
> 524288000 bytes transferred in 3.304514 secs (158658118 bytes/sec
> 
> [root@git ~]# dd if=/dev/zero of=/var/testfile bs=1M count=500
> 500+0 records in
> 500+0 records out
> 524288000 bytes transferred in 52.083421 secs (10066313 bytes/sec)
I assume you are using UFS and SU? What tunables have you tried?

It's unlikely they will help, but try:

vfs.read_max=32

for read speeds (but test using the UFS file system, not as a raw device
like above), and:

vfs.hirunningspace=8388608
vfs.lorunningspace=4194304

for writes. Again, it's unlikely but I'm interested in results you
achieve.

Ivan Voras

2010-Aug-07 18:23 UTC

head link

8-STABLE Slow Write Speeds on ESXI 4.0

On 7 August 2010 19:03, Joshua Boyd <boydjd@jbip.net>
wrote:> On Sat, Aug 7, 2010 at 7:57 AM, Ivan Voras <ivoras@freebsd.org>
wrote:
>> It's unlikely they will help, but try:
>>
>> vfs.read_max=32
>>
>> for read speeds (but test using the UFS file system, not as a raw
device
>> like above), and:
>>
>> vfs.hirunningspace=8388608
>> vfs.lorunningspace=4194304
>>
>> for writes. Again, it's unlikely but I'm interested in results
you
>> achieve.
>>
>
> This is interesting. Write speeds went up to 40MBish. Still slow, but 4x
> faster than before.
> [root@git ~]# dd if=/dev/zero of=/var/testfile bs=1M count=250
> 250+0 records in
> 250+0 records out
> 262144000 bytes transferred in 6.185955 secs (42377288 bytes/sec)
> [root@git ~]# dd if=/var/testfile of=/dev/null
> 512000+0 records in
> 512000+0 records out
> 262144000 bytes transferred in 0.811397 secs (323077424 bytes/sec)
> So read speeds are up to what they should be, but write speeds are still
> significantly below what they should be.
Well, you *could* double the size of "runningspace" tunables and try
that :)

Basically, in tuning these two settings we are cheating: increasing
read-ahead (read_max) and write in-flight buffering (runningspace) in
order to offload as much IO to the controller (in this case vmware) as
soon as possible, so to reschedule horrible IO-caused context switches
vmware has. It will help sequential performance, but nothing can help
random IOs.

freebsd stable - Aug 2010 - 8-STABLE Slow Write Speeds on ESXI 4.0

8-STABLE Slow Write Speeds on ESXI 4.0

8-STABLE Slow Write Speeds on ESXI 4.0

8-STABLE Slow Write Speeds on ESXI 4.0