Hi all,
After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) i
did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions on the
4 disks.
root@antsrv4 [/data/nocompression] # bonnie++ -u root -d .
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
antsrv4.ant.uni 16G 94 81 151823 39 100780 27 303 99 239853 30 148.3
4
Latency 4733ms 6051ms 8173ms 37504us 917ms 1030ms
Version 1.96 ------Sequential Create------ --------Random Create--------
antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 24183 97 +++++ +++ 21715 98 17209 98 +++++ +++ 21291 97
Latency 19779us 128us 162us 39671us 38us 77us
1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,,,,,24183,97,+++++,+++,21715,98,17209,98,+++++,+++,21291,97,
733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us
I am somewhat concerned about the numbers for per-char-output and
per-char-input. In fact, i have never before seen that low numbers in a bonnie
test. Using a single disk with UFS yields about 6 times as much.
I know that this is not crucial for a normal file server, but i want to rule out
any configuration errors.
Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt any
tuning yet.
BTW: Running OpenSolaris on the same hardware yields 110306 for per-char-write
and 94698 for per-char-read.
--Heinrich
Heinrich Rebehn
University of Bremen
Physics / Electrical and Electronics Engineering
- Department of Telecommunications -
Phone : +49/421/218-62394
Fax : -3341
2010/8/20 Heinrich Rebehn <rebehn@ant.uni-bremen.de>:> Hi all, > > After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions on the 4 disks. > > root@antsrv4 [/data/nocompression] # bonnie++ -u root -d . > Using uid:0, gid:0. > Writing a byte at a time...done > Writing intelligently...done > Rewriting...done > Reading a byte at a time...done > Reading intelligently...done > start 'em...done...done...done...done...done... > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...done. > Create files in random order...done. > Stat files in random order...done. > Delete files in random order...done. > Version ?1.96 ? ? ? ------Sequential Output------ --Sequential Input- --Random- > Concurrency ? 1 ? ? -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine ? ? ? ?Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP ?/sec %CP > antsrv4.ant.uni 16G ? ?94 ?81 151823 ?39 100780 ?27 ? 303 ?99 239853 ?30 148.3 ? 4 > Latency ? ? ? ? ? ? ?4733ms ? ?6051ms ? ?8173ms ? 37504us ? ? 917ms ? ?1030ms > Version ?1.96 ? ? ? ------Sequential Create------ --------Random Create-------- > antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > ? ? ? ? ? ? ?files ?/sec %CP ?/sec %CP ?/sec %CP ?/sec %CP ?/sec %CP ?/sec %CP > ? ? ? ? ? ? ? ? 16 24183 ?97 +++++ +++ 21715 ?98 17209 ?98 +++++ +++ 21291 ?97 > Latency ? ? ? ? ? ? 19779us ? ? 128us ? ? 162us ? 39671us ? ? ?38us ? ? ?77us > 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,,,,,24183,97,+++++,+++,21715,98,17209,98,+++++,+++,21291,97, 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us > > > I am somewhat concerned about the numbers for per-char-output and per-char-input. In fact, i have never before seen that low numbers in a bonnie test. Using a single disk with UFS yields about 6 times as much. > > > I know that this is not crucial for a normal file server, but i want to rule out any configuration errors. > > Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt any tuning yet.Can you try without AHCI ? http://www.phoronix.com/scan.php?page=article&item=freebsd_zfs_cam&num=5> BTW: Running OpenSolaris on the same hardware yields 110306 for per-char-write and 94698 for per-char-read. > > --Heinrich > > > Heinrich Rebehn > > University of Bremen > Physics / Electrical and Electronics Engineering > - Department of Telecommunications - > > Phone : +49/421/218-62394 > Fax ? : ? ? ? ? ? ?-3341 > > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >-- Olivier Smedts? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? _ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ASCII ribbon campaign ( ) e-mail: olivier@gid0.org? ? ? ? - against HTML email & vCards? X www: http://www.gid0.org? ? - against proprietary attachments / \ ? "Il y a seulement 10 sortes de gens dans le monde : ? ceux qui comprennent le binaire, ? et ceux qui ne le comprennent pas."
On 20.08.2010, at 14:52, Vladislav V. Prodan wrote:> 20.08.2010 15:12, Heinrich Rebehn wrote: >> root@antsrv4 [/data/nocompression] # kldstat > > #kldstat -v-------------- next part -------------- Id Refs Address Size Name 1 9 0xffffffff80100000 d6aa98 kernel (/boot/kernel/kernel) Contains modules: Id Name 93 ataraid 340 if_lo 407 elf32 328 shell 327 elf64 313 pseudofs 280 uether 341 if_tun 339 if_gif 342 if_vlan 350 mld 349 igmp 337 if_faith 332 sysvmsg 333 sysvsem 335 sem 334 sysvshm 356 nfslockd 351 nfs_common 355 nfssvc 354 nfsserver 360 krpc 361 ufs 312 procfs 352 nfs 311 msdosfs 310 devfs 326 cd9660 6 cam 63 ata 16 sa 15 pass 13 ada 322 g_part_mbr 321 g_part_gpt 320 g_part_ebr 319 g_part_bsd 10 probe 12 ch 14 da 7 xpt 11 cd 8 aprobe 17 ses 9 pmp 221 pci/ppc 220 isa/ppc 219 acpi/ppc 218 ppbus/ppi 113 pci/dc 217 ppc/ppbus 216 ppbus/lpt 112 dc/miibus 215 ppbus/plip 214 pci/pcn 213 pcn/miibus 111 pccard/cs 212 pci/vgapci 211 pci/pcib 210 pcib/pci 110 isa/cs 53 alc/miibus 52 ahc 109 cpu/ichss 51 ahd 108 pci/ciss 50 pci/ahd 209 pci/isab 208 pci/ignore_pci 207 pci/hostb 206 pci/fixup_pci 205 pci/cbb 49 pci/ahc_pci 48 isa/ahc_isa 204 isa/cbb 47 pccard/aic 22 nexus/acpi 31 pcib/acpi_pci 107 cbb/cardbus 106 pci/bt 105 isa/bt 203 pcic/pccard 411 cpu/p4tcc 410 cpu/hwpstate 202 cbb/pccard 201 null 409 cpu/est 408 cpu/powernow 200 pci/nge 199 nge/miibus 104 pci/bge 103 bge/miibus 20 pci/aacch 406 isa/vga 405 vgapci/vgapm 198 pci/mskc 404 isa/sc 403 isa/atrtc 402 acpi/atrtc 401 scrndr-vga 400 scterm-scteken 197 mskc/msk 196 msk/miibus 102 pci/bfe 399 pci/nfe 398 nfe/miibus 101 bfe/miibus 397 pci/hptrr 396 pci/hptmv 395 pccard/fdc 394 isa/fdc 393 acpi/fdc 392 fdc/fd 391 io 100 pci/bce 390 isa/ed 193 pci/mpt 99 bce/miibus 389 atkbdc/psm 388 isa/psmcpnp 387 acpi/psmcpnp 46 pci/age 386 isa/atkbdc 385 acpi/atkbdc 384 atkbdc/atkbd 383 pci/arcmsr 382 hostb/agp_via 381 hostb/agp_intel 380 vgapci/agp_i810 379 hostb/agp_amd64 45 age/miibus 190 pci/mly 378 legacy/pcib 377 isa/pcibus_pnp 376 isa/atdma 375 acpi/atdma 374 legacy/isa 189 pci/mlx 188 mlx/mlxd 373 isa/attimer 372 acpi/attimer 187 miibus/xmphy 186 miibus/ukphy 185 miibus/truephy 371 root/nexus 370 nexus/ram 369 isa/sysresource 184 miibus/tlphy 183 miibus/tdkphy 182 miibus/smcphy 181 miibus/ruephy 180 miibus/rlphy 179 miibus/rgephy 178 miibus/qsphy 177 miibus/pnaphy 176 miibus/nsphyter 175 miibus/nsphy 368 nexus/legacy 367 legacy/cpu 174 miibus/nsgphy 173 miibus/mlphy 366 pci/ioapic 365 nexus/apic 172 miibus/lxtphy 364 acpi/fpupnp 171 miibus/jmphy 170 miibus/ip1000phy 169 miibus/inphy 168 miibus/icsphy 167 miibus/gentbi 166 miibus/xlphy 165 miibus/e1000phy 164 miibus/ciphy 163 miibus/brgphy 162 miibus/bmtphy 363 root/nexus_acpi 161 miibus/axphy 160 miibus/atphy 159 miibus/amphy 158 miibus/acphy 157 mfi/mfid 156 pci/mfi 44 pci/ae 98 pci/ath 43 ae/miibus 30 acpi/acpi_lid 155 mem 42 pci/adw 41 pci/adv 97 ata/ast 153 pci/lge 152 lge/miibus 40 acpi/acpi_timer 151 pci/le 96 ata/afd 150 kbdmux 39 cpu/acpi_throttle 95 ata/acd 149 pci/jme 329 cpu/cpufreq 148 jme/miibus 147 pci/ixgbe 38 acpi/acpi_tz 94 ad/subdisk 29 acpi/acpi_isab 92 ata/ad 91 pci/ata_via 90 pci/ata_sis 89 pci/ata_sii 88 pci/ata_serverworks 146 pci/isp 145 pci/ips 144 ips/ipsd 87 pci/ata_promise 143 pci/iir 86 pci/ata_nvidia 142 pci/ida 325 isa/orm 141 ida/idad 324 isab/isa 323 eisab/isa 140 pci/hptiop 139 pci/fxp 138 fxp/miibus 359 pci/rl 358 cardbus/rl 357 rl/miibus 85 pci/ata_netcell 84 pci/ata_national 137 firewire/fwip 83 pci/ata_micron 82 pci/ata_marvell 81 pci/ata_jmicron 80 pci/ata_ite 136 firewire/fwe 135 pci/fwohci 353 nfslock 79 pci/ata_intel 78 pci/ata_highpoint 77 pci/ata_cyrix 76 pci/ata_cypress 134 fwohci/firewire 133 pccard/fe 132 exca 131 pccard/ex 130 isa/ex 129 pccard/ep 128 isa/ep 75 pci/ata_cenatek 127 pci/et 126 et/miibus 74 pci/ata_ati 73 pci/ata_amd 72 pci/ata_adaptec 71 pci/ata_ali 125 pci/igb 70 pci/ata_acard 69 pci/ata_ahci 68 atapci/ata_ahci_ata 25 acpi/acpi_cmbat 309 pci/xl 308 xl/miibus 307 pccard/xe 306 pci/wi 305 pccard/wi 304 pci/wb 303 wb/miibus 302 watchdog 301 pci/vx 300 pci/vr 299 vr/miibus 67 pci/atapci 298 pci/vge 297 vge/miibus 66 atapci/ata 296 uhub/ums 65 isa/ata 64 pccard/ata 295 uhub/ukbd 294 uhub/uhid 124 pci/lem 37 acpi/acpi_smbat 36 acpi/acpi_sysresource 293 uhub/uvscom 292 uhub/uvisor 291 uhub/uslcom 290 uhub/uplcom 289 uhub/ulpt 288 uhub/uipaq 287 uhub/uftdi 286 uhub/ubsa 285 uhub/uark 284 uhub/zyd 283 uhub/ural 28 acpi/acpi_hpet 282 uhub/uath 281 uhub/rum 279 uhub/udav 278 udav/miibus 277 uhub/rue 276 rue/miibus 275 uhub/kue 274 uhub/cue 273 uhub/cdce 272 uhub/axe 271 axe/miibus 270 uhub/aue 269 aue/miibus 24 acpi/acpi_button 19 aac/aacd 268 usbus/uhub 267 uhub/uhub 35 cpu/acpi_perf 23 acpi/acpi_acad 34 pci/acpi_pcib 62 pci/an 61 pccard/an 123 pci/em 60 isa/an 33 acpi/acpi_pcib 59 pci/amr 58 amr/amrd 266 uhub/usb_linux 265 uhub/urio 18 aac/aacp 264 uhub/umass 57 pci/amd 263 ohci/usbus 262 uhci/usbus 261 ehci/usbus 260 at91_udp/usbus 259 uss820/usbus 258 pci/uhci 27 acpi/acpi_ec 21 pci/aac 257 pci/ohci 56 pci/ale 256 pci/ehci 55 ale/miibus 26 acpi/cpu 32 acpi/acpi_pci_link 122 pci/ed 121 pccard/ed 255 pci/uart 254 pccard/uart 253 isa/uart 252 acpi/uart 120 ed/miibus 251 pci/txp 250 pci/tx 249 tx/miibus 248 pci/twe 247 twe/twed 119 pci/dpt 118 pci/de 246 pci/twa 245 pci/trm 244 pci/tl 243 tl/miibus 242 pci/ti 117 dcons 241 pci/sym 240 pci/stge 239 stge/miibus 238 pci/ste 237 ste/miibus 236 pccard/sn 235 isa/sn 54 pci/alc 234 pci/skc 233 skc/sk 232 sk/miibus 231 pci/sis 230 sis/miibus 229 pci/sge 228 sge/miibus 227 pci/sf 226 sf/miibus 116 firewire/dcons_crom 115 miibus/pnphy 225 pci/re 224 re/miibus 114 miibus/dcphy 223 random 222 pci/ral 194 mpt_raid 195 mpt_user 192 mpt_cam 315 g_disk 314 g_dev 317 g_label 348 wlan_sta 330 rootbus 316 g_vfs 191 mpt_core 331 firmware 154 g_md 362 g_class 347 wlan 318 g_part 346 wlan_wep 345 wlan_tkip 344 wlan_ccmp 343 wlan_amrr 338 if_firewire 336 ether 412 x86bios 2 1 0xffffffff80e6b000 19eb18 zfs.ko (/boot/kernel/zfs.ko) Contains modules: Id Name 2 zfsctrl 3 zfs 5 zfs_vdev 4 zfs_zvol 3 2 0xffffffff8100a000 3868 opensolaris.ko (/boot/kernel/opensolaris.ko) Contains modules: Id Name 1 opensolaris -------------- next part --------------> _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
On 08/20/10 12:30, Heinrich Rebehn wrote:> I am somewhat concerned about the numbers for per-char-output and per-char-input. In fact, i have never before seen that low numbers in a bonnie test. Using a single disk with UFS yields about 6 times as much. >> BTW: Running OpenSolaris on the same hardware yields 110306 for per-char-write and 94698 for per-char-read."per-char" stats are different between different operating systems because of how they are implemented. Apparently, bonnie++ forces full disk writes (fsyncs) for each byte written on BSDs, but Linux (and apparently Solaris) somehow manage to write-cache this (or at least - cache it much more). It only matters if you have software which depends on this caching and performs slowly otherwise.