Hello, I''m curious if anyone would mind sharing their experiences with zvol''s. I recently started using zvol as an iSCSI backend and was supprised by the performance I was getting. Further testing revealed that it wasn''t an iSCSI performance issue but a zvol issue. Testing on a SATA disk locally, I get these numbers (sequentual write): UFS: 38MB/s ZFS: 38MB/s Zvol UFS: 6MB/s Zvol Raw: ~6MB/s ZFS is nice and fast but Zvol performance just drops off a cliff. Suggestion or observations by others using zvol would be extremely helpful. My current testing is being done using a debug build of B44 (NV 6/10/06). benr. This message posted from opensolaris.org
> Further testing revealed > that it wasn''t an iSCSI performance issue but a zvol > issue. Testing on a SATA disk locally, I get these > numbers (sequentual write): > > UFS: 38MB/s > ZFS: 38MB/s > Zvol UFS: 6MB/s > Zvol Raw: ~6MB/s > > ZFS is nice and fast but Zvol performance just drops > off a cliff. Suggestion or observations by others > using zvol would be extremely helpful.# zfs create -V 1g data/zvol-test # time dd if=/data/media/sol-10-u2-ga-x86-dvd.iso of=/dev/zvol/rdsk/data/zvol-test bs=32k count=10000 10000+0 records in 10000+0 records out 0.08u 9.37s 2:21.56 6.6% That''s ~ 2.3 MB/s. I do see *frequent* DKIOCFLUSHWRITECACHE ioctls (one flush write cache ioctl after writing ~36KB of data, needs ~6-7 milliseconds per flush): 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02778, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 5736778 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e027c0, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6209599 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02808, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6572132 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02850, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6732316 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02898, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6175876 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e028e0, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6251611 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02928, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 7756397 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02970, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6393356 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e029b8, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6147003 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02a00, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6247036 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02a48, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6061991 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02a90, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6284297 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02ad8, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6174818 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02b20, count 9000 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6245923 nsec, error 0 dtrace with stack backtraces: 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5d1ec10, count 9000 0 39404 zio_ioctl:entry zfs`zil_flush_vdevs+0x144 zfs`zil_commit+0x311 zfs`zvol_strategy+0x4bc genunix`default_physio+0x308 genunix`physio+0x1d zfs`zvol_write+0x22 genunix`cdev_write+0x25 specfs`spec_write+0x4d6 genunix`fop_write+0x2e genunix`write+0x2ae unix`sys_sysenter+0x104 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6638189 nsec, error 0 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5d1ec58, count 9000 0 39404 zio_ioctl:entry zfs`zil_flush_vdevs+0x144 zfs`zil_commit+0x311 zfs`zvol_strategy+0x4bc genunix`default_physio+0x308 genunix`physio+0x1d zfs`zvol_write+0x22 genunix`cdev_write+0x25 specfs`spec_write+0x4d6 genunix`fop_write+0x2e genunix`write+0x2ae unix`sys_sysenter+0x104 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 7881400 nsec, error 0 ====================================================================#!/usr/sbin/dtrace -s BEGIN { DKIOC = 0x04 << 8; DKIOCFLUSHWRITECACHE = DKIOC|34; } fbt::bdev_strategy:entry { bp = (struct buf *)arg0; printf("edev %x, flags %x, bn %x, count %x", bp->b_edev, bp->b_flags, bp->_b_blkno._p._l, bp->b_bcount); } fbt::zio_ioctl:entry /arg3 == DKIOCFLUSHWRITECACHE/ { flushtime = timestamp; } fbt::vdev_disk_ioctl_done:entry /flushtime/ { this->zio = (zio_t *)arg0; printf("%s time: %llu nsec, error %d", this->zio->io_cmd == DKIOCFLUSHWRITECACHE ? "DKIOCFLUSHWRITECACHE" : "???", timestamp - flushtime, arg1); flushtime = 0; } This message posted from opensolaris.org
This is change request: 6428639 large writes to zvol synchs too much, better cut down a little which I have a fix for, but it hasn''t been put back. Neil. J?rgen Keil wrote On 07/17/06 04:18,:>>Further testing revealed >>that it wasn''t an iSCSI performance issue but a zvol >>issue. Testing on a SATA disk locally, I get these >> numbers (sequentual write): >> >>UFS: 38MB/s >>ZFS: 38MB/s >>Zvol UFS: 6MB/s >>Zvol Raw: ~6MB/s >> >>ZFS is nice and fast but Zvol performance just drops >>off a cliff. Suggestion or observations by others >>using zvol would be extremely helpful. > > > # zfs create -V 1g data/zvol-test > # time dd if=/data/media/sol-10-u2-ga-x86-dvd.iso of=/dev/zvol/rdsk/data/zvol-test bs=32k count=10000 > 10000+0 records in > 10000+0 records out > 0.08u 9.37s 2:21.56 6.6% > > That''s ~ 2.3 MB/s. > > I do see *frequent* DKIOCFLUSHWRITECACHE ioctls > (one flush write cache ioctl after writing ~36KB of data, needs ~6-7 milliseconds per flush): > > > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02778, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 5736778 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e027c0, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6209599 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02808, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6572132 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02850, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6732316 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02898, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6175876 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e028e0, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6251611 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02928, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 7756397 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02970, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6393356 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e029b8, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6147003 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02a00, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6247036 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02a48, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6061991 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02a90, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6284297 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02ad8, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6174818 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5e02b20, count 9000 > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6245923 nsec, error 0 > > > > dtrace with stack backtraces: > > > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5d1ec10, count 9000 > 0 39404 zio_ioctl:entry > zfs`zil_flush_vdevs+0x144 > zfs`zil_commit+0x311 > zfs`zvol_strategy+0x4bc > genunix`default_physio+0x308 > genunix`physio+0x1d > zfs`zvol_write+0x22 > genunix`cdev_write+0x25 > specfs`spec_write+0x4d6 > genunix`fop_write+0x2e > genunix`write+0x2ae > unix`sys_sysenter+0x104 > > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 6638189 nsec, error 0 > 0 12308 bdev_strategy:entry edev 1980047, flags 1080101, bn 5d1ec58, count 9000 > 0 39404 zio_ioctl:entry > zfs`zil_flush_vdevs+0x144 > zfs`zil_commit+0x311 > zfs`zvol_strategy+0x4bc > genunix`default_physio+0x308 > genunix`physio+0x1d > zfs`zvol_write+0x22 > genunix`cdev_write+0x25 > specfs`spec_write+0x4d6 > genunix`fop_write+0x2e > genunix`write+0x2ae > unix`sys_sysenter+0x104 > > 0 38530 vdev_disk_ioctl_done:entry DKIOCFLUSHWRITECACHE time: 7881400 nsec, error 0 > > > ====================================================================> #!/usr/sbin/dtrace -s > > > BEGIN > { > DKIOC = 0x04 << 8; > DKIOCFLUSHWRITECACHE = DKIOC|34; > } > > > fbt::bdev_strategy:entry > { > bp = (struct buf *)arg0; > printf("edev %x, flags %x, bn %x, count %x", > bp->b_edev, bp->b_flags, bp->_b_blkno._p._l, bp->b_bcount); > } > > > fbt::zio_ioctl:entry > /arg3 == DKIOCFLUSHWRITECACHE/ > { > flushtime = timestamp; > } > > > fbt::vdev_disk_ioctl_done:entry > /flushtime/ > { > this->zio = (zio_t *)arg0; > printf("%s time: %llu nsec, error %d", > this->zio->io_cmd == DKIOCFLUSHWRITECACHE ? "DKIOCFLUSHWRITECACHE" : "???", > timestamp - flushtime, arg1); > flushtime = 0; > } > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Neil