Hi Guys,
Has somebody encountered (significantly) different read/write speeds
when using gmirror?
I have 2xWD WD30EFRX RED drives which are configured as follows:
$ gmirror status
Name Status Components
mirror/root COMPLETE ada0p2 (ACTIVE)
ada1p2 (ACTIVE)
mirror/data COMPLETE ada0p4 (ACTIVE)
ada1p4 (ACTIVE)
mirror/root is mounted as the root fs (UFS2).
Doing write:
$ time dd if=/dev/zero of=/IMAGE bs=1024k count=`expr 4 \* 1024`
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 29.326044 secs (146455733 bytes/sec)
Doing read:
$ time dd if=/IMAGE of=/dev/null bs=1024k count=`expr 4 \* 1024`
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 48.821649 secs (87972598 bytes/sec)
As you can see, read is much slower than write (87 vs 146 MB/s). Why?
Any help would be appreciated.
Best regards,
Andrew
PS: Partition layout (partitions are 4k aligned):
$ gpart show ada0 ada1
=> 34 5860533101 ada0 GPT (2.7T)
34 6 - free - (3.0k)
40 1024 1 freebsd-boot (512k)
1064 16777216 2 freebsd-ufs (8.0G)
16778280 16777216 3 freebsd-swap (8.0G)
33555496 5826977632 4 freebsd-ufs (2.7T)
5860533128 7 - free - (3.5k)
=> 34 5860533101 ada1 GPT (2.7T)
34 6 - free - (3.0k)
40 1024 1 freebsd-boot (512k)
1064 16777216 2 freebsd-ufs (8.0G)
16778280 16777216 3 freebsd-swap (8.0G)
33555496 5826977632 4 freebsd-ufs (2.7T)
5860533128 7 - free - (3.5k)
On Thu, 28 Nov 2013 21:47:17 +0100 Szalai Andr?s <szalai.bandi at gmail.com> wrote:> Hi Guys, > > Has somebody encountered (significantly) different read/write speeds > when using gmirror? > > I have 2xWD WD30EFRX RED drives which are configured as follows: > > $ gmirror status > Name Status Components > mirror/root COMPLETE ada0p2 (ACTIVE) > ada1p2 (ACTIVE) > mirror/data COMPLETE ada0p4 (ACTIVE) > ada1p4 (ACTIVE) > > mirror/root is mounted as the root fs (UFS2). > > Doing write: > > $ time dd if=/dev/zero of=/IMAGE bs=1024k count=`expr 4 \* 1024` > 4096+0 records in > 4096+0 records out > 4294967296 bytes transferred in 29.326044 secs (146455733 bytes/sec) > > Doing read: > > $ time dd if=/IMAGE of=/dev/null bs=1024k count=`expr 4 \* 1024` > 4096+0 records in > 4096+0 records out > 4294967296 bytes transferred in 48.821649 secs (87972598 bytes/sec) > > As you can see, read is much slower than write (87 vs 146 MB/s). Why? > > Any help would be appreciated. > > Best regards, > Andrew > > PS: Partition layout (partitions are 4k aligned): > > $ gpart show ada0 ada1 > => 34 5860533101 ada0 GPT (2.7T) > 34 6 - free - (3.0k) > 40 1024 1 freebsd-boot (512k) > 1064 16777216 2 freebsd-ufs (8.0G) > 16778280 16777216 3 freebsd-swap (8.0G) > 33555496 5826977632 4 freebsd-ufs (2.7T) > 5860533128 7 - free - (3.5k) > > => 34 5860533101 ada1 GPT (2.7T) > 34 6 - free - (3.0k) > 40 1024 1 freebsd-boot (512k) > 1064 16777216 2 freebsd-ufs (8.0G) > 16778280 16777216 3 freebsd-swap (8.0G) > 33555496 5826977632 4 freebsd-ufs (2.7T) > 5860533128 7 - free - (3.5k)Modern HDDs have both Command Queuing and Excessive Cache Memory. Using them a spindle disk can cache multiple write requests and do them all in one revolution. While multiple read requests can also be done at once chances are the on-disk cache is not usefull (because the requested data is only resident if it was accessed short before, and then it's also availible in the kernel's larger Filesystem/GEOM caches which reside in main memory and were consulted prior to disk accesses) and the GEOM layer might not have issued them yet. IIRC the UFS subsystem will perform no read requests smaller than 512kB by default, which means it does some readahead just in case the issuing application wants to read more data soon - however you have used read blocks which are exact multiples of 512kB, so there is no gain in this. readahead is the buzzword for tuning large sequencial reads, and I had thought there was a sysctl for it, though I can't find it now. Alonso
On Thu, 2013-11-28 at 21:47 +0100, Szalai Andr?s wrote:> Hi Guys, > > Has somebody encountered (significantly) different read/write speeds > when using gmirror? > > I have 2xWD WD30EFRX RED drives which are configured as follows: > > $ gmirror status > Name Status Components > mirror/root COMPLETE ada0p2 (ACTIVE) > ada1p2 (ACTIVE) > mirror/data COMPLETE ada0p4 (ACTIVE) > ada1p4 (ACTIVE) > > mirror/root is mounted as the root fs (UFS2). > > Doing write: > > $ time dd if=/dev/zero of=/IMAGE bs=1024k count=`expr 4 \* 1024` > 4096+0 records in > 4096+0 records out > 4294967296 bytes transferred in 29.326044 secs (146455733 bytes/sec) > > Doing read: > > $ time dd if=/IMAGE of=/dev/null bs=1024k count=`expr 4 \* 1024` > 4096+0 records in > 4096+0 records out > 4294967296 bytes transferred in 48.821649 secs (87972598 bytes/sec) > > As you can see, read is much slower than write (87 vs 146 MB/s). Why? > > Any help would be appreciated.You can't use dd to a file to measure performance -- you're just measuring the behavior of the filesystem caching. Much, perhaps most, of the actual writing to disk happens long after the dd command has ended. If you followed the dd command with an unmount of the filesystem and included the time until the umount command completes, you'd have a much closer measure. -- Ian