Hi,
I''m showing slow zfs send on pool v29. About 25MB/sec
bash-3.2# zpool status vdipool
pool: vdipool
state: ONLINE
scan: scrub repaired 86.5K in 7h15m with 0 errors on Mon Feb 6
01:36:23 2012
config:
NAME STATE READ WRITE CKSUM
vdipool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c0t5000C500103F2057d0 ONLINE 0 0 0
(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod
c0t5000C5000440AA0Bd0 ONLINE 0 0 0
(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod
c0t5000C500103E9FFBd0 ONLINE 0 0
0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod
c0t5000C500103E370Fd0 ONLINE 0 0
0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod
c0t5000C500103E120Fd0 ONLINE 0 0
0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod
logs
mirror-1 ONLINE 0 0 0
c0t500151795955D430d0 ONLINE 0 0 0(ATA-INTEL
SSDSA2VP02-02M5-18.64GB) onboard drive on x4140
c0t500151795955BDB6d0 ONLINE 0 0 0
(ATA-INTEL SSDSA2VP02-02M5-18.64GB)onboard drive on x4140
cache
c0t5001517BB271845Dd0 ONLINE 0 0 0
(ATA-INTEL SSDSA2CW16-0362-149.05GB)onboard drive on x4140
spares
c0t5000C500103E368Fd0 AVAIL
(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod
The drives are in an external promise 12 drive jbod. The jbod is also
connected to another server that uses the other 6 SEAGATE ST31000640SS
drives.
This on Solaris 10 8/11 (Generic_147441-01). I''m using LSI 9200 for the
external promise jbod and an internal 9200 for the zli and l2arc which
also uses rpool.
FW versions on both cards are MPTFW-12.00.00.00-IT and MPT2BIOS-7.23.01.00.
I''m wondering why the zfs send could be so slow. Could the other
server
be slowing down the sas bus?
Karl
CONFIDENTIALITY NOTICE: This communication (including all attachments) is
confidential and is intended for the use of the named addressee(s) only and
may contain information that is private, confidential, privileged, and
exempt from disclosure under law. All rights to privilege are expressly
claimed and reserved and are not waived. Any use, dissemination,
distribution, copying or disclosure of this message and any attachments, in
whole or in part, by anyone other than the intended recipient(s) is strictly
prohibited. If you have received this communication in error, please notify
the sender immediately, delete this communication from all data storage
devices and destroy all hard copies.
2012-05-07 20:45, Karl Rossing ?????:> I''m wondering why the zfs send could be so slow. Could the other server > be slowing down the sas bus?I hope other posters would have more relevant suggestions, but you can see if the buses are contended by dd''ing from the drives. At least that would give you the measure of available sequential throughput. During the send you can also monitor "zpool iostat 1" and usual "iostat -xnz 1" in order to see how busy the disks are and how many IO requests are issued. The snapshots are likely sent in the order of block age (TXG number), which for a busy pool may mean heavy fragmentation and lots of random small IOs... HTH, //Jim
Hi Karl, I like to verify that no dead or dying disk is killing pool performance and your zpool status looks good. Jim has replied with some ideas to check your individual device performance. Otherwise, you might be impacted by this CR: 7060894 zfs recv is excruciatingly slow This CR covers both zfs send/recv ops and should be resolved in an upcoming Solaris 10 release. Its already available in an s11 SRU. Thanks, Cindy On 5/7/12 10:45 AM, Karl Rossing wrote:> Hi, > > I''m showing slow zfs send on pool v29. About 25MB/sec > bash-3.2# zpool status vdipool > pool: vdipool > state: ONLINE > scan: scrub repaired 86.5K in 7h15m with 0 errors on Mon Feb 6 > 01:36:23 2012 > config: > > NAME STATE READ WRITE CKSUM > vdipool ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > c0t5000C500103F2057d0 ONLINE 0 0 0 > (SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod > c0t5000C5000440AA0Bd0 ONLINE 0 0 0 > (SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod > c0t5000C500103E9FFBd0 ONLINE 0 0 > 0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod > c0t5000C500103E370Fd0 ONLINE 0 0 > 0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod > c0t5000C500103E120Fd0 ONLINE 0 0 > 0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod > logs > mirror-1 ONLINE 0 0 0 > c0t500151795955D430d0 ONLINE 0 0 > 0(ATA-INTEL SSDSA2VP02-02M5-18.64GB) onboard drive on x4140 > c0t500151795955BDB6d0 ONLINE 0 0 0 > (ATA-INTEL SSDSA2VP02-02M5-18.64GB)onboard drive on x4140 > cache > c0t5001517BB271845Dd0 ONLINE 0 0 0 > (ATA-INTEL SSDSA2CW16-0362-149.05GB)onboard drive on x4140 > spares > c0t5000C500103E368Fd0 AVAIL > (SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod > > The drives are in an external promise 12 drive jbod. The jbod is also > connected to another server that uses the other 6 SEAGATE ST31000640SS > drives. > > This on Solaris 10 8/11 (Generic_147441-01). I''m using LSI 9200 for > the external promise jbod and an internal 9200 for the zli and l2arc > which also uses rpool. > FW versions on both cards are MPTFW-12.00.00.00-IT and > MPT2BIOS-7.23.01.00. > > I''m wondering why the zfs send could be so slow. Could the other > server be slowing down the sas bus? > > Karl > > > > > CONFIDENTIALITY NOTICE: This communication (including all > attachments) is > confidential and is intended for the use of the named addressee(s) > only and > may contain information that is private, confidential, privileged, and > exempt from disclosure under law. All rights to privilege are expressly > claimed and reserved and are not waived. Any use, dissemination, > distribution, copying or disclosure of this message and any > attachments, in > whole or in part, by anyone other than the intended recipient(s) is > strictly > prohibited. If you have received this communication in error, please > notify > the sender immediately, delete this communication from all data storage > devices and destroy all hard copies. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Hi Karl, Someone sitting across the table from me (who saw my posting) informs me that CR 7060894 would not impact Solaris 10 releases, so kindly withdrawn my comment about CR 7060894. Thanks, Cindy On 5/7/12 11:35 AM, Cindy Swearingen wrote:> Hi Karl, > > I like to verify that no dead or dying disk is killing pool > performance and your zpool status looks good. Jim has replied > with some ideas to check your individual device performance. > > Otherwise, you might be impacted by this CR: > > 7060894 zfs recv is excruciatingly slow > > This CR covers both zfs send/recv ops and should be resolved > in an upcoming Solaris 10 release. Its already available in an > s11 SRU. > > Thanks, > > Cindy > > On 5/7/12 10:45 AM, Karl Rossing wrote: >> Hi, >> >> I''m showing slow zfs send on pool v29. About 25MB/sec >> bash-3.2# zpool status vdipool >> pool: vdipool >> state: ONLINE >> scan: scrub repaired 86.5K in 7h15m with 0 errors on Mon Feb 6 >> 01:36:23 2012 >> config: >> >> NAME STATE READ WRITE CKSUM >> vdipool ONLINE 0 0 0 >> raidz1-0 ONLINE 0 0 0 >> c0t5000C500103F2057d0 ONLINE 0 0 0 >> (SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod >> c0t5000C5000440AA0Bd0 ONLINE 0 0 0 >> (SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod >> c0t5000C500103E9FFBd0 ONLINE 0 0 >> 0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod >> c0t5000C500103E370Fd0 ONLINE 0 0 >> 0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod >> c0t5000C500103E120Fd0 ONLINE 0 0 >> 0(SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod >> logs >> mirror-1 ONLINE 0 0 0 >> c0t500151795955D430d0 ONLINE 0 0 >> 0(ATA-INTEL SSDSA2VP02-02M5-18.64GB) onboard drive on x4140 >> c0t500151795955BDB6d0 ONLINE 0 0 0 >> (ATA-INTEL SSDSA2VP02-02M5-18.64GB)onboard drive on x4140 >> cache >> c0t5001517BB271845Dd0 ONLINE 0 0 0 >> (ATA-INTEL SSDSA2CW16-0362-149.05GB)onboard drive on x4140 >> spares >> c0t5000C500103E368Fd0 AVAIL >> (SEAGATE-ST31000640SS-0003-931.51GB) Promise Jbod >> >> The drives are in an external promise 12 drive jbod. The jbod is also >> connected to another server that uses the other 6 SEAGATE >> ST31000640SS drives. >> >> This on Solaris 10 8/11 (Generic_147441-01). I''m using LSI 9200 for >> the external promise jbod and an internal 9200 for the zli and l2arc >> which also uses rpool. >> FW versions on both cards are MPTFW-12.00.00.00-IT and >> MPT2BIOS-7.23.01.00. >> >> I''m wondering why the zfs send could be so slow. Could the other >> server be slowing down the sas bus? >> >> Karl >> >> >> >> >> CONFIDENTIALITY NOTICE: This communication (including all >> attachments) is >> confidential and is intended for the use of the named addressee(s) >> only and >> may contain information that is private, confidential, privileged, and >> exempt from disclosure under law. All rights to privilege are expressly >> claimed and reserved and are not waived. Any use, dissemination, >> distribution, copying or disclosure of this message and any >> attachments, in >> whole or in part, by anyone other than the intended recipient(s) is >> strictly >> prohibited. If you have received this communication in error, please >> notify >> the sender immediately, delete this communication from all data storage >> devices and destroy all hard copies. >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 12-05-07 12:18 PM, Jim Klimov wrote:> During the send you can also monitor "zpool iostat 1" and usual > "iostat -xnz 1" in order to see how busy the disks are and how > many IO requests are issued. The snapshots are likely sent in > the order of block age (TXG number), which for a busy pool may > mean heavy fragmentation and lots of random small IOs..I have been able to verify that I can get a zfs send at 135MB/sec for a striped pool with 2 internal drives on the same server. Each dataset had about 3-4 snapshots. There were about 36 datasets I deleted the snapshots and the speed may have increased slightly. Given "iostat -xnz 1" it looks like the IO''s are very high. So I guess the drives are badly fragmented. So is fixing this going to require a zfs pool rebuild? Karl extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 4.0 0.0 16.0 0.0 0.0 0.0 0.0 0 0 c0t500151795955D430d0 0.0 4.0 0.0 16.0 0.0 0.0 0.0 0.0 0 0 c0t500151795955BDB6d0 0.0 1.0 0.0 8.0 0.0 0.0 0.0 0.1 0 0 c0t5001517BB271845Dd0 759.0 0.0 4800.0 0.0 0.0 2.9 0.0 3.8 0 75 c0t5000C500103F2057d0 887.0 0.0 4738.0 0.0 0.0 1.6 0.0 1.8 0 42 c0t5000C500103E9FFBd0 915.0 0.0 4628.5 0.0 0.0 1.5 0.0 1.6 0 30 c0t5000C5000440AA0Bd0 922.0 0.0 4676.5 0.0 0.0 1.0 0.0 1.1 0 26 c0t5000C500103E120Fd0 970.0 0.0 4276.0 0.0 0.0 1.0 0.0 1.0 0 20 c0t5000C500103E370Fd0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 4.0 0.0 32.0 0.0 0.0 0.0 0.1 0 0 c0t5001517BB271845Dd0 1363.0 0.0 9007.8 0.0 0.0 2.0 0.0 1.5 1 54 c0t5000C500103F2057d0 1405.0 0.0 10169.2 0.0 0.0 1.8 0.0 1.3 1 37 c0t5000C500103E9FFBd0 1448.0 0.0 9884.2 0.0 0.0 1.7 0.0 1.2 1 40 c0t5000C5000440AA0Bd0 1264.0 0.0 9537.3 0.0 0.0 2.1 0.0 1.7 0 51 c0t5000C500103E120Fd0 1260.0 0.0 9749.8 0.0 0.0 1.9 0.0 1.5 0 44 c0t5000C500103E370Fd0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 6.0 0.0 24.0 0.0 0.0 0.0 0.0 0 0 c0t500151795955D430d0 0.0 6.0 0.0 24.0 0.0 0.0 0.0 0.0 0 0 c0t500151795955BDB6d0 1023.0 0.0 5131.6 0.0 0.0 1.6 0.0 1.6 0 45 c0t5000C500103F2057d0 1003.0 0.0 5040.1 0.0 0.0 1.5 0.0 1.5 0 36 c0t5000C500103E9FFBd0 959.0 0.0 5069.1 0.0 0.0 1.7 0.0 1.8 0 46 c0t5000C5000440AA0Bd0 941.0 0.0 5117.6 0.0 0.0 1.7 0.0 1.8 0 45 c0t5000C500103E120Fd0 1043.0 0.0 5034.1 0.0 0.0 1.0 0.0 1.0 0 24 c0t5000C500103E370Fd0 CONFIDENTIALITY NOTICE: This communication (including all attachments) is confidential and is intended for the use of the named addressee(s) only and may contain information that is private, confidential, privileged, and exempt from disclosure under law. All rights to privilege are expressly claimed and reserved and are not waived. Any use, dissemination, distribution, copying or disclosure of this message and any attachments, in whole or in part, by anyone other than the intended recipient(s) is strictly prohibited. If you have received this communication in error, please notify the sender immediately, delete this communication from all data storage devices and destroy all hard copies.
On Mon, 7 May 2012, Karl Rossing wrote:> On 12-05-07 12:18 PM, Jim Klimov wrote: >> During the send you can also monitor "zpool iostat 1" and usual >> "iostat -xnz 1" in order to see how busy the disks are and how >> many IO requests are issued. The snapshots are likely sent in >> the order of block age (TXG number), which for a busy pool may >> mean heavy fragmentation and lots of random small IOs.. > I have been able to verify that I can get a zfs send at 135MB/sec for a > striped pool with 2 internal drives on the same server.I see that there are a huge number of reads and hardy any reads. Are you SURE that deduplication was not enabled for this pool? This is the sort of behavior that one might expect if deduplication was enabled without enough RAM or L2 read cache. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On 12-05-07 8:45 PM, Bob Friesenhahn wrote:> I see that there are a huge number of reads and hardy any reads. Are > you SURE that deduplication was not enabled for this pool? This is > the sort of behavior that one might expect if deduplication was > enabled without enough RAM or L2 read cache. > > BobAfter hours the pool is pretty quiet. zpool history does not have dedup. zfs get dedup shows dedup off. Karl CONFIDENTIALITY NOTICE: This communication (including all attachments) is confidential and is intended for the use of the named addressee(s) only and may contain information that is private, confidential, privileged, and exempt from disclosure under law. All rights to privilege are expressly claimed and reserved and are not waived. Any use, dissemination, distribution, copying or disclosure of this message and any attachments, in whole or in part, by anyone other than the intended recipient(s) is strictly prohibited. If you have received this communication in error, please notify the sender immediately, delete this communication from all data storage devices and destroy all hard copies.