I was running zpool iostat on a pool comprising a stripe of raidz2 vdevs
that appears to be writing slowly and I notice a considerable imbalance
of both free space and write operations. The pool is currently feeding
a tape backup while receiving a large filesystem.
Is this imbalance normal? I would expect a more even distribution as
the poll configuration hasn''t been changed since creation.
The system is running Solaris 10 update 7.
capacity operations bandwidth
pool used avail read write read write
------------ ----- ----- ----- ----- ----- -----
tank 15.9T 2.19T 87 119 2.34M 1.88M
raidz2 2.90T 740G 24 27 762K 95.5K
c0t1d0 - - 14 13 273K 18.6K
c1t1d0 - - 15 13 263K 18.3K
c4t2d0 - - 17 14 288K 18.2K
spare - - 17 20 104K 17.2K
c5t2d0 - - 16 13 277K 17.6K
c7t5d0 - - 0 14 0 17.6K
c6t3d0 - - 15 12 242K 18.7K
c7t3d0 - - 15 12 242K 17.6K
c6t4d0 - - 16 12 272K 18.1K
c1t0d0 - - 15 13 275K 16.8K
raidz2 3.59T 37.8G 20 0 546K 0
c0t2d0 - - 11 0 184K 361
c1t3d0 - - 10 0 182K 361
c4t5d0 - - 14 0 237K 361
c5t5d0 - - 13 0 220K 361
c6t6d0 - - 12 0 155K 361
c7t6d0 - - 11 0 149K 361
c7t4d0 - - 14 0 219K 361
c4t0d0 - - 14 0 213K 361
raidz2 3.58T 44.1G 27 0 1.01M 0
c0t5d0 - - 16 0 290K 361
c1t6d0 - - 15 0 301K 361
c4t7d0 - - 20 0 375K 361
c5t1d0 - - 19 0 374K 361
c6t7d0 - - 17 0 285K 361
c7t7d0 - - 15 0 253K 361
c0t0d0 - - 18 0 328K 361
c6t0d0 - - 18 0 348K 361
raidz2 3.05T 587G 7 47 24.9K 1.07M
c0t4d0 - - 3 21 254K 187K
c1t2d0 - - 3 22 254K 187K
c4t3d0 - - 5 22 350K 187K
c5t3d0 - - 5 21 350K 186K
c6t2d0 - - 4 22 265K 187K
c7t1d0 - - 4 21 271K 187K
c6t1d0 - - 5 22 345K 186K
c4t1d0 - - 5 24 333K 184K
raidz2 2.81T 835G 8 45 30.9K 733K
c0t3d0 - - 5 16 339K 126K
c1t5d0 - - 5 16 333K 126K
c4t6d0 - - 6 16 441K 127K
c5t6d0 - - 6 17 435K 126K
c6t5d0 - - 4 18 294K 126K
c7t2d0 - - 4 18 282K 124K
c0t6d0 - - 7 19 446K 124K
c5t7d0 - - 7 21 452K 122K
------------ ----- ----- ----- ----- ----- -----
--
Ian.
Ian Collins wrote:> I was running zpool iostat on a pool comprising a stripe of raidz2 > vdevs that appears to be writing slowly and I notice a considerable > imbalance of both free space and write operations. The pool is > currently feeding a tape backup while receiving a large filesystem. > > Is this imbalance normal? I would expect a more even distribution as > the poll configuration hasn''t been changed since creation.The second and third ones are pretty much full, with the others having well over 10 times more free space, so I wouldn''t expect many writes to the full ones. Have the others ever been in a degraded state? That might explain why the fill level has become unbalanced.> > The system is running Solaris 10 update 7. > > capacity operations bandwidth > pool used avail read write read write > ------------ ----- ----- ----- ----- ----- ----- > tank 15.9T 2.19T 87 119 2.34M 1.88M > raidz2 2.90T 740G 24 27 762K 95.5K > c0t1d0 - - 14 13 273K 18.6K > c1t1d0 - - 15 13 263K 18.3K > c4t2d0 - - 17 14 288K 18.2K > spare - - 17 20 104K 17.2K > c5t2d0 - - 16 13 277K 17.6K > c7t5d0 - - 0 14 0 17.6K > c6t3d0 - - 15 12 242K 18.7K > c7t3d0 - - 15 12 242K 17.6K > c6t4d0 - - 16 12 272K 18.1K > c1t0d0 - - 15 13 275K 16.8K > raidz2 3.59T 37.8G 20 0 546K 0 > c0t2d0 - - 11 0 184K 361 > c1t3d0 - - 10 0 182K 361 > c4t5d0 - - 14 0 237K 361 > c5t5d0 - - 13 0 220K 361 > c6t6d0 - - 12 0 155K 361 > c7t6d0 - - 11 0 149K 361 > c7t4d0 - - 14 0 219K 361 > c4t0d0 - - 14 0 213K 361 > raidz2 3.58T 44.1G 27 0 1.01M 0 > c0t5d0 - - 16 0 290K 361 > c1t6d0 - - 15 0 301K 361 > c4t7d0 - - 20 0 375K 361 > c5t1d0 - - 19 0 374K 361 > c6t7d0 - - 17 0 285K 361 > c7t7d0 - - 15 0 253K 361 > c0t0d0 - - 18 0 328K 361 > c6t0d0 - - 18 0 348K 361 > raidz2 3.05T 587G 7 47 24.9K 1.07M > c0t4d0 - - 3 21 254K 187K > c1t2d0 - - 3 22 254K 187K > c4t3d0 - - 5 22 350K 187K > c5t3d0 - - 5 21 350K 186K > c6t2d0 - - 4 22 265K 187K > c7t1d0 - - 4 21 271K 187K > c6t1d0 - - 5 22 345K 186K > c4t1d0 - - 5 24 333K 184K > raidz2 2.81T 835G 8 45 30.9K 733K > c0t3d0 - - 5 16 339K 126K > c1t5d0 - - 5 16 333K 126K > c4t6d0 - - 6 16 441K 127K > c5t6d0 - - 6 17 435K 126K > c6t5d0 - - 4 18 294K 126K > c7t2d0 - - 4 18 282K 124K > c0t6d0 - - 7 19 446K 124K > c5t7d0 - - 7 21 452K 122K > ------------ ----- ----- ----- ----- ----- ------- Andrew
Andrew Gabriel wrote:> Ian Collins wrote: >> I was running zpool iostat on a pool comprising a stripe of raidz2 >> vdevs that appears to be writing slowly and I notice a considerable >> imbalance of both free space and write operations. The pool is >> currently feeding a tape backup while receiving a large filesystem. >> >> Is this imbalance normal? I would expect a more even distribution as >> the poll configuration hasn''t been changed since creation. > > The second and third ones are pretty much full, with the others having > well over 10 times more free space, so I wouldn''t expect many writes > to the full ones. > > Have the others ever been in a degraded state? That might explain why > the fill level has become unbalanced. >We had to swap a drive in the second one and I''ve seen hot spares kick in as in the first one here. These have always been as a result of phantom errors from that wretched Marvell driver (the box is an x4500). Nothing has been degraded for long and I stop all copies to the box when scrubbing or resilvering is in progress (receives still restart scrubs in update 7).>> >> The system is running Solaris 10 update 7. >> >> capacity operations bandwidth >> pool used avail read write read write >> ------------ ----- ----- ----- ----- ----- ----- >> tank 15.9T 2.19T 87 119 2.34M 1.88M >> raidz2 2.90T 740G 24 27 762K 95.5K >> c0t1d0 - - 14 13 273K 18.6K >> c1t1d0 - - 15 13 263K 18.3K >> c4t2d0 - - 17 14 288K 18.2K >> spare - - 17 20 104K 17.2K >> c5t2d0 - - 16 13 277K 17.6K >> c7t5d0 - - 0 14 0 17.6K >> c6t3d0 - - 15 12 242K 18.7K >> c7t3d0 - - 15 12 242K 17.6K >> c6t4d0 - - 16 12 272K 18.1K >> c1t0d0 - - 15 13 275K 16.8K >> raidz2 3.59T 37.8G 20 0 546K 0 >> c0t2d0 - - 11 0 184K 361 >> c1t3d0 - - 10 0 182K 361 >> c4t5d0 - - 14 0 237K 361 >> c5t5d0 - - 13 0 220K 361 >> c6t6d0 - - 12 0 155K 361 >> c7t6d0 - - 11 0 149K 361 >> c7t4d0 - - 14 0 219K 361 >> c4t0d0 - - 14 0 213K 361 >> raidz2 3.58T 44.1G 27 0 1.01M 0 >> c0t5d0 - - 16 0 290K 361 >> c1t6d0 - - 15 0 301K 361 >> c4t7d0 - - 20 0 375K 361 >> c5t1d0 - - 19 0 374K 361 >> c6t7d0 - - 17 0 285K 361 >> c7t7d0 - - 15 0 253K 361 >> c0t0d0 - - 18 0 328K 361 >> c6t0d0 - - 18 0 348K 361 >> raidz2 3.05T 587G 7 47 24.9K 1.07M >> c0t4d0 - - 3 21 254K 187K >> c1t2d0 - - 3 22 254K 187K >> c4t3d0 - - 5 22 350K 187K >> c5t3d0 - - 5 21 350K 186K >> c6t2d0 - - 4 22 265K 187K >> c7t1d0 - - 4 21 271K 187K >> c6t1d0 - - 5 22 345K 186K >> c4t1d0 - - 5 24 333K 184K >> raidz2 2.81T 835G 8 45 30.9K 733K >> c0t3d0 - - 5 16 339K 126K >> c1t5d0 - - 5 16 333K 126K >> c4t6d0 - - 6 16 441K 127K >> c5t6d0 - - 6 17 435K 126K >> c6t5d0 - - 4 18 294K 126K >> c7t2d0 - - 4 18 282K 124K >> c0t6d0 - - 7 19 446K 124K >> c5t7d0 - - 7 21 452K 122K >> ------------ ----- ----- ----- ----- ----- ------- Ian.
On 02/28/10 08:09 PM, Ian Collins wrote:> I was running zpool iostat on a pool comprising a stripe of raidz2 > vdevs that appears to be writing slowly and I notice a considerable > imbalance of both free space and write operations. The pool is > currently feeding a tape backup while receiving a large filesystem. > > Is this imbalance normal? I would expect a more even distribution as > the poll configuration hasn''t been changed since creation. > > The system is running Solaris 10 update 7. > > capacity operations bandwidth > pool used avail read write read write > ------------ ----- ----- ----- ----- ----- ----- > tank 15.9T 2.19T 87 119 2.34M 1.88M > raidz2 2.90T 740G 24 27 762K 95.5K > raidz2 3.59T 37.8G 20 0 546K 0 > raidz2 3.58T 44.1G 27 0 1.01M 0 > raidz2 3.05T 587G 7 47 24.9K 1.07M > raidz2 2.81T 835G 8 45 30.9K 733K > ------------ ----- ----- ----- ----- ----- ----- >This system has since been upgraded, but the imbalance in getting worse: zpool iostat -v tank | grep raid raidz2 3.60T 28.5G 166 41 6.97M 764K raidz2 3.59T 33.3G 170 35 7.35M 709K raidz2 3.60T 26.1G 173 35 7.36M 658K raidz2 1.69T 1.93T 129 46 6.70M 610K raidz2 2.25T 1.38T 124 54 5.77M 967K Is there any way to determine how this is happening? I may have to resort to destroying and recreating some large filesystems, but there''s no way to determine which ones to target... -- Ian.
Hi, As far as i know this is a "normal" behaviour in ZFS... So what we need is somesort of "rebalance" task what moves data around multiple vdevs in order to achieve the best performance possible... Take a look to http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855425 Bruno On 24-3-2010 22:29, Ian Collins wrote:> On 02/28/10 08:09 PM, Ian Collins wrote: >> I was running zpool iostat on a pool comprising a stripe of raidz2 >> vdevs that appears to be writing slowly and I notice a considerable >> imbalance of both free space and write operations. The pool is >> currently feeding a tape backup while receiving a large filesystem. >> >> Is this imbalance normal? I would expect a more even distribution as >> the poll configuration hasn''t been changed since creation. >> >> The system is running Solaris 10 update 7. >> >> capacity operations bandwidth >> pool used avail read write read write >> ------------ ----- ----- ----- ----- ----- ----- >> tank 15.9T 2.19T 87 119 2.34M 1.88M >> raidz2 2.90T 740G 24 27 762K 95.5K >> raidz2 3.59T 37.8G 20 0 546K 0 >> raidz2 3.58T 44.1G 27 0 1.01M 0 >> raidz2 3.05T 587G 7 47 24.9K 1.07M >> raidz2 2.81T 835G 8 45 30.9K 733K >> ------------ ----- ----- ----- ----- ----- ----- >> > This system has since been upgraded, but the imbalance in getting worse: > > zpool iostat -v tank | grep raid > raidz2 3.60T 28.5G 166 41 6.97M 764K > raidz2 3.59T 33.3G 170 35 7.35M 709K > raidz2 3.60T 26.1G 173 35 7.36M 658K > raidz2 1.69T 1.93T 129 46 6.70M 610K > raidz2 2.25T 1.38T 124 54 5.77M 967K > > Is there any way to determine how this is happening? > > I may have to resort to destroying and recreating some large > filesystems, but there''s no way to determine which ones to target... >-------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/93f1a2b7/attachment.bin>
> This system has since been upgraded, but the imbalance in getting worse: > > zpool iostat -v tank | grep raid > raidz2 3.60T 28.5G 166 41 6.97M 764K > raidz2 3.59T 33.3G 170 35 7.35M 709K > raidz2 3.60T 26.1G 173 35 7.36M 658K > raidz2 1.69T 1.93T 129 46 6.70M 610K > raidz2 2.25T 1.38T 124 54 5.77M 967K > > Is there any way to determine how this is happening? > > I may have to resort to destroying and recreating some large > filesystems, but there''s no way to determine which ones to target... > > -- > Ian.Hi, if you have had faulted disks in some raidsets that would explain imbalance as zfs "avoids" writing to them while they are in faulted state. I''ve encountered similar imbalance but that is due later changes in pool configuration where vdev''s were added after first one''s got too full. Anyway, this is an issue, as your writes will definitely get slower after first raidsets get more full, as mine did, writes went from 1.2GB/s to 40-50KB/s and freeing up some space made problem go away (total pool capacity was around 60%). Yours Markus Kovero
On 03/25/10 09:32 PM, Bruno Sousa wrote:> On 24-3-2010 22:29, Ian Collins wrote: > >> On 02/28/10 08:09 PM, Ian Collins wrote: >> >>> I was running zpool iostat on a pool comprising a stripe of raidz2 >>> vdevs that appears to be writing slowly and I notice a considerable >>> imbalance of both free space and write operations. The pool is >>> currently feeding a tape backup while receiving a large filesystem. >>> >>> Is this imbalance normal? I would expect a more even distribution as >>> the poll configuration hasn''t been changed since creation. >>> >>> The system is running Solaris 10 update 7. >>> >>> capacity operations bandwidth >>> pool used avail read write read write >>> ------------ ----- ----- ----- ----- ----- ----- >>> tank 15.9T 2.19T 87 119 2.34M 1.88M >>> raidz2 2.90T 740G 24 27 762K 95.5K >>> raidz2 3.59T 37.8G 20 0 546K 0 >>> raidz2 3.58T 44.1G 27 0 1.01M 0 >>> raidz2 3.05T 587G 7 47 24.9K 1.07M >>> raidz2 2.81T 835G 8 45 30.9K 733K >>> ------------ ----- ----- ----- ----- ----- ----- >>> >>> >> This system has since been upgraded, but the imbalance in getting worse: >> >> zpool iostat -v tank | grep raid >> raidz2 3.60T 28.5G 166 41 6.97M 764K >> raidz2 3.59T 33.3G 170 35 7.35M 709K >> raidz2 3.60T 26.1G 173 35 7.36M 658K >> raidz2 1.69T 1.93T 129 46 6.70M 610K >> raidz2 2.25T 1.38T 124 54 5.77M 967K >> >> Is there any way to determine how this is happening? >> >> I may have to resort to destroying and recreating some large >> filesystems, but there''s no way to determine which ones to target.. > Hi, > > As far as i know this is a "normal" behaviour in ZFS... > So what we need is somesort of "rebalance" task what moves data around > multiple vdevs in order to achieve the best performance possible... > Take a look to > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855425 > >It would be if drives had been added, but this pool hasn''t been changed since it was created. -- Ian.
Hi, You never experienced any faulted drives, or something similar? So far i only saw imbalance if the vdevs add changed, if a hotspare is used and i think even during a replacement of one disk of a raidz2 group. I Bruno On 25-3-2010 9:46, Ian Collins wrote:> On 03/25/10 09:32 PM, Bruno Sousa wrote: >> On 24-3-2010 22:29, Ian Collins wrote: >> >>> On 02/28/10 08:09 PM, Ian Collins wrote: >>> >>>> I was running zpool iostat on a pool comprising a stripe of raidz2 >>>> vdevs that appears to be writing slowly and I notice a considerable >>>> imbalance of both free space and write operations. The pool is >>>> currently feeding a tape backup while receiving a large filesystem. >>>> >>>> Is this imbalance normal? I would expect a more even distribution as >>>> the poll configuration hasn''t been changed since creation. >>>> >>>> The system is running Solaris 10 update 7. >>>> >>>> capacity operations bandwidth >>>> pool used avail read write read write >>>> ------------ ----- ----- ----- ----- ----- ----- >>>> tank 15.9T 2.19T 87 119 2.34M 1.88M >>>> raidz2 2.90T 740G 24 27 762K 95.5K >>>> raidz2 3.59T 37.8G 20 0 546K 0 >>>> raidz2 3.58T 44.1G 27 0 1.01M 0 >>>> raidz2 3.05T 587G 7 47 24.9K 1.07M >>>> raidz2 2.81T 835G 8 45 30.9K 733K >>>> ------------ ----- ----- ----- ----- ----- ----- >>>> >>>> >>> This system has since been upgraded, but the imbalance in getting >>> worse: >>> >>> zpool iostat -v tank | grep raid >>> raidz2 3.60T 28.5G 166 41 6.97M 764K >>> raidz2 3.59T 33.3G 170 35 7.35M 709K >>> raidz2 3.60T 26.1G 173 35 7.36M 658K >>> raidz2 1.69T 1.93T 129 46 6.70M 610K >>> raidz2 2.25T 1.38T 124 54 5.77M 967K >>> >>> Is there any way to determine how this is happening? >>> >>> I may have to resort to destroying and recreating some large >>> filesystems, but there''s no way to determine which ones to target.. >> Hi, >> >> As far as i know this is a "normal" behaviour in ZFS... >> So what we need is somesort of "rebalance" task what moves data around >> multiple vdevs in order to achieve the best performance possible... >> Take a look to >> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855425 >> >> > It would be if drives had been added, but this pool hasn''t been > changed since it was created. >-------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/ed1a3123/attachment.bin>
On 03/25/10 11:23 PM, Bruno Sousa wrote:> On 25-3-2010 9:46, Ian Collins wrote: > >> On 03/25/10 09:32 PM, Bruno Sousa wrote: >> >>> On 24-3-2010 22:29, Ian Collins wrote: >>> >>> >>>> On 02/28/10 08:09 PM, Ian Collins wrote: >>>> >>>> >>>>> I was running zpool iostat on a pool comprising a stripe of raidz2 >>>>> vdevs that appears to be writing slowly and I notice a considerable >>>>> imbalance of both free space and write operations. The pool is >>>>> currently feeding a tape backup while receiving a large filesystem. >>>>> >>>>> Is this imbalance normal? I would expect a more even distribution as >>>>> the poll configuration hasn''t been changed since creation. >>>>> >>>>> The system is running Solaris 10 update 7. >>>>> >>>>> capacity operations bandwidth >>>>> pool used avail read write read write >>>>> ------------ ----- ----- ----- ----- ----- ----- >>>>> tank 15.9T 2.19T 87 119 2.34M 1.88M >>>>> raidz2 2.90T 740G 24 27 762K 95.5K >>>>> raidz2 3.59T 37.8G 20 0 546K 0 >>>>> raidz2 3.58T 44.1G 27 0 1.01M 0 >>>>> raidz2 3.05T 587G 7 47 24.9K 1.07M >>>>> raidz2 2.81T 835G 8 45 30.9K 733K >>>>> ------------ ----- ----- ----- ----- ----- ----- >>>>> >>>>> >>>>> >>>> This system has since been upgraded, but the imbalance in getting >>>> worse: >>>> >>>> zpool iostat -v tank | grep raid >>>> raidz2 3.60T 28.5G 166 41 6.97M 764K >>>> raidz2 3.59T 33.3G 170 35 7.35M 709K >>>> raidz2 3.60T 26.1G 173 35 7.36M 658K >>>> raidz2 1.69T 1.93T 129 46 6.70M 610K >>>> raidz2 2.25T 1.38T 124 54 5.77M 967K >>>> >>>> Is there any way to determine how this is happening? >>>> >>>> I may have to resort to destroying and recreating some large >>>> filesystems, but there''s no way to determine which ones to target.. >>>> >>> Hi, >>> >>> As far as i know this is a "normal" behaviour in ZFS... >>> So what we need is somesort of "rebalance" task what moves data around >>> multiple vdevs in order to achieve the best performance possible... >>> Take a look to >>> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855425 >>> >>> >>> >> It would be if drives had been added, but this pool hasn''t been >> changed since it was created > Hi, > > You never experienced any faulted drives, or something similar? So far i > only saw imbalance if the vdevs add changed, if a hotspare is used and i > think even during a replacement of one disk of a raidz2 group. >There has been one faulted drive, but a hot spare kicked in. A the moment, I have another replaced by a spare, but I/O to that vdev is unaffected: raidz2 1.69T 1.94T 126 46 6.54M 598K spare - - 121 34 1.09M 31.4K c0t4d0 - - 34 23 1.37M 98.7K c7t5d0 - - 0 78 0 786K c1t2d0 - - 34 23 1.37M 98.8K c4t3d0 - - 37 24 1.44M 98.9K c5t3d0 - - 36 24 1.44M 98.8K c6t2d0 - - 36 23 1.42M 98.9K c7t1d0 - - 36 24 1.42M 98.8K c6t1d0 - - 36 23 1.44M 98.7K c4t1d0 - - 36 23 1.43M 98.6K The vdev with the most free space has never lost a drive. Cheers, -- Ian.
Well...i''m pretty much certain that at my job i faced something similar.. We had a server with 2 raidz2 groups each with 3 drives, and one drive has failed and replaced by a hot spare. However, the balance of data between the 2 groups of raidz2 start to be imbalance. I assume this is due the fact that during the resilvering process (due to the hot spare thing) the vdev was somehow "halted" of writting new data..therefore during that time only 1 group was getting new data, and that lead to an imbalance data across the 2 raidz2 groups.. Maybe you have something similar, or maybe i''m talking a huge mistake. If someone with more knowledge about zfs would like to comment, please do so.. It''s always a learning experience. Bruno On 25-3-2010 11:53, Ian Collins wrote:> On 03/25/10 11:23 PM, Bruno Sousa wrote: >> On 25-3-2010 9:46, Ian Collins wrote: >> >>> On 03/25/10 09:32 PM, Bruno Sousa wrote: >>> >>>> On 24-3-2010 22:29, Ian Collins wrote: >>>> >>>> >>>>> On 02/28/10 08:09 PM, Ian Collins wrote: >>>>> >>>>> >>>>>> I was running zpool iostat on a pool comprising a stripe of raidz2 >>>>>> vdevs that appears to be writing slowly and I notice a considerable >>>>>> imbalance of both free space and write operations. The pool is >>>>>> currently feeding a tape backup while receiving a large filesystem. >>>>>> >>>>>> Is this imbalance normal? I would expect a more even >>>>>> distribution as >>>>>> the poll configuration hasn''t been changed since creation. >>>>>> >>>>>> The system is running Solaris 10 update 7. >>>>>> >>>>>> capacity operations bandwidth >>>>>> pool used avail read write read write >>>>>> ------------ ----- ----- ----- ----- ----- ----- >>>>>> tank 15.9T 2.19T 87 119 2.34M 1.88M >>>>>> raidz2 2.90T 740G 24 27 762K 95.5K >>>>>> raidz2 3.59T 37.8G 20 0 546K 0 >>>>>> raidz2 3.58T 44.1G 27 0 1.01M 0 >>>>>> raidz2 3.05T 587G 7 47 24.9K 1.07M >>>>>> raidz2 2.81T 835G 8 45 30.9K 733K >>>>>> ------------ ----- ----- ----- ----- ----- ----- >>>>>> >>>>>> >>>>>> >>>>> This system has since been upgraded, but the imbalance in getting >>>>> worse: >>>>> >>>>> zpool iostat -v tank | grep raid >>>>> raidz2 3.60T 28.5G 166 41 6.97M 764K >>>>> raidz2 3.59T 33.3G 170 35 7.35M 709K >>>>> raidz2 3.60T 26.1G 173 35 7.36M 658K >>>>> raidz2 1.69T 1.93T 129 46 6.70M 610K >>>>> raidz2 2.25T 1.38T 124 54 5.77M 967K >>>>> >>>>> Is there any way to determine how this is happening? >>>>> >>>>> I may have to resort to destroying and recreating some large >>>>> filesystems, but there''s no way to determine which ones to target.. >>>>> >>>> Hi, >>>> >>>> As far as i know this is a "normal" behaviour in ZFS... >>>> So what we need is somesort of "rebalance" task what moves data around >>>> multiple vdevs in order to achieve the best performance possible... >>>> Take a look to >>>> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855425 >>>> >>>> >>>> >>> It would be if drives had been added, but this pool hasn''t been >>> changed since it was created >> Hi, >> >> You never experienced any faulted drives, or something similar? So far i >> only saw imbalance if the vdevs add changed, if a hotspare is used and i >> think even during a replacement of one disk of a raidz2 group. >> > There has been one faulted drive, but a hot spare kicked in. A the > moment, I have another replaced by a spare, but I/O to that vdev is > unaffected: > > raidz2 1.69T 1.94T 126 46 6.54M 598K > spare - - 121 34 1.09M 31.4K > c0t4d0 - - 34 23 1.37M 98.7K > c7t5d0 - - 0 78 0 786K > c1t2d0 - - 34 23 1.37M 98.8K > c4t3d0 - - 37 24 1.44M 98.9K > c5t3d0 - - 36 24 1.44M 98.8K > c6t2d0 - - 36 23 1.42M 98.9K > c7t1d0 - - 36 24 1.42M 98.8K > c6t1d0 - - 36 23 1.44M 98.7K > c4t1d0 - - 36 23 1.43M 98.6K > > The vdev with the most free space has never lost a drive. > > Cheers, >-------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/26730c41/attachment.bin>
On 03/26/10 12:16 AM, Bruno Sousa wrote:> Well...i''m pretty much certain that at my job i faced something similar.. > We had a server with 2 raidz2 groups each with 3 drives, and one drive > has failed and replaced by a hot spare. However, the balance of data > between the 2 groups of raidz2 start to be imbalance. >You were right. I cleared about 25% of the data from the pool (2TB) and started to load it back. Using zpool iostat I could see the vdev with the active spare was performing significantly less writes then the others, even though it had the most free space. So I detached the faulted drive to bring the spare into the pool and the write load balance is now biased to that vdev: raidz2 2.91T 730G 0 75 0 3.14M raidz2 2.93T 715G 0 125 76 3.42M raidz2 2.92T 721G 0 140 25 3.46M raidz2 1.54T 2.08T 0 391 0 5.24M raidz2 2.36T 1.26T 0 116 0 3.25M> I assume this is due the fact that during the resilvering process (due > to the hot spare thing) the vdev was somehow "halted" of writting new > data..therefore during that time only 1 group was getting new data, and > that lead to an imbalance data across the 2 raidz2 groups.. > >Or could the writes be reduced to cut the eventual resilver time when the faulted drive is replaced? -- Ian.