Another hang on zpool import thread, I''m afraid, because I don''t seem to have observed any great successes in the others and I hope there''s a way of saving my data ... In March, using OpenSolaris build 134, I created a zpool, some zfs filesystems, enabled dedup on them, moved content into them and promptly discovered how slow it was because I only have 4GB RAM. Even with 30GB L2ARC, the performance was unacceptable. The trouble started when the machine hung one day. Ever since, I''ve been unable to import my pool without it hanging again. At the time I saw posts from others who had run into similar problems, so I thought it best that I wait until a later build, on the assumption that some ZFS dedup bug would be fixed and I could see my data again. I''ve been waiting ever since, and only just had a chance to try build 147, thanks to illumos and a schillix live CD. However, the pool still won''t import, so I''d much appreciate any troubleshooting hints and tips to help me on my way. schillix b147i My process is: 1. boot the live CD. 2. on the console session, run vmstat 1 3. from another machine, SSH in with multiple sessions and: vmstat 60 vmstat 1 zpool import -f zp zpool iostat zp 1 zpool iostat zp -v 5 4. wait until it all stops What I observe is that the zpool import command never finishes, there will be a lengthy period of read activity made up of very small reads which then stops before an even longer period of what looks like no disk activity. zp 512G 1.31T 0 0 0 0 The box will be responsive for quite some time, seemingly doing not a great deal: kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr cd cd rm s0 in sy cs us sy id 0 0 0 2749064 3122988 0 7 0 0 0 0 0 0 1 0 0 365 218 714 0 1 99 Then after a matter of hours it''ll hang. SSH sessions are no longer responsive. On the console I can press return which creates a new line, but vmstat will have stopped updating. Interestingly, what I observed in b134 was the same thing, however the free memory would slowly decrease over the course of hours, before a sudden nose-dive right before the lock up. Now it appears to hang without that same effect. While the import appears to be working, I can cd to /zp and look at content of the filesystems of 5 of the 9 "esx*" directories. Coincidence or not, it''s the last four which appear to be empty - esx_prod onward. # zfs list NAME USED AVAIL REFER MOUNTPOINT zp 905G 1.28T 23K /zp zp/nfs 889G 1.28T 32K /zp/nfs zp/nfs/esx_dev 264G 1.28T 264G /zp/nfs/esx_dev zp/nfs/esx_hedgehog 25.8G 1.28T 25.8G /zp/nfs/esx_hedgehog zp/nfs/esx_meerkat 223G 1.28T 223G /zp/nfs/esx_meerkat zp/nfs/esx_meerkat_dedup 938M 1.28T 938M /zp/nfs/esx_meerkat_dedup zp/nfs/esx_page 8.90G 1.28T 8.90G /zp/nfs/esx_page zp/nfs/esx_prod 306G 1.28T 306G /zp/nfs/esx_prod zp/nfs/esx_skunk 21K 1.28T 21K /zp/nfs/esx_skunk zp/nfs/esx_temp 45.5G 1.28T 45.5G /zp/nfs/esx_temp zp/nfs/esx_template 15.2G 1.28T 15.2G /zp/nfs/esx_template Any help would be appreciated. What could be going wrong here? Is it getting progressively closer to becoming imported each time I try this, or will it be starting from scratch? Feels to me like there''s an action in the /zp/nfs/esx_prod filesystem it''s trying to replay and never getting to the end of, for some reason. In case it was getting in a muddle with the l2arc, I removed the cache device a matter of minutes into this run. It hasn''t hung yet, vmstat is still updating, but I tried a ''zpool import'' in one of the windows to see if I could even see a pool on another disk, and that hasn''t returned me back to the prompt yet. Also tried to SSH in with another session, and that hasn''t produced the login prompt. Thanks in advance, Chris -- This message posted from opensolaris.org
Chris Murray wrote:> Another hang on zpool import thread, I''m afraid, because I don''t seem to have observed any great successes in the others and I hope there''s a way of saving my data ... > > In March, using OpenSolaris build 134, I created a zpool, some zfs filesystems, enabled dedup on them, moved content into them and promptly discovered how slow it was because I only have 4GB RAM. Even with 30GB L2ARC, the performance was unacceptable. The trouble started when the machine hung one day. Ever since, I''ve been unable to import my pool without it hanging again. At the time I saw posts from others who had run into similar problems, so I thought it best that I wait until a later build, on the assumption that some ZFS dedup bug would be fixed and I could see my data again. I''ve been waiting ever since, and only just had a chance to try build 147, thanks to illumos and a schillix live CD. > > However, the pool still won''t import, so I''d much appreciate any troubleshooting hints and tips to help me on my way. > > schillix b147i > > My process is: > 1. boot the live CD. > 2. on the console session, run vmstat 1 > 3. from another machine, SSH in with multiple sessions and: > vmstat 60 > vmstat 1 > zpool import -f zp > zpool iostat zp 1 > zpool iostat zp -v 5 > 4. wait until it all stops > > What I observe is that the zpool import command never finishes, there will be a lengthy period of read activity made up of very small reads which then stops before an even longer period of what looks like no disk activity. > > zp 512G 1.31T 0 0 0 0 > > The box will be responsive for quite some time, seemingly doing not a great deal: > > kthr memory page disk faults cpu > r b w swap free re mf pi po fr de sr cd cd rm s0 in sy cs us sy id > 0 0 0 2749064 3122988 0 7 0 0 0 0 0 0 1 0 0 365 218 714 0 1 99 > > Then after a matter of hours it''ll hang. SSH sessions are no longer responsive. On the console I can press return which creates a new line, but vmstat will have stopped updating. > > Interestingly, what I observed in b134 was the same thing, however the free memory would slowly decrease over the course of hours, before a sudden nose-dive right before the lock up. Now it appears to hang without that same effect. > > While the import appears to be working, I can cd to /zp and look at content of the filesystems of 5 of the 9 "esx*" directories. > Coincidence or not, it''s the last four which appear to be empty - esx_prod onward. > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > zp 905G 1.28T 23K /zp > zp/nfs 889G 1.28T 32K /zp/nfs > zp/nfs/esx_dev 264G 1.28T 264G /zp/nfs/esx_dev > zp/nfs/esx_hedgehog 25.8G 1.28T 25.8G /zp/nfs/esx_hedgehog > zp/nfs/esx_meerkat 223G 1.28T 223G /zp/nfs/esx_meerkat > zp/nfs/esx_meerkat_dedup 938M 1.28T 938M /zp/nfs/esx_meerkat_dedup > zp/nfs/esx_page 8.90G 1.28T 8.90G /zp/nfs/esx_page > zp/nfs/esx_prod 306G 1.28T 306G /zp/nfs/esx_prod > zp/nfs/esx_skunk 21K 1.28T 21K /zp/nfs/esx_skunk > zp/nfs/esx_temp 45.5G 1.28T 45.5G /zp/nfs/esx_temp > zp/nfs/esx_template 15.2G 1.28T 15.2G /zp/nfs/esx_template > > Any help would be appreciated. What could be going wrong here? Is it getting progressively closer to becoming imported each time I try this, or will it be starting from scratch? Feels to me like there''s an action in the /zp/nfs/esx_prod filesystem it''s trying to replay and never getting to the end of, for some reason. In case it was getting in a muddle with the l2arc, I removed the cache device a matter of minutes into this run. It hasn''t hung yet, vmstat is still updating, but I tried a ''zpool import'' in one of the windows to see if I could even see a pool on another disk, and that hasn''t returned me back to the prompt yet. Also tried to SSH in with another session, and that hasn''t produced the login prompt. > > Thanks in advance, > ChrisIt looks like you may be past the import phase and into the mounting phase. What I would recommend is that you ''zpool import -N zp'' so that none of the datasets get mounted and only the import happens. Then one by one you can mount the datasets in order (starting with ''zp'') so you can find out which one maybe hanging. - George
Absolutely spot on George. The import with -N took seconds. Working on the assumption that esx_prod is the one with the problem, I bumped that to the bottom of the list. Each mount was done in a second: # zfs mount zp # zfs mount zp/nfs # zfs mount zp/nfs/esx_dev # zfs mount zp/nfs/esx_hedgehog # zfs mount zp/nfs/esx_meerkat # zfs mount zp/nfs/esx_meerkat_dedup # zfs mount zp/nfs/esx_page # zfs mount zp/nfs/esx_skunk # zfs mount zp/nfs/esx_temp # zfs mount zp/nfs/esx_template And those directories have the content in them that I''d expect. Good! So now I try to mount esx_prod, and the influx of reads has started in zpool iostat zp 1 This is the filesystem with the issue, but what can I do now? Thanks again. -- This message posted from opensolaris.org
Pawel Jakub Dawidek
2010-Sep-13 13:06 UTC
[zfs-discuss] Hang on zpool import (dedup related)
On Sun, Sep 12, 2010 at 11:24:06AM -0700, Chris Murray wrote:> Absolutely spot on George. The import with -N took seconds. > > Working on the assumption that esx_prod is the one with the problem, I bumped that to the bottom of the list. Each mount was done in a second: > > # zfs mount zp > # zfs mount zp/nfs > # zfs mount zp/nfs/esx_dev > # zfs mount zp/nfs/esx_hedgehog > # zfs mount zp/nfs/esx_meerkat > # zfs mount zp/nfs/esx_meerkat_dedup > # zfs mount zp/nfs/esx_page > # zfs mount zp/nfs/esx_skunk > # zfs mount zp/nfs/esx_temp > # zfs mount zp/nfs/esx_template > > And those directories have the content in them that I''d expect. Good! > > So now I try to mount esx_prod, and the influx of reads has started in zpool iostat zp 1 > > This is the filesystem with the issue, but what can I do now?You could try to snapshot it (but keep it unmounted), then zfs send it and zfs recv it to eg. zp/foo. Use -u option for zfs recv too, then try to mount what you received. -- Pawel Jakub Dawidek http://www.wheelsystems.com pjd at FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100913/87965e43/attachment.bin>