Hi everyone, I have a serious problem and need some assistance. I was doing a rolling upgrade of a raidz1, replacing 320GB drives with 1.5TB drives (ie. zpool replace). I had replaced three of the drives and they had resilvered without errors and then I started on the fourth one. It went downhill from there... while the fourth disk was resilvering, one of the others started throwing errors... lots and lots of them... to the point that the system is unusable. So, what I have are the four original 320GB drives that I would like to bring back online (on another server). The pool was unmounted when I was replacing the old units with the new ones, so the data should have been static and I would expect to be able to bring the pool online on another server. The problem I am running into is that the pool seems to be remembering the 1.5TB drive names/locations/etc. So, instead of using the new controller locations in the new system, the pool is insisting on looking for the old devices. For example: In the new system, the drives are c5d0 c5d1 c6d0 c6d1 -- in the old system, the drives were c2d0s0 c3d0s0 c4d0s0 c5d0s0. I have tried everything I can come up with to get the pool to import, but it just wont do it. I need to know how to get zpool to simply import them as I specify the new devices. When I try it, it complains that the pool doesn''t exist... that''s because it wont let me import it! I''m stuck and I need help. I have tried import -f, import -d, import -f -d ... nothing works. I have a lot of data on those drives that I couldn''t back up because I don''t have a backup system big enough to handle it all. I''d really like to salvage that data if at all possible. I was trusting ZFS to handle it and until now, it did an excellent job! It seems pretty simple... shouldn''t each drive be labeled as part of a pool? When I do a zdb -l, I see the labels and pool identifiers, but I also see the old device names hard-coded into the label. How can I get zpool to simply see that all four drives are members of the same pool and build new device locations? Any help would be greatly appreciated. Thanks to you all in advance, -Michael -- This message posted from opensolaris.org
Michael McKnight wrote:> Hi everyone, > > I have a serious problem and need some assistance. I was doing a rolling upgrade of a raidz1, replacing 320GB drives with 1.5TB drives (ie. zpool replace). I had replaced three of the drives and they had resilvered without errors and then I started on the fourth one. It went downhill from there... while the fourth disk was resilvering, one of the others started throwing errors... lots and lots of them... to the point that the system is unusable. > > So, what I have are the four original 320GB drives that I would like to bring back online (on another server). The pool was unmounted when I was replacing the old units with the new ones, so the data should have been static and I would expect to be able to bring the pool online on another server. > > The problem I am running into is that the pool seems to be remembering the 1.5TB drive names/locations/etc. So, instead of using the new controller locations in the new system, the pool is insisting on looking for the old devices. For example: > > In the new system, the drives are c5d0 c5d1 c6d0 c6d1 -- in the old system, the drives were c2d0s0 c3d0s0 c4d0s0 c5d0s0. > > I have tried everything I can come up with to get the pool to import, but it just wont do it. I need to know how to get zpool to simply import them as I specify the new devices. When I try it, it complains that the pool doesn''t exist... that''s because it wont let me import it! I''m stuck and I need help. > > I have tried import -f, import -d, import -f -d ... nothing works. > > I have a lot of data on those drives that I couldn''t back up because I don''t have a backup system big enough to handle it all. I''d really like to salvage that data if at all possible. I was trusting ZFS to handle it and until now, it did an excellent job! > > It seems pretty simple... shouldn''t each drive be labeled as part of a pool? When I do a zdb -l, I see the labels and pool identifiers, but I also see the old device names hard-coded into the label. How can I get zpool to simply see that all four drives are members of the same pool and build new device locations? >If you were doing a rolling upgrade, I suspect the old disks are all horribly out of sync with each other? If that is the problem, then if the filesystem(s) have a snapshot that existed when all the old disks were still online, I wonder if it might be possible to roll them back to it by hand, so it looks like the current live filesystem? I don''t know if this is possible with zdb. -- Andrew
--- On Tue, 12/30/08, Andrew Gabriel <agabriel at opensolaris.org> wrote:>If you were doing a rolling upgrade, I suspect the old disks are all >horribly out of sync with each other? > >If that is the problem, then if the filesystem(s) have a snapshot that >existed when all the old disks were still online, I wonder if it might >be possible to roll them back to it by hand, so it looks like the >current live filesystem? I don''t know if this is possible with zdb.What I meant is that I rolled the new drives in one at a time... replace a drive, let it resilver, replace another, let it resilver. etc.? The filesystems were not mounted at the time so the data should have been static.? The only thing that changed (from what I can tell) is the zpool labels. Thanks, Michael
If memory serves me right, sometime around 12:34am, Michael McKnight told me:> > I have tried import -f, import -d, import -f -d ... nothing works. >Did you try zpool export 1st?
On Tue, Dec 30, 2008 at 3:32 PM, Weldon S Godfrey 3 <weldon at excelsus.com>wrote:> > > > If memory serves me right, sometime around 12:34am, Michael McKnight told > me: > > > > > I have tried import -f, import -d, import -f -d ... nothing works. > > > > Did you try zpool export 1st? >He did say he was doing zpool replace commands when "it went downhill" -- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke My blog: http://initialprogramload.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081230/901c4669/attachment.html>
Yes, but I can''t export a pool that has never been imported. These drives are no longer connected to their original system, and at this point, when I connect them to their original system, the results are the same. Thanks, Michael --- On Tue, 12/30/08, Weldon S Godfrey 3 <weldon at excelsus.com> wrote:> > Did you try zpool export 1st?
I''m not an expert but for what it''s worth- 1. Try the original system. It might be a fluke/bad cable or anything else intermittent. I''ve seen it happen here. If so, your pool may be alright. 2. For the (defunct) originals, I''d say we''d need to take a look into the sources to find if something needs to be done. AFAIK, device paths aren''t hard-coded. ZFS doesn''t care where the disks are as long as it finds them and they contain the right label. -- This message posted from opensolaris.org
> I''m not an expert but for what it''s worth- > > 1. Try the original system. It might be a fluke/bad > cable or anything else intermittent. I''ve seen it > happen here. If so, your pool may be alright. > > 2. For the (defunct) originals, I''d say we''d need to > take a look into the sources to find if something > needs to be done. AFAIK, device paths aren''t > hard-coded. ZFS doesn''t care where the disks are as > long as it finds them and they contain the right > label.I tried the original system and it had much the same reaction. The cables, etc. are all fine. The new system sees the drives and they check out in drive-testing utilities. I don''t think we''re dealing with a hardware issue. I agree that the problem is most likely the labels. When I look at zdb -l output for each of the drives, I can see that they all show the correct pool name and numeric identifier. I think the problem is that they have "children" defined that no longer exist -- at least not at the locations indicated in the label. The question is... how do update the labels so the pool members are all reflected with their new identifications? Thanks, Michael -- This message posted from opensolaris.org