Greetings, all. I put myself into a bit of a predicament, and I''m hoping there''s a way out. I had a drive (EIDE) in a ZFS mirror die on me. Not a big deal, right? Well, I bought two SATA drives to build a new mirror. Since they were about the same size (I wanted bigger drives, but they were out of stock), I decided to mirror the still-good EIDE drive with one of the new SATA drives. When that was finished, I detached the EIDE drive from the ZFS mirror, and attached the other SATA drive. The first disk started giving a bunch of errors almost immediately, and the mirror is having big problems reslivering. I suspect I need to replace the disk. The problem is, with the resilver incomplete, I can''t rely solely on the second SATA drive. How can I revert back to the original EIDE drive? I cannot attach it to the mirror, as the "master" is now the bad SATA drive. I also get an error saying it is too small (the drives differ by just a GB or two) when I enter the attach option listing the EIDE drive first. I _really_ don''t want to lose the data in this pool. So, is there a way to blow away the new mirror (that''s easy, actually) and somehow pull in the old EIDE drive as a new zpool without destroying the data? I can''t see anything in the command options that seems suitable for this. Any recommendations I can try when I get home after work would be greatly appreciated. Rainer This message posted from opensolaris.org
So, from the deafening silence, am I to assume there''s no way to tell ZFS that the EIDE drive was a zpool, and pull it into a new pool in a manner that I can (once again) see the data that''s on the drive? :-( Rainer This message posted from opensolaris.org
Rainer Heilke wrote:> Greetings, all. > > I put myself into a bit of a predicament, and I''m hoping there''s a way out. > > I had a drive (EIDE) in a ZFS mirror die on me. Not a big deal, right? Well, I bought two SATA drives to build a new mirror. Since they were about the same size (I wanted bigger drives, but they were out of stock), I decided to mirror the still-good EIDE drive with one of the new SATA drives. When that was finished, I detached the EIDE drive from the ZFS mirror, and attached the other SATA drive. > > The first disk started giving a bunch of errors almost immediately, and the mirror is having big problems reslivering. I suspect I need to replace the disk. The problem is, with the resilver incomplete, I can''t rely solely on the second SATA drive. > > How can I revert back to the original EIDE drive? I cannot attach it to the mirror, as the "master" is now the bad SATA drive. I also get an error saying it is too small (the drives differ by just a GB or two) when I enter the attach option listing the EIDE drive first. I _really_ don''t want to lose the data in this pool. > > So, is there a way to blow away the new mirror (that''s easy, actually) and somehow pull in the old EIDE drive as a new zpool without destroying the data? I can''t see anything in the command options that seems suitable for this. Any recommendations I can try when I get home after work would be greatly appreciated. > > Rainer >Are you able to hook up the original EIDE drive to the same machine/different machine and do a ''zpool import''? I assume no new data has been create/modified since the switch to some broken SATA devices? eric
Nope. I get "no pools available to import". I think that detaching the drive cleared any pool information/headers on the drive, which is why I can''t figure out a way to get the data/pool back. There is some new data on the SATA drives, but I''ve also kept a copy of it elsewhere. I don''t mind losing any new data on the SATA drives as I can lay it down again from the other copy. Rainer This message posted from opensolaris.org
On 11/11/06, Rainer Heilke <rheilke at dragonhearth.com> wrote:> Nope. I get "no pools available to import". I think that detaching the drive cleared any pool information/headers on the drive, which is why I can''t figure out a way to get the data/pool back.Did you also export the original pool before you tried this? I believe it was said that you can''t import a pool if one of the same name already exists on the system. (Of course, you should pull the other disks as well, or it may not import the right pool.) In any case, I don''t think this is expected behavior. It should be possible to remove part of a mirror or simply pull a disk without affecting the contents. (Assuming that the pool is a single N-way mirror vdev.) The manual page for zpool offline indicates that no further attempts are made to read or write the device, so the data should still be there. While it does not elaborate on the result of a zpool detach, I would expect it to behave the same way, by leaving the data intact. If it does not work that way, that seems like a serious bug. Removing a disk should not destroy a complete replica, wether it is through zpool detach and attach, or zpool offline and replace. Chris
Hello Rainer,
Saturday, November 11, 2006, 2:27:46 AM, you wrote:
RH> So, from the deafening silence, am I to assume there''s no way to
RH> tell ZFS that the EIDE drive was a zpool, and pull it into a new
RH> pool in a manner that I can (once again) see the data that''s on
the drive? :-(
Right now there''s no such method - at least not out of the box.
(not that I''m suggesting you can do it easily manually).
However after detaching sub-mirror I think it would be useful to be
able to import it as separate pool - as many people do with SVM.
I haven''t check a code so I don''t know how hard would it be to
implement it. I wanted to look for it for months now but had no
time.
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
Hello Chris, Saturday, November 11, 2006, 9:44:26 AM, you wrote: CC> On 11/11/06, Rainer Heilke <rheilke at dragonhearth.com> wrote:>> Nope. I get "no pools available to import". I think that detaching the drive cleared any pool information/headers on the drive, which is why I can''t figure out a way to get the data/pool back.CC> Did you also export the original pool before you tried this? I CC> believe it was said that you can''t import a pool if one of the same CC> name already exists on the system. (Of course, you should pull the CC> other disks as well, or it may not import the right pool.) CC> In any case, I don''t think this is expected behavior. It should be CC> possible to remove part of a mirror or simply pull a disk without CC> affecting the contents. (Assuming that the pool is a single N-way CC> mirror vdev.) CC> The manual page for zpool offline indicates that no further attempts CC> are made to read or write the device, so the data should still be CC> there. While it does not elaborate on the result of a zpool detach, I CC> would expect it to behave the same way, by leaving the data intact. He did use detach not offline. Also I''m not sure offline works the way you describe (but I guess it does). If it does ''zpool import'' should show a pool to import however I''m not sure if there won''t be a problem with pool id (not pool name). -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
On 11/11/06, Robert Milkowski <rmilkowski at task.gda.pl> wrote:> CC> The manual page for zpool offline indicates that no further attempts > CC> are made to read or write the device, so the data should still be > CC> there. While it does not elaborate on the result of a zpool detach, I > CC> would expect it to behave the same way, by leaving the data intact. > > He did use detach not offline. > Also I''m not sure offline works the way you describe (but I guess it > does). If it does ''zpool import'' should show a pool to import however > I''m not sure if there won''t be a problem with pool id (not pool name).Perhaps I have confused the issue of identical pool id and identical pool names. Still, I expect there will be issues trying to import an orphaned part of an existing pool. This seems like an area which could use a bit of work. While a single mirror vdev pool is a corner case, it probably will be fairly common. If a disk is intact when removed from the mirror, though detach, offline, or simply being pulled, it should remain importable somehow. (Perhaps it does, after addressing the identical pool id issue, though I haven''t tried.) In a similar way, it may be nice to allow detach to work on multiple devices atomically. For instance, if you have a set of mirror vdevs, you could then split off an entire replica of the pool, and move it to another machine. I think you can do this today by simply exporting the pool though, so it is not a major inconvenience. Chris
After exporting the pool on the two SATA drives, shutting down and disconnecting them, I tried importing the pool on the EIDE drive. I get the message about there being no pools to import. This was done using both "zpool import" and "zpool import <poolname>". So, it does seem that something gets cleared when the drive is detached or offlined. I agree that, while a non-standard case, the ability to import the pool on such a disk would be a good thing. In my case specifically, the (first) SATA drive did not exhibit errors until I tried to mirror it to the second SATA drive. In fact, I am now wondering if both drives have issues. I cannot clear the read errors from the first drive, and now have write errors (checksum) on the second disk. These checksum errors will not clear even after detaching the first SATA disk from the mirror and performing two scrub operations. Some of my data is now irretrievably corrupted, I believe, though I have no way of knowing which files are affected. The ability to export the pool on the two SATA drives and retrieve the pool on the EIDE drive is the only way to get back to a clean state. I think I''m out of luck. :-( Rainer This message posted from opensolaris.org
Replying to myself here... ZFS is now in a totally confused state. Trying to attach SATA disk 4 to the pool, I get an error saying a zpool exists on c4d0s0. Yet, when I export the pool on SATA disk 5 and disconnect the drive, and try to import the pool on disk 4, I''m told there aren''t any. zpool did tell me I could use -f to force the zpool mirror operation. I''m thinking I will have to do this. My remaining problem, though, is that I am completely unable to get the pool to a non-errored state. I have to accept that some of the data is now corrupt and I won''t know which until I try to access it, but it would be nice to at least scrub the pool to a clean state, even if that means deleting the errored data. Rainer This message posted from opensolaris.org
Hello Rainer,
Saturday, November 11, 2006, 10:32:42 PM, you wrote:
RH> After exporting the pool on the two SATA drives, shutting down
RH> and disconnecting them, I tried importing the pool on the EIDE
RH> drive. I get the message about there being no pools to import.
RH> This was done using both "zpool import" and "zpool import
RH> <poolname>". So, it does seem that something gets cleared when
the drive is detached or offlined.
Actually it should work.
Can you see disks in format(1M)? If not try to run devfsadm and try
them. First make sure that system created entries in /dev for disks
then try to import.
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
Sorry for the delay... No, it doesn''t. The format command shows the drive, but zpool import does not find any pools. I''ve also used the detached bad SATA drive for testing; no go. Once a drive is detached, there seems to be no (not enough?) information about the pool that allows import. I have also still not been able to clear the pool errors. There doesn''t seem to be any way to relate the errors to the relevant files to fix/delete them. Rainer This message posted from opensolaris.org
Robert Milkowski
2006-Nov-14 06:39 UTC
[zfs-discuss] Re: Re: Re[2]: Re: Dead drives and ZFS
Hello Rainer,
Tuesday, November 14, 2006, 4:43:32 AM, you wrote:
RH> Sorry for the delay...
RH> No, it doesn''t. The format command shows the drive, but zpool
RH> import does not find any pools. I''ve also used the detached bad
RH> SATA drive for testing; no go. Once a drive is detached, there
RH> seems to be no (not enough?) information about the pool that allows
import.
Aha, you did zpool detach - sorry I missed it. Then zpool import won''t
show you any pools to import from such disk. I agree with you it would
be useful to do so.
RH> I have also still not been able to clear the pool errors. There
RH> doesn''t seem to be any way to relate the errors to the relevant
files to fix/delete them.
zpool clear pool_name
?
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
On 11/14/06, Robert Milkowski <rmilkowski at task.gda.pl> wrote:> Hello Rainer, > > Tuesday, November 14, 2006, 4:43:32 AM, you wrote: > > RH> Sorry for the delay... > > RH> No, it doesn''t. The format command shows the drive, but zpool > RH> import does not find any pools. I''ve also used the detached bad > RH> SATA drive for testing; no go. Once a drive is detached, there > RH> seems to be no (not enough?) information about the pool that allows import. > > Aha, you did zpool detach - sorry I missed it. Then zpool import won''t > show you any pools to import from such disk. I agree with you it would > be useful to do so.After examining the source, it clearly wipes the vdev label during a detach. I suppose it does this so that the machine can''t get confused at a later date. It would be nice if the detach simply renamed something, rather than destroying the pool though. At the very least, the manual page ought to reflect the destructive nature of the detach command. That said, it looks as if the code only zeros the first uberblock, so the data may yet be recoverable. In order to reconstruct the pool, I think you would need to replace the vdev labels with ones from another of your mirrors, and possibly the EFI label so that the GUID matched. Then, corrupt the first uberblock, and pray that it imports. (It may be necessary to modify the txg in the labels as well, though I have already speculated enough...) Can anyone say for certain? Chris
On Tue, 2006-11-14 at 03:50 -0600, Chris Csanady wrote:> After examining the source, it clearly wipes the vdev label during a detach. > I suppose it does this so that the machine can''t get confused at a later date. > It would be nice if the detach simply renamed something, rather than > destroying the pool though. At the very least, the manual page ought > to reflect the destructive nature of the detach command.rather than patch it up after the detach, why not have the filesystem do it? seems like the problem would be solved by something looking vaguely like: zpool fork -p poolname -n newpoolname [devname ...] Create the new exported pool "newpoolname" from poolname by detaching one side from each mirrored vdev, starting with the device names listed on the command line. Fails if the pool does not consist exclusively of mirror vdevs, if any device listed on the command line is not part of the pool, or if there is a scrub or resilver necessary or in progress. Use on bootable pools not recommended. For best results, snapshot filesystems you care about before the fork. (just a concept... ) - Bill
On 11/14/06, Bill Sommerfeld <sommerfeld at sun.com> wrote:> On Tue, 2006-11-14 at 03:50 -0600, Chris Csanady wrote: > > After examining the source, it clearly wipes the vdev label during a detach. > > I suppose it does this so that the machine can''t get confused at a later date. > > It would be nice if the detach simply renamed something, rather than > > destroying the pool though. At the very least, the manual page ought > > to reflect the destructive nature of the detach command. > > rather than patch it up after the detach, why not have the filesystem do > it? > > seems like the problem would be solved by something looking vaguely > like: > > zpool fork -p poolname -n newpoolname [devname ...] > > Create the new exported pool "newpoolname" from poolname by detaching > one side from each mirrored vdev, starting with the > device names listed on the command line. Fails if the pool does not > consist exclusively of mirror vdevs, if any device listed on the > command line is not part of the pool, or if there is a scrub or resilver > necessary or in progress. Use on bootable pools not recommended. > For best results, snapshot filesystems you care about before the fork. >I''m more inclined to "split" instead of "fork". ;) -- Regards, Jeremy
On 11/14/06, Jeremy Teo <white.wristband at gmail.com> wrote:> I''m more inclined to "split" instead of "fork". ;)I prefer "split" too since that''s what most of the storage guys are using for mirrors. Still, we are not making any progress on helping Rainer out of his predicaments. -- Just me, Wire ...
>> zpool fork -p poolname -n newpoolname [devname ...] >> >> Create the new exported pool "newpoolname" from poolname by detaching >> one side from each mirrored vdev, starting with the >> device names listed on the command line. Fails if the pool does not >> consist exclusively of mirror vdevs, if any device listed on the >> command line is not part of the pool, or if there is a scrub or resilver >> necessary or in progress. Use on bootable pools not recommended. >> For best results, snapshot filesystems you care about before the fork. >> >I''m more inclined to "split" instead of "fork". ;)Seems that "break" is a more obvious thing to do with mirrors; does this allow me to peel of one bit of a three-way mirror? Casper
Hello Bill, Tuesday, November 14, 2006, 2:31:11 PM, you wrote: BS> On Tue, 2006-11-14 at 03:50 -0600, Chris Csanady wrote:>> After examining the source, it clearly wipes the vdev label during a detach. >> I suppose it does this so that the machine can''t get confused at a later date. >> It would be nice if the detach simply renamed something, rather than >> destroying the pool though. At the very least, the manual page ought >> to reflect the destructive nature of the detach command.BS> rather than patch it up after the detach, why not have the filesystem do BS> it? BS> seems like the problem would be solved by something looking vaguely BS> like: BS> zpool fork -p poolname -n newpoolname [devname ...] BS> Create the new exported pool "newpoolname" from poolname by detaching BS> one side from each mirrored vdev, starting with the BS> device names listed on the command line. Fails if the pool does not BS> consist exclusively of mirror vdevs, if any device listed on the BS> command line is not part of the pool, or if there is a scrub or resilver BS> necessary or in progress. Use on bootable pools not recommended. BS> For best results, snapshot filesystems you care about before the fork. BS> (just a concept... ) Could you please create an RFE for it and give us id? I would immediately add a call record to it :) -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
On Tue, 2006-11-14 at 17:10 +0100, Robert Milkowski wrote:> BS> zpool fork -p poolname -n newpoolname [devname ...]> BS> (just a concept... ) > > Could you please create an RFE for it and give us id? > I would immediately add a call record to it :)6493559 need way to clone a pool by splitting mirrors
Rainer Heilke
2006-Nov-14 16:55 UTC
[zfs-discuss] Re: Re: Re: Re[2]: Re: Dead drives and ZFS
Neither clear nor scrub clean up the errors on the pool. I''ve done this about a dozen times in the past several days, without success. Rainer This message posted from opensolaris.org
This makes sense for the most part (and yes, I think it should be done by the file system, not a manual grovelling through vdev labels). The one difference I would make is that it should not fail if the pool _requires_ a scrub (but yes, if a scrub is in progress...). I worry about this requirement, as my pool has had errors since the second SATA drive was attached (admitedly, it was clean as I detached the EIDE drive). If a scrub cannot clean up the errors on the (bad) disk, the inability to cleanly detach the good disk in a mirror and import the pool from that disk on another system leaves you in the same limbo that I am in now. Thus, fail on the normal attempt, but allow a force if the scrub or resilver are finished but you still have errors on what would be the last (disk) mirror on the system. This would also provide a way to offline a fixed file system state. That is, dettach the mirror, power down, and pull the disk. Some time later, put the disk into a system and pull in the pool(s), creating a data pool of a known, clean state, with known files. One could think of this as a special case of the export function, where one only exports one side of a mirrored pool.. Rainer This message posted from opensolaris.org
On 11/14/06, Rainer Heilke <rheilke at dragonhearth.com> wrote:> This makes sense for the most part (and yes, I think it should be done by the file system, not a manual grovelling through vdev labels).I agree, this should be done with a new command, as has been suggested. However, what I was suggesting with respect to copying the vdev labels is a possible way to recover the the already detached (but otherwise good) mirror from your pool. If you have overwritten the disk though, it no longer matters. Chris
Bill Sommerfeld wrote:> On Tue, 2006-11-14 at 17:10 +0100, Robert Milkowski wrote: > >> BS> zpool fork -p poolname -n newpoolname [devname ...] > >> BS> (just a concept... ) >> >> Could you please create an RFE for it and give us id? >> I would immediately add a call record to it :) > > 6493559 need way to clone a pool by splitting mirrorsFYI, this is a duplicate of: 5097228 provide ''zpool split'' to create new pool by breaking all mirrors --matt
Well, I haven''t overwritten the disk, in the hopes that I can get the data back. So, how do I go about copying or otherwise repairing the vdevs? Rainer This message posted from opensolaris.org
> Seems that "break" is a more obvious thing to do with > mirrors; does this > allow me to peel of one bit of a three-way mirror? > > CasperI would think that this makes sense, and splitting off one side of a two-way mirror is more the edge case (though emphatically required/desired). Rainer This message posted from opensolaris.org