zfs discuss - Oct 2006 - recover zfs data from a crashed system?

Hi,

I am trying to experiment a scenario that we would like to find a
possible solution. Is there anyone out there experienced or analyzed
before the scenario given below?

Scenario: The system is attached to an array. The array type is really
doesn''t matter, i,e., it can be a JBOD or a RAID array. Needless to
say that ZFS is used to access the array. Note that the array is
exclusively used to store data for the database.

The question is that if the system is crashed, can I still use the
array (or rather data) on a different system?

Assume the system is crashed and it can''t come up. I have to work with
support to bring the system up and access the array is one way. My
question is really that is it possible that the array can be detached
from the failed system and attached to another system and get the data
to reduce the downtime? To keep our discussion simpler, lets consider
the target (good) system is exactly similar to the source (failed)
system.

Would ZFS-snapshot or ZFS-clone work? Any pointers/input would be
greatly appreciated. Please don''t hesitate to suggest me RTFM if it
has any good solution. :o)

thanks.

senthil

Hello senthil,

Monday, October 30, 2006, 1:12:28 PM, you wrote:

sr> Hi,

sr> I am trying to experiment a scenario that we would like to find a
sr> possible solution. Is there anyone out there experienced or analyzed
sr> before the scenario given below?

sr> Scenario: The system is attached to an array. The array type is really
sr> doesn''t matter, i,e., it can be a JBOD or a RAID array. Needless
to
sr> say that ZFS is used to access the array. Note that the array is
sr> exclusively used to store data for the database.

sr> The question is that if the system is crashed, can I still use the
sr> array (or rather data) on a different system?

sr> Assume the system is crashed and it can''t come up. I have to
work with
sr> support to bring the system up and access the array is one way. My
sr> question is really that is it possible that the array can be detached
sr> from the failed system and attached to another system and get the data
sr> to reduce the downtime? To keep our discussion simpler, lets consider
sr> the target (good) system is exactly similar to the source (failed)
sr> system.

sr> Would ZFS-snapshot or ZFS-clone work? Any pointers/input would be
sr> greatly appreciated. Please don''t hesitate to suggest me RTFM if
it
sr> has any good solution. :o)

Of course it will work just OOTB.
All you will have to do is to manually import pool(s) on new system.

In case the array has LUN masking feature then probably array
reconfiguration will be needed. But other than that it just work.

So, lets say you''ve got SCSI JBOD connected to host A. Now host A is
down, you re-connecy JBOD to host B, do zpool import pool_a
and that''s it.

Now if you do not use legacy mounts and use sharenfs property instead
/etc/dfs/dfstab then you don''t even have to worry about mountpoints,
fs parameters, nfs shares, etc.


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

senthil ramanujam wrote:
> Hi,
> 
> I am trying to experiment a scenario that we would like to find a
> possible solution. Is there anyone out there experienced or analyzed
> before the scenario given below?
> 
> Scenario: The system is attached to an array. The array type is really
> doesn''t matter, i,e., it can be a JBOD or a RAID array. Needless
to
> say that ZFS is used to access the array. Note that the array is
> exclusively used to store data for the database.
> 
> The question is that if the system is crashed, can I still use the
> array (or rather data) on a different system?

this should work by simply issuing "zfs import" on the "new"
system (you may
need to add "-f"). As long as you don''t reattach the crashed
machine to the
storage, all should be fine.

HTH
-- 
Michael Schuster                  +49 89 46008-2974 / x62974
visit the online support center:  http://www.sun.com/osc/

Recursion, n.: see ''Recursion''

Thanks Robert, Michael.

I guess that has answered my question. I now have got to do a couple
of experiments and get this under control. I will keep you posted if I
see something strange, which I don''t hope for. ;o)

senthil


On 10/30/06, Michael Schuster <Michael.Schuster at sun.com>
wrote:
> senthil ramanujam wrote:
> > Hi,
> >
> > I am trying to experiment a scenario that we would like to find a
> > possible solution. Is there anyone out there experienced or analyzed
> > before the scenario given below?
> >
> > Scenario: The system is attached to an array. The array type is really
> > doesn''t matter, i,e., it can be a JBOD or a RAID array.
Needless to
> > say that ZFS is used to access the array. Note that the array is
> > exclusively used to store data for the database.
> >
> > The question is that if the system is crashed, can I still use the
> > array (or rather data) on a different system?
>
>
> this should work by simply issuing "zfs import" on the
"new" system (you may
> need to add "-f"). As long as you don''t reattach the
crashed machine to the
> storage, all should be fine.
>
> HTH
> --
> Michael Schuster                  +49 89 46008-2974 / x62974
> visit the online support center:  http://www.sun.com/osc/
>
> Recursion, n.: see ''Recursion''
>

Hi Senthil,

We experienced a situation very close to this. Due to some instabilities, we
weren''t able to export the zpool safely from the distressed system (a
T2000 running SXb41). The only free system we had was an X4100, which was
running S10 6/06. Both were SAN attached. The filesystem imported like a champ
onto the X4100 (we had to force the import since we didn''t cleanly
export). We had no corruption issues at all, verified by a full scrub. We went
system to system in about 10 minutes. Most of that was spent re-configuring the
LUN masking on the SAN array. Can''t vouch you''ll have the same
experience, but we were very impressed. Particularly, going between different
Solaris versions and different CPU architectures.

Best Regards,
Jason
 
 
This message posted from opensolaris.org

I was doing some experimentation of my own, using SCSI attached JBOD.

I built a test zpool spanning 7 drives (raidz) on S10U2.  The 7 disks were split
between 3 controllers.
I then started replacing the 18GB drives with 36GB drives, one at a time, and
watched it rebuild the zpool, growing as it did.
Finally, when I had all the drives replaced, I took the system down, moved the
drives around on the controllers, reloaded the OS onto the internal drives on my
Sun Blade 1000 workstation.  I then forced a write of random data to one of the
drives in the 7 disk raidz array.   Then I did the zpool import - it reported
the array, with errors.  I did the zpool import <pool name>, and it came
right in, and mounted up.   From there I did a scrub (and yes, it did
essentially chew up all available system resources, while it scrubbed the 200+GB
pool on this small and not so mighty system), and it corrected all the issues.


All in all it was a good test, and I was rather impressed that it was able to
juggle re-ordering of the drives, re-enumeration of the controllers they were
attached to, erasure of one of the drives, all while importing the pool on a
rebuilt system.

Nice work!

Message was edited by: 
        guyverdh
 
 
This message posted from opensolaris.org