Lustre discuss - Nov 2007 - Copying data from one Lustre system to an other lustre system

Hi;

 

Customer have 2 x lustre system. Two independen MDS clusters looking after
their individual OSS''s. 

We need to copy 80 TB of data from one system to the other system. One full
copy + 6 incremental copies per week. 

 

I plan on offering a few data mover servers. 

 

Can I use one IB HCA dedicated to source lustre and another HCA dedicated to
the target lustre, so that systems qill not have to share bandwith through
one HCA?

 

By the way do anybody have any scripts ready to do the job above?

 

regards

 


 <http://www.sun.com/> http://www.sun.com/emrkt/sigs/6g_top.gif

Mertol Ozyoney 
Storage Practice - Sales Manager

Sun Microsystems, TR
Istanbul TR
Phone +902123352200
Mobile +905339310752
Fax +902123352222
Email  <mailto:Ayca.Yalcin at Sun.COM> mertol.ozyoney at Sun.COM

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071106/5a4db9c2/attachment-0002.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 1257 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071106/5a4db9c2/attachment-0002.gif

Mertol,

You probably want to check out the "e2scan" work that is being done
atm. There''s a bugzilla entry for it (NERSC). As this was not available
to us a couple of months ago we came up with a way to snapshot both MDSs and
then run "rsync --itemize-changes" between them. This builds a
filelist that we can then split into deletes and copies. For the copies you can
split it N times and use multiple "rsync --files-from=" instance
across many clients.

The filelist generation takes around 6-8 hours and the copies around 5 hours for
our 72Tb filesystem. Fast enough for us to get a backup through every 24 hours.
But like I said a better/faster longterm solution is to use e2scan.

Daire


----- "Mertol Ozyoney" <Mertol.Ozyoney at Sun.COM>
wrote:
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss