Hello, I have a large migration of a single system going from windows to linux. We are moving 9 TB and 60 million files. I am planning on using rsync to move this data. Are there any special considerations I should have in mind when moving this much data and more importantly, this many small files? I am mounting the volumes out of fstab like this: //dss01/f$ /mnt/dss01/f cifs credentials=/etc/asdf,ro 0 0 And rsync command is: rsync -rt --delete --stats /mnt/dss01/f /data00 Any ideas? Thanks! Terry
On Mon, 25 Feb 2008 14:11:48 -0600 Terry <td3201@gmail.com> wrote:> Are there any special considerations I > should have in mind when moving this much data and more importantly, > this many small files?The short answer is "maybe". :) If you take a peek through the (recent) list archives, you'll note that there has been a lot of discussion surrounding large amount of /files/, and how rsync handles the building of enormous file lists. Several factors will be at play, including the OS of your source and target systems, and which version of rsync you plan to use. I would suggest browsing the thread view from about November 2007 on forward... http://lists.samba.org/archive/rsync/ -- Daniel Maher <dma AT witbe.net> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20080226/1a225c3c/signature.bin
On Mon, 2008-02-25 at 14:11 -0600, Terry wrote:> I have a large migration of a single system going from windows to > linux. We are moving 9 TB and 60 million files. I am planning on > using rsync to move this data. Are there any special considerations I > should have in mind when moving this much data and more importantly, > this many small files? > > I am mounting the volumes out of fstab like this: > //dss01/f$ /mnt/dss01/f cifs credentials=/etc/asdf,ro 0 0 > > And rsync command is: > rsync -rt --delete --stats /mnt/dss01/f /data00In short: use rsync 3.0.0 (or a prerelease of it). The incremental recursion mode will make processing such an enormous number of files much faster and will keep the memory usage reasonable. Matt