Hello, I'm having an issue with one particular server and am hoping someone here has dealt with this. I'm not sure whether this is a strictly samba issue or relates to the way rsync walks the file list. Basically after mounting a Windows 2000 file system using and then rsyncing the contents of this mount, it seems to take 5 - 8 hours to complete. I've checked on the log periodically and determined that it's the 'building of the file list' that is taking 95% of the time. We are only talking about 140,000 files. I do many samba shares and not of them have this issue. When doing a manual 'ls' command in various directories on the mount, I encounter no slowness or anything out of the ordinary. The samba log doesn't give much of a clue either. Has anybody come across this? Or does anybody have any ideas of how to troubleshoot? Oh, and I'm using Rsync 2.6 Thanks, Max
Max - what is the OS of the machine you're using to access the Win2K SMB share from? I ask because the reference implementation of Samba - ie, the ftp-alike smbclient - works just dandy under every OS I've looked at, but the kernel implementation of smbfs under FreeBSD (and possibly other BSDs, I'm not sure) is severely broken - it will work fine for small operations from an interactive prompt, but if you ask it to handle several thousand files, it slows to an absolute crawl. This is not an rsync-samba interaction problem, it occurs even with a simple cp /mnt/smbshare/* /home/whoever type command as well, if significant numbers of files are involved. Hope this helps.> Hello, > > > > I'm having an issue with one particular server and am hoping someone > here has dealt with this. > > > > I'm not sure whether this is a strictly samba issue or relates to the > way rsync walks the file list. > > > > Basically after mounting a Windows 2000 file system using and then > rsyncing the contents of this mount, it seems to take 5 - 8 hours to > complete. I've checked on the log periodically and determined that it's > the 'building of the file list' that is taking 95% of the time. We are > only talking about 140,000 files. I do many samba shares and not of them > have this issue. When doing a manual 'ls' command in various directories > on the mount, I encounter no slowness or anything out of the ordinary. > The samba log doesn't give much of a clue either. > > > > Has anybody come across this? Or does anybody have any ideas of how to > troubleshoot? > > > > Oh, and I'm using Rsync 2.6 > > > > Thanks, > > Max >
> I'm having an issue with one particular server and am hoping someone > here has dealt with this. > > > > I'm not sure whether this is a strictly samba issue or relates to the > way rsync walks the file list. > > > > Basically after mounting a Windows 2000 file system using and then > rsyncing the contents of this mount, it seems to take 5 - 8 hours to > complete. I've checked on the log periodically and determined thatit's> the 'building of the file list' that is taking 95% of the time. We are > only talking about 140,000 files. I do many samba shares and not ofthem> have this issue. When doing a manual 'ls' command in variousdirectories> on the mount, I encounter no slowness or anything out of the ordinary. > The samba log doesn't give much of a clue either. > > > > Has anybody come across this? Or does anybody have any ideas of how to > troubleshoot? > > > > Oh, and I'm using Rsync 2.6Max - what is the OS of the machine you're using to access the Win2K SMB share from? I ask because the reference implementation of Samba - ie, the ftp-alike smbclient - works just dandy under every OS I've looked at, but the kernel implementation of smbfs under FreeBSD (and possibly other BSDs, I'm not sure) is severely broken - it will work fine for small operations from an interactive prompt, but if you ask it to handle several thousand files, it slows to an absolute crawl. This is not an rsync-samba interaction problem, it occurs even with a simple cp /mnt/smbshare/* /home/whoever type command as well, if significant numbers of files are involved. Hope this helps. ------------------------------------------------------ I'm actually using RedHat 9 with Samba 2.2.8a. 3.0 was giving me a slight issue with hanging on the mount command. I've got this same basic setup on several servers but this is the only one giving me this issue. I have one that does 130,000 from a samba mount that takes 8 minutes for both the 'file walk' and the actual syncing of files. I've tried a -vv and it doesn't show much of interest. I may try a triple v. Thanks, Max
I assume - the remote server is Win2K - the remote server does not have an rsync daemon - the remote server is accessed using Samba The problem, I reckon, is that your entire set of files is being dragged across the network to the local machine during the "building file list" before it properly gets into copying files. My simple understanding of what rsync does is this: Frist: Rsync works out what to send by firstly determining which files exist on the source (this is a quick "ls" type operation). Second: assume a file has an identical file size then rsync must examine the file to see if it is the same. It does, in essence, by doing a checksum on the file; I am not sure quite how rsync works, but let's assume it divides the file into blocks and does a checksum on the first block of bytes and compares that to the local copy's first block checksum. How can it do that on the remote Win2K? If you machine has no rsync daemon then your LOCAL machine needs to calculate the checksum for the REMOTE file. The checksum is small to transmit over the network, but if the block is, say 100k, we incur a 100k(+a little bit) network transmission. If the file is actually identical then WHOLE file has to be transmitted before it knows that that file does not need to be transferred! So why use rsync if it is this "bad"? Well much of the time (a) the remote runs an rsync daemon so the client can ask the remote to calculate the checksum, (b) the local can invoke on the remote a shell in which it can invoke rsync. In the case of Win2K and a Samba share you don't have this possibility. How should you solve it? One way is to use "Cygwin" (www.cygwin.com) which allows you to run Unix commands (including rysnc) on a Windows platform. I use rsync CLIENT on a Win2K box fine. What I have not tried is running the daemon on the Win2K box. Perhaps someone else can confirm this works ok. At 18:43 16/02/2004, Max Kipness wrote:>Hello, > > > >I'm having an issue with one particular server and am hoping someone >here has dealt with this. > > > >I'm not sure whether this is a strictly samba issue or relates to the >way rsync walks the file list. > > > >Basically after mounting a Windows 2000 file system using and then >rsyncing the contents of this mount, it seems to take 5 - 8 hours to >complete. I've checked on the log periodically and determined that it's >the 'building of the file list' that is taking 95% of the time. We are >only talking about 140,000 files. I do many samba shares and not of them >have this issue. When doing a manual 'ls' command in various directories >on the mount, I encounter no slowness or anything out of the ordinary. >The samba log doesn't give much of a clue either. > > > >Has anybody come across this? Or does anybody have any ideas of how to >troubleshoot? > > > >Oh, and I'm using Rsync 2.6 > > > >Thanks, > >Max > >-- >To unsubscribe or change options: >http://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
On Mon, Feb 16, 2004 at 11:19:25PM +0000, Andrew Liles wrote:> Second: assume a file has an identical file size then rsync must > examine the file to see if it is the same. It does, in essence, by > doing a checksum on the file;This is not correct. The default "is unchanged" algorithm uses just the size and the modify-time of the files, so it only needs to stat the files. If you specify -c, rsync checksums _all_ the files. There are also other options to modify how rsync decides a file is unchanged (e.g. -I and --size-only), but these are specialty options that are not typically used. ..wayne..
So...assuming that I am using Samba to rsync a share. And using "--archive --compress --times --delete --stat -e ssh" as my option, will the assumption by Andrew Liles is still true? Wayne Davison wrote:> On Mon, Feb 16, 2004 at 11:19:25PM +0000, Andrew Liles wrote: > > Second: assume a file has an identical file size then rsync must > > examine the file to see if it is the same. It does, in essence, by > > doing a checksum on the file; > > This is not correct. The default "is unchanged" algorithm uses just the > size and the modify-time of the files, so it only needs to stat the > files. If you specify -c, rsync checksums _all_ the files. There are > also other options to modify how rsync decides a file is unchanged (e.g. > -I and --size-only), but these are specialty options that are not > typically used. > > ...wayne.. >
On Wed, Feb 18, 2004 at 11:37:48AM -0600, Adnan wrote:> So...assuming that I am using Samba to rsync a share. And using > "--archive --compress --times --delete --stat -e ssh" as my option, will > the assumption by Andrew Liles is still true?Those options use the default "is updated" check (size and mtime). ..wayne..
Well, this is what I noticed on a Samba share. Using the options below, I ran 2 tests and the results are below: Test 1: Number of files: 95000 Number of bytes: 593MB Total time for building file list was 70 secs Test2: Number of files: 2093 Number of bytes: 3.14GB Total time for building file list was 5 secs Is there a way or option to minimize building the file list? The reason I am asking is because I have shares that have more than 200,000 files and about 100GB of data and it is taking awfully slow on a 1GB network card. Thanks Adnan -----Original Message----- From: Wayne Davison [mailto:wayned@samba.org] Sent: Wednesday, February 18, 2004 11:42 AM To: Adnan Cc: rsync@lists.samba.org Subject: Re: File walking issue? On Wed, Feb 18, 2004 at 11:37:48AM -0600, Adnan wrote:> So...assuming that I am using Samba to rsync a share. And using > "--archive --compress --times --delete --stat -e ssh" as my option,will> the assumption by Andrew Liles is still true?Those options use the default "is updated" check (size and mtime). ..wayne..