rsync - Aug 2004 - bash: /usr/local/bin/rsync: Argument list too long

I get this error when I try to copy a directory with a lot of files: 
"bash: /usr/local/bin/rsync: Argument list too long"

The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r 
--delete --perms --owner --group /mail/spool/imap/user/wex/* 
root@192.168.250.68:/mail/spool/imap/user/wex".

BUT, if I try tris command it works: "/usr/local/bin/rsync 
-rsh=/usr/bin/rsh -r --delete --perms --owner --group 
/mail/spool/imap/user/* root@192.168.250.68:/mail/spool/imap/user/wex".

So I see that rsync does not copies the files if in the root of the 
source directory are too many files. But if they are too meny files in 
any oter directory in works.

I need the first command to work.
How can I do this?

Thank you.

In article <4118B482.90900@ambra.ro> (at Tue, 10 Aug 2004 14:41:54 +0300),
victor <victor@ambra.ro> says:
> I get this error when I try to copy a directory with a lot of files: 
> "bash: /usr/local/bin/rsync: Argument list too long"
> 
> The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r
> --delete --perms --owner --group /mail/spool/imap/user/wex/*
> root@192.168.250.68:/mail/spool/imap/user/wex".
:
> So I see that rsync does not copies the files if in the root of the 
> source directory are too many files. But if they are too meny files in 
> any oter directory in works.
> 
> I need the first command to work.
> How can I do this?
This is nothing to do with rsync, but your shell.
Anyway, RTFM, or you may want to try xargs(1).

--yoshfuji

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I get this error when I try to copy a directory with a lot of files:
> "bash: /usr/local/bin/rsync: Argument list too long"
>
> The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r
> --delete --perms --owner --group /mail/spool/imap/user/wex/*
> root@192.168.250.68:/mail/spool/imap/user/wex".
>
> BUT, if I try tris command it works: "/usr/local/bin/rsync
> -rsh=/usr/bin/rsh -r --delete --perms --owner --group
> /mail/spool/imap/user/*
root@192.168.250.68:/mail/spool/imap/user/wex".
>
> So I see that rsync does not copies the files if in the root of the
> source directory are too many files. But if they are too meny files in
> any oter directory in works.
>
> I need the first command to work.
> How can I do this?
>
> Thank you.
Your shell is expanding the * to create an argument list that is too long.
It's an issue with your shell, not rsync.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBGLsPBn4EFUVUIO0RAkNkAKDzsNLYV5jUCzf+MH0z3yML/50RVQCg8r+9
hByFgOrVt9zXmGxftbmIVzI=OtuC
-----END PGP SIGNATURE-----

On Tue, 2004-08-10 14:41:54 +0300, victor <victor@ambra.ro>
wrote in message <4118B482.90900@ambra.ro>:
> I get this error when I try to copy a directory with a lot of files: 
> "bash: /usr/local/bin/rsync: Argument list too long"
> 
> The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r 
> --delete --perms --owner --group /mail/spool/imap/user/wex/* 
> root@192.168.250.68:/mail/spool/imap/user/wex".
The "*" you supply is expanded by the shell to a lot of filenames.
Your
shell uses some internal buffer (which may grow to several megabytes),
but upon exec*(), the kernel cannot copy all the list to the (child's)
argv[] buffer, thus rejecting to exec() at all.

So this isn't actually a limitation of your shell, but of your operating
system. IIRC Linux will grant you some 128 KB on systems using 4KB pages
(that is, the famous PeeCee).

You've got several ways to work around that:

	- Try to split your single rsync call into severals:

		rsync a*
		rsync b*
		rsync c*
		...

	- Hack your operating system's kernel to allow a larger buffer
	  for argv[]. (For Linux, you'll need to edit
	  ./include/linux/binfmts.h; change MAX_ARG_PAGES to whatever
	  you like better)

	- Try to use xargs, but that may be tricky...

MfG, JBG

-- 
Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481             _ O _
"Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg 
_ _ O
 fuer einen Freien Staat voll Freier B?rger" | im Internet! |   im Irak!  
O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url :
http://lists.samba.org/archive/rsync/attachments/20040810/6fdc7a49/attachment.bin

Thank you.
I will try the first sugestion.

However, can you explain me why this command works?
/usr/local/bin/rsync -r --delete --perms --owner --group 
/mail/spool/imap/user root@192.168.250.68:/mail/spool/imap
In /mail/spool/imap/user I have a lot of subdirectories with >1000 files.

What I mean is that rsync can make sutch a transfer(no mather the 
kernel), but for some reason he does not.

Jan-Benedict Glaw wrote:
> On Tue, 2004-08-10 14:41:54 +0300, victor <victor@ambra.ro>
> wrote in message <4118B482.90900@ambra.ro>:
>  
>
>> I get this error when I try to copy a directory with a lot of files: 
>> "bash: /usr/local/bin/rsync: Argument list too long"
>>
>> The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r 
>> --delete --perms --owner --group /mail/spool/imap/user/wex/* 
>> root@192.168.250.68:/mail/spool/imap/user/wex".
>>   
>
>
> The "*" you supply is expanded by the shell to a lot of
filenames. Your
> shell uses some internal buffer (which may grow to several megabytes),
> but upon exec*(), the kernel cannot copy all the list to the (child's)
> argv[] buffer, thus rejecting to exec() at all.
>
> So this isn't actually a limitation of your shell, but of your
operating
> system. IIRC Linux will grant you some 128 KB on systems using 4KB pages
> (that is, the famous PeeCee).
>
> You've got several ways to work around that:
>
>     - Try to split your single rsync call into severals:
>
>         rsync a*
>         rsync b*
>         rsync c*
>         ...
>
>     - Hack your operating system's kernel to allow a larger buffer
>       for argv[]. (For Linux, you'll need to edit
>       ./include/linux/binfmts.h; change MAX_ARG_PAGES to whatever
>       you like better)
>
>     - Try to use xargs, but that may be tricky...
>
> MfG, JBG
>
>  
>

You want everything in wex into wex on the remote.  Gotcha.
Let's take a simple case.
wex contains a b c d e.
"/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner 
--group /mail/spool/imap/user/wex/* 
root@192.168.250.68:/mail/spool/imap/user/wex" on the commandline becomes 
"/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner 
--group /mail/spool/imap/user/wex/a /mail/spool/imap/user/wex/b 
/mail/spool/imap/user/wex/c /mail/spool/imap/user/wex/d 
/mail/spool/imap/user/wex/e 
root@192.168.250.68:/mail/spool/imap/user/wex".  The parameter ending in 
"*" is replaced by as many entries as there are in the directory,
which
can be quite a loth, and is your problem.  rsync is perfectly happy to 
handle freaking enormous numbers of files, but it has to find out about 
them.
Unless there's some valid reason why you want to avoid applying 
perms,owner,group to the wex directory itself, "/usr/local/bin/rsync 
-rsh=/usr/bin/rsh -r --delete --perms --owner --group 
/mail/spool/imap/user/wex/. 
root@192.168.250.68:/mail/spool/imap/user/wex/." will work nicely, or 
perhaps "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms 
--owner --group /mail/spool/imap/user/wex 
root@192.168.250.68:/mail/spool/imap/user".  These will let rsync build 
the filelist itself, avoiding problems (and plain old inefficiencies) of 
argument passing.
Speaking of inefficiencies, unless you want to avoid maintaining symlinks 
devices (not likely to be there anyway) and times, you can improve 
readability by changing your commandline to "/usr/local/bin/rsync 
-rsh=/usr/bin/rsh -a --delete /mail/spool/imap/user/wex/. 
root@192.168.250.68:/mail/spool/imap/user/wex/."  Letting it keep times 
synced lets it use them to optimize future syncs by not checksumming files 
that match in name/timestamp/size.  "-a" is a lot faster to type than 
"--owner --group --perms --times --links --recursive --devices"

Tim Conway
Unix System Administration
Contractor - IBM Global Services
desk:3032734776
conway@us.ibm.com



I get this error when I try to copy a directory with a lot of files: 
"bash: /usr/local/bin/rsync: Argument list too long"

The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r 
--delete --perms --owner --group /mail/spool/imap/user/wex/* 
root@192.168.250.68:/mail/spool/imap/user/wex".

BUT, if I try tris command it works: "/usr/local/bin/rsync 
-rsh=/usr/bin/rsh -r --delete --perms --owner --group 
/mail/spool/imap/user/* root@192.168.250.68:/mail/spool/imap/user/wex".