I get this error when I try to copy a directory with a lot of files: "bash: /usr/local/bin/rsync: Argument list too long" The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/wex/* root@192.168.250.68:/mail/spool/imap/user/wex". BUT, if I try tris command it works: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/* root@192.168.250.68:/mail/spool/imap/user/wex". So I see that rsync does not copies the files if in the root of the source directory are too many files. But if they are too meny files in any oter directory in works. I need the first command to work. How can I do this? Thank you.
YOSHIFUJI Hideaki / 吉藤英明
2004-Aug-10 12:05 UTC
bash: /usr/local/bin/rsync: Argument list too long
In article <4118B482.90900@ambra.ro> (at Tue, 10 Aug 2004 14:41:54 +0300), victor <victor@ambra.ro> says:> I get this error when I try to copy a directory with a lot of files: > "bash: /usr/local/bin/rsync: Argument list too long" > > The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r > --delete --perms --owner --group /mail/spool/imap/user/wex/* > root@192.168.250.68:/mail/spool/imap/user/wex".:> So I see that rsync does not copies the files if in the root of the > source directory are too many files. But if they are too meny files in > any oter directory in works. > > I need the first command to work. > How can I do this?This is nothing to do with rsync, but your shell. Anyway, RTFM, or you may want to try xargs(1). --yoshfuji
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> I get this error when I try to copy a directory with a lot of files: > "bash: /usr/local/bin/rsync: Argument list too long" > > The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r > --delete --perms --owner --group /mail/spool/imap/user/wex/* > root@192.168.250.68:/mail/spool/imap/user/wex". > > BUT, if I try tris command it works: "/usr/local/bin/rsync > -rsh=/usr/bin/rsh -r --delete --perms --owner --group > /mail/spool/imap/user/* root@192.168.250.68:/mail/spool/imap/user/wex". > > So I see that rsync does not copies the files if in the root of the > source directory are too many files. But if they are too meny files in > any oter directory in works. > > I need the first command to work. > How can I do this? > > Thank you.Your shell is expanding the * to create an argument list that is too long. It's an issue with your shell, not rsync. - -- Mark Watts Senior Systems Engineer QinetiQ Trusted Information Management Trusted Solutions and Services group GPG Public Key ID: 455420ED -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBGLsPBn4EFUVUIO0RAkNkAKDzsNLYV5jUCzf+MH0z3yML/50RVQCg8r+9 hByFgOrVt9zXmGxftbmIVzI=OtuC -----END PGP SIGNATURE-----
Jan-Benedict Glaw
2004-Aug-10 12:36 UTC
bash: /usr/local/bin/rsync: Argument list too long
On Tue, 2004-08-10 14:41:54 +0300, victor <victor@ambra.ro> wrote in message <4118B482.90900@ambra.ro>:> I get this error when I try to copy a directory with a lot of files: > "bash: /usr/local/bin/rsync: Argument list too long" > > The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r > --delete --perms --owner --group /mail/spool/imap/user/wex/* > root@192.168.250.68:/mail/spool/imap/user/wex".The "*" you supply is expanded by the shell to a lot of filenames. Your shell uses some internal buffer (which may grow to several megabytes), but upon exec*(), the kernel cannot copy all the list to the (child's) argv[] buffer, thus rejecting to exec() at all. So this isn't actually a limitation of your shell, but of your operating system. IIRC Linux will grant you some 128 KB on systems using 4KB pages (that is, the famous PeeCee). You've got several ways to work around that: - Try to split your single rsync call into severals: rsync a* rsync b* rsync c* ... - Hack your operating system's kernel to allow a larger buffer for argv[]. (For Linux, you'll need to edit ./include/linux/binfmts.h; change MAX_ARG_PAGES to whatever you like better) - Try to use xargs, but that may be tricky... MfG, JBG -- Jan-Benedict Glaw jbglaw@lug-owl.de . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://lists.samba.org/archive/rsync/attachments/20040810/6fdc7a49/attachment.bin
Thank you. I will try the first sugestion. However, can you explain me why this command works? /usr/local/bin/rsync -r --delete --perms --owner --group /mail/spool/imap/user root@192.168.250.68:/mail/spool/imap In /mail/spool/imap/user I have a lot of subdirectories with >1000 files. What I mean is that rsync can make sutch a transfer(no mather the kernel), but for some reason he does not. Jan-Benedict Glaw wrote:> On Tue, 2004-08-10 14:41:54 +0300, victor <victor@ambra.ro> > wrote in message <4118B482.90900@ambra.ro>: > > >> I get this error when I try to copy a directory with a lot of files: >> "bash: /usr/local/bin/rsync: Argument list too long" >> >> The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r >> --delete --perms --owner --group /mail/spool/imap/user/wex/* >> root@192.168.250.68:/mail/spool/imap/user/wex". >> > > > The "*" you supply is expanded by the shell to a lot of filenames. Your > shell uses some internal buffer (which may grow to several megabytes), > but upon exec*(), the kernel cannot copy all the list to the (child's) > argv[] buffer, thus rejecting to exec() at all. > > So this isn't actually a limitation of your shell, but of your operating > system. IIRC Linux will grant you some 128 KB on systems using 4KB pages > (that is, the famous PeeCee). > > You've got several ways to work around that: > > - Try to split your single rsync call into severals: > > rsync a* > rsync b* > rsync c* > ... > > - Hack your operating system's kernel to allow a larger buffer > for argv[]. (For Linux, you'll need to edit > ./include/linux/binfmts.h; change MAX_ARG_PAGES to whatever > you like better) > > - Try to use xargs, but that may be tricky... > > MfG, JBG > > >
You want everything in wex into wex on the remote. Gotcha. Let's take a simple case. wex contains a b c d e. "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/wex/* root@192.168.250.68:/mail/spool/imap/user/wex" on the commandline becomes "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/wex/a /mail/spool/imap/user/wex/b /mail/spool/imap/user/wex/c /mail/spool/imap/user/wex/d /mail/spool/imap/user/wex/e root@192.168.250.68:/mail/spool/imap/user/wex". The parameter ending in "*" is replaced by as many entries as there are in the directory, which can be quite a loth, and is your problem. rsync is perfectly happy to handle freaking enormous numbers of files, but it has to find out about them. Unless there's some valid reason why you want to avoid applying perms,owner,group to the wex directory itself, "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/wex/. root@192.168.250.68:/mail/spool/imap/user/wex/." will work nicely, or perhaps "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/wex root@192.168.250.68:/mail/spool/imap/user". These will let rsync build the filelist itself, avoiding problems (and plain old inefficiencies) of argument passing. Speaking of inefficiencies, unless you want to avoid maintaining symlinks devices (not likely to be there anyway) and times, you can improve readability by changing your commandline to "/usr/local/bin/rsync -rsh=/usr/bin/rsh -a --delete /mail/spool/imap/user/wex/. root@192.168.250.68:/mail/spool/imap/user/wex/." Letting it keep times synced lets it use them to optimize future syncs by not checksumming files that match in name/timestamp/size. "-a" is a lot faster to type than "--owner --group --perms --times --links --recursive --devices" Tim Conway Unix System Administration Contractor - IBM Global Services desk:3032734776 conway@us.ibm.com I get this error when I try to copy a directory with a lot of files: "bash: /usr/local/bin/rsync: Argument list too long" The exact command is: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/wex/* root@192.168.250.68:/mail/spool/imap/user/wex". BUT, if I try tris command it works: "/usr/local/bin/rsync -rsh=/usr/bin/rsh -r --delete --perms --owner --group /mail/spool/imap/user/* root@192.168.250.68:/mail/spool/imap/user/wex".