Hi folks, I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my first backup yesterday. But I notice that some hidden (or dot files) are not copied to the other host on the network. And group and ownership of the files are changed to ?nobody? while files and folders permission are copied properly. So what am mission here can anybody shed light please? Here is the result; --------------------------------------------------------------------------- Number of files: 101558 Number of files transferred: 94209 Total file size: 21140359756 bytes Total transferred file size: 21140357580 bytes Literal data: 21063075446 bytes Matched data: 0 bytes File list size: 7819534 File list generation time: 62.368 seconds File list transfer time: 0.000 seconds Total bytes sent: 21077482260 Total bytes received: 1884192 sent 21077482260 bytes received 1884192 bytes 3231792.48 bytes/sec total size is 21140359756 speedup is 1.00 rsync warning: some files vanished before they could be transferred (code 24) at main.c(977) [sender=2.6.9] ----------------------------------------------------------------------------- And here is the log; ----------------------------------------------------------------------------- [root@mail email_backup]# tail /var/log/rsyncd.log 2007/07/18 19:03:19 [5675] skipping non-regular file "domains/fleetwoodmetal.com/.qmail-windsorusers-return-default" 2007/07/18 19:17:37 [5675] skipping non-regular file "domains/fleetwoodmetal.com/0/tinag/IMAPdir/INBOX" 2007/07/18 19:27:49 [5675] skipping non-regular file "domains/fleetwoodmetal.com/akhan/IMAPdir/INBOX" 2007/07/18 20:39:18 [5675] skipping non-regular file "domains/fleetwoodmetal.com/lisat/IMAPdir/INBOX" 2007/07/18 20:43:43 [5675] skipping non-regular file "domains/fleetwoodmetal.com/paulc/IMAPdir/INBOX" 2007/07/18 20:46:53 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi" 2007/07/18 20:47:13 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi-default" 2007/07/18 20:47:13 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi-owner" 2007/07/18 20:47:13 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi-return-default" 2007/07/18 20:50:58 [5675] wrote 28 bytes read 21077482315 bytes total size 21140357580 ------------------------------------------------------------------------------ Permission, ownership and group on the source box (before rsync) drwx------ 38 vpopmail vchkpw 4096 Jul 9 13:58 0 drwx------ 6 vpopmail vchkpw 4096 Sep 18 2006 akhan drwx------ 3 vpopmail vchkpw 4096 Feb 23 11:09 alb drwx------ 8 vpopmail vchkpw 4096 Jun 27 13:37 allusers drwx------ 3 vpopmail vchkpw 4096 Sep 18 2006 alm drwx------ 3 vpopmail vchkpw 4096 Feb 22 08:17 angilad drwx------ 3 vpopmail vchkpw 4096 Jul 4 11:48 barbm drwx------ 3 vpopmail vchkpw 4096 Jul 16 08:00 billj drwx------ 3 vpopmail vchkpw 4096 Apr 10 11:04 bills drwx------ 3 vpopmail vchkpw 4096 Sep 18 2006 bonitap drwx------ 3 vpopmail vchkpw 4096 Feb 23 11:07 brianm --------------------------------------------------------------------------------- Permission, ownership and group after rsync --------------------------------------------------------------------------------- drwx------ 38 nobody nobody 4096 Jul 18 15:27 0 drwx------ 6 nobody nobody 4096 Jul 18 15:34 akhan drwx------ 3 nobody nobody 4096 Jul 18 15:35 alb drwx------ 8 nobody nobody 4096 Jul 18 15:35 allusers drwx------ 3 nobody nobody 4096 Jul 18 15:35 alm drwx------ 3 nobody nobody 4096 Jul 18 15:35 angilad drwx------ 3 nobody nobody 4096 Jul 18 15:36 barbm drwx------ 3 nobody nobody 4096 Jul 18 15:37 billj drwx------ 3 nobody nobody 4096 Jul 18 15:37 bills drwx------ 3 nobody nobody 4096 Jul 18 15:38 bonitap drwx------ 3 nobody nobody 4096 Jul 18 15:38 brianm
Matt McCutchen
2007-Jul-19 13:38 UTC
files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan <akhan@systems.fleetwoodmetal.com> wrote:> I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my > first backup yesterday.In the future and if you need more help on this issue than the below, please send the exact rsync command you used!> But I notice that some hidden (or dot files) are not copied to the other > host on the network.Rsync said "skipping non-regular file". That means the files aren't regular and you haven't passed the option to tell rsync to copy files of that type. What are those files? Symlinks? If so, then pass -l.> And group and ownership of the files are changed to > "nobody" while files and folders permission are copied properly. So what > am mission here can anybody shed light please?By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a "vpopmail" user and a "vchkpw" group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent "nobody" and "nobody" on the destination, the files get that ownership. Creating "vpopmail" and "vchkpw" on the destination machine should fix the problem. Matt
On 7/19/07, Abdul Khan <akhan@systems.fleetwoodmetal.com> wrote:> >> I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my > >> first backup yesterday. > > > In the future and if you need more help on this issue than the below, > please send the exact rsync command you used! >> >> But I notice that some hidden (or dot files) are not copied to the other > >> host on the network. > > > Rsync said "skipping non-regular file". That means the files aren't > regular and you haven't passed the option to tell rsync to copy files > of that type. What are those files? Symlinks? If so, then pass -l. >> >> And group and ownership of the files are changed to > >> "nobody" while files and folders permission are copied properly. So what > >> am mission here can anybody shed light please? > > > By default, rsync preserves owners and groups by name if possible or > otherwise by number. That means if the destination machine lacks a > "vpopmail" user and a "vchkpw" group, rsync sets the same numerical > IDs on the destination as on the source. If those IDs represent > "nobody" and "nobody" on the destination, the files get that > ownership. Creating "vpopmail" and "vchkpw" on the destination > machine should fix the problem. > > Matt >Hi Matt, Thank you for response.> > In the future and if you need more help on this issue than the below, > > please send the exact rsync command you used! >Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [root@mail admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), so is there any thing specific that I have to add in the command to copy those as well?> > By default, rsync preserves owners and groups by name if possible or > > otherwise by number. That means if the destination machine lacks a > > "vpopmail" user and a "vchkpw" group, rsync sets the same numerical > > IDs on the destination as on the source. If those IDs represent > > "nobody" and "nobody" on the destination, the files get that > > ownership. Creating "vpopmail" and "vchkpw" on the destination > > machine should fix the problem. >The destination machine is actually the mirror of the source so both mochines are identical and both "vpopmail" user and "vchkpw" group do exist on the destinition. here is the evidence of "vpopmail" and "vchpkw" on the destinition; [root@mail admin]# cat /etc/group | grep vchk* vchkpw:x:89: [root@mail admin]# cat /etc/passwd | grep vpopmail vpopmail:x:89:89::/home/vpopmail:/bin/bash Thanks in advance. ak -------------- next part -------------- HTML attachment scrubbed and removed
Matt McCutchen
2007-Jul-19 14:58 UTC
files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan <akhan@systems.fleetwoodmetal.com> wrote:> > In the future and if you need more help on this issue than the below, > > please send the exact rsync command you used! > > Here is the command that I ran from the destinition machine which is also > running the rsync daemon or the rsync server; > > [root@mail admin]# rsync --verbose --progress --stats --recursive > /home/vpopmail/ 192.167.1.61::email_backup/ > > they are not symlinks as log mentioned they are hidden files (or dot > files),"Hidden file" is not a file type. On Unix-like systems, the concept of hidden-ness does not exist at the filesystem level or in most filesystem tools (e.g., rsync). That a file whose name begins with a dot is considered hidden is merely a convention among the user interfaces of most file listers/browsers and a few other tools.> so is there any thing specific that I have to add in the command > to copy those as well?As per the above, dot files are nothing out of the ordinary to rsync and no special option is needed to make rsync copy them. Please send the "ls -la" output showing the dot files so I can see what their file type is (as indicated by the first character of each line of the listing).> > By default, rsync preserves owners and groups by name if possible or > > otherwise by number. That means if the destination machine lacks a > > "vpopmail" user and a "vchkpw" group, rsync sets the same numerical > > IDs on the destination as on the source. If those IDs represent > > "nobody" and "nobody" on the destination, the files get that > > ownership. Creating "vpopmail" and "vchkpw" on the destination > > machine should fix the problem. > > The destination machine is actually the mirror of the source so both > mochines are identical and both "vpopmail" user and "vchkpw" group do > exist on the destinition. > > here is the evidence of "vpopmail" and "vchpkw" on the destinition; > > [root@mail admin]# cat /etc/group | grep vchk* > vchkpw:x:89: > [root@mail admin]# cat /etc/passwd | grep vpopmail > vpopmail:x:89:89::/home/vpopmail:/bin/bashI just realized a much more likely cause of the problem. If the rsync daemon on the destination is started as "root", by default it changes IDs to "nobody:nobody" before accessing the module and thus loses the power to set the owners of received files arbitrarily (in this case, to "vpopmail:vchkpw"). To make the daemon stay as root so it can chown the files, explicitly say "uid = root" and "gid = root" in the rsyncd.conf . Matt
> > In the future and if you need more help on this issue than the below, > > please send the exact rsync command you used! > > Here is the command that I ran from the destinition machine which is > also > running the rsync daemon or the rsync server; > > [root@mail admin]# rsync --verbose --progress --stats --recursive > /home/vpopmail/ 192.167.1.61::email_backup/ > > they are not symlinks as log mentioned they are hidden files (or dot > files),"Hidden file" is not a file type. On Unix-like systems, the concept of hidden-ness does not exist at the filesystem level or in most filesystem tools (e.g., rsync). That a file whose name begins with a dot is considered hidden is merely a convention among the user interfaces of most file listers/browsers and a few other tools.> so is there any thing specific that I have to add in the command > to copy those as well?As per the above, dot files are nothing out of the ordinary to rsync and no special option is needed to make rsync copy them. >Please send the "ls -la" output showing the dot files so I can see >what their file type is (as indicated by the first character of each >line of the listing). Matt you are right that the hidden files are not file types. I certainly overlooked at them and they are basically Symlinks. (sorry for my previous claim) I was basically mixing with one of our other box that I had to create some hidden files (dot files) manually. Applology. Since that is out of the way I can pass --links with rsync command to rsync the symlinks. Here is the ls -la output of few of the files that you asked for; lrwxrwxrwx 1 vpopmail vchkpw 57 Oct 23 2006 .qmail-allusers -> /home/vpopmail/domains/f leetwoodmetal.com/allusers/editor lrwxrwxrwx 1 vpopmail vchkpw 60 Oct 23 2006 .qmail-allusers-accept-default -> /home/vpo pmail/domains/fleetwoodmetal.com/allusers/moderator lrwxrwxrwx 1 vpopmail vchkpw 58 Oct 23 2006 .qmail-allusers-default -> /home/vpopmail/d omains/fleetwoodmetal.com/allusers/manager >I just realized a much more likely cause of the problem. If the rsync >daemon on the destination is started as "root", by default it changes >IDs to "nobody:nobody" before accessing the module and thus loses the >power to set the owners of received files arbitrarily (in this case, >to "vpopmail:vchkpw"). To make the daemon stay as root so it can >chown the files, explicitly say "uid = root" and "gid = root" in the >rsyncd.conf . Here is the existing /etc/rsyncd.conf file; [root@mail fleetwoodmetal.com]# cat /etc/rsyncd.conf log file = /var/log/rsyncd.log log format = %h %o %f %l %b [email_backup] path = /email_backup comment = email backup list = yes read only = no Which I am going to change it to ; log file = /var/log/rsyncd.log log format = %h %o %f %l %b uid = root gid = root [email_backup] path = /email_backup comment = email backup list = yes read only = no Thank you for your help again. -------------- next part -------------- HTML attachment scrubbed and removed
Matt McCutchen
2007-Jul-20 14:05 UTC
files permission and hidden files are not rsync properly
On 7/20/07, Abdul Khan <akhan@systems.fleetwoodmetal.com> wrote:> Thanks for the reply, the permission issue is resolved after I > reconfigured and reinstalled rsync. The only issue I have now is user > "vpopmail" and group "vchkpw" is not retaining. Its just not retaining > the user and the group of files. > I even tried to explicitly put "uid=root" and "gid=root" in the > /etc/rsync.conf file but it does not work.Did you add something to the client's command line to tell it to preserve users and groups (either -og, or -a to preserve everything as Wayne suggested)? If so, what users and groups are being set on the destination? Still nobody:nobody?> Another question: i have to manually put list's email address > "rsync@lists.samba.org" to send mail to the list instead of just hit > the reply. I mean it should go to the list instead of any member. Can > anybody just guide me?If your email client has a "Reply to All", use that instead of "Reply". "Reply" replies only to the From address of the original message, while "Reply to All" also sends a copy to everyone else who was listed in the To or Cc of the original message. The rsync@lists.samba.org address, since it never sends messages itself, would be among these "everyone else". If there's no "Reply to All", get a better email client or create a separate webmail account (e.g., Gmail ;) ) for communications with the rsync list. Matt