Nate Case
2003-Jun-17 02:53 UTC
rsync error: error in rsync protocol data stream (Broken pipe)
I'm experiencing some unexpected behavior with rsync v2.5.6. After looking around it seems like several people have experienced similar problems in the past, but those appear to be old bugs that are solved now. Also doesn't appear to be any direct bugfixes made for this issue since 2.5.6. The strange thing is that things seem to work when I use ssh for transport. However now I need to switch to using the rsync daemon directly and this problem has arose. I'm pushing local data to a remote rsync backup server: ---[snip]--- # rsync -a -v --modify-window=2 /backup/Source/Dir/ user@xx.xx.xx.xx::backup/dest Password: building file list ... rsync: writefd_unbuffered failed to write 4092 bytes: phase "send_file_entry": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(515) ---[snip]--- With more -v's, I end up with the same message at the end, but it's preceeded by a bunch of lines like this (filenames altered): make_file(3,file1) make_file(3,file2) make_file(3,file3) make_file(3,file4) It stops on the same file each time. I tried deleting this file to see if it's just choking on a particular file (or type of file), but in that case it just chokes on the next file (different type) in the directory. The files in this directory are no bigger than 2MB. If I remove the subdirectory entirely, it still just dies on the next file it encounters. The directories that I'm syncing do have a large number of files, but as I mentioned earlier, it does seem to work when using "-e ssh". I do not seem to be running out of memory. I also tried the same parameters but breaking it up into smaller subdirectories, and it still failed after X amount of files with the same error (I did not determine whether or not X is always the same with using other directories). Note that this is always occuring in the "building file list" stage, and not the actual file transfer. After looking at io.c, it seems like the problem (rather, the symptom) for this error message is a failed write() call (which explains the Broken Pipe message to stderr). I thought maybe --blocking-io would avoid this issue, but it still happened when using --blocking-io. Does anyone have any ideas about what could be causing this? Thanks, - Nate Case <nd@kracked.com>
jw schultz
2003-Jun-17 09:20 UTC
rsync error: error in rsync protocol data stream (Broken pipe)
On Mon, Jun 16, 2003 at 11:53:04AM -0500, Nate Case wrote:> I'm experiencing some unexpected behavior with rsync v2.5.6. After > looking around it seems like several people have experienced similar > problems in the past, but those appear to be old bugs that are solved now. > Also doesn't appear to be any direct bugfixes made for this issue since > 2.5.6. > > The strange thing is that things seem to work when I use ssh for > transport. However now I need to switch to using the rsync daemon > directly and this problem has arose. > > I'm pushing local data to a remote rsync backup server: > > ---[snip]--- > # rsync -a -v --modify-window=2 /backup/Source/Dir/ > user@xx.xx.xx.xx::backup/dest > Password: > building file list ... rsync: writefd_unbuffered failed to write 4092 > bytes: phase "send_file_entry": Broken pipe > rsync error: error in rsync protocol data stream (code 12) at io.c(515) > ---[snip]--- > > With more -v's, I end up with the same message at the end, but it's > preceeded by a bunch of lines like this (filenames altered): > > make_file(3,file1) > make_file(3,file2) > make_file(3,file3) > make_file(3,file4) > > It stops on the same file each time. I tried deleting this file to see if > it's just choking on a particular file (or type of file), but in that case > it just chokes on the next file (different type) in the directory. The > files in this directory are no bigger than 2MB. If I remove the > subdirectory entirely, it still just dies on the next file it encounters. > > The directories that I'm syncing do have a large number of files, but as I > mentioned earlier, it does seem to work when using "-e ssh". I do not > seem to be running out of memory. I also tried the same parameters but > breaking it up into smaller subdirectories, and it still failed after X > amount of files with the same error (I did not determine whether or not X > is always the same with using other directories). Note that this is > always occuring in the "building file list" stage, and not the actual file > transfer. > > After looking at io.c, it seems like the problem (rather, the symptom) for > this error message is a failed write() call (which explains the Broken > Pipe message to stderr). I thought maybe --blocking-io would avoid this > issue, but it still happened when using --blocking-io.It would seem that the server process (forked off from the daemon) is dying. Look in the rsyncd log file (may be syslog) for clues. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
Nathaniel Case
2003-Jun-18 00:10 UTC
rsync error: error in rsync protocol data stream (Broken pipe)
On Mon, 16 Jun 2003 at 16:24:25 -0700, jw schultz wrote:> > > After looking at io.c, it seems like the problem (rather, the symptom) > > for > > this error message is a failed write() call (which explains the Broken > > Pipe message to stderr). I thought maybe --blocking-io would avoid this > > issue, but it still happened when using --blocking-io. > > It would seem that the server process (forked off from the > daemon) is dying. Look in the rsyncd log file (may be > syslog) for clues.Yes, this does appear to be what is happening. I actually had tried looking at the daemon logs, but there didn't appear to be anything remotely interesting in it. This is the only line I see in the log: 2003/06/16 16:23:48 [17924] rsync to blah/blah/Blah/ from backup@host (10.x.x.x) Is there any way to increase the verbosity for this? Adding -v's to the daemon process didn't appear to affect it. Oh well, I guess it's time to start debugging. - Nate Case <nd@kracked.com> (not subscribed)
Maybe Matching Threads
- 'error in rsync protocol data stream' - Tim Conway
- rsync out of memory problem
- [Bug 1959] New: writefd_unbuffered failed to write 4092 bytes phase send_file_entry broken pipe
- DO NOT REPLY [Bug 3979] New: writefd_unbuffered failed to write 4092 bytes phase send_file_entry broken pipe
- RSYNC Hang on client match_sums