rsync - Jul 2015 - [Bug 11378] Please add a '--line-buffered' option to rsync to make logging/output more friendly with pipes/syslog/CI systems/etc.

https://bugzilla.samba.org/show_bug.cgi?id=11378

            Bug ID: 11378
           Summary: Please add a '--line-buffered' option to rsync to
make
                    logging/output more friendly with pipes/syslog/CI
                    systems/etc.
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: nneul at neulinger.org
        QA Contact: rsync-qa at samba.org

Created attachment 11225
  --> https://bugzilla.samba.org/attachment.cgi?id=11225&action=edit
patch to implement --line-buffered option

Behavior change with --line-buffered would be primarily to --progress - which
would output a newline after percentage update instead of just a
carriage-return.

During a normal operation with smaller files you'd never notice the
difference,
but with large files (say recurrent sync of ISO images or similar) - you'd
get
a MUCH more usable output trace in the build system and logs instead of it all
being merged onto one "line" of output.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

How about

<@andrewTO> alias unbuf='stdbuf -i0 -o0 -e0'

then unbuf rsync

i have not tested this in any way.

--progress would be some interesting stuff to parse, esp with all the screen
redrawing
of the K/s line as well as background deletes and scans overwriting while
--progress of the previous file occurs. Ever try to parse ANSI screendraw
output?

/kc


On Thu, Jul 02, 2015 at 06:00:19PM +0000, samba-bugs at samba.org said:
  >https://bugzilla.samba.org/show_bug.cgi?id=11378
  >
  >            Bug ID: 11378
  >           Summary: Please add a '--line-buffered' option to rsync
to make
  >                    logging/output more friendly with pipes/syslog/CI
  >                    systems/etc.
  >           Product: rsync
  >           Version: 3.1.1
  >          Hardware: All
  >                OS: All
  >            Status: NEW
  >          Severity: enhancement
  >          Priority: P5
  >         Component: core
  >          Assignee: wayned at samba.org
  >          Reporter: nneul at neulinger.org
  >        QA Contact: rsync-qa at samba.org
  >
  >Created attachment 11225
  >  -->
https://bugzilla.samba.org/attachment.cgi?id=11225&action=edit
  >patch to implement --line-buffered option
  >
  >Behavior change with --line-buffered would be primarily to --progress -
which
  >would output a newline after percentage update instead of just a
  >carriage-return.
  >
  >During a normal operation with smaller files you'd never notice the
difference,
  >but with large files (say recurrent sync of ISO images or similar) -
you'd get
  >a MUCH more usable output trace in the build system and logs instead of it
all
  >being merged onto one "line" of output.
  >
  >-- 
  >You are receiving this mail because:
  >You are the QA Contact for the bug.
  >-- 
  >Please use reply-all for most replies to avoid omitting the mailing list.
  >To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
  >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Ken Chase - ken at heavycomputing.ca Toronto Canada
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front
St. W.

https://bugzilla.samba.org/show_bug.cgi?id=11378

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #1 from Wayne Davison <wayned at samba.org> ---
            --outbuf=N|L|B          set out buffering to None, Line, or Block

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11378

Nathan Neulinger <nneul at neulinger.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |---

--- Comment #2 from Nathan Neulinger <nneul at neulinger.org> ---
Perhaps the naming is not correct on my suggested option (and I'll admit, I
completely missed the outbuf option) - unfortunately, outbuf doesn't
actually
solve the problem. 

The goal is to get incremental progress output while running rsync through a
build system or similar.

As near as I can tell, with outbuf, it still writes out a "single
line" into
the output stream, so you get result like this:

custom.iso
^M         32,768   0%    0.00kB/s    0:00:00  ^M    264,765,440  29% 
252.47MB/s    0:00:02  ^M    508,592,128  57%  242.62MB/s    0:00:01  ^M   
778,338,304  87%  247.58MB/s    0:00:00  ^M    890,107,904 100%  250.03MB/s   
0:00:03 (xfr#1, to-chk=0/1)


Which doesn't mesh nicely with build tools, or simple stuff like:

   rsync ..... | logger -t nightly-sync

Which results in a mess like:

...
Jul  4 12:52:59 skyhawk rsync-test: sending incremental file list
Jul  4 12:52:59 skyhawk rsync-test: custom.iso
Jul  4 12:53:01 skyhawk CROND[13401]: (nneul) CMD
(/local/thermo/grab-status.pl
>> /local/thermo/status.log 2>&1)
     17,764,896   1%  597.84kB/s    0:24:20  0%    0.00kB/s    0:00:00  
     33,587,200   3%  641.25kB/s    0:22:16  %  613.17kB/s    0:23:42  
Jul  4 12:54:01 skyhawk CROND[13448]: (nneul) CMD
(/local/thermo/grab-status.pl
>> /local/thermo/status.log 2>&1)
...


Instead of (output with my suggested patch):

Jul  4 12:54:46 skyhawk rsync-test: custom.iso
Jul  4 12:54:46 skyhawk rsync-test: 32,768   0%    0.00kB/s    0:00:00 (xfr#1,
to-chk=0/1)
Jul  4 12:54:47 skyhawk rsync-test: 2,818,048   0%    2.31MB/s    0:00:01
(xfr#1, to-chk=0/1)
Jul  4 12:54:48 skyhawk rsync-test: 3,538,944   0%    1.56MB/s    0:00:02
(xfr#1, to-chk=0/1)
Jul  4 12:54:49 skyhawk rsync-test: 4,128,768   0%    1.24MB/s    0:00:03
(xfr#1, to-chk=0/1)
Jul  4 12:54:50 skyhawk rsync-test: 4,784,128   0%    1.06MB/s    0:00:04
(xfr#1, to-chk=0/1)
Jul  4 12:54:51 skyhawk rsync-test: 5,439,488   0% 1003.42kB/s    0:00:05
(xfr#1, to-chk=0/1)
Jul  4 12:54:52 skyhawk rsync-test: 6,094,848   0%  945.38kB/s    0:00:06
(xfr#1, to-chk=0/1)


Perhaps "line-progress" or something?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

On Sat, 04 Jul 2015 17:56:25 +0000
samba-bugs at samba.org wrote:
> --- Comment #2 from Nathan Neulinger <nneul at neulinger.org> ---
> Perhaps the naming is not correct on my suggested option (and I'll
> admit, I completely missed the outbuf option) - unfortunately, outbuf
> doesn't actually solve the problem. 
> 
> The goal is to get incremental progress output while running rsync
> through a build system or similar.
What would happen if you piped the rsync output through
tr and changed \r to \n?



Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

https://bugzilla.samba.org/show_bug.cgi?id=11378

--- Comment #3 from Karl O. Pinc <kop at meme.com> ---
On Sat, 04 Jul 2015 17:56:25 +0000
samba-bugs at samba.org wrote:
> --- Comment #2 from Nathan Neulinger <nneul at neulinger.org> ---
> Perhaps the naming is not correct on my suggested option (and I'll
> admit, I completely missed the outbuf option) - unfortunately, outbuf
> doesn't actually solve the problem. 
> 
> The goal is to get incremental progress output while running rsync
> through a build system or similar.
What would happen if you piped the rsync output through
tr and changed \r to \n?



Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

Imagine it, all those updates to a transfering large files with -P , 100,000
lines
of progress PER file...

/kc


On Sat, Jul 04, 2015 at 06:56:21PM +0000, samba-bugs at samba.org said:
  >https://bugzilla.samba.org/show_bug.cgi?id=11378
  >
  >--- Comment #3 from Karl O. Pinc <kop at meme.com> ---
  >On Sat, 04 Jul 2015 17:56:25 +0000
  >samba-bugs at samba.org wrote:
  >
  >> --- Comment #2 from Nathan Neulinger <nneul at neulinger.org>
---
  >> Perhaps the naming is not correct on my suggested option (and
I'll
  >> admit, I completely missed the outbuf option) - unfortunately, outbuf
  >> doesn't actually solve the problem. 
  >> 
  >> The goal is to get incremental progress output while running rsync
  >> through a build system or similar.
  >
  >What would happen if you piped the rsync output through
  >tr and changed \r to \n?
  >
  >
  >
  >Karl <kop at meme.com>
  >Free Software:  "You don't pay back, you pay forward."
  >                 -- Robert A. Heinlein
  >
  >-- 
  >You are receiving this mail because:
  >You are the QA Contact for the bug.
  >-- 
  >Please use reply-all for most replies to avoid omitting the mailing list.
  >To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
  >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto
Canada
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front
St. W.

https://bugzilla.samba.org/show_bug.cgi?id=11378

--- Comment #4 from Nathan Neulinger <nneul at neulinger.org> ---
Unfortunately, that just moves the buffering issue to tr instead of rsync.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11378

Nathan Neulinger <nneul at neulinger.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |WONTFIX

--- Comment #5 from Nathan Neulinger <nneul at neulinger.org> ---
With some other tweaks to my local scripts I was able to make outbuf=L work.
I'd still like to recommend that there be some way to request that the
--progress output use CR instead of LF for piped output though.

Maybe a special "--outbuf=M" for "machine processed" - which
would both set to
line buffered AND use full CR.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.