Hi, I've discovered a bug in rsync which leads to increased CPU usage and slower transfers in many situations. When syncing with compression (-z), certain file types should not be compressed during the transfer because they are already compressed. The file types which are not to be compressed can be seen in the man page section --skip-compress. Unfortunately skipping the default file types doesn't work and all transferred data is being compressed during the transfer. This is true for all versions since 3.1.0. Steps to Reproduce: 1. run 'rm -f Z ; rsync -azv alpha:z.gz Z' Actual results, transferred data was compressed during the transfer: sent 43 bytes received 63,873 bytes 25,566.40 bytes/sec total size is 628,952 speedup is 9.84 Expected results: No compression should happen for .gz file. Additional info: Note that the source file 'z.gz' was an ascii text file to show clearly that compression took place. Please the following BZ for more info: https://bugzilla.samba.org/show_bug.cgi?id=14323 https://bugzilla.redhat.com/show_bug.cgi?id=1816528 I tried to locate the bug in the code but failed. That said I'm not a C developer and therefore need some help from someone who understands C better than me. Any help is much appreciated! Thanks and kind regards, Simon
On Wed, 25 Mar 2020 14:49:24 +0100 Simon Matter via CentOS <centos at centos.org> wrote:> Hi, > > I've discovered a bug in rsync which leads to increased CPU usage and > slower transfers in many situations. > > When syncing with compression (-z)Tbh, using -z with rsync is almost always a bad idea (unless you're on some pre-historic type of network link..). /Peter
Since you state that using -z is almost always a bad idea, could you provide the rationale for that? I must be missing something. ________________________________ From: CentOS <centos-bounces at centos.org> on behalf of Peter Kjellstr?m <cap at nsc.liu.se> Sent: Wednesday, March 25, 2020 9:34 AM To: Simon Matter <simon.matter at invoca.ch> Cc: centos at centos.org <centos at centos.org> Subject: [EXTERNAL] Re: [CentOS] Need help to fix bug in rsync On Wed, 25 Mar 2020 14:49:24 +0100 Simon Matter via CentOS <centos at centos.org> wrote:> Hi, > > I've discovered a bug in rsync which leads to increased CPU usage and > slower transfers in many situations. > > When syncing with compression (-z)Tbh, using -z with rsync is almost always a bad idea (unless you're on some pre-historic type of network link..). /Peter _______________________________________________ CentOS mailing list CentOS at centos.org https://lists.centos.org/mailman/listinfo/centos Harriscomputer Leroy Tennison Network Information/Cyber Security Specialist E: leroy at datavoiceint.com [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] 2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.com<http://www..com> This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. If you prefer not to be contacted by Harris Operating Group please notify us<http://subscribe.harriscomputer.com/>. This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
On Wed, Mar 25, 2020 at 02:49:24PM +0100, Simon Matter via CentOS wrote:> Hi, > > I've discovered a bug in rsync which leads to increased CPU usage and > slower transfers in many situations. > > When syncing with compression (-z), certain file types should not be > compressed during the transfer because they are already compressed. The > file types which are not to be compressed can be seen in the man page > section --skip-compress. > > Unfortunately skipping the default file types doesn't work and all > transferred data is being compressed during the transfer. This is true for > all versions since 3.1.0. > > Steps to Reproduce: > 1. run 'rm -f Z ; rsync -azv alpha:z.gz Z' > > Actual results, transferred data was compressed during the transfer: > sent 43 bytes received 63,873 bytes 25,566.40 bytes/sec > total size is 628,952 speedup is 9.84 > > Expected results: > No compression should happen for .gz file. > > Additional info: > Note that the source file 'z.gz' was an ascii text file to show clearly > that compression took place.That may invalidate your testing. rsync may not depend upon the filename extension but instead check the magic numbers within the file to determine whether to compress or not. Jon -- Jon H. LaBadie jon at jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
> On Wed, Mar 25, 2020 at 02:49:24PM +0100, Simon Matter via CentOS wrote: >> Hi, >> >> I've discovered a bug in rsync which leads to increased CPU usage and >> slower transfers in many situations. >> >> When syncing with compression (-z), certain file types should not be >> compressed during the transfer because they are already compressed. The >> file types which are not to be compressed can be seen in the man page >> section --skip-compress. >> >> Unfortunately skipping the default file types doesn't work and all >> transferred data is being compressed during the transfer. This is true >> for >> all versions since 3.1.0. >> >> Steps to Reproduce: >> 1. run 'rm -f Z ; rsync -azv alpha:z.gz Z' >> >> Actual results, transferred data was compressed during the transfer: >> sent 43 bytes received 63,873 bytes 25,566.40 bytes/sec >> total size is 628,952 speedup is 9.84 >> >> Expected results: >> No compression should happen for .gz file. >> >> Additional info: >> Note that the source file 'z.gz' was an ascii text file to show clearly >> that compression took place. > > That may invalidate your testing. > > rsync may not depend upon the filename extension > but instead check the magic numbers within the file > to determine whether to compress or not.One could think so but it's not the case, rsync just uses the file suffix to decide which files to compress. It even lower cases them, leaving good old UNIX .Z file behind... Also a bug IMHO but not so important. Simon