Kevin Korb
2021-Nov-28 21:59 UTC
Copying a large file from an USB 3 disk (NTFS or EXT4) to another one (UFS/FreeBSD) is a very slow process....
Rsync is designed to reduce the amount of data transmitted over the network. If rsync isn't networking it can't do that. However, it still uses the same code so it is still using a sender and a receiver rather than simply reading and writing as cp does. Also, rsync forces --whole-file because using rsync's algorithm to delta-copy is slower than just re-copying a file (especially if one of the local paths is really a network mount and double especially if that is the writing end). On 11/28/21 16:44, Harry Mangalam wrote:> Can you elaborate on why this is? > > I wrote a parallel rsync wrapper that works very?well over networks but > is similarly very slow over local disks. I thought it was a bug in my > code but didn't get around to tracking it down since my use cases were > all network/ parallel file systems. > Harry > > On Sun, Nov 28, 2021, 11:43 AM Kevin Korb via rsync > <rsync at lists.samba.org <mailto:rsync at lists.samba.org>> wrote: > > rsync is terribly slow at local copies.? Also, it doesn't do its normal > optimizing (see --whole-file).? Just use cp. > > On 11/28/21 13:38, Mario Marietto via rsync wrote: > > Hello to everyone. > > > > I'm copying a large file from a NTFS formatted disk to another > > one,UFS/FreeBSD disk,both are removable disks attached to the USB 3 > > port. The file is 200 GB large and it is copied very slowly. Why > it is > > so slow ? I don't know where to store my virtual machines. I > tried to > > save them on the ext4 disk because I wanted to share them easily > between > > Linux and FreeBSD but I've realized that when I mount the disk in > > FreeBSD after some time it corrupts. I tried to store it on the NTFS > > disk but it happens the same. So,now I'm on FreeBSD and I'm > copying them > > to a dedicated UFS/FreeBSD style disk,but as I said,the speed is > very > > slow. How can I increase the speed ?. Actually I'm using this > > command,because I want to resume the uploading if it breaks at > some point : > > > > > > root at marietto:/mnt/da3p2/bhyve/Ubuntu # rsync -avAXEWSlHh > > /mnt/da0p1/Backups/OS/bhyve/Ubuntu/im* . --no-compress > --info=progress2 > > > > sending incremental file list > > impish-cuda-11-4-nvidia-470.img > > > > 2.13M 0% 9.49kB/s 6284:55:38 > > > > > > and : where do you save large files ? what's the procedure that > you use > > to copy large files with a decent speed ? Unfortunately under > Linux is > > not safe to use a RW ufs disk access. So,I'm out of solutions. > > > > - > > Mario. > > > > -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > ? ? ? ? Kevin Korb? ? ? ? ? ? ? ? ? ? ? Phone:? ? (407) 252-6853 > ? ? ? ? Systems Administrator? ? ? ? ? ?Internet: > ? ? ? ? FutureQuest, Inc.? ? ? ? ? ? ? ?Kevin at FutureQuest.net? (work) > ? ? ? ? Orlando, Florida kmk at sanitarium.net > <mailto:kmk at sanitarium.net> (personal) > ? ? ? ? Web page: https://sanitarium.net/ <https://sanitarium.net/> > ? ? ? ? PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > <https://lists.samba.org/mailman/listinfo/rsync> > Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > <http://www.catb.org/~esr/faqs/smart-questions.html> >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Harry Mangalam
2021-Nov-28 22:09 UTC
Copying a large file from an USB 3 disk (NTFS or EXT4) to another one (UFS/FreeBSD) is a very slow process....
Ok, thanks for that explanation. H On Sun, Nov 28, 2021, 1:59 PM Kevin Korb <kmk at sanitarium.net> wrote:> Rsync is designed to reduce the amount of data transmitted over the > network. If rsync isn't networking it can't do that. However, it still > uses the same code so it is still using a sender and a receiver rather > than simply reading and writing as cp does. Also, rsync forces > --whole-file because using rsync's algorithm to delta-copy is slower > than just re-copying a file (especially if one of the local paths is > really a network mount and double especially if that is the writing end). > > On 11/28/21 16:44, Harry Mangalam wrote: > > Can you elaborate on why this is? > > > > I wrote a parallel rsync wrapper that works very well over networks but > > is similarly very slow over local disks. I thought it was a bug in my > > code but didn't get around to tracking it down since my use cases were > > all network/ parallel file systems. > > Harry > > > > On Sun, Nov 28, 2021, 11:43 AM Kevin Korb via rsync > > <rsync at lists.samba.org <mailto:rsync at lists.samba.org>> wrote: > > > > rsync is terribly slow at local copies. Also, it doesn't do its > normal > > optimizing (see --whole-file). Just use cp. > > > > On 11/28/21 13:38, Mario Marietto via rsync wrote: > > > Hello to everyone. > > > > > > I'm copying a large file from a NTFS formatted disk to another > > > one,UFS/FreeBSD disk,both are removable disks attached to the USB > 3 > > > port. The file is 200 GB large and it is copied very slowly. Why > > it is > > > so slow ? I don't know where to store my virtual machines. I > > tried to > > > save them on the ext4 disk because I wanted to share them easily > > between > > > Linux and FreeBSD but I've realized that when I mount the disk in > > > FreeBSD after some time it corrupts. I tried to store it on the > NTFS > > > disk but it happens the same. So,now I'm on FreeBSD and I'm > > copying them > > > to a dedicated UFS/FreeBSD style disk,but as I said,the speed is > > very > > > slow. How can I increase the speed ?. Actually I'm using this > > > command,because I want to resume the uploading if it breaks at > > some point : > > > > > > > > > root at marietto:/mnt/da3p2/bhyve/Ubuntu # rsync -avAXEWSlHh > > > /mnt/da0p1/Backups/OS/bhyve/Ubuntu/im* . --no-compress > > --info=progress2 > > > > > > sending incremental file list > > > impish-cuda-11-4-nvidia-470.img > > > > > > 2.13M 0% 9.49kB/s 6284:55:38 > > > > > > > > > and : where do you save large files ? what's the procedure that > > you use > > > to copy large files with a decent speed ? Unfortunately under > > Linux is > > > not safe to use a RW ufs disk access. So,I'm out of solutions. > > > > > > - > > > Mario. > > > > > > > -- > > > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > > Kevin Korb Phone: (407) 252-6853 > > Systems Administrator Internet: > > FutureQuest, Inc. Kevin at FutureQuest.net > (work) > > Orlando, Florida kmk at sanitarium.net > > <mailto:kmk at sanitarium.net> (personal) > > Web page: https://sanitarium.net/ <https://sanitarium.net/> > > PGP public key available on web site. > > > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > > > > -- > > Please use reply-all for most replies to avoid omitting the mailing > > list. > > To unsubscribe or change options: > > https://lists.samba.org/mailman/listinfo/rsync > > <https://lists.samba.org/mailman/listinfo/rsync> > > Before posting, read: > > http://www.catb.org/~esr/faqs/smart-questions.html > > <http://www.catb.org/~esr/faqs/smart-questions.html> > > > > -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. Kevin at FutureQuest.net (work) > Orlando, Florida kmk at sanitarium.net (personal) > Web page: https://sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20211128/0807702c/attachment.htm>