Is anyone aware of any freeware program that can speed up copying tons of data (2 TB) from UFS to ZFS on same server? Thanks.
On Fri, Apr 29, 2011 at 10:53 AM, Dan Shelton <dan.shelton at oracle.com> wrote:> Is anyone aware of any freeware program that can speed up copying tons of > data (2 TB) from UFS to ZFS on same server?rsync, with --whole-file --inplace (and other options), works well for the initial copy. rsync, with --no-whole-file --inplace (and other options), works extremely fast for updates. -- Freddie Cash fjwcash at gmail.com
Dan Shelton <dan.shelton at oracle.com> wrote:> Is anyone aware of any freeware program that can speed up copying tons > of data (2 TB) from UFS to ZFS on same server?Try star -copy .... Note that due to the problems on ZFS to deal with stable states, I recommend to use -no-fsync and it may of course help to specify a larger FIFO than the default in case you have plenty of RAM. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
On 04/30/11 06:00 AM, Freddie Cash wrote:> On Fri, Apr 29, 2011 at 10:53 AM, Dan Shelton<dan.shelton at oracle.com> wrote: >> Is anyone aware of any freeware program that can speed up copying tons of >> data (2 TB) from UFS to ZFS on same server? > rsync, with --whole-file --inplace (and other options), works well for > the initial copy. >Is rsync ACL aware yet? I always use find piped to cpio. -- Ian.
On Fri, Apr 29, 2011 at 10:53 AM, Dan Shelton <dan.shelton at oracle.com> wrote:> Is anyone aware of any freeware program that can speed up copying tons of > data (2 TB) from UFS to ZFS on same server?Setting ''sync=disabled'' for the initial copy will help, since it will make all writes asynchronous. You will probably want to set it back to default after you''re done. -B -- Brandon High : bhigh at freaks.com
On Apr 29, 2011, at 1:37 PM, Brandon High wrote:> On Fri, Apr 29, 2011 at 10:53 AM, Dan Shelton <dan.shelton at oracle.com> wrote: >> Is anyone aware of any freeware program that can speed up copying tons of >> data (2 TB) from UFS to ZFS on same server? > > Setting ''sync=disabled'' for the initial copy will help, since it will > make all writes asynchronous.Few local commands are sync. cp, cpio, tar, rsync, etc are all effectively async for local copies. Disabling the ZIL will do nothing to speed these. -- richard
Freddie Cash <fjwcash at gmail.com> wrote:> On Fri, Apr 29, 2011 at 10:53 AM, Dan Shelton <dan.shelton at oracle.com> wrote: > > Is anyone aware of any freeware program that can speed up copying tons of > > data (2 TB) from UFS to ZFS on same server? > > rsync, with --whole-file --inplace (and other options), works well for > the initial copy.But this is most likely slower than star and does rsync support sparse files? J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Dan Shelton wrote:> Is anyone aware of any freeware program that can speed up copying tons > of data (2 TB) from UFS to ZFS on same server?I use ''ufsdump | ufsrestore''*. I would also suggest try setting ''sync=disabled'' during the operation, and reverting it afterwards. Certainly, fastfs (a similar although more dangerous option for ufs) makes ufs to ufs copying significantly faster. *ufsrestore works fine on ZFS filesystems (although I haven''t tried it with any POSIX ACLs on the original ufs filesystem, which would probably simply get lost). -- Andrew Gabriel
Andrew Gabriel <Andrew.Gabriel at oracle.com> wrote:> Dan Shelton wrote: > > Is anyone aware of any freeware program that can speed up copying tons > > of data (2 TB) from UFS to ZFS on same server? > > I use ''ufsdump | ufsrestore''*. I would also suggest try setting > ''sync=disabled'' during the operation, and reverting it afterwards. > Certainly, fastfs (a similar although more dangerous option for ufs) > makes ufs to ufs copying significantly faster. > > *ufsrestore works fine on ZFS filesystems (although I haven''t tried it > with any POSIX ACLs on the original ufs filesystem, which would probably > simply get lost).star -copy -no-fsync .... is typically 30% faster that ufsdump | ufsrestore. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
On Tue, May 3, 2011 at 5:47 AM, Joerg Schilling <Joerg.Schilling at fokus.fraunhofer.de> wrote:> But this is most likely slower than star and does rsync support sparse files?''rsync -ASHXavP'' -A: ACLs -S: Sparse files -H: Hard links -X: Xattrs -a: archive mode; equals -rlptgoD (no -H,-A,-X) You don''t need to specify --whole-file, it''s implied when copying on the same system. --inplace can play badly with hard links and shouldn''t be used. It probably will be slower than other options but it may be more accurate, especially with -H -B -- Brandon High : bhigh at freaks.com
On 5/3/2011 8:55 AM, Brandon High wrote:> On Tue, May 3, 2011 at 5:47 AM, Joerg Schilling > <Joerg.Schilling at fokus.fraunhofer.de> wrote: >> But this is most likely slower than star and does rsync support sparse files? > ''rsync -ASHXavP'' > > -A: ACLs > -S: Sparse files > -H: Hard links > -X: Xattrs > -a: archive mode; equals -rlptgoD (no -H,-A,-X) > > You don''t need to specify --whole-file, it''s implied when copying on > the same system. --inplace can play badly with hard links and > shouldn''t be used. > > It probably will be slower than other options but it may be more > accurate, especially with -H > > -Brsync is indeed slower than star; so far as I can tell, this is due almost exclusively to the fact that rsync needs to build an in-memory table of all work being done *before* it starts to copy. After that, it copies at about the same rate as star (my observations). I''d have to look at the code, but rsync appears to internally buffer a signification amount (due to its expect network use pattern), which helps for ZFS copying. The one thing I''m not sure of is whether rsync uses a socket, pipe, or semaphore method when doing same-host copying. I presume socket (which would slightly slow it down vs star). That said, rsync is really the only solution if you have a partial or interrupted copy. It''s also really the best method to do verification. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
On 05/ 4/11 01:35 AM, Joerg Schilling wrote:> Andrew Gabriel<Andrew.Gabriel at oracle.com> wrote: > >> Dan Shelton wrote: >>> Is anyone aware of any freeware program that can speed up copying tons >>> of data (2 TB) from UFS to ZFS on same server? >> I use ''ufsdump | ufsrestore''*. I would also suggest try setting >> ''sync=disabled'' during the operation, and reverting it afterwards. >> Certainly, fastfs (a similar although more dangerous option for ufs) >> makes ufs to ufs copying significantly faster. >> >> *ufsrestore works fine on ZFS filesystems (although I haven''t tried it >> with any POSIX ACLs on the original ufs filesystem, which would probably >> simply get lost). > star -copy -no-fsync .... is typically 30% faster that ufsdump | ufsrestore. >Does it preserve ACLs? -- Ian.
On Tue, May 3, 2011 at 12:36 PM, Erik Trimble <erik.trimble at oracle.com> wrote:> On 5/3/2011 8:55 AM, Brandon High wrote: >> >> On Tue, May 3, 2011 at 5:47 AM, Joerg Schilling >> <Joerg.Schilling at fokus.fraunhofer.de> ?wrote: >>> >>> But this is most likely slower than star and does rsync support sparse >>> files? >> >> ''rsync -ASHXavP'' >> >> -A: ACLs >> -S: Sparse files >> -H: Hard links >> -X: Xattrs >> -a: archive mode; equals -rlptgoD (no -H,-A,-X) >> >> You don''t need to specify --whole-file, it''s implied when copying on >> the same system. --inplace can play badly with hard links and >> shouldn''t be used. >> >> It probably will be slower than other options but it may be more >> accurate, especially with -H >> >> -B > > rsync is indeed slower than star; so far as I can tell, this is due almost > exclusively to the fact that rsync needs to build an in-memory table of all > work being done *before* it starts to copy.rsync 2.x works that way., building a complete list of files/directories to copy before starting the copy. rsync 3.x doesn''t. 3.x builds an initial file list for the first directory and then starts copying files while continuing to build the list of files, so there''s only a small pause at the beginning. -- Freddie Cash fjwcash at gmail.com
On Tue, May 3, 2011 at 12:36 PM, Erik Trimble <erik.trimble at oracle.com> wrote:> rsync is indeed slower than star; so far as I can tell, this is due almost > exclusively to the fact that rsync needs to build an in-memory table of all > work being done *before* it starts to copy. After that, it copies at aboutrsync 3.0+ will start copying almost immediately, so it''s much better in that respect than previous versions. It continues to scan update the list of files while sending data.> network use pattern), which helps for ZFS copying. ?The one thing I''m not > sure of is whether rsync uses a socket, pipe, or semaphore method when doing > same-host copying. I presume socket (which would slightly slow it down vsIt creates a socketpair() before clone()ing itself and uses the socket for communications.> That said, rsync is really the only solution if you have a partial or > interrupted copy. ?It''s also really the best method to do verification.For verification you should specify -c (checksums), otherwise it will only look at the size, permissions, owner and date and if they all match it will not look at the file contents. It can take as long (or longer) to complete than the original copy, since files on both side need to be read and checksummed. -B -- Brandon High : bhigh at freaks.com
Erik Trimble <erik.trimble at oracle.com> wrote:> rsync is indeed slower than star; so far as I can tell, this is due > almost exclusively to the fact that rsync needs to build an in-memory > table of all work being done *before* it starts to copy. After that, it > copies at about the same rate as star (my observations). I''d have to > look at the code, but rsync appears to internally buffer a signification > amount (due to its expect network use pattern), which helps for ZFS > copying. The one thing I''m not sure of is whether rsync uses a socket, > pipe, or semaphore method when doing same-host copying. I presume socket > (which would slightly slow it down vs star).The reason why star is faster than any other copy method is based on the fact that star is not implemented like historical tar or cpio implementations. Since around 1990, star forks into two processes unless you forbid this by an option. In the normal modes, one of them is the "archive process" that just reads or writes from/to the archive file or tape, the other is the tar process that understands the archive content and deals with the filesystem (the direction of the filesystem operation depends on whether it is in extract or create mode). Between both processes, there is a large FIFO of shared memory that is used to share the data. If the FIFO has much free space, star will read files in one single chunk into the FIFO, this is another reason for it''s speed. Another advantage in star is that it reads every directory in one large chunk and thus allows the OS to do optimization at this place. BTW: An OS that floods (and probably overflows) the stat/vnode cache in such a case may cause an unneeded slow down. In copy mode, star starts two archive processes and a FIFO between them. The create process tries to keep the FIFO as full as possible and as is makes sense to use a FIFO size up to aprox. half of the real system memory, this FIFO may be really huge, so it will even be able to keep modern tapes streaming for at least 30-60 seconds. Ufsdump only allows a small number of 126 kB buffers (I belive is it 6 buffer) and thus ufsdump | ufsrestore is tightly coupled while star allows to freely run both creation and extration of the internal virtual archive nearly independent from each other. This way, star does not need to wait every time extraction slows down but just fills the FIFO instead. Before SEEK_HOLE/SEEK_DATA existed, the only place where ufsdump was faster than star have been sparse files. This is why I talked with Jeff Bonwick in September 2004 to find a useful interface for user space programs (in special star) that do not read the filesystem at block level (like ufsdump) but cleanly in the documented POSIX way. Since SEEK_HOLE/SEEK_DATA have been introduced, there is no single known case, where star is not at least 30% faster than ufsdump. BTW: ufsdump is another implementation that first sits and collects all filenames before it starts to read file content.> That said, rsync is really the only solution if you have a partial or > interrupted copy. It''s also really the best method to do verification.Star offers another method to continue interrupted extracts or copies: Star sets the time stamp of an incomplete file to 0 (1.1.1970 GMT). As star does not overwrite files in case they are not newer in the archive, star can skip the other files in extract mode and continue with the missing files or with the file(s) that have the time stamp 0. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Ian Collins <ian at ianshome.com> wrote:> >> *ufsrestore works fine on ZFS filesystems (although I haven''t tried it > >> with any POSIX ACLs on the original ufs filesystem, which would probably > >> simply get lost). > > star -copy -no-fsync .... is typically 30% faster that ufsdump | ufsrestore. > > > Does it preserve ACLs?Star supports ACLs from the withdrawn POSIX draft. Star could already support ZFS ACLs in case that Sun had offered a correctly working ACL support library when they introdiced ZFS ACLs. Unfortunately it took some time until this lib was fixed and since then, I had other projects that took my time. ZFS ACLs are not fogetten however. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
On 05/ 5/11 10:02 PM, Joerg Schilling wrote:> Ian Collins<ian at ianshome.com> wrote: > >>>> *ufsrestore works fine on ZFS filesystems (although I haven''t tried it >>>> with any POSIX ACLs on the original ufs filesystem, which would probably >>>> simply get lost). >>> star -copy -no-fsync .... is typically 30% faster that ufsdump | ufsrestore. >>> >> Does it preserve ACLs? > Star supports ACLs from the withdrawn POSIX draft. >So star would work moving data from UFS to ZFS, assuming it uses acl_get/set to read and write the ACLs.> Star could already support ZFS ACLs in case that Sun had offered a correctly > working ACL support library when they introdiced ZFS ACLs. Unfortunately it > took some time until this lib was fixed and since then, I had other projects > that took my time. ZFS ACLs are not fogetten however. >Um, I thought the acl_totext and acl_fromtext functions had been around for many years. -- Ian.
Op 03-05-11 17:55, Brandon High schreef:> -H: Hard linksIf you''re going to this for 2 TB of data, remember to expand your swap space first (or have tons of memory). Rsync will need it to store every inode number in the directory. -- No part of this copyright message may be reproduced, read or seen, dead or alive or by any means, including but not limited to telepathy without the benevolence of the author.
On Thu, 19 May 2011 15:39:50 +0200, Frank Van Damme wrote:> Op 03-05-11 17:55, Brandon High schreef: >> -H: Hard links > > If you''re going to this for 2 TB of data, remember to expand your > swap > space first (or have tons of memory). Rsync will need it to store > every > inode number in the directory.No, only for files with a link count greater than 1 - unless you are using a very old version of rsync (<2.6.1). -- Daniel