Hello All, I have been using rsync to backup several filesystems by using Mike Rubel's hard link method (http://www.mikerubel.org/computers/rsync_snapshots/). The problem is, I am backing up a lot of ASCII .log, csv, and .txt files. These files are large and can range anywhere from 1GB to 30GB. I was wondering if on the target side (the backup side), if I can use some sort of compression. I am using ext3 filesystem. Any ideas? TIA
Thanks all. I figured this was the only solution available. Too bad I am using Linux and don't think my RAID controller is supported under Solaris. On Mon, Jan 19, 2009 at 10:41 AM, Kyle Lanclos <lanclos@ucolick.org> wrote:> You wrote: >> The problem is, I am backing up a lot of ASCII .log, csv, and .txt >> files. These files are large and can range anywhere from 1GB to 30GB. >> I was wondering if on the target side (the backup side), if I can use >> some sort of compression. I am using ext3 filesystem. > > One could always switch to the ZFS filesystem; compression is but one > of many good reasons to do so. > > I'm not sure what an equivalent Linux-based solution would be. > > --Kyle >
On Mon, Jan 19, 2009 at 12:33 PM, Ryan Malayter <malayter@gmail.com> wrote:> You can switch to a filesystem that supports transparent encrytpion > (Reiser, ZFS, NTFS, others depending on your OS). Rsync would be > completely unaware of any file-system level compression in that case.Oops. I meant "transparent compression", not "transparent encryption". -- RPM
yep. ZFS on fuse is just too slow. I suppose I will wait for ZFS on Linux (pipe dream) or try to switch to Solaris 10 on x86 On Mon, Jan 19, 2009 at 1:34 PM, Ryan Malayter <malayter@gmail.com> wrote:> On Mon, Jan 19, 2009 at 12:33 PM, Ryan Malayter <malayter@gmail.com> wrote: >> You can switch to a filesystem that supports transparent encrytpion >> (Reiser, ZFS, NTFS, others depending on your OS). Rsync would be >> completely unaware of any file-system level compression in that case. > > Oops. I meant "transparent compression", not "transparent encryption". > -- > RPM > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >
You can switch to a filesystem that supports transparent encrytpion (Reiser, ZFS, NTFS, others depending on your OS). Rsync would be completely unaware of any file-system level compression in that case. Or you can use gzip with the --rsyncable option. Not all distributions of gzip support --rsyncable, as the last officially "stable" release of gzip was way back in 2003. On Mon, Jan 19, 2009 at 9:14 AM, Mag Gam <magawake@gmail.com> wrote:> Hello All, > > I have been using rsync to backup several filesystems by using Mike > Rubel's hard link method > (http://www.mikerubel.org/computers/rsync_snapshots/). > > The problem is, I am backing up a lot of ASCII .log, csv, and .txt > files. These files are large and can range anywhere from 1GB to 30GB. > I was wondering if on the target side (the backup side), if I can use > some sort of compression. I am using ext3 filesystem. > > Any ideas? > > TIA > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >-- RPM
On Mon, Jan 19, 2009 at 2:34 PM, Mag Gam <magawake@gmail.com> wrote:> ZFS on fuse is just too slow. I suppose I will wait for ZFS on Linux > (pipe dream) or try to switch to Solaris 10 on x86 >There will never be ZFS in the Linux kernel because of license incompatibilites. The linux answer to ZFS is btrfs, which is still in development, and not much of an answer in my opinion ;-). Also, there does not appear to be any "stock" linux kernel filesystem that supports transparent compression read/write. SquashFS is read-only. What Linux distribution are you using? It might bundle a patch or other filesystems. I would suggest trying gzip --rsyncable. Compress the files with gzip --rsyncable at the source, and rsync should be able to find significant matches (especially for updates of log files). -- RPM