Hourly I have an rsync job backup /home to /home/backup. I have 24
directories (one for each hour):
home.0
...
home.23
Here is the script I am running via cron:
#! /usr/local/bin/bash
dest=`date +%k | sed 's/ //g'`
linkdir=`date -v-1H +%k | sed 's/ //g'`
chflags -R noschg /home/backup
rm -rf /home/backup/home.$dest
rsync -ahHP --numeric-ids --delete --stats --link-dest=../
home."$linkdir"\
--exclude=/backup/* /home/ /home/backup/home."$dest"/ >
/var/
rsync.log
sleep 2
chflags -R schg /home/backup
Per reading this list and other google finds, it was stated that for
link-dest to work, the destination must be empty, which I do.
However, let's say a 2PM backup runs. The 2PM backup has 2.7GiB (the
current size of /home). The 1PM backup directory is also 2.7GiB.
[root at arthur /home/backup/home.14]# du -hcd0
2.7G .
2.7G total
[root at arthur /home/backup/home.14]#
[root at arthur /home/backup/home.13]# !du
du -hcd0
2.7G .
2.7G total
[root at arthur /home/backup/home.13]#
For a specific file:
[root at arthur /home/backup/home.14/archon/public_html]# ls -li d2x.jpg
22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg
[root at arthur /home/backup/home.14/archon/public_html]#
[root at arthur /home/backup/home.13/archon/public_html]# !ls
ls -li d2x.jpg
22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg
[root at arthur /home/backup/home.13/archon/public_html]#
Both files share the same inode number, so I'm assuming it's a
hardlink, but why is the home.14 directory still saying 2.7GiB used?
Shouldn't it only be using whatever the delta is between home.13 and
home.14 (in this case, rsync reported copying over about 40 MiB of
data).
df -h shows that /home is 8.6GiB when only ~3 GiB is actually being
used, so it's still creating fresh copies.
I'm guessing I'm misunderstanding what --link-dest does, as I
understood it to create a hardlink to the file in the link-dest
directory if the file does not exist in the destination directory.
Here's a log from the 2PM script run:
[root at arthur /var]# cat rsync.log
sending incremental file list
created directory /home/backup/home.14
archon/IMAPdir/Apple Mail To Do/
archon/IMAPdir/Deleted Messages/
archon/IMAPdir/Deleted Messages/bincimap-cache
2.22K 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1943/2336)
archon/IMAPdir/Deleted Messages/bincimap-uidvalidity
75 100% 0.60kB/s 0:00:00 (xfer#2, to-check=1942/2336)
archon/IMAPdir/Deleted Messages/cur/
archon/IMAPdir/Deleted Messages/cur/
1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S
24.44K 100% 194.07kB/s 0:00:00 (xfer#3, to-check=1921/2336)
archon/IMAPdir/Deleted Messages/new/
archon/IMAPdir/Deleted Messages/tmp/
archon/IMAPdir/Freecycle/
archon/IMAPdir/Freecycle/bincimap-cache
876 100% 6.96kB/s 0:00:00 (xfer#4, to-check=1890/2336)
archon/IMAPdir/Freecycle/bincimap-uidvalidity
74 100% 0.59kB/s 0:00:00 (xfer#5, to-check=1889/2336)
archon/IMAPdir/Freecycle/cur/
archon/IMAPdir/Freecycle/cur/
1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2,
5.20K 100% 40.95kB/s 0:00:00 (xfer#6, to-check=1879/2336)
archon/IMAPdir/Freecycle/new/
archon/IMAPdir/Freecycle/tmp/
archon/Maildir/
archon/Maildir/bincimap-cache
1.13K 100% 2.14kB/s 0:00:00 (xfer#7, to-check=1041/3870)
archon/Maildir/bincimap-uidvalidity
75 100% 0.14kB/s 0:00:00 (xfer#8, to-check=1040/3870)
archon/Maildir/cur/
archon/Maildir/cur/1250021559.7658.arthur.silvertree.org:2,S
7.60K 100% 13.45kB/s 0:00:00 (xfer#9, to-check=1026/3870)
archon/Maildir/cur/1250022468.8904.arthur.silvertree.org:2,S
7.76K 100% 13.71kB/s 0:00:00 (xfer#10, to-check=1025/3870)
archon/Maildir/cur/1250022542.9042.arthur.silvertree.org:2,ST =>
archon/IMAPdir/Deleted Messages/cur/
1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S
archon/Maildir/cur/1250023069.9684.arthur.silvertree.org:2,ST =>
archon/IMAPdir/Freecycle/cur/
1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2,
archon/Maildir/cur/1250023397.10077.arthur.silvertree.org:2,S
4.68K 100% 8.26kB/s 0:00:00 (xfer#11, to-check=1022/3870)
archon/Maildir/cur/1250023861.10654.arthur.silvertree.org:2,S
6.45K 100% 11.38kB/s 0:00:00 (xfer#12, to-check=1021/3870)
archon/Maildir/new/
archon/Maildir/tmp/
backup/
pinnacle/IMAPdir/Apple Mail To Do/
romany/Maildir/new/
romany/Maildir/new/1250023382.10044.arthur.silvertree.org
2.49K 100% 0.00kB/s 0:00:00 (xfer#13, to-check=1001/22172)
romany/Maildir/tmp/
scott.schappell/IMAPdir/Apple Mail To Do/
scott.schappell/Maildir/
scott.schappell/Maildir/bincimap-cache
46.29K 100% 279.03kB/s 0:00:00 (xfer#14, to-check=1004/24511)
scott.schappell/Maildir/bincimap-uidvalidity
75 100% 0.30kB/s 0:00:00 (xfer#15, to-check=1003/24511)
scott.schappell/Maildir/cur/
scott.schappell/Maildir/cur/1250023996.10809.arthur.silvertree.org:2,
20.12K 100% 66.38kB/s 0:00:00 (xfer#16, to-check=1024/24943)
scott.schappell/Maildir/new/
scott.schappell/Maildir/tmp/
soucon/3.1/game/data/sc.gdbm
34.97M 100% 16.10MB/s 0:00:02 (xfer#17, to-check=1014/28108)
Number of files: 32052
Number of files transferred: 17
Total file size: 2.79G bytes
Total transferred file size: 35.10M bytes
Literal data: 35.10M bytes
Matched data: 0 bytes
File list size: 991.74K
File list generation time: 0.009 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 36.10M
Total bytes received: 2.71K
sent 36.10M bytes received 2.71K bytes 4.81M bytes/sec
total size is 2.79G speedup is 77.36
rsync version:
[root at arthur /var]# rsync --version
rsync version 3.0.6 protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, no iconv, symtimes
Installed from FreeBSD 7.2 ports collection.
Thanks for any help in enabling me to understand how this works.
Scott
And now df is reporting proper usage of 5.4 GiB (which is what I expected). Maybe I just wasn't being patient enough and there's some weird df lag or something. Anyway, seems like it's working OK, but if anyone has any pointers on doing this more efficiently, I'd be more than happy to hear it. I'm still puzzed as to why du shows 2.7 GiB in both directories, though. Scott On Aug 11, 2009, at 14:25:46, I wrote:> Hourly I have an rsync job backup /home to /home/backup. I have 24 > directories (one for each hour): > > home.0 > ... > home.23 > > Here is the script I am running via cron: > > #! /usr/local/bin/bash > dest=`date +%k | sed 's/ //g'` > linkdir=`date -v-1H +%k | sed 's/ //g'` > chflags -R noschg /home/backup > rm -rf /home/backup/home.$dest > rsync -ahHP --numeric-ids --delete --stats --link-dest=../ > home."$linkdir"\ > --exclude=/backup/* /home/ /home/backup/home."$dest"/ > /var/ > rsync.log > sleep 2 > chflags -R schg /home/backup > > Per reading this list and other google finds, it was stated that for > link-dest to work, the destination must be empty, which I do. > However, let's say a 2PM backup runs. The 2PM backup has 2.7GiB > (the current size of /home). The 1PM backup directory is also 2.7GiB. > > [root at arthur /home/backup/home.14]# du -hcd0 > 2.7G . > 2.7G total > [root at arthur /home/backup/home.14]# > > [root at arthur /home/backup/home.13]# !du > du -hcd0 > 2.7G . > 2.7G total > [root at arthur /home/backup/home.13]# > > For a specific file: > > [root at arthur /home/backup/home.14/archon/public_html]# ls -li d2x.jpg > 22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg > [root at arthur /home/backup/home.14/archon/public_html]# > > [root at arthur /home/backup/home.13/archon/public_html]# !ls > ls -li d2x.jpg > 22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg > [root at arthur /home/backup/home.13/archon/public_html]# > > Both files share the same inode number, so I'm assuming it's a > hardlink, but why is the home.14 directory still saying 2.7GiB used? > Shouldn't it only be using whatever the delta is between home.13 and > home.14 (in this case, rsync reported copying over about 40 MiB of > data). > > df -h shows that /home is 8.6GiB when only ~3 GiB is actually being > used, so it's still creating fresh copies. > > I'm guessing I'm misunderstanding what --link-dest does, as I > understood it to create a hardlink to the file in the link-dest > directory if the file does not exist in the destination directory. > > Here's a log from the 2PM script run: > > [root at arthur /var]# cat rsync.log > sending incremental file list > created directory /home/backup/home.14 > archon/IMAPdir/Apple Mail To Do/ > archon/IMAPdir/Deleted Messages/ > archon/IMAPdir/Deleted Messages/bincimap-cache > 2.22K 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1943/2336) > archon/IMAPdir/Deleted Messages/bincimap-uidvalidity > 75 100% 0.60kB/s 0:00:00 (xfer#2, to-check=1942/2336) > archon/IMAPdir/Deleted Messages/cur/ > archon/IMAPdir/Deleted Messages/cur/ > 1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S > 24.44K 100% 194.07kB/s 0:00:00 (xfer#3, to-check=1921/2336) > archon/IMAPdir/Deleted Messages/new/ > archon/IMAPdir/Deleted Messages/tmp/ > archon/IMAPdir/Freecycle/ > archon/IMAPdir/Freecycle/bincimap-cache > 876 100% 6.96kB/s 0:00:00 (xfer#4, to-check=1890/2336) > archon/IMAPdir/Freecycle/bincimap-uidvalidity > 74 100% 0.59kB/s 0:00:00 (xfer#5, to-check=1889/2336) > archon/IMAPdir/Freecycle/cur/ > archon/IMAPdir/Freecycle/cur/ > 1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2, > 5.20K 100% 40.95kB/s 0:00:00 (xfer#6, to-check=1879/2336) > archon/IMAPdir/Freecycle/new/ > archon/IMAPdir/Freecycle/tmp/ > archon/Maildir/ > archon/Maildir/bincimap-cache > 1.13K 100% 2.14kB/s 0:00:00 (xfer#7, to-check=1041/3870) > archon/Maildir/bincimap-uidvalidity > 75 100% 0.14kB/s 0:00:00 (xfer#8, to-check=1040/3870) > archon/Maildir/cur/ > archon/Maildir/cur/1250021559.7658.arthur.silvertree.org:2,S > 7.60K 100% 13.45kB/s 0:00:00 (xfer#9, to-check=1026/3870) > archon/Maildir/cur/1250022468.8904.arthur.silvertree.org:2,S > 7.76K 100% 13.71kB/s 0:00:00 (xfer#10, to-check=1025/3870) > archon/Maildir/cur/1250022542.9042.arthur.silvertree.org:2,ST => > archon/IMAPdir/Deleted Messages/cur/ > 1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S > archon/Maildir/cur/1250023069.9684.arthur.silvertree.org:2,ST => > archon/IMAPdir/Freecycle/cur/ > 1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2, > archon/Maildir/cur/1250023397.10077.arthur.silvertree.org:2,S > 4.68K 100% 8.26kB/s 0:00:00 (xfer#11, to-check=1022/3870) > archon/Maildir/cur/1250023861.10654.arthur.silvertree.org:2,S > 6.45K 100% 11.38kB/s 0:00:00 (xfer#12, to-check=1021/3870) > archon/Maildir/new/ > archon/Maildir/tmp/ > backup/ > pinnacle/IMAPdir/Apple Mail To Do/ > romany/Maildir/new/ > romany/Maildir/new/1250023382.10044.arthur.silvertree.org > 2.49K 100% 0.00kB/s 0:00:00 (xfer#13, to-check=1001/22172) > romany/Maildir/tmp/ > scott.schappell/IMAPdir/Apple Mail To Do/ > scott.schappell/Maildir/ > scott.schappell/Maildir/bincimap-cache > 46.29K 100% 279.03kB/s 0:00:00 (xfer#14, to-check=1004/24511) > scott.schappell/Maildir/bincimap-uidvalidity > 75 100% 0.30kB/s 0:00:00 (xfer#15, to-check=1003/24511) > scott.schappell/Maildir/cur/ > scott.schappell/Maildir/cur/1250023996.10809.arthur.silvertree.org:2, > 20.12K 100% 66.38kB/s 0:00:00 (xfer#16, to-check=1024/24943) > scott.schappell/Maildir/new/ > scott.schappell/Maildir/tmp/ > soucon/3.1/game/data/sc.gdbm > 34.97M 100% 16.10MB/s 0:00:02 (xfer#17, to-check=1014/28108) > > Number of files: 32052 > Number of files transferred: 17 > Total file size: 2.79G bytes > Total transferred file size: 35.10M bytes > Literal data: 35.10M bytes > Matched data: 0 bytes > File list size: 991.74K > File list generation time: 0.009 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 36.10M > Total bytes received: 2.71K > > sent 36.10M bytes received 2.71K bytes 4.81M bytes/sec > total size is 2.79G speedup is 77.36 > > rsync version: > > [root at arthur /var]# rsync --version > rsync version 3.0.6 protocol version 30 > Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. > Web site: http://rsync.samba.org/ > Capabilities: > 64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints, > socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, > append, ACLs, xattrs, no iconv, symtimes > > Installed from FreeBSD 7.2 ports collection. > > Thanks for any help in enabling me to understand how this works. > > Scott > > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
I think your problem is with reading the correct size of folders as there have hard links. To do this with du command, try: du -sh /home/backup/* As far as I know, du command will only report the "real" disk size if the 2 hard links are in the du "scoop". Otherwise, running 2 times du on the 2 separate directories (wich have hardlinked files from one to the other into them) will show the size of those directories, counting the hardlinked as it's normal size. Am I clear enough? Hardlinked files are actually one unique file viewed from different places. If you look at each place at once they will show you the entire file size. To see different sizes you need to see both places at same time with a command that handle hardlinks (as du does). From man du: "Files having multiple hard links are counted (and displayed) a single time per du execution." Mind the "du execution", so you have to have both directories on the same command :) Best regards, Vitorio Le 11 ao?t 09 ? 23:25, Scott Schappell a ?crit :> Hourly I have an rsync job backup /home to /home/backup. I have 24 > directories (one for each hour): > > home.0 > ... > home.23 > > Here is the script I am running via cron: > > #! /usr/local/bin/bash > dest=`date +%k | sed 's/ //g'` > linkdir=`date -v-1H +%k | sed 's/ //g'` > chflags -R noschg /home/backup > rm -rf /home/backup/home.$dest > rsync -ahHP --numeric-ids --delete --stats --link-dest=../ > home."$linkdir"\ > --exclude=/backup/* /home/ /home/backup/home."$dest"/ > /var/ > rsync.log > sleep 2 > chflags -R schg /home/backup > > Per reading this list and other google finds, it was stated that for > link-dest to work, the destination must be empty, which I do. > However, let's say a 2PM backup runs. The 2PM backup has 2.7GiB > (the current size of /home). The 1PM backup directory is also 2.7GiB. > > [root at arthur /home/backup/home.14]# du -hcd0 > 2.7G . > 2.7G total > [root at arthur /home/backup/home.14]# > > [root at arthur /home/backup/home.13]# !du > du -hcd0 > 2.7G . > 2.7G total > [root at arthur /home/backup/home.13]# > > For a specific file: > > [root at arthur /home/backup/home.14/archon/public_html]# ls -li d2x.jpg > 22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg > [root at arthur /home/backup/home.14/archon/public_html]# > > [root at arthur /home/backup/home.13/archon/public_html]# !ls > ls -li d2x.jpg > 22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg > [root at arthur /home/backup/home.13/archon/public_html]# > > Both files share the same inode number, so I'm assuming it's a > hardlink, but why is the home.14 directory still saying 2.7GiB used? > Shouldn't it only be using whatever the delta is between home.13 and > home.14 (in this case, rsync reported copying over about 40 MiB of > data). > > df -h shows that /home is 8.6GiB when only ~3 GiB is actually being > used, so it's still creating fresh copies. > > I'm guessing I'm misunderstanding what --link-dest does, as I > understood it to create a hardlink to the file in the link-dest > directory if the file does not exist in the destination directory. > > Here's a log from the 2PM script run: > > [root at arthur /var]# cat rsync.log > sending incremental file list > created directory /home/backup/home.14 > archon/IMAPdir/Apple Mail To Do/ > archon/IMAPdir/Deleted Messages/ > archon/IMAPdir/Deleted Messages/bincimap-cache > 2.22K 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1943/2336) > archon/IMAPdir/Deleted Messages/bincimap-uidvalidity > 75 100% 0.60kB/s 0:00:00 (xfer#2, to-check=1942/2336) > archon/IMAPdir/Deleted Messages/cur/ > archon/IMAPdir/Deleted Messages/cur/ > 1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S > 24.44K 100% 194.07kB/s 0:00:00 (xfer#3, to-check=1921/2336) > archon/IMAPdir/Deleted Messages/new/ > archon/IMAPdir/Deleted Messages/tmp/ > archon/IMAPdir/Freecycle/ > archon/IMAPdir/Freecycle/bincimap-cache > 876 100% 6.96kB/s 0:00:00 (xfer#4, to-check=1890/2336) > archon/IMAPdir/Freecycle/bincimap-uidvalidity > 74 100% 0.59kB/s 0:00:00 (xfer#5, to-check=1889/2336) > archon/IMAPdir/Freecycle/cur/ > archon/IMAPdir/Freecycle/cur/ > 1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2, > 5.20K 100% 40.95kB/s 0:00:00 (xfer#6, to-check=1879/2336) > archon/IMAPdir/Freecycle/new/ > archon/IMAPdir/Freecycle/tmp/ > archon/Maildir/ > archon/Maildir/bincimap-cache > 1.13K 100% 2.14kB/s 0:00:00 (xfer#7, to-check=1041/3870) > archon/Maildir/bincimap-uidvalidity > 75 100% 0.14kB/s 0:00:00 (xfer#8, to-check=1040/3870) > archon/Maildir/cur/ > archon/Maildir/cur/1250021559.7658.arthur.silvertree.org:2,S > 7.60K 100% 13.45kB/s 0:00:00 (xfer#9, to-check=1026/3870) > archon/Maildir/cur/1250022468.8904.arthur.silvertree.org:2,S > 7.76K 100% 13.71kB/s 0:00:00 (xfer#10, to-check=1025/3870) > archon/Maildir/cur/1250022542.9042.arthur.silvertree.org:2,ST => > archon/IMAPdir/Deleted Messages/cur/ > 1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S > archon/Maildir/cur/1250023069.9684.arthur.silvertree.org:2,ST => > archon/IMAPdir/Freecycle/cur/ > 1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2, > archon/Maildir/cur/1250023397.10077.arthur.silvertree.org:2,S > 4.68K 100% 8.26kB/s 0:00:00 (xfer#11, to-check=1022/3870) > archon/Maildir/cur/1250023861.10654.arthur.silvertree.org:2,S > 6.45K 100% 11.38kB/s 0:00:00 (xfer#12, to-check=1021/3870) > archon/Maildir/new/ > archon/Maildir/tmp/ > backup/ > pinnacle/IMAPdir/Apple Mail To Do/ > romany/Maildir/new/ > romany/Maildir/new/1250023382.10044.arthur.silvertree.org > 2.49K 100% 0.00kB/s 0:00:00 (xfer#13, to-check=1001/22172) > romany/Maildir/tmp/ > scott.schappell/IMAPdir/Apple Mail To Do/ > scott.schappell/Maildir/ > scott.schappell/Maildir/bincimap-cache > 46.29K 100% 279.03kB/s 0:00:00 (xfer#14, to-check=1004/24511) > scott.schappell/Maildir/bincimap-uidvalidity > 75 100% 0.30kB/s 0:00:00 (xfer#15, to-check=1003/24511) > scott.schappell/Maildir/cur/ > scott.schappell/Maildir/cur/1250023996.10809.arthur.silvertree.org:2, > 20.12K 100% 66.38kB/s 0:00:00 (xfer#16, to-check=1024/24943) > scott.schappell/Maildir/new/ > scott.schappell/Maildir/tmp/ > soucon/3.1/game/data/sc.gdbm > 34.97M 100% 16.10MB/s 0:00:02 (xfer#17, to-check=1014/28108) > > Number of files: 32052 > Number of files transferred: 17 > Total file size: 2.79G bytes > Total transferred file size: 35.10M bytes > Literal data: 35.10M bytes > Matched data: 0 bytes > File list size: 991.74K > File list generation time: 0.009 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 36.10M > Total bytes received: 2.71K > > sent 36.10M bytes received 2.71K bytes 4.81M bytes/sec > total size is 2.79G speedup is 77.36 > > rsync version: > > [root at arthur /var]# rsync --version > rsync version 3.0.6 protocol version 30 > Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. > Web site: http://rsync.samba.org/ > Capabilities: > 64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints, > socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, > append, ACLs, xattrs, no iconv, symtimes > > Installed from FreeBSD 7.2 ports collection. > > Thanks for any help in enabling me to understand how this works. > > Scott > > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html