Hey, I have two identical machines setup with a RAID 5 array. One of them is used for failovers and data from the master is synced everyday using rsync to the failover machine. The data on this disks are usually intranet KB's, DB's etc.. The RAID 5 arrays are formatted using the default options i,e mkfs.ext3 /dev/Xda. The RAID controller is 3ware escalade and each disk member in the RAID 5 array are 400Gb IDE. Now the wierd part is, after syncing the failover with the master and comparing the size of each dir and file I find some files where the size mismatches.. [root at storage-master repositories]# du --si "/store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" 8.2k /store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt root at storage-slave compare]# du --si "/store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" 4.1k /store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt stat on the same file shows.. [root at storage-master repositories]# stat "/store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" File: `/store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt' Size: 1126 Blocks: 16 IO Block: 4096 regular file Device: 801h/2049d Inode: 10403842 Links: 1 Access: (0775/-rwxrwxr-x) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2006-09-11 12:22:24.000000000 +0530 Modify: 2004-09-23 16:45:31.000000000 +0530 Change: 2006-02-23 18:31:42.000000000 +0530 root at storage-slave compare]# stat "/store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" File: `/store1/SystemAdministration-OldVideos/SysAd Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt' Size: 1126 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 23019536 Links: 1 Access: (0775/-rwxrwxr-x) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2001-01-28 21:10:14.000000000 +0530 Modify: 2004-09-23 16:45:31.000000000 +0530 Change: 2001-01-28 21:10:14.000000000 +0530 The number of blocks allocated on the master seems to be 16 and the failover is 8. Is this the reason for the file size difference even though the content is the same ? I rsynced the same file from the master to a different server and the file size matched. Any reason why the no. of blocks allocated is different across both this machines ? The file i gave above is just a example and there are many more files like this. Also only 10% of the files have different sizes. I.e out of 263032 files/folders only 17655 have the above problem. Below is the ext3 filesystem info on both the master and failover. [root at storage-master repositories]# dumpe2fs -h /dev/sda1 dumpe2fs 1.35 (28-Feb-2004) Filesystem volume name: /store1 Last mounted on: <not available> Filesystem UUID: 2368a03d-f21f-4c5e-b12a-cbd2c726237c Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode filetype needs_recovery sparse_super large_file Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 97681408 Block count: 195354408 Reserved block count: 9767720 Free blocks: 22200635 Free inodes: 97329015 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 1024 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16384 Inode blocks per group: 512 Filesystem created: Tue Jun 28 17:06:41 2005 Last mount time: Tue Oct 10 20:22:02 2006 Last write time: Tue Oct 10 20:22:02 2006 Mount count: 93 Maximum mount count: -1 Last checked: Thu Oct 20 19:03:56 2005 Check interval: 0 (<none>) Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 First orphan inode: 52691033 Default directory hash: tea Directory Hash Seed: 0449f257-e47d-4faf-92fa-fa497efab3a1 Journal backup: inode blocks [root at storage-slave compare]# dumpe2fs -h /dev/sda1 dumpe2fs 1.35 (28-Feb-2004) Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: b003440d-d153-4cec-a668-94f5482d54cf Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode filetype needs_recovery sparse_super large_file Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 97681408 Block count: 195354408 Reserved block count: 9767720 Free blocks: 26532722 Free inodes: 97327187 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 1024 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16384 Inode blocks per group: 512 Filesystem created: Tue Jun 28 14:37:12 2005 Last mount time: Thu Nov 16 01:11:30 2006 Last write time: Thu Nov 16 01:11:30 2006 Mount count: 65 Maximum mount count: -1 Last checked: Thu Oct 6 15:11:41 2005 Check interval: 0 (<none>) Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: tea Directory Hash Seed: fa6b4317-d51d-4050-b0f3-c72b45148777 Journal backup: inode blocks tia dushy
Have you MD5SUM'd the file on both sides? If it is the same, then you have no problems. % md5sum filename On each side, compare output. Justin. On Wed, 6 Dec 2006, dushy wrote:> Hey, > > I have two identical machines setup with a RAID 5 array. One of them is used for > failovers and data from the master is synced everyday using rsync to the > failover machine. The data on this disks are usually intranet KB's, DB's etc.. > > The RAID 5 arrays are formatted using the default options i,e mkfs.ext3 > /dev/Xda. The RAID controller is 3ware escalade and each disk member in the RAID > 5 array are 400Gb IDE. > > Now the wierd part is, after syncing the failover with the master and comparing > the size of each dir and file I find some files where the size mismatches.. > > [root at storage-master repositories]# du --si > "/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" > 8.2k /store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt > > root at storage-slave compare]# du --si > "/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" > 4.1k /store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt > > stat on the same file shows.. > > [root at storage-master repositories]# stat > "/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" > File: `/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt' > Size: 1126 Blocks: 16 IO Block: 4096 regular file > Device: 801h/2049d Inode: 10403842 Links: 1 > Access: (0775/-rwxrwxr-x) Uid: ( 48/ apache) Gid: ( 48/ apache) > Access: 2006-09-11 12:22:24.000000000 +0530 > Modify: 2004-09-23 16:45:31.000000000 +0530 > Change: 2006-02-23 18:31:42.000000000 +0530 > > root at storage-slave compare]# stat "/store1/SystemAdministration-OldVideos/SysAd > Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" > File: `/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt' > Size: 1126 Blocks: 8 IO Block: 4096 regular file > Device: 801h/2049d Inode: 23019536 Links: 1 > Access: (0775/-rwxrwxr-x) Uid: ( 48/ apache) Gid: ( 48/ apache) > Access: 2001-01-28 21:10:14.000000000 +0530 > Modify: 2004-09-23 16:45:31.000000000 +0530 > Change: 2001-01-28 21:10:14.000000000 +0530 > > The number of blocks allocated on the master seems to be 16 and the failover is > 8. Is this the reason for the file size difference even though the content is > the same ? > > I rsynced the same file from the master to a different server and the file size > matched. Any reason why the no. of blocks allocated is different across both > this machines ? > > The file i gave above is just a example and there are many more files like this. > Also only 10% of the files have different sizes. I.e out of 263032 > files/folders only 17655 have the above problem. > > Below is the ext3 filesystem info on both the master and failover. > > [root at storage-master repositories]# dumpe2fs -h /dev/sda1 > dumpe2fs 1.35 (28-Feb-2004) > Filesystem volume name: /store1 > Last mounted on: <not available> > Filesystem UUID: 2368a03d-f21f-4c5e-b12a-cbd2c726237c > Filesystem magic number: 0xEF53 > Filesystem revision #: 1 (dynamic) > Filesystem features: has_journal ext_attr resize_inode filetype > needs_recovery sparse_super large_file > Default mount options: (none) > Filesystem state: clean > Errors behavior: Continue > Filesystem OS type: Linux > Inode count: 97681408 > Block count: 195354408 > Reserved block count: 9767720 > Free blocks: 22200635 > Free inodes: 97329015 > First block: 0 > Block size: 4096 > Fragment size: 4096 > Reserved GDT blocks: 1024 > Blocks per group: 32768 > Fragments per group: 32768 > Inodes per group: 16384 > Inode blocks per group: 512 > Filesystem created: Tue Jun 28 17:06:41 2005 > Last mount time: Tue Oct 10 20:22:02 2006 > Last write time: Tue Oct 10 20:22:02 2006 > Mount count: 93 > Maximum mount count: -1 > Last checked: Thu Oct 20 19:03:56 2005 > Check interval: 0 (<none>) > Reserved blocks uid: 0 (user root) > Reserved blocks gid: 0 (group root) > First inode: 11 > Inode size: 128 > Journal inode: 8 > First orphan inode: 52691033 > Default directory hash: tea > Directory Hash Seed: 0449f257-e47d-4faf-92fa-fa497efab3a1 > Journal backup: inode blocks > > [root at storage-slave compare]# dumpe2fs -h /dev/sda1 > dumpe2fs 1.35 (28-Feb-2004) > Filesystem volume name: <none> > Last mounted on: <not available> > Filesystem UUID: b003440d-d153-4cec-a668-94f5482d54cf > Filesystem magic number: 0xEF53 > Filesystem revision #: 1 (dynamic) > Filesystem features: has_journal ext_attr resize_inode filetype > needs_recovery sparse_super large_file > Default mount options: (none) > Filesystem state: clean > Errors behavior: Continue > Filesystem OS type: Linux > Inode count: 97681408 > Block count: 195354408 > Reserved block count: 9767720 > Free blocks: 26532722 > Free inodes: 97327187 > First block: 0 > Block size: 4096 > Fragment size: 4096 > Reserved GDT blocks: 1024 > Blocks per group: 32768 > Fragments per group: 32768 > Inodes per group: 16384 > Inode blocks per group: 512 > Filesystem created: Tue Jun 28 14:37:12 2005 > Last mount time: Thu Nov 16 01:11:30 2006 > Last write time: Thu Nov 16 01:11:30 2006 > Mount count: 65 > Maximum mount count: -1 > Last checked: Thu Oct 6 15:11:41 2005 > Check interval: 0 (<none>) > Reserved blocks uid: 0 (user root) > Reserved blocks gid: 0 (group root) > First inode: 11 > Inode size: 128 > Journal inode: 8 > Default directory hash: tea > Directory Hash Seed: fa6b4317-d51d-4050-b0f3-c72b45148777 > Journal backup: inode blocks > > tia > dushy > > > > > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users >
On Dec 06, 2006 15:38 +0000, dushy wrote:> [root at storage-master repositories]# stat > "/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" > File: `/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt' > Size: 1126 Blocks: 16 IO Block: 4096 regular file > Device: 801h/2049d Inode: 10403842 Links: 1 > Access: (0775/-rwxrwxr-x) Uid: ( 48/ apache) Gid: ( 48/ apache) > Access: 2006-09-11 12:22:24.000000000 +0530 > Modify: 2004-09-23 16:45:31.000000000 +0530 > Change: 2006-02-23 18:31:42.000000000 +0530 > > root at storage-slave compare]# stat "/store1/SystemAdministration-OldVideos/SysAd > Training/Technology Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt" > File: `/store1/SystemAdministration-OldVideos/SysAd Training/Technology > Basics/000_READ_ME_FIRST_FOR_INDEX_OF_VIDEOS.txt' > Size: 1126 Blocks: 8 IO Block: 4096 regular file > Device: 801h/2049d Inode: 23019536 Links: 1 > Access: (0775/-rwxrwxr-x) Uid: ( 48/ apache) Gid: ( 48/ apache) > Access: 2001-01-28 21:10:14.000000000 +0530 > Modify: 2004-09-23 16:45:31.000000000 +0530 > Change: 2001-01-28 21:10:14.000000000 +0530I'd suspect you have SELinux enabled on one of the nodes and not the other? Could also be ACLs. It is likely adding a 4kB EA block to each file. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
On Wed, Dec 06, 2006 at 03:38:08PM +0000, dushy wrote:> Now the wierd part is, after syncing the failover with the master and comparing > the size of each dir and file I find some files where the size mismatches.. > Size: 1126 Blocks: 16 IO Block: 4096 regular file > Size: 1126 Blocks: 8 IO Block: 4096 regular filemaybe those files contain enough zero-bytes, and rsync has made a sparse file ? -- Opinions above are GNU-copylefted.