We have been trying to use e2scan to generate a list of files that were changed since a specific date. We are seeing that it has been missing some of the modified files and was wondering if anyone else has seen this behavior. I executed: e2scan -N "Mar 27 08:00:00 2009" /dev/mapper/mdt_scratch > chad.out [root at abe-mds04 projects_sync]# grep ILRocstar chad.out ./ROOT/.projects/roc/fnajjar/ILRocstar [root at abe-mds04 projects_sync]# The following list is a very small subset of files within this directory that were changed after the scan date: [root at abe-bk1 ILRocstar]# pwd /cfs/scratch/.projects/roc/fnajjar/ILRocstar [root at abe-bk1 ILRocstar]# ls -lR | grep "Mar 28" drwxr-x--- 4 fnajjar gac 4096 Mar 28 00:56 RSRM105_3D_HG drwxr-x--- 2 fnajjar gac 4096 Mar 28 00:56 Turb_LES -rw------- 1 fnajjar gac 4424 Mar 28 01:05 IVHG_j01.o863001 -rw------- 1 fnajjar gac 4412 Mar 28 02:00 IVHG_j02.o863005 -rw------- 1 fnajjar gac 4427 Mar 28 05:53 IVHG_j03.o863017 -rw------- 1 fnajjar gac 5616 Mar 28 06:54 IVHG_j04.o863026 -rw------- 1 fnajjar gac 5084 Mar 28 11:47 IVHG_j05.o863591 -rw------- 1 fnajjar gac 4489 Mar 28 11:58 IVHG_j06.o863592 -rw------- 1 fnajjar gac 4428 Mar 28 20:55 IVHG_j07.o863637 [root at abe-bk1 ILRocstar]# ls -lR | grep "Mar 28" | wc -l 164 [root at abe-bk1 ILRocstar]# We are wanting to use e2scan to generate a list of files to back up with tivoli, because it is much faster to pass tivoli a list than have it traverse the entire tree. However, it looks like e2scan can''t be trusted to generate a valid list. The version of e2scan we are running is: [root at abe-mds04 projects_sync]# rpm -qilf `which e2scan` Name : e2fsprogs Relocations: (not relocatable) Version : 1.40.11.sun1 Vendor: (none) Release : 0redhat Build Date: Thu 10 Jul 2008 11:21:23 AM CDT Install Date: Sat 20 Dec 2008 04:07:31 AM CST Build Host: lts-build-x86-64-0.co.cfs Group : System Environment/Base Source RPM: e2fsprogs-1.40.11.sun1-0redhat.src.rpm Size : 2085263 License: GPLv2 Signature : (none) URL : ftp://ftp.lustre.org/pub/lustre/other/e2fsprogs/ Summary : Utilities for managing the second and third extended (ext2/ext3) filesystems Description : Any thoughts or suggestions would be greately appreciated. Thanks, Chad -- Chad Kerner - ckerner at ncsa.uiuc.edu Systems Engineer, Storage Enabling Technologies National Center for Supercomputing Applications http://www.ncsa.uiuc.edu/~ckerner
Reading through the Lustre documentation, it is still unclear to me about how to best set up the underlying RAID arrays. Three OSSes connect to 3 disk drawers with 15 drives. These are directly connected SATA drawers with no connections between the OSSes. My initial feeling was to set these up as 4+2 (RAID6), two per drawer, as the documentation states that more OSTs are better than fewer. This would allow 256K blocks with the stride of 1M. So far, so good. The argument has been made that more spindles provide better performance. So 8+2. The 128K blocks still allow 1M stride across the 8 spindles. Because we have 15 drives though, a 12+2 (with always 1 hot spare) and a possible block size of 256K, stride 3M, this might be a much better option. I''m kinda stuck in this 1M block though that Lustre seems to want to throw around. Is there any reason that the underlying disk structure needs to match this exactly to get the best performance? Or am I believing in some urban legend here? Does the Raid card, with it''s cache just hide this all from Lustre anyway? Any insight would be appreciated. Thanks. Bill
We are using e2scan since few days and we have noticed that date
specification is not being processed correctly by e2scan.
> date
Fri Apr 3 15:56:49 BST 2009
> /usr/sbin/e2scan -C /ROOT -l -N "2009-03-29 19:44:00" /dev/dm-0
>
file_list
generating list of files with
mtime newer than Sun Mar 29 15:56:54 2009
ctime newer than Sun Mar 29 15:56:54 2009
inode bitmap is read, 0 seconds
visible root: "/ROOT"
scanning inode tables ..
However solution for us is to use epoch time format.
Let me know if it works for you.
Cheers
Wojciech
Chad Kerner wrote:> We have been trying to use e2scan to generate a list of files that were
> changed since a specific date. We are seeing that it has been missing
> some of the modified files and was wondering if anyone else has seen
> this behavior.
>
> I executed:
> e2scan -N "Mar 27 08:00:00 2009" /dev/mapper/mdt_scratch >
chad.out
>
> [root at abe-mds04 projects_sync]# grep ILRocstar chad.out
> ./ROOT/.projects/roc/fnajjar/ILRocstar
> [root at abe-mds04 projects_sync]#
>
> The following list is a very small subset of files within this directory
> that were changed after the scan date:
> [root at abe-bk1 ILRocstar]# pwd
> /cfs/scratch/.projects/roc/fnajjar/ILRocstar
> [root at abe-bk1 ILRocstar]# ls -lR | grep "Mar 28"
> drwxr-x--- 4 fnajjar gac 4096 Mar 28 00:56 RSRM105_3D_HG
> drwxr-x--- 2 fnajjar gac 4096 Mar 28 00:56 Turb_LES
> -rw------- 1 fnajjar gac 4424 Mar 28 01:05 IVHG_j01.o863001
> -rw------- 1 fnajjar gac 4412 Mar 28 02:00 IVHG_j02.o863005
> -rw------- 1 fnajjar gac 4427 Mar 28 05:53 IVHG_j03.o863017
> -rw------- 1 fnajjar gac 5616 Mar 28 06:54 IVHG_j04.o863026
> -rw------- 1 fnajjar gac 5084 Mar 28 11:47 IVHG_j05.o863591
> -rw------- 1 fnajjar gac 4489 Mar 28 11:58 IVHG_j06.o863592
> -rw------- 1 fnajjar gac 4428 Mar 28 20:55 IVHG_j07.o863637
>
> [root at abe-bk1 ILRocstar]# ls -lR | grep "Mar 28" | wc -l
> 164
> [root at abe-bk1 ILRocstar]#
>
>
> We are wanting to use e2scan to generate a list of files to back up with
> tivoli, because it is much faster to pass tivoli a list than have it
> traverse the entire tree. However, it looks like e2scan can''t be
> trusted to generate a valid list.
>
> The version of e2scan we are running is:
> [root at abe-mds04 projects_sync]# rpm -qilf `which e2scan`
> Name : e2fsprogs Relocations: (not relocatable)
> Version : 1.40.11.sun1 Vendor: (none)
> Release : 0redhat Build Date: Thu 10 Jul 2008
11:21:23 AM CDT
> Install Date: Sat 20 Dec 2008 04:07:31 AM CST Build Host:
lts-build-x86-64-0.co.cfs
> Group : System Environment/Base Source RPM:
e2fsprogs-1.40.11.sun1-0redhat.src.rpm
> Size : 2085263 License: GPLv2
> Signature : (none)
> URL : ftp://ftp.lustre.org/pub/lustre/other/e2fsprogs/
> Summary : Utilities for managing the second and third extended
> (ext2/ext3) filesystems
> Description :
>
>
> Any thoughts or suggestions would be greately appreciated.
>
> Thanks,
> Chad
> --
> Chad Kerner - ckerner at ncsa.uiuc.edu
> Systems Engineer, Storage Enabling Technologies
> National Center for Supercomputing Applications
> http://www.ncsa.uiuc.edu/~ckerner
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>