Hello, We would like to build an informational page that will provide users with stats (size, name, path, mod time,...) on files they have in the lustre filesystem (and maybe store it in a db). I can use "lfs" to get strip and OST info but not "stat" info. We could build some perl tools (probably already out there) which would traverse the filesystem and give us this information, but I would like to know if there is a better or efficient way to do this. Or maybe there is already a lustre utility that can give me the stat info on a file just by querying the MDS server instead of traversing the entire lustre filesystem? Or maybe this is not a good approach? Something better? thanks -k
The problem becomes that it is very time consuming to run stat on every file of the MDS. We have used a multi-threaded python program to walk the entire filesystem fairly effectively on our two large Lustre systems. I released one of them a while back to the mailing list, and it should be in the archives somewhere. If you cant locate it send me a mail, and I''ll send it on to you. The released version was designed to expire files when they have been on the file system for a long time, but it doesn''t have to, as it also collects usage data for files, and directories. I''ve heard rumors that someone in CFS is working on and MDS scanning utility that should be more efficient, but until we see it I have no idea how well it works.. Evan> -----Original Message----- > From: lustre-discuss-bounces@clusterfs.com > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of > Kaizaad Bilimorya > Sent: Thursday, January 11, 2007 7:31 AM > To: lustre-discuss@clusterfs.com > Subject: [Lustre-discuss] file stat in lustre > > Hello, > > We would like to build an informational page that will > provide users with stats (size, name, path, mod time,...) on > files they have in the lustre filesystem (and maybe store it > in a db). I can use "lfs" to get strip and OST info but not > "stat" info. We could build some perl tools (probably already > out there) which would traverse the filesystem and give us > this information, but I would like to know if there is a > better or efficient way to do this. > > Or maybe there is already a lustre utility that can give me > the stat info on a file just by querying the MDS server > instead of traversing the entire lustre filesystem? Or maybe > this is not a good approach? > Something better? > > thanks > -k > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
CFS recently posted a C program that can scan the disks (even while Lustre is mounted) and generate pathname lists. Our main purpose was backups. This program should be usable for other purposes. Huangwei, could you point out how this works? - peter - Felix, Evan J wrote:> The problem becomes that it is very time consuming to run stat on every > file of the MDS. We have used a multi-threaded python program to walk > the entire filesystem fairly effectively on our two large Lustre > systems. I released one of them a while back to the mailing list, and > it should be in the archives somewhere. If you cant locate it send me a > mail, and I''ll send it on to you. The released version was designed to > expire files when they have been on the file system for a long time, but > it doesn''t have to, as it also collects usage data for files, and > directories. > > I''ve heard rumors that someone in CFS is working on and MDS scanning > utility that should be more efficient, but until we see it I have no > idea how well it works.. > > Evan > > >> -----Original Message----- >> From: lustre-discuss-bounces@clusterfs.com >> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of >> Kaizaad Bilimorya >> Sent: Thursday, January 11, 2007 7:31 AM >> To: lustre-discuss@clusterfs.com >> Subject: [Lustre-discuss] file stat in lustre >> >> Hello, >> >> We would like to build an informational page that will >> provide users with stats (size, name, path, mod time,...) on >> files they have in the lustre filesystem (and maybe store it >> in a db). I can use "lfs" to get strip and OST info but not >> "stat" info. We could build some perl tools (probably already >> out there) which would traverse the filesystem and give us >> this information, but I would like to know if there is a >> better or efficient way to do this. >> >> Or maybe there is already a lustre utility that can give me >> the stat info on a file just by querying the MDS server >> instead of traversing the entire lustre filesystem? Or maybe >> this is not a good approach? >> Something better? >> >> thanks >> -k >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss@clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070113/3d1f080b/attachment.html
Peter Braam wrote:> CFS recently posted a C program that can scan the disks (even while > Lustre is mounted) and generate pathname lists. Our main purpose was > backups. This program should be usable for other purposes. > > Huangwei, could you point out how this works?yes, this program is named e2scan, which is a program using libext2fs to find inodes with ctime or mtime newer than a given time and print out their pathname. this program can be found in bug10929, i also attached it here. Simple usage example for e2scan: e2scan -u -f filename dev this will create a file named modified_pathnames include all modified file''s pathname. more detail of e2scan can be found in their man page.> > - peter - > > Felix, Evan J wrote: >> The problem becomes that it is very time consuming to run stat on every >> file of the MDS. We have used a multi-threaded python program to walk >> the entire filesystem fairly effectively on our two large Lustre >> systems. I released one of them a while back to the mailing list, and >> it should be in the archives somewhere. If you cant locate it send me a >> mail, and I''ll send it on to you. The released version was designed to >> expire files when they have been on the file system for a long time, but >> it doesn''t have to, as it also collects usage data for files, and >> directories. >> >> I''ve heard rumors that someone in CFS is working on and MDS scanning >> utility that should be more efficient, but until we see it I have no >> idea how well it works.. >> >> Evan >> >> >>> -----Original Message----- >>> From: lustre-discuss-bounces@clusterfs.com >>> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of >>> Kaizaad Bilimorya >>> Sent: Thursday, January 11, 2007 7:31 AM >>> To: lustre-discuss@clusterfs.com >>> Subject: [Lustre-discuss] file stat in lustre >>> >>> Hello, >>> >>> We would like to build an informational page that will >>> provide users with stats (size, name, path, mod time,...) on >>> files they have in the lustre filesystem (and maybe store it >>> in a db). I can use "lfs" to get strip and OST info but not >>> "stat" info. We could build some perl tools (probably already >>> out there) which would traverse the filesystem and give us >>> this information, but I would like to know if there is a >>> better or efficient way to do this. >>> >>> Or maybe there is already a lustre utility that can give me >>> the stat info on a file just by querying the MDS server >>> instead of traversing the entire lustre filesystem? Or maybe >>> this is not a good approach? >>> Something better? >>> >>> thanks >>> -k >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss@clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>> >>> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss@clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>-------------- next part -------------- A non-text attachment was scrubbed... Name: e2scan.tar.gz Type: application/x-gzip Size: 8398 bytes Desc: not available Url : http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070115/682f2a5f/e2scan.tar.gz
Huangwei: Could you post the man page here? - peter - huangwei wrote:> Peter Braam wrote: >> CFS recently posted a C program that can scan the disks (even while >> Lustre is mounted) and generate pathname lists. Our main purpose was >> backups. This program should be usable for other purposes. >> >> Huangwei, could you point out how this works? > yes, this program is named e2scan, which is a program using libext2fs > to find inodes with ctime or mtime newer than a given time and print > out their pathname. this program can be found in bug10929, i also > attached it here. > Simple usage example for e2scan: > e2scan -u -f filename dev > this will create a file named modified_pathnames include all modified > file''s pathname. > more detail of e2scan can be found in their man page. > >> >> - peter - >> >> Felix, Evan J wrote: >>> The problem becomes that it is very time consuming to run stat on every >>> file of the MDS. We have used a multi-threaded python program to walk >>> the entire filesystem fairly effectively on our two large Lustre >>> systems. I released one of them a while back to the mailing list, and >>> it should be in the archives somewhere. If you cant locate it send >>> me a >>> mail, and I''ll send it on to you. The released version was designed to >>> expire files when they have been on the file system for a long time, >>> but >>> it doesn''t have to, as it also collects usage data for files, and >>> directories. >>> >>> I''ve heard rumors that someone in CFS is working on and MDS scanning >>> utility that should be more efficient, but until we see it I have no >>> idea how well it works.. >>> >>> Evan >>> >>> >>>> -----Original Message----- >>>> From: lustre-discuss-bounces@clusterfs.com >>>> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of Kaizaad >>>> Bilimorya >>>> Sent: Thursday, January 11, 2007 7:31 AM >>>> To: lustre-discuss@clusterfs.com >>>> Subject: [Lustre-discuss] file stat in lustre >>>> >>>> Hello, >>>> >>>> We would like to build an informational page that will provide >>>> users with stats (size, name, path, mod time,...) on files they >>>> have in the lustre filesystem (and maybe store it in a db). I can >>>> use "lfs" to get strip and OST info but not "stat" info. We could >>>> build some perl tools (probably already out there) which would >>>> traverse the filesystem and give us this information, but I would >>>> like to know if there is a better or efficient way to do this. >>>> >>>> Or maybe there is already a lustre utility that can give me the >>>> stat info on a file just by querying the MDS server instead of >>>> traversing the entire lustre filesystem? Or maybe this is not a >>>> good approach? Something better? >>>> >>>> thanks >>>> -k >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss@clusterfs.com >>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>>> >>>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss@clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070115/e8479a69/attachment.html
Peter Braam wrote:> Huangwei: > > Could you post the man page here?e2scan(1) backup utilities e2scan(1) NAME e2scan - Ext2 filesystem modified inode scan program SYNOPSIS e2scan [options] [-f file] block_device DESCRIPTION e2scan , when invoked, iterates all inodes on the block device, find modified inodes, print their inode numbers.A similar iterator, using libext2fs(5) , will build a table(called par- ent database) which for each inode lists the parent inode. With a lookup function lookup ino in a directory, one can reconstruct modified pathnames from root. To manage hardlinks it is probably a good idea to write in this file <ino> <pathname>, so hard linked files will appear multiple times and all pathnames of the link are present OPTIONS -b inode buffer blocks set readahead inode blocks to get excellent performance when scanning block device. -o output file if output file is set, modified pathnames is written in to this file, otherwise to stdout. -t inode | pathname set e2scan type, if type is "inode", e2scan just print modified inode ino?s to std- out. By default, or type is set "pathname", e2scan will list out modified pathnames based on modified inode ino?s. -u build parent database is a little time consume, if parent database is not up-to-date, e2scan use this option to rebuild parent database from scratch. Otherwise, current parent database is used. BUGS None are known. Lustre 2006 Sep 26 e2scan(1)> > - peter - > > huangwei wrote: >> Peter Braam wrote: >>> CFS recently posted a C program that can scan the disks (even while >>> Lustre is mounted) and generate pathname lists. Our main purpose >>> was backups. This program should be usable for other purposes. >>> >>> Huangwei, could you point out how this works? >> yes, this program is named e2scan, which is a program using libext2fs >> to find inodes with ctime or mtime newer than a given time and print >> out their pathname. this program can be found in bug10929, i also >> attached it here. >> Simple usage example for e2scan: >> e2scan -u -f filename dev >> this will create a file named modified_pathnames include all modified >> file''s pathname. >> more detail of e2scan can be found in their man page. >> >>> >>> - peter - >>> >>> Felix, Evan J wrote: >>>> The problem becomes that it is very time consuming to run stat on >>>> every >>>> file of the MDS. We have used a multi-threaded python program to walk >>>> the entire filesystem fairly effectively on our two large Lustre >>>> systems. I released one of them a while back to the mailing list, and >>>> it should be in the archives somewhere. If you cant locate it send >>>> me a >>>> mail, and I''ll send it on to you. The released version was >>>> designed to >>>> expire files when they have been on the file system for a long >>>> time, but >>>> it doesn''t have to, as it also collects usage data for files, and >>>> directories. >>>> >>>> I''ve heard rumors that someone in CFS is working on and MDS scanning >>>> utility that should be more efficient, but until we see it I have no >>>> idea how well it works.. >>>> >>>> Evan >>>> >>>> >>>>> -----Original Message----- >>>>> From: lustre-discuss-bounces@clusterfs.com >>>>> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of Kaizaad >>>>> Bilimorya >>>>> Sent: Thursday, January 11, 2007 7:31 AM >>>>> To: lustre-discuss@clusterfs.com >>>>> Subject: [Lustre-discuss] file stat in lustre >>>>> >>>>> Hello, >>>>> >>>>> We would like to build an informational page that will provide >>>>> users with stats (size, name, path, mod time,...) on files they >>>>> have in the lustre filesystem (and maybe store it in a db). I can >>>>> use "lfs" to get strip and OST info but not "stat" info. We could >>>>> build some perl tools (probably already out there) which would >>>>> traverse the filesystem and give us this information, but I would >>>>> like to know if there is a better or efficient way to do this. >>>>> >>>>> Or maybe there is already a lustre utility that can give me the >>>>> stat info on a file just by querying the MDS server instead of >>>>> traversing the entire lustre filesystem? Or maybe this is not a >>>>> good approach? Something better? >>>>> >>>>> thanks >>>>> -k >>>>> >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss@clusterfs.com >>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss@clusterfs.com >>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>>> >>
Hello,
I am not sure of how to further debug this problem. Here is the situation.
Feb 6 15:58:00 nar2 kernel: LustreError:
2939:0:(client.c:442:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err ==
-107 req@00000100b5cd1a00 x3573956/t0
o400->nar-sfs-ost103_UUID@NID_3712500356_UUID:28 lens 64/64 ref 1 fl
Rpc:RN/0/0 rc 0/-107
Feb 6 15:58:00 nar2 kernel: LustreError: Connection to service nar-sfs-ost103
via nid 0:3712500356 was lost; in progress operations using this service will
wait for recovery to complete.
Feb 6 15:58:00 nar2 kernel: Lustre:
2939:0:(import.c:139:ptlrpc_set_import_discon())
OSC_nar2_nar-sfs-ost103_MNT_client_gm: connection lost to
nar-sfs-ost103_UUID@NID_3712500356_UUID
Feb 6 15:58:00 nar2 kernel: Lustre:
2939:0:(import.c:288:import_select_connection())
OSC_nar2_nar-sfs-ost103_MNT_client_gm: Using connectionNID_3712497273_UUID
Feb 6 15:58:00 nar2 kernel: Lustre:
2939:0:(import.c:288:import_select_connection()) skipped 2 similar messages
(ending 354162.872 seconds ago)
Feb 6 15:58:11 nar2 kernel: LustreError: This client was evicted by
nar-sfs-ost103; in progress operations using this service will be reattempted.
Feb 6 15:58:11 nar2 kernel: LustreError:
5652:0:(ldlm_resource.c:361:ldlm_namespace_cleanup()) Namespace
OSC_nar2_nar-sfs-ost103_MNT_client_gm resource refcount 4 after lock cleanup
Feb 6 15:58:11 nar2 kernel: LustreError:
5646:0:(llite_mmap.c:208:ll_tree_unlock()) couldn''t unlock -5
Feb 6 15:58:11 nar2 kernel: Lustre: Connection restored to service
nar-sfs-ost103 using nid 0:3712500356.
Feb 6 15:58:11 nar2 kernel: Lustre:
5652:0:(import.c:692:ptlrpc_import_recovery_state_machine())
OSC_nar2_nar-sfs-ost103_MNT_client_gm: connection restored to
nar-sfs-ost103_UUID@NID_3712500356_UUID
[root@nar2 ~]# ps -lfu dgerbasi
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
5 S dgerbasi 5642 5639 0 76 0 - 8211 - Feb06 ? 00:01:21
slurmd: [164600.1]
0 D dgerbasi 5643 5642 0 76 0 - 6473 lock_p Feb06 ? 00:00:00
/nar_sfs/dgerbasi/programs/siesta/water/./siesta
0 D dgerbasi 5644 5642 0 76 0 - 6473 wait_o Feb06 ? 00:00:00
/nar_sfs/dgerbasi/programs/siesta/water/./siesta
0 D dgerbasi 5645 5642 0 76 0 - 6473 lock_p Feb06 ? 00:00:00
/nar_sfs/dgerbasi/programs/siesta/water/./siesta
there is a file that seems to be "locked" on the problem node:
[root@nar2 water]# ls -al O.POT.CONF
-rw-r--r-- 1 dgerbasi twoo 62275 Feb 6 15:58 O.POT.CONF
[root@nar2 water]# stat O.POT.CONF
File: `O.POT.CONF''
Size: 62275 Blocks: 128 IO Block: 2097152 regular file
Device: f908b518h/-116869864d Inode: 1441916 Links: 1
Access: (0644/-rw-r--r--) Uid: (130408/dgerbasi) Gid: (130023/ twoo)
Access: 2007-02-06 15:57:05.507257800 -0500
Modify: 2007-02-06 15:58:11.711026503 -0500
Change: 2007-02-06 15:58:11.711026503 -0500
[root@nar2 water]# file O.POT.CONF
...hangs
while on another node, it is fine but an older version:
[root@nar320 water]# ls -al O.POT.CONF
-rw-r--r-- 1 dgerbasi twoo 62275 Feb 6 15:57 O.POT.CONF
[root@nar320 water]# stat O.POT.CONF
File: `O.POT.CONF''
Size: 62275 Blocks: 128 IO Block: 2097152 regular file
Device: f908b518h/-116869864d Inode: 1441916 Links: 1
Access: (0644/-rw-r--r--) Uid: (130408/dgerbasi) Gid: (130023/ twoo)
Access: 2007-02-07 10:50:21.299914088 -0500
Modify: 2007-02-06 15:57:05.000000000 -0500
Change: 2007-02-06 15:57:05.000000000 -0500
[root@nar320 water]# file O.POT.CONF
O.POT.CONF: ASCII text
So it seems, the problem node (nar2) lost connectivity to the ost103
service, got evicted, then reconnected ("lfs check servers" now shows
active). It is now waiting on some type of page lock to be released before
it can update the file to disk ("lfs getstripe O.POT.CONF" confirmed
file
is on ost103).
Any advise or points in the right direction would be appreciated. We see
this pop up every now and then on our nodes with a variety of user codes.
thanks
-k
This is a HP SFS system (based on lustre 1.4.2)
client version:
[root@nar320 water]# cat /proc/fs/lustre/version
1.4.2-20051219152732-CHANGED-.bld.PER_RC3_2.xc.src.lustre........obj.x86_64.kernel-2.6.9.linux-2.6.9-2.6.9-22.7hp.XCsmp
server version:
V2.1-0 (build nlH8hp, 2005-12-22) (filesystem 1.4.2)