Mark Seger wrote:> > > Tom.Wang wrote: >> Hi, Mark >> Mark Seger wrote: >>> I recently got an email from a collectl/lustre user who reported I''m >>> not handling all the data for an MDS properly and showed me the >>> following from his stats file: >>> >>> snapshot_time 1217282595.543548 secs.usecs >>> req_waittime 170215294 samples [usec] 0 1992338 >>> 23146560348 30473383620091658 >>> req_qdepth 170215294 samples [reqs] 0 9444 124827689 >>> 348732141299 >>> req_active 170215294 samples [reqs] 1 127 352457248 >>> 5984384772 >>> req_timeout 170215294 samples [sec] 1 301 442496292 >>> 28785551992 >>> reqbuf_avail 410936563 samples [bufs] 128 1024 >>> 420313410439 430054594284593 >>> ldlm_flock_enqueue 4 samples [reqs] 1 1 4 4 >>> ldlm_ibits_enqueue 139193009 samples [reqs] 1 1 139193009 >>> 139193009 >>> mds_reint_create 11018837 samples [reqs] 1 1 11018837 11018837 >>> mds_reint_link 51315 samples [reqs] 1 1 51315 51315 >>> mds_reint_setattr 395400 samples [reqs] 1 1 395400 395400 >>> mds_reint_rename 224241 samples [reqs] 1 1 224241 224241 >>> mds_reint_unlink 13109877 samples [reqs] 1 1 13109877 13109877 >>> mds_getattr 49739 samples [usec] 10 271559 7792733 >>> 655474619433 >>> mds_connect 2373 samples [usec] 12 1300137 5532759 >>> 3705296256699 >>> mds_disconnect 291 samples [usec] 18 3047 76662 88841758 >>> mds_getstatus 899 samples [usec] 6 41 9824 114260 >>> mds_statfs 4090 samples [usec] 5 79121 1205331 >>> 45689777475 >>> mds_sync 1190 samples [usec] 1843 5014449 236091207 >>> 450545148159775 >>> mds_quotactl 2049 samples [usec] 7 883434 7687131 >>> 3232395885317 >>> mds_getxattr 36089 samples [usec] 9 8996 675208 252525110 >>> mds_setxattr 1230 samples [usec] 123 10110 263367 >>> 225741995 >>> obd_ping 6124258 samples [usec] 0 30366 67518884 >>> 4513131130 >>> >>> and I''ve never heard of all these ''reint'' variable now have see some >>> many of the others either and so thought they were recently added. >>> the interesting thing is I have 1.6.5.1 installed and my stats file >>> shows: >>> >>> snapshot_time 1217344821.777664 secs.usecs >>> req_waittime 5500 samples [usec] 6 399 57198 831384 >>> req_qdepth 5500 samples [reqs] 0 2 16 18 >>> req_active 5500 samples [reqs] 1 2 5514 5542 >>> req_timeout 5500 samples [sec] 1 10 5572 6292 >>> reqbuf_avail 12650 samples [bufs] 511 512 6475895 >>> 3315195785 >>> ldlm_ibits_enqueue 1646 samples [reqs] 1 1 1646 1646 >>> mds_reint_unlink 8 samples [reqs] 1 1 8 8 >>> mds_getattr 17 samples [usec] 11 14567 15655 212882791 >>> mds_connect 24 samples [usec] 33 1418 2948 2142530 >>> mds_getstatus 17 samples [usec] 8 15 165 1657 >>> mds_statfs 55 samples [usec] 7 48 762 15484 >>> mds_sync 800 samples [usec] 803 40698 2188133 >>> 15429580807 >>> obd_ping 2933 samples [usec] 6 48 48843 1130361 >> MDS_REINT and LDLM req information update /proc has been detailed >> since 1.6.5, see bug 14184. > I don''t think so. It looks like a bunch of patches and random > comments. Are we both talking about - > https://bugzilla.lustre.org/show_bug.cgi?id=14184 ? > The first thing I did was to see what it said about those 5 reint > counters and so searched for "mds_reint_" and found none of them. > Perhaps there are some clues to the meanings to some of the counters > but certainly not all of them.What I mean here is that the original MDS_REINT req has been divided to 5 sub-req status(which I mean detailed here, sorry about the confusion). In original implementation, there is only 1 "mds_reint" req counter in the stats file, and we update that and change it to five specific mds_reint req, Then MDS would know what kind of reint request it has been handled. Not sure whether lustre manual include all the statements of these stats, you might check that?> > Also, you didn''t answer my question about what types of operations > will cause each of the different counters to increment.Hmm, I assume you only ask mds_reint req ? These counters will be incremented when mds identify the request and ready to handle it. So mds_reint_unlink: unlink a file mds_reint_create: mknod or mkdir mds_reint_link: link a file mds_reint_setattr: chmod/chacl or other setattr meta ops. mds_reint_rename: rename a file. Hope I did not miss your questions this time.> Would it be easier and more beneficial to other users to move this to > lustre-discuss?Yes.> > -mark > >Thanks WangDi -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com
Thanks for posting this... Just to be a little clearer, what I''m trying to do is multipurpose. My main objective is to make sure collectl gathers and reports comprehensive/accurate lustre data. The next most important thing is that users of collectl (including me) understand what that data means. As for making sure I collect the correct data, it sounds like you''re saying the only thing that changed in 1.6.5.1 is splitting reint into 5 new counters, which is certainly something I can do as well but it will require a code change and will also result in incorrect data being reported in earlier versions of collectl. From the perspective of understanding what the data means, I''d like to make sure the meanings of the different fields are properly documented in collectl and since none of this seems to be written down anywhere else could actually become a very useful document to others. Right now it you look at http://collectl.sourceforge.net/Data.html you can see the data definitions for all collectl data, including lustre. Some definitions are good while other could use more work. One thing that confuses me about lustre counters, and maybe others, is I don''t really know what they mean, when they change and in fact how to stimulate them to change. For example, on my system I''m doing a watch of /proc/fs/lustre/mdt/MDS/mds/stats and only see 1 reint counter, because the others are all 0. So I went and did some file renames, and chmods and sure enough, the other counters did appear. Cool! The easiest thing for me to do is to simply say that reint_setattr counts the number of setattrs, but that would be a pretty weak definition. When I changed did a single chmod to 100 files, setattr only incremented by 1 and I expected it to increment by 100. This means that the counter is not of how many files actually had there attributes changes, which is what I had thought. It would seem to me that this level of detail should be captured somewhere. And why didn''t reint_setattr get called when I created the files? Shouldn''t it have been? And if not, shouldn''t that be noted somewhere as well. That''s just 2 reint counters. How about all the other mds counters? Are there any missing from the list below - remember, since you only see then after they''re incremented you don''t always see them and that''s apparently what happened to me with the reint counters. I really don''t want to do in this thread is get into a low level discussion about each counter and what they mean, but if someone is willing to write the words, I''m willing to put them in my document. If people want to discuss the finer points of any of the definitions in a different thread (or borrow this one) that''d be fine too, but I don''t want to be the one responsible for the words or all you''re going to see is ''reint_setattr counts the number of setattr calls'' and I really don''t think that would be all that useful to anyone. -mark Tom.Wang wrote:> Mark Seger wrote: >> >> >> Tom.Wang wrote: >>> Hi, Mark >>> Mark Seger wrote: >>>> I recently got an email from a collectl/lustre user who reported >>>> I''m not handling all the data for an MDS properly and showed me the >>>> following from his stats file: >>>> >>>> snapshot_time 1217282595.543548 secs.usecs >>>> req_waittime 170215294 samples [usec] 0 1992338 >>>> 23146560348 30473383620091658 >>>> req_qdepth 170215294 samples [reqs] 0 9444 124827689 >>>> 348732141299 >>>> req_active 170215294 samples [reqs] 1 127 352457248 >>>> 5984384772 >>>> req_timeout 170215294 samples [sec] 1 301 442496292 >>>> 28785551992 >>>> reqbuf_avail 410936563 samples [bufs] 128 1024 >>>> 420313410439 430054594284593 >>>> ldlm_flock_enqueue 4 samples [reqs] 1 1 4 4 >>>> ldlm_ibits_enqueue 139193009 samples [reqs] 1 1 139193009 >>>> 139193009 >>>> mds_reint_create 11018837 samples [reqs] 1 1 11018837 >>>> 11018837 >>>> mds_reint_link 51315 samples [reqs] 1 1 51315 51315 >>>> mds_reint_setattr 395400 samples [reqs] 1 1 395400 395400 >>>> mds_reint_rename 224241 samples [reqs] 1 1 224241 224241 >>>> mds_reint_unlink 13109877 samples [reqs] 1 1 13109877 >>>> 13109877 >>>> mds_getattr 49739 samples [usec] 10 271559 7792733 >>>> 655474619433 >>>> mds_connect 2373 samples [usec] 12 1300137 5532759 >>>> 3705296256699 >>>> mds_disconnect 291 samples [usec] 18 3047 76662 88841758 >>>> mds_getstatus 899 samples [usec] 6 41 9824 114260 >>>> mds_statfs 4090 samples [usec] 5 79121 1205331 >>>> 45689777475 >>>> mds_sync 1190 samples [usec] 1843 5014449 >>>> 236091207 450545148159775 >>>> mds_quotactl 2049 samples [usec] 7 883434 7687131 >>>> 3232395885317 >>>> mds_getxattr 36089 samples [usec] 9 8996 675208 252525110 >>>> mds_setxattr 1230 samples [usec] 123 10110 263367 >>>> 225741995 >>>> obd_ping 6124258 samples [usec] 0 30366 67518884 >>>> 4513131130 >>>> >>>> and I''ve never heard of all these ''reint'' variable now have see >>>> some many of the others either and so thought they were recently >>>> added. >>>> the interesting thing is I have 1.6.5.1 installed and my stats file >>>> shows: >>>> >>>> snapshot_time 1217344821.777664 secs.usecs >>>> req_waittime 5500 samples [usec] 6 399 57198 831384 >>>> req_qdepth 5500 samples [reqs] 0 2 16 18 >>>> req_active 5500 samples [reqs] 1 2 5514 5542 >>>> req_timeout 5500 samples [sec] 1 10 5572 6292 >>>> reqbuf_avail 12650 samples [bufs] 511 512 6475895 >>>> 3315195785 >>>> ldlm_ibits_enqueue 1646 samples [reqs] 1 1 1646 1646 >>>> mds_reint_unlink 8 samples [reqs] 1 1 8 8 >>>> mds_getattr 17 samples [usec] 11 14567 15655 212882791 >>>> mds_connect 24 samples [usec] 33 1418 2948 2142530 >>>> mds_getstatus 17 samples [usec] 8 15 165 1657 >>>> mds_statfs 55 samples [usec] 7 48 762 15484 >>>> mds_sync 800 samples [usec] 803 40698 2188133 >>>> 15429580807 >>>> obd_ping 2933 samples [usec] 6 48 48843 1130361 >>> MDS_REINT and LDLM req information update /proc has been detailed >>> since 1.6.5, see bug 14184. >> I don''t think so. It looks like a bunch of patches and random >> comments. Are we both talking about - >> https://bugzilla.lustre.org/show_bug.cgi?id=14184 ? >> The first thing I did was to see what it said about those 5 reint >> counters and so searched for "mds_reint_" and found none of them. >> Perhaps there are some clues to the meanings to some of the counters >> but certainly not all of them. > What I mean here is that the original MDS_REINT req has been divided > to 5 sub-req status(which I mean detailed here, sorry about the > confusion). > In original implementation, there is only 1 "mds_reint" req counter in > the stats file, and we update that and change it to five specific > mds_reint req, > Then MDS would know what kind of reint request it has been handled. > Not sure whether lustre manual include all the statements of these stats, > you might check that? > >> >> Also, you didn''t answer my question about what types of operations >> will cause each of the different counters to increment. > Hmm, I assume you only ask mds_reint req ? These counters will be > incremented when mds identify the request and > ready to handle it. So > > mds_reint_unlink: unlink a file > mds_reint_create: mknod or mkdir > mds_reint_link: link a file > mds_reint_setattr: chmod/chacl or other setattr meta ops. > mds_reint_rename: rename a file. > > Hope I did not miss your questions this time. > >> Would it be easier and more beneficial to other users to move this to >> lustre-discuss? > Yes. >> >> -mark >> >> > Thanks > WangDi > >
On Jul 29, 2008 14:36 -0400, Mark Seger wrote:> One thing that confuses me about lustre counters, and maybe others, is I > don''t really know what they mean, when they change and in fact how to > stimulate them to change. For example, on my system I''m doing a watch > of /proc/fs/lustre/mdt/MDS/mds/stats and only see 1 reint counter, > because the others are all 0. So I went and did some file renames, and > chmods and sure enough, the other counters did appear. Cool!Yes, this is expected. We dropped the "0" counters because they are very noisy and useless in most contexts.> The easiest thing for me to do is to simply say that reint_setattr > counts the number of setattrs, but that would be a pretty weak > definition. When I changed did a single chmod to 100 files, setattr only > incremented by 1 and I expected it to increment by 100.It should have been incremented by 100, and if it didn''t it is possibly a bug.> want to be the one responsible for the words or all you''re going to see > is ''reint_setattr counts the number of setattr calls'' and I really don''t > think that would be all that useful to anyone."reint_setattr" includes all operations that modify inode attributes, including chmod, chown, touch, etc.>>>>> mds_reint_create 11018837 samples [reqs] 1 1 11018837For mknod and mkdir operations, also used by NFS servers internally when creating files.>>>>> mds_reint_link 51315 samples [reqs] 1 1 51315 51315For hard or symbolic links, like with "ln">>>>> mds_reint_rename 224241 samples [reqs] 1 1 224241 224241For file and directory renames, like with "mv".>>>>> mds_reint_unlink 13109877 samples [reqs] 1 1 13109877For removing files and directories, like with "rm" or "rmdir".>>>>> mds_getxattr 36089 samples [usec] 9 8996 675208 252525110For extended attributes and ACLs, like with "getfattr" or "getfacl".>>>>> mds_setxattr 1230 samples [usec] 123 10110 263367For extended attributes and ACLs, like with "setfattr" or "setfacl". Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
your definition are perfect and I''ll add them to my documentation. With respect to my comment about chmod on 100 files only incrementing the counter once, it did it 100 times this time when I tired it, so never mind... one counter you didn''t mention is getstatus. when does that get updated? -mark Andreas Dilger wrote:> On Jul 29, 2008 14:36 -0400, Mark Seger wrote: > >> One thing that confuses me about lustre counters, and maybe others, is I >> don''t really know what they mean, when they change and in fact how to >> stimulate them to change. For example, on my system I''m doing a watch >> of /proc/fs/lustre/mdt/MDS/mds/stats and only see 1 reint counter, >> because the others are all 0. So I went and did some file renames, and >> chmods and sure enough, the other counters did appear. Cool! >> > > Yes, this is expected. We dropped the "0" counters because they are very > noisy and useless in most contexts. > > >> The easiest thing for me to do is to simply say that reint_setattr >> counts the number of setattrs, but that would be a pretty weak >> definition. When I changed did a single chmod to 100 files, setattr only >> incremented by 1 and I expected it to increment by 100. >> > > It should have been incremented by 100, and if it didn''t it is possibly > a bug. > > >> want to be the one responsible for the words or all you''re going to see >> is ''reint_setattr counts the number of setattr calls'' and I really don''t >> think that would be all that useful to anyone. >> > > "reint_setattr" includes all operations that modify inode attributes, > including chmod, chown, touch, etc. > > >>>>>> mds_reint_create 11018837 samples [reqs] 1 1 11018837 >>>>>> > > For mknod and mkdir operations, also used by NFS servers internally > when creating files. > > >>>>>> mds_reint_link 51315 samples [reqs] 1 1 51315 51315 >>>>>> > > For hard or symbolic links, like with "ln" > > >>>>>> mds_reint_rename 224241 samples [reqs] 1 1 224241 224241 >>>>>> > > For file and directory renames, like with "mv". > > >>>>>> mds_reint_unlink 13109877 samples [reqs] 1 1 13109877 >>>>>> > > For removing files and directories, like with "rm" or "rmdir". > > >>>>>> mds_getxattr 36089 samples [usec] 9 8996 675208 252525110 >>>>>> > > For extended attributes and ACLs, like with "getfattr" or "getfacl". > > >>>>>> mds_setxattr 1230 samples [usec] 123 10110 263367 >>>>>> > > For extended attributes and ACLs, like with "setfattr" or "setfacl". > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. >
On Jul 29, 2008 18:43 -0400, Mark Seger wrote:> your definition are perfect and I''ll add them to my documentation. With > respect to my comment about chmod on 100 files only incrementing the > counter once, it did it 100 times this time when I tired it, so never > mind... > > one counter you didn''t mention is getstatus. when does that get updated?That one is only used once at mount...> -mark > > Andreas Dilger wrote: >> On Jul 29, 2008 14:36 -0400, Mark Seger wrote: >> >>> One thing that confuses me about lustre counters, and maybe others, >>> is I don''t really know what they mean, when they change and in fact >>> how to stimulate them to change. For example, on my system I''m >>> doing a watch of /proc/fs/lustre/mdt/MDS/mds/stats and only see 1 >>> reint counter, because the others are all 0. So I went and did some >>> file renames, and chmods and sure enough, the other counters did >>> appear. Cool! >>> >> >> Yes, this is expected. We dropped the "0" counters because they are very >> noisy and useless in most contexts. >> >> >>> The easiest thing for me to do is to simply say that reint_setattr >>> counts the number of setattrs, but that would be a pretty weak >>> definition. When I changed did a single chmod to 100 files, setattr >>> only incremented by 1 and I expected it to increment by 100. >>> >> >> It should have been incremented by 100, and if it didn''t it is possibly >> a bug. >> >> >>> want to be the one responsible for the words or all you''re going to >>> see is ''reint_setattr counts the number of setattr calls'' and I >>> really don''t think that would be all that useful to anyone. >>> >> >> "reint_setattr" includes all operations that modify inode attributes, >> including chmod, chown, touch, etc. >> >> >>>>>>> mds_reint_create 11018837 samples [reqs] 1 1 >>>>>>> 11018837 >> >> For mknod and mkdir operations, also used by NFS servers internally >> when creating files. >> >> >>>>>>> mds_reint_link 51315 samples [reqs] 1 1 51315 51315 >>>>>>> >> >> For hard or symbolic links, like with "ln" >> >> >>>>>>> mds_reint_rename 224241 samples [reqs] 1 1 224241 224241 >>>>>>> >> >> For file and directory renames, like with "mv". >> >> >>>>>>> mds_reint_unlink 13109877 samples [reqs] 1 1 >>>>>>> 13109877 >> >> For removing files and directories, like with "rm" or "rmdir". >> >> >>>>>>> mds_getxattr 36089 samples [usec] 9 8996 675208 252525110 >>>>>>> >> >> For extended attributes and ACLs, like with "getfattr" or "getfacl". >> >> >>>>>>> mds_setxattr 1230 samples [usec] 123 10110 >>>>>>> 263367 >> >> For extended attributes and ACLs, like with "setfattr" or "setfacl". >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Sr. Staff Engineer, Lustre Group >> Sun Microsystems of Canada, Inc. >>Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Andreas Dilger wrote:> On Jul 29, 2008 18:43 -0400, Mark Seger wrote: > >> your definition are perfect and I''ll add them to my documentation. With >> respect to my comment about chmod on 100 files only incrementing the >> counter once, it did it 100 times this time when I tired it, so never >> mind... >> >> one counter you didn''t mention is getstatus. when does that get updated? >> > > That one is only used once at mount... >ahhh, I was playing with mounting/unmounting, watching connect/disconnect change and did see that one change with mounts. Back on the subject of collectl, the general philosophy is to show rates as counters/sec and so when they hit an unexpected high or low, they then to jump out at you. Given that mount/unmounts typically happen in large chunks and not all that often, I''m not sure showing the rates of them are all that useful. I suppose one might also make that argument about things like statfs, getattr - the only time I was able to make them change was in response to lfs commands. Might that logic also be applied to extended attributes and acl counters which I suspect also fall into the category of slowly changing counters? In fact if you''re sampling data once every 10 seconds (which is the daemon default) and a counter is incremented by 4 or less during that sample, it will show up at a rate of 0/sec! On the other hand, it seems like the ''reint'' counters are the ones that tend to change a lot. Perhaps a clue is they''re all prefaced with reint which leads me to ask if there is some simple definition of what reint actually means other than ''reintegrated operations''? Perhaps such a definition will help explain why setattr is a reint counter but getattr is not. In fact, I have seen getattr_lock change a lot more than getattr. What is the difference between the 2 (obviously the latter is some sort of lock but it must be used more than just when incrementing getattr since they don''t change together)? That all said, it feels like the data to report is all the reints, getattr, getattr_lock and sync. As a side note, collectl will collect all the mds data saving it in its ''raw'' file, still making it possible to still get at even if not reported in a standard display format. -mark>> -mark >> >> Andreas Dilger wrote: >> >>> On Jul 29, 2008 14:36 -0400, Mark Seger wrote: >>> >>> >>>> One thing that confuses me about lustre counters, and maybe others, >>>> is I don''t really know what they mean, when they change and in fact >>>> how to stimulate them to change. For example, on my system I''m >>>> doing a watch of /proc/fs/lustre/mdt/MDS/mds/stats and only see 1 >>>> reint counter, because the others are all 0. So I went and did some >>>> file renames, and chmods and sure enough, the other counters did >>>> appear. Cool! >>>> >>>> >>> Yes, this is expected. We dropped the "0" counters because they are very >>> noisy and useless in most contexts. >>> >>> >>> >>>> The easiest thing for me to do is to simply say that reint_setattr >>>> counts the number of setattrs, but that would be a pretty weak >>>> definition. When I changed did a single chmod to 100 files, setattr >>>> only incremented by 1 and I expected it to increment by 100. >>>> >>>> >>> It should have been incremented by 100, and if it didn''t it is possibly >>> a bug. >>> >>> >>> >>>> want to be the one responsible for the words or all you''re going to >>>> see is ''reint_setattr counts the number of setattr calls'' and I >>>> really don''t think that would be all that useful to anyone. >>>> >>>> >>> "reint_setattr" includes all operations that modify inode attributes, >>> including chmod, chown, touch, etc. >>> >>> >>> >>>>>>>> mds_reint_create 11018837 samples [reqs] 1 1 >>>>>>>> 11018837 >>>>>>>> >>> For mknod and mkdir operations, also used by NFS servers internally >>> when creating files. >>> >>> >>> >>>>>>>> mds_reint_link 51315 samples [reqs] 1 1 51315 51315 >>>>>>>> >>>>>>>> >>> For hard or symbolic links, like with "ln" >>> >>> >>> >>>>>>>> mds_reint_rename 224241 samples [reqs] 1 1 224241 224241 >>>>>>>> >>>>>>>> >>> For file and directory renames, like with "mv". >>> >>> >>> >>>>>>>> mds_reint_unlink 13109877 samples [reqs] 1 1 >>>>>>>> 13109877 >>>>>>>> >>> For removing files and directories, like with "rm" or "rmdir". >>> >>> >>> >>>>>>>> mds_getxattr 36089 samples [usec] 9 8996 675208 252525110 >>>>>>>> >>>>>>>> >>> For extended attributes and ACLs, like with "getfattr" or "getfacl". >>> >>> >>> >>>>>>>> mds_setxattr 1230 samples [usec] 123 10110 >>>>>>>> 263367 >>>>>>>> >>> For extended attributes and ACLs, like with "setfattr" or "setfacl". >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger >>> Sr. Staff Engineer, Lustre Group >>> Sun Microsystems of Canada, Inc. >>> >>> > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. >
Hi Mark,> useful. I suppose one might also make that argument about things like > statfs, getattr - the only time I was able to make them change was in > response to lfs commands. Might that logic also be applied to > extended attributes and acl counters which I suspect also fall into > the category of slowly changing counters?If you have ACLs enabled on your MDS, then every "ls -l" will induce getxattr()s and the mds_getxattr counter will be increased by as much. So this can change quickly. mds_setxattr, on the other hand, may change less often, since you usually set ACLs less often than you list files. But it can still be interesting to see if mds_setxattr goes through the roof.> On the other hand, it seems like the ''reint'' counters are the ones > that tend to change a lot. Perhaps a clue is they''re all prefaced > with reint which leads me to ask if there is some simple definition > of what reint actually means other than ''reintegrated operations''?I''d bet on "request identification" or something along those lines.> Perhaps such a definition will help explain why setattr is a reint > counter but getattr is not. In fact, I have seen getattr_lock change > a lot more than getattr. What is the difference between the 2 > (obviously the latter is some sort of lock but it must be used more > than just when incrementing getattr since they don''t change > together)?I''m only speculating here, but I believe that extended attributes which are modifiable by a user on a client (like ACLs) are counted in *_xattr, while internal extended attributes used by the MDS, are counted in gettatr.> That all said, it feels like the data to report is all the reints, > getattr, getattr_lock and sync.I would also be interested in seeing (dis)connect (this can probably reveal network problems, if it increases too much), as well as quotactl and get/setxattr, since I use quotas and ACLs. :) Cheers, -- Kilian
I just did an interesting experiment. If I run the command ''lfs df'' in a tight loop, the reint_statfs counters increments once but if I stick in a sleep 1, it increments it for every call. is this a bug or a feature? Is it doing some kind of short-lived cache lookup in the first can but not the second? -mark Kilian CAVALOTTI wrote:> Hi Mark, > > >> useful. I suppose one might also make that argument about things like >> statfs, getattr - the only time I was able to make them change was in >> response to lfs commands. Might that logic also be applied to >> extended attributes and acl counters which I suspect also fall into >> the category of slowly changing counters? >> > > If you have ACLs enabled on your MDS, then every "ls -l" will induce > getxattr()s and the mds_getxattr counter will be increased by as much. > So this can change quickly. mds_setxattr, on the other hand, may change > less often, since you usually set ACLs less often than you list files. > But it can still be interesting to see if mds_setxattr goes through the > roof. > > >> On the other hand, it seems like the ''reint'' counters are the ones >> that tend to change a lot. Perhaps a clue is they''re all prefaced >> with reint which leads me to ask if there is some simple definition >> of what reint actually means other than ''reintegrated operations''? >> > > I''d bet on "request identification" or something along those lines. > > >> Perhaps such a definition will help explain why setattr is a reint >> counter but getattr is not. In fact, I have seen getattr_lock change >> a lot more than getattr. What is the difference between the 2 >> (obviously the latter is some sort of lock but it must be used more >> than just when incrementing getattr since they don''t change >> together)? >> > > I''m only speculating here, but I believe that extended attributes which > are modifiable by a user on a client (like ACLs) are counted in > *_xattr, while internal extended attributes used by the MDS, are > counted in gettatr. > > >> That all said, it feels like the data to report is all the reints, >> getattr, getattr_lock and sync. >> > > I would also be interested in seeing (dis)connect (this can probably > reveal network problems, if it increases too much), as well as quotactl > and get/setxattr, since I use quotas and ACLs. :) > > > Cheers, >
On Jul 30, 2008 14:52 -0400, Mark Seger wrote:> I just did an interesting experiment. If I run the command ''lfs df'' in > a tight loop, the reint_statfs counters increments once but if I stick > in a sleep 1, it increments it for every call. is this a bug or a > feature? Is it doing some kind of short-lived cache lookup in the first > can but not the second?Yes, the client will cache the "df" data for up to a second, to allow something like "grep [0-9] /proc/fs/lustre/{mdc,osc}/*/{kbytes,files}*" to avoid doing STATFS RPCs for each file. There was a bug fixed recently in the caching of statfs data, that if the statfs was requested too frequently it won''t refresh properly from the servers.> Kilian CAVALOTTI wrote: >>> useful. I suppose one might also make that argument about things like >>> statfs, getattr - the only time I was able to make them change was in >>> response to lfs commands. Might that logic also be applied to >>> extended attributes and acl counters which I suspect also fall into >>> the category of slowly changing counters? >> >> If you have ACLs enabled on your MDS, then every "ls -l" will induce >> getxattr()s and the mds_getxattr counter will be increased by as much. >> So this can change quickly. mds_setxattr, on the other hand, may change >> less often, since you usually set ACLs less often than you list files. >> But it can still be interesting to see if mds_setxattr goes through the >> roof. >> >> >>> On the other hand, it seems like the ''reint'' counters are the ones >>> that tend to change a lot. Perhaps a clue is they''re all prefaced >>> with reint which leads me to ask if there is some simple definition >>> of what reint actually means other than ''reintegrated operations''? >>> >> >> I''d bet on "request identification" or something along those lines. >> >> >>> Perhaps such a definition will help explain why setattr is a reint >>> counter but getattr is not. In fact, I have seen getattr_lock change >>> a lot more than getattr. What is the difference between the 2 >>> (obviously the latter is some sort of lock but it must be used more >>> than just when incrementing getattr since they don''t change >>> together)? >>> >> >> I''m only speculating here, but I believe that extended attributes which >> are modifiable by a user on a client (like ACLs) are counted in >> *_xattr, while internal extended attributes used by the MDS, are >> counted in gettatr. >> >> >>> That all said, it feels like the data to report is all the reints, >>> getattr, getattr_lock and sync. >> >> I would also be interested in seeing (dis)connect (this can probably >> reveal network problems, if it increases too much), as well as quotactl >> and get/setxattr, since I use quotas and ACLs. :) >> >> >> Cheers, >>Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Kilian CAVALOTTI wrote:> Hi Mark, > > >> useful. I suppose one might also make that argument about things like >> statfs, getattr - the only time I was able to make them change was in >> response to lfs commands. Might that logic also be applied to >> extended attributes and acl counters which I suspect also fall into >> the category of slowly changing counters? >> > > If you have ACLs enabled on your MDS, then every "ls -l" will induce > getxattr()s and the mds_getxattr counter will be increased by as much. > So this can change quickly. mds_setxattr, on the other hand, may change > less often, since you usually set ACLs less often than you list files. > But it can still be interesting to see if mds_setxattr goes through the > roof. >so here''s the test I did. I created 200 files and set acls on all of them. setxattr changed by 200. then I did an ls -l on the directory with the files in it and getxattr changed by 1. when I removed all the files reint changed by 200. I should quickly point out this was done on a 1.6.4 system as I''m waiting for access to remount the mds on my 1.6.5 systems with acls enabled. -mark
On Jul 30, 2008 18:15 -0400, Mark Seger wrote:> so here''s the test I did. I created 200 files and set acls on all of > them. setxattr changed by 200. then I did an ls -l on the directory > with the files in it and getxattr changed by 1. when I removed all the > files reint changed by 200. I should quickly point out this was done on > a 1.6.4 system as I''m waiting for access to remount the mds on my 1.6.5 > systems with acls enabled.Normally the MDS will send the ACL data along with the first "getattr" RPC, to avoid the overhead of sending a separate RPC for each stat(). Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.