Ross, Justin
2012-May-21 08:32 UTC
[Lustre-discuss] lfs find doesn''t find files but lustre seems to be using them
I am stuck on an issue with an OST. I am in the process of replacing some disks in the Lustre file system under a few OSTs. The process has been to drain the OSTs by marking the OST inactive, lfs find, copy to a new name, move to the old name. It has been working very well for all of the OSTs. I have one particular OST that has about 4.5TB of data on it and lfs find doesn''t find any files on it at all. My assumption was that Lustre doesn''t know about any files on that OST. I assumed a full fsck is in order. But the real weird catch is that just watching the OST it is in use. I have set it inactive so no new files are written to it. There are a good bit of reads coming from that OST. I watch the OSS with iostat and all of the other OSTs on that system are idle, already drained. This one OST has plenty of read activity. So it seems that Lustre does know about files on this OST, but just not with lfs find... Any ideas how to find these files and move them? I have mounted the OST with ldiskfs and everything looks ok. There are plenty of files there. I have file striping set to one OST. I am running 1.8.5. I will upgrade to 1.8.7 as soon as I can, but am curious if anyone else has seen anything like this. I can''t find any similar threads. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120521/e062833a/attachment.html
Michael Barnes
2012-May-21 13:14 UTC
[Lustre-discuss] lfs find doesn''t find files but lustre seems to be using them
Justin, Do you have holes in your OST list? Meaning, is this OST something like: lfs df for me shows: ? lustre-OST001b_UUID 15378613900 8631157344 5966252368 59% /lustre[OST:27] lustre-OST0029_UUID 7689826700 5611663868 1687542044 77% /lustre[OST:41] ? Note, there are no OST:28,29,? and no OST001c,1d,? I can lfs find files on lustre-OST001b_UUID but never on lustre-OST0029_UUID. I did some reconfiguring of OSTs a while back and holes got into my lustre system, and the OST after hole is never found with lfs find. lfs getstripe shows that the files are indeed there and on the OST indexed as 41. -mb On May 21, 2012, at 4:32 AM, Ross, Justin wrote:> I am stuck on an issue with an OST. I am in the process of replacing some disks in the Lustre file system under a few OSTs. The process has been to drain the OSTs by marking the OST inactive, lfs find, copy to a new name, move to the old name. It has been working very well for all of the OSTs. I have one particular OST that has about 4.5TB of data on it and lfs find doesn''t find any files on it at all. My assumption was that Lustre doesn''t know about any files on that OST. I assumed a full fsck is in order. But the real weird catch is that just watching the OST it is in use. I have set it inactive so no new files are written to it. There are a good bit of reads coming from that OST. I watch the OSS with iostat and all of the other OSTs on that system are idle, already drained. This one OST has plenty of read activity. So it seems that Lustre does know about files on this OST, but just not with lfs find... Any ideas how to find these files and move them? I have mounted the OST with ldiskfs and everything looks ok. There are plenty of files there. > > I have file striping set to one OST. I am running 1.8.5. I will upgrade to 1.8.7 as soon as I can, but am curious if anyone else has seen anything like this. I can''t find any similar threads. > > Thanks, > Justin > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- +----------------------------------------------- | Michael Barnes | | Thomas Jefferson National Accelerator Facility | Scientific Computing Group | 12000 Jefferson Ave. | Newport News, VA 23606 | (757) 269-7634 +-----------------------------------------------
Ross, Justin
2012-May-21 13:17 UTC
[Lustre-discuss] lfs find doesn''t find files but lustre seems to be using them
Yes. Actually that is exactly it. As you say lfs getstripe works just fine. So I am finding all of the files that way. Thanks for the info on that one. I was worried that things were very corrupted... But I guess just a bug. Thanks, Justin -----Original Message----- From: Michael Barnes [mailto:Michael.Barnes at jlab.org] Sent: Monday, May 21, 2012 8:15 AM To: Ross, Justin Cc: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] lfs find doesn''t find files but lustre seems to be using them Justin, Do you have holes in your OST list? Meaning, is this OST something like: lfs df for me shows: ... lustre-OST001b_UUID 15378613900 8631157344 5966252368 59% /lustre[OST:27] lustre-OST0029_UUID 7689826700 5611663868 1687542044 77% /lustre[OST:41] ... Note, there are no OST:28,29,... and no OST001c,1d,... I can lfs find files on lustre-OST001b_UUID but never on lustre-OST0029_UUID. I did some reconfiguring of OSTs a while back and holes got into my lustre system, and the OST after hole is never found with lfs find. lfs getstripe shows that the files are indeed there and on the OST indexed as 41. -mb On May 21, 2012, at 4:32 AM, Ross, Justin wrote:> I am stuck on an issue with an OST. I am in the process of replacing some disks in the Lustre file system under a few OSTs. The process has been to drain the OSTs by marking the OST inactive, lfs find, copy to a new name, move to the old name. It has been working very well for all of the OSTs. I have one particular OST that has about 4.5TB of data on it and lfs find doesn''t find any files on it at all. My assumption was that Lustre doesn''t know about any files on that OST. I assumed a full fsck is in order. But the real weird catch is that just watching the OST it is in use. I have set it inactive so no new files are written to it. There are a good bit of reads coming from that OST. I watch the OSS with iostat and all of the other OSTs on that system are idle, already drained. This one OST has plenty of read activity. So it seems that Lustre does know about files on this OST, but just not with lfs find... Any ideas how to find these files and move them? I have mounted the OST with ldiskfs and everything looks ok. There are plenty of files there. > > I have file striping set to one OST. I am running 1.8.5. I will upgrade to 1.8.7 as soon as I can, but am curious if anyone else has seen anything like this. I can''t find any similar threads. > > Thanks, > Justin > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- +----------------------------------------------- | Michael Barnes | | Thomas Jefferson National Accelerator Facility Scientific Computing | Group | 12000 Jefferson Ave. | Newport News, VA 23606 | (757) 269-7634 +-----------------------------------------------
Andreas Dilger
2012-May-21 17:31 UTC
[Lustre-discuss] lfs find doesn''t find files but lustre seems to be using them
Could you please file a ticket in Jira for this problem. Cheers, Andreas On 2012-05-21, at 7:17, "Ross, Justin" <ross at smu.edu> wrote:> Yes. Actually that is exactly it. As you say lfs getstripe works just fine. So I am finding all of the files that way. > > Thanks for the info on that one. I was worried that things were very corrupted... But I guess just a bug. > > Thanks, > Justin > > -----Original Message----- > From: Michael Barnes [mailto:Michael.Barnes at jlab.org] > Sent: Monday, May 21, 2012 8:15 AM > To: Ross, Justin > Cc: lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] lfs find doesn''t find files but lustre seems to be using them > > > Justin, > > Do you have holes in your OST list? > > Meaning, is this OST something like: > > lfs df for me shows: > > ... > lustre-OST001b_UUID 15378613900 8631157344 5966252368 59% /lustre[OST:27] > lustre-OST0029_UUID 7689826700 5611663868 1687542044 77% /lustre[OST:41] > ... > > Note, there are no OST:28,29,... and no OST001c,1d,... > > I can lfs find files on lustre-OST001b_UUID but never on lustre-OST0029_UUID. I did some reconfiguring of OSTs a while back and holes got into my lustre system, and the OST after hole is never found with lfs find. lfs getstripe shows that the files are indeed there and on the OST indexed as 41. > > -mb > > > On May 21, 2012, at 4:32 AM, Ross, Justin wrote: > >> I am stuck on an issue with an OST. I am in the process of replacing some disks in the Lustre file system under a few OSTs. The process has been to drain the OSTs by marking the OST inactive, lfs find, copy to a new name, move to the old name. It has been working very well for all of the OSTs. I have one particular OST that has about 4.5TB of data on it and lfs find doesn''t find any files on it at all. My assumption was that Lustre doesn''t know about any files on that OST. I assumed a full fsck is in order. But the real weird catch is that just watching the OST it is in use. I have set it inactive so no new files are written to it. There are a good bit of reads coming from that OST. I watch the OSS with iostat and all of the other OSTs on that system are idle, already drained. This one OST has plenty of read activity. So it seems that Lustre does know about files on this OST, but just not with lfs find... Any ideas how to find these files and move them? I have > mounted the OST with ldiskfs and everything looks ok. There are plenty of files there. >> >> I have file striping set to one OST. I am running 1.8.5. I will upgrade to 1.8.7 as soon as I can, but am curious if anyone else has seen anything like this. I can''t find any similar threads. >> >> Thanks, >> Justin >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > -- > +----------------------------------------------- > | Michael Barnes > | > | Thomas Jefferson National Accelerator Facility Scientific Computing > | Group > | 12000 Jefferson Ave. > | Newport News, VA 23606 > | (757) 269-7634 > +----------------------------------------------- > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss