Hi, Is there a lustre tool similar to LSOF -N (for NFS) to help identify open files on an OSS/OST ? We are seeing some unusual load patterns and have run out of ideas on diagnosing it and tracing the issue back to the source. thanks in advance, Mark -- mark day | mark.day at rsp.com.au -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20111215/228d2e9f/attachment.html
On Thursday, December 15, 2011 02:49:41 AM Mark Day wrote:> Hi, > > Is there a lustre tool similar to LSOF -N (for NFS) to help identify open > files on an OSS/OST ? We are seeing some unusual load patterns and have > run out of ideas on diagnosing it and tracing the issue back to the > source.Exactly what are you looking for? The -N option to lsof is nothing special (adds no NFS specific data) other than displaying only NFS files. If this is what you''re looking for then you can simply give lsof the mountpoint to your lustre filesystem: "lsof /path/to/lustre" (Note: trailing / not allowed) and only open files on that filesystem will be shown. Without any options lsof will show open files on any type of filesystem. /Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20111215/cde38da1/attachment-0001.bin
Sure, I was vague. We''d like to know what files are currently being accessed on an OST from all clients. Is it possible to query the MDS/T or OSS/T for this information? ta, Mark ----- Original Message ----- From: "Peter Kjellstr?m" <cap at nsc.liu.se> To: lustre-discuss at lists.lustre.org Sent: Thursday, 15 December, 2011 11:29:46 PM Subject: Re: [Lustre-discuss] LSOF for Lustre On Thursday, December 15, 2011 02:49:41 AM Mark Day wrote:> Hi, > > Is there a lustre tool similar to LSOF -N (for NFS) to help identify open > files on an OSS/OST ? We are seeing some unusual load patterns and have > run out of ideas on diagnosing it and tracing the issue back to the > source.Exactly what are you looking for? The -N option to lsof is nothing special (adds no NFS specific data) other than displaying only NFS files. If this is what you''re looking for then you can simply give lsof the mountpoint to your lustre filesystem: "lsof /path/to/lustre" (Note: trailing / not allowed) and only open files on that filesystem will be shown. Without any options lsof will show open files on any type of filesystem. /Peter _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20111216/a713f71f/attachment.html
On 2011-12-15, at 4:34 PM, Mark Day wrote:> We''d like to know what files are currently being accessed on an OST from all clients. > Is it possible to query the MDS/T or OSS/T for this information?The OST doesn''t have any open/close information. As yet there is no open file information accessible on the MDT by userspace, though this is something that has been requested in the past and probably makes sense to implement at some point.> From: "Peter Kjellstr?m" <cap at nsc.liu.se> > To: lustre-discuss at lists.lustre.org > Sent: Thursday, 15 December, 2011 11:29:46 PM > Subject: Re: [Lustre-discuss] LSOF for Lustre > > On Thursday, December 15, 2011 02:49:41 AM Mark Day wrote: > > Hi, > > > > Is there a lustre tool similar to LSOF -N (for NFS) to help identify open > > files on an OSS/OST ? We are seeing some unusual load patterns and have > > run out of ideas on diagnosing it and tracing the issue back to the > > source. > > Exactly what are you looking for? > > The -N option to lsof is nothing special (adds no NFS specific data) other > than displaying only NFS files. If this is what you''re looking for then you > can simply give lsof the mountpoint to your lustre filesystem: > > "lsof /path/to/lustre" (Note: trailing / not allowed) > > and only open files on that filesystem will be shown. Without any options lsof > will show open files on any type of filesystem. > > /Peter > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc.
peter.chiu at stfc.ac.uk
2011-Dec-16 13:05 UTC
[Lustre-discuss] How to re-activate "inactive device" on client
Dear all, We have an odd problem on a lustre client in that a couple of OST elements have become "inactive". However, lctl dl shows all devices are in "UP" state. The "UP" status is also echoed on the mds as well as on another client - please see below. I have tried rebooting this client, but with no joy. I have also searched around the discussion list, and can find a few others had experienced similar problems before. But unfortunatley none of them has reported back with a good recovery procedure, other than a complete reformatting of the lustre storage. So can I trouble you experts to give me some clue as to how to recover from this? Many thanks. Regards, Peter PS. Client: SLES 11 with 2.6.32.29 kernel, Lustre 1.8.5 ================================================================================== Client Capuchin: capuchin:~ # lfs df -h UUID bytes Used Available Use% Mounted on ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0] ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0] ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1] ceda3-OST0002_UUID : inactive device ceda3-OST0003_UUID : inactive device ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4] ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5] ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6] filesystem summary: 35.8T 34.0T 32.5G 94% /disks/ceda3 capuchin:~ # lctl dl 0 UP mgc MGC130.246.191.64 at tcp be6c0da2-275f-c59d-fa3a-c792134a0d9d 5 1 UP lov ceda3-clilov-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 4 2 UP mdc ceda3-MDT0000-mdc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 3 UP osc ceda3-OST0000-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 4 UP osc ceda3-OST0001-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 5 UP osc ceda3-OST0002-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 6 UP osc ceda3-OST0003-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 7 UP osc ceda3-OST0004-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 8 UP osc ceda3-OST0005-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 9 UP osc ceda3-OST0006-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 capuchin:~ # Client2 cmip-dn1: [root at aspre script]# ssh cmip-dn1 lfs df -h UUID bytes Used Available Use% Mounted on ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0] ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0] ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1] ceda3-OST0002_UUID 7.2T 451.6M 6.8T 0% /disks/ceda3[OST:2] ceda3-OST0003_UUID 7.2T 451.8M 6.8T 0% /disks/ceda3[OST:3] ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4] ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5] ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6] filesystem summary: 50.1T 34.0T 13.6T 67% /disks/ceda3 Mds: [root at mds02 ~]# lctl dl 0 UP mgc MGC130.246.191.64 at tcp 8aa29420-11f1-8ca3-a361-ce5135a09be2 5 1 UP mdt MDS MDS_uuid 3 2 UP lov ceda3-mdtlov ceda3-mdtlov_UUID 4 3 UP mds ceda3-MDT0000 ceda3-MDT0000_UUID 9 4 UP osc ceda3-OST0000-osc ceda3-mdtlov_UUID 5 5 UP osc ceda3-OST0001-osc ceda3-mdtlov_UUID 5 6 UP osc ceda3-OST0002-osc ceda3-mdtlov_UUID 5 7 UP osc ceda3-OST0003-osc ceda3-mdtlov_UUID 5 8 UP osc ceda3-OST0004-osc ceda3-mdtlov_UUID 5 9 UP osc ceda3-OST0005-osc ceda3-mdtlov_UUID 5 10 UP osc ceda3-OST0006-osc ceda3-mdtlov_UUID 5 11 UP lov ceda4-mdtlov ceda4-mdtlov_UUID 4 12 UP mds ceda4-MDT0000 ceda4-MDT0000_UUID 9 13 UP osc ceda4-OST0000-osc ceda4-mdtlov_UUID 5 14 UP osc ceda4-OST0001-osc ceda4-mdtlov_UUID 5 15 UP osc ceda4-OST0002-osc ceda4-mdtlov_UUID 5 16 UP osc ceda4-OST0003-osc ceda4-mdtlov_UUID 5 17 UP osc ceda4-OST0004-osc ceda4-mdtlov_UUID 5 18 UP osc ceda4-OST0005-osc ceda4-mdtlov_UUID 5 [root at mds02 ~]# -- Scanned by iCritical.