Hi list ! We have a very busy webserver hosted in a clustered environment where the document root and data is on a GFS2 partition off a fiber-attached disk array. Now on busy moments, I can see in htop, nmon that there is a fair percentage of cpu that is waiting for I/O. In nmon, I can spot that the most busy block device correspond to our gfs2 partition where many times, it shows that it's 100% busy and is read all along. Now, I want to know what files are being waited for. With lsof I can get a listing of open files, but it doesn't gives me if a file is just opended in ram or if it's being waited for... What tools besides lsof, nmon, htop, atop can help me find that info ? I am under RHEL/CentOS 6.1. Thanks
Hi Nicolas, While this doesn't exactly answer your question, I was wondering what scheduler you were using on your GFS2 (Note: I have not used this file system before) block. You can find this by issuing 'cat /sys/block/<insert block device>/queue/scheduler' ? By default the system uses cfq, which will show up as [cfq] when catting the scheduler as I showed above. This is not the most optimal scheduler for a webserver. In most cases you'd be better off with deadline or noop. Not being familiar with GFS2 myself, I did skim this article, which makes me think noop would be the better choice: http://www.redhat.com/archives/linux-cluster/2010-June/msg00027.html This could be why you are seeing the processes waiting on I/O. Chad M. Gross On Tue, Sep 20, 2011 at 2:55 PM, Nicolas Ross <rossnick-lists at cybercat.ca>wrote:> Hi list ! > > We have a very busy webserver hosted in a clustered environment where the > document root and data is on a GFS2 partition off a fiber-attached disk > array. > > Now on busy moments, I can see in htop, nmon that there is a fair > percentage > of cpu that is waiting for I/O. In nmon, I can spot that the most busy > block > device correspond to our gfs2 partition where many times, it shows that > it's > 100% busy and is read all along. > > Now, I want to know what files are being waited for. With lsof I can get a > listing of open files, but it doesn't gives me if a file is just opended in > ram or if it's being waited for... > > What tools besides lsof, nmon, htop, atop can help me find that info ? > > I am under RHEL/CentOS 6.1. > > Thanks > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20110920/b199f267/attachment-0002.html>
> Hi Nicolas, > > While this doesn't exactly answer your question, I was wondering what > scheduler you were using on your GFS2 (Note: I have not used this file > system before) block. You can find this by issuing 'cat /sys/block/<insert > block device>/queue/scheduler' ? > > By default the system uses cfq, which will show up as [cfq] when catting > the scheduler as I showed above. This is not the most optimal scheduler > for a webserver. In most cases you'd be better off with deadline or noop. > Not being familiar with GFS2 myself, I did skim this article, which makes > me think noop would be the better choice: > > http://www.redhat.com/archives/linux-cluster/2010-June/msg00027.html > > This could be why you are seeing the processes waiting on I/O. >In my case, /sys/block/dm-9/queue/scheduler show : none and /sys/block/sdb/queue/scheduler shows "noop anticipatory deadline [cfq]". Since this is a production cluster, I do not want to make changes to it just now. I will ask advice from RHEL support for setting this. But that seems logical. In the meen time, I'd still like to find a tool to know what files are requeted to the filesystem and what ones are being waited for... Thanks