Sridharan Ramaswamy (srramasw)
2007-Jan-23 11:39 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
I''m tring to measure the overhead of having Lustre on the system. Among other things, I noticed it starts lot of processes, root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] I can account about 200 processes related to Lustre on an OST node. That seems a lot considering we would like to run other applications on the same node. Is it possible to tune these number of processes down? Say reduce 32 ldlm process to about 8. What kind of performance degradation to expect if we can do that? thanks, Sridharan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070123/e7df51a8/attachment.html
Sridharan Ramaswamy (srramasw)
2007-Feb-05 12:13 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
Hi Andreas, I didn''t see any reply earlier. Can you please comment on this question on number of Lustre processes? - Sridharan ________________________________ From: lustre-discuss-bounces@clusterfs.com [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of Sridharan Ramaswamy (srramasw) Sent: Tuesday, January 23, 2007 10:39 AM To: lustre-discuss@clusterfs.com Subject: [Lustre-discuss] Number of processes in Lustre tunable? I''m tring to measure the overhead of having Lustre on the system. Among other things, I noticed it starts lot of processes, root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] I can account about 200 processes related to Lustre on an OST node. That seems a lot considering we would like to run other applications on the same node. Is it possible to tune these number of processes down? Say reduce 32 ldlm process to about 8. What kind of performance degradation to expect if we can do that? thanks, Sridharan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070205/84dc26ae/attachment.html
Sridharan Ramaswamy (srramasw)
2007-Feb-07 12:48 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
Hi Aaron, Thanks for the reply. I agree ultimately the amount of CPU usage is what that matters. But still seems high occupying kernel data structures. Infact we are planning to run multiple MDTs for different logical-volumes on same node. The worry is the number of processes might dramatically increase, like 400 processes for two Lustre MDT set. We are designing for a low-end system with system memory as low as 256MB, these resources are precious :-) - Sridharan> -----Original Message----- > From: Aaron Knister [mailto:aaron@iges.org] > Sent: Wednesday, February 07, 2007 8:35 AM > To: Sridharan Ramaswamy (srramasw) > Cc: Andreas Dilger; lustre-discuss@clusterfs.com > Subject: Re: [Lustre-discuss] Number of processes in Lustre tunable? > > There is no real reason that I see to reduce the number of Lustre > processes. It won''t cause any ill effects on other processes running. > Lustre is heavily multi threaded hence the large number of > lustre procs. > > Sridharan Ramaswamy (srramasw) wrote: > > Hi Andreas, > > > > I didn''t see any reply earlier. Can you please comment on this > > question on number of Lustre processes? > > > > - Sridharan > > > > > -------------------------------------------------------------- > ---------- > > *From:* lustre-discuss-bounces@clusterfs.com > > [mailto:lustre-discuss-bounces@clusterfs.com] *On Behalf Of > > *Sridharan Ramaswamy (srramasw) > > *Sent:* Tuesday, January 23, 2007 10:39 AM > > *To:* lustre-discuss@clusterfs.com > > *Subject:* [Lustre-discuss] Number of processes in > Lustre tunable? > > > > > > I''m tring to measure the overhead of having Lustre on the > > system. Among other things, I noticed it starts lot of > processes, > > > > root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] > > root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] > > root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] > > root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] > > root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] > > I can account about 200 processes related to Lustre on an OST > > node. That seems a lot considering we would like to run other > > applications on the same node. Is it possible to tune > these number > > of processes down? Say reduce 32 ldlm process to about 8. What > > kind of performance degradation to expect if we can do that? > > > > thanks, > > Sridharan > > > > > -------------------------------------------------------------- > ---------- > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss@clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > >
Aaron Knister
2007-Feb-07 22:48 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
There is no real reason that I see to reduce the number of Lustre processes. It won''t cause any ill effects on other processes running. Lustre is heavily multi threaded hence the large number of lustre procs. Sridharan Ramaswamy (srramasw) wrote:> Hi Andreas, > > I didn''t see any reply earlier. Can you please comment on this > question on number of Lustre processes? > > - Sridharan > > ------------------------------------------------------------------------ > *From:* lustre-discuss-bounces@clusterfs.com > [mailto:lustre-discuss-bounces@clusterfs.com] *On Behalf Of > *Sridharan Ramaswamy (srramasw) > *Sent:* Tuesday, January 23, 2007 10:39 AM > *To:* lustre-discuss@clusterfs.com > *Subject:* [Lustre-discuss] Number of processes in Lustre tunable? > > > I''m tring to measure the overhead of having Lustre on the > system. Among other things, I noticed it starts lot of processes, > > root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] > root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] > root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] > root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] > root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] > I can account about 200 processes related to Lustre on an OST > node. That seems a lot considering we would like to run other > applications on the same node. Is it possible to tune these number > of processes down? Say reduce 32 ldlm process to about 8. What > kind of performance degradation to expect if we can do that? > > thanks, > Sridharan > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
Andreas Dilger
2007-Feb-12 00:39 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
On Jan 23, 2007 10:39 -0800, Sridharan Ramaswamy (srramasw) wrote:> I''m tring to measure the overhead of having Lustre on the system. Among > other things, I noticed it starts lot of processes, > > root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] > root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] > root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] > root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] > root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] > > I can account about 200 processes related to Lustre on an OST node. That > seems a lot considering we would like to run other applications on the > same node. Is it possible to tune these number of processes down? Say > reduce 32 ldlm process to about 8. What kind of performance degradation > to expect if we can do that?You can reduce the MDS and OST process count via modprobe.conf: option mds mds_num_threads=NNN option ost ost_num_threads=NNN If this isn''t good enough, you can change the count at compile time in lustre/include/linux/lustre-net.h In 1.6 the number of threads will be more dynamic, depending on load. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Sridharan Ramaswamy (srramasw)
2007-Feb-12 00:51 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
Thanks Andreas. I''ll try a lower value for the num_threads=NNN. How low can these values be? - Sridharan> -----Original Message----- > From: Andreas Dilger [mailto:adilger@clusterfs.com] > Sent: Sunday, February 11, 2007 11:39 PM > To: Sridharan Ramaswamy (srramasw) > Cc: lustre-discuss@clusterfs.com > Subject: Re: [Lustre-discuss] Number of processes in Lustre tunable? > > On Jan 23, 2007 10:39 -0800, Sridharan Ramaswamy (srramasw) wrote: > > I''m tring to measure the overhead of having Lustre on the > system. Among > > other things, I noticed it starts lot of processes, > > > > root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] > > root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] > > root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] > > root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] > > root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] > > > > I can account about 200 processes related to Lustre on an > OST node. That > > seems a lot considering we would like to run other > applications on the > > same node. Is it possible to tune these number of processes > down? Say > > reduce 32 ldlm process to about 8. What kind of performance > degradation > > to expect if we can do that? > > You can reduce the MDS and OST process count via modprobe.conf: > > option mds mds_num_threads=NNN > option ost ost_num_threads=NNN > > If this isn''t good enough, you can change the count at compile time > in lustre/include/linux/lustre-net.h > > In 1.6 the number of threads will be more dynamic, depending on load. > > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. >
Andreas Dilger
2007-Feb-12 09:50 UTC
[Lustre-discuss] Number of processes in Lustre tunable?
On Feb 11, 2007 23:51 -0800, Sridharan Ramaswamy (srramasw) wrote:> Thanks Andreas. I''ll try a lower value for the num_threads=NNN. How low > can these values be?On a very small UML system there are only 2 threads per service.> > -----Original Message----- > > From: Andreas Dilger [mailto:adilger@clusterfs.com] > > Sent: Sunday, February 11, 2007 11:39 PM > > To: Sridharan Ramaswamy (srramasw) > > Cc: lustre-discuss@clusterfs.com > > Subject: Re: [Lustre-discuss] Number of processes in Lustre tunable? > > > > On Jan 23, 2007 10:39 -0800, Sridharan Ramaswamy (srramasw) wrote: > > > I''m tring to measure the overhead of having Lustre on the > > system. Among > > > other things, I noticed it starts lot of processes, > > > > > > root 4565 1 0 Jan03 ? 00:00:00 [ll_mdt_attr_14] > > > root 4636 1 0 Jan03 ? 00:00:01 [ll_ost_13] > > > root 4651 1 0 Jan03 ? 00:00:08 [ll_ost_io_13] > > > root 4550 1 0 Jan03 ? 00:00:03 [ll_mdt_14] > > > root 4516 1 0 Jan03 ? 00:00:00 [ldlm_cb_31] > > > > > > I can account about 200 processes related to Lustre on an > > OST node. That > > > seems a lot considering we would like to run other > > applications on the > > > same node. Is it possible to tune these number of processes > > down? Say > > > reduce 32 ldlm process to about 8. What kind of performance > > degradation > > > to expect if we can do that? > > > > You can reduce the MDS and OST process count via modprobe.conf: > > > > option mds mds_num_threads=NNN > > option ost ost_num_threads=NNN > > > > If this isn''t good enough, you can change the count at compile time > > in lustre/include/linux/lustre-net.h > > > > In 1.6 the number of threads will be more dynamic, depending on load. > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Principal Software Engineer > > Cluster File Systems, Inc. > >Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.