If our IO servers are seeing extended periods of socknal_sd00 at 100% cpu, Would this cause a bottle neck? If so its a single homed hosts, would adding another interface to the host help? Is there threading anyplace? Or is faster cpu the only way out? Brock Palen Center for Advanced Computing brockp at umich.edu (734)936-1985
Brock,> If our IO servers are seeing extended periods of socknal_sd00 at 100% > cpu, Would this cause a bottle neck?Yes, I think so.> If so its a single homed hosts, would adding another interface to the > host help? >Probably, no. It could only help in the case you have several CPUs but something prevents ksocklnd to spread the load over them.> Is there threading anyplace?Yes, ksocklnd spawns separate socknal_sd thread for each CPU/core that you have. There are two algorithms of spreading the load - you can play with enable_irq_affinity modparam flag.> Or is faster cpu the only way out? >I believe you either need faster CPU or faster system bus. If slow system bus isn''t your case, increasing number of CPUs also will do. Sincerely, Maxim
On Mar 7, 2008, at 8:51 AM, Maxim V. Patlasov wrote:> Brock, > >> If our IO servers are seeing extended periods of socknal_sd00 at >> 100% cpu, Would this cause a bottle neck? > Yes, I think so. > >> If so its a single homed hosts, would adding another interface to >> the host help? >> > Probably, no. It could only help in the case you have several CPUs > but something prevents ksocklnd to spread the load over them.The servers are dual cpu systems. But I only see a single socknal_sd thread.> >> Is there threading anyplace? > Yes, ksocklnd spawns separate socknal_sd thread for each CPU/core > that you have. There are two algorithms of spreading the load - > you can play with enable_irq_affinity modparam flag.I see some things in logs about setting cpu affinity, Ill check out the manual some more,> >> Or is faster cpu the only way out? >> > I believe you either need faster CPU or faster system bus. If slow > system bus isn''t your case, increasing number of CPUs also will do.Ok> > Sincerely, > Maxim > >
Brock,> The servers are dual cpu systems. But I only see a single socknal_sd > thread. >It can''t be so. Please provide output of ''ps aux | grep socknal'' and ''cat /proc/cpuinfo''. Sincerely, Maxim
On Fri, Mar 07, 2008 at 12:53:52AM -0500, Brock Palen wrote:> If our IO servers are seeing extended periods of socknal_sd00 at 100% > cpu, Would this cause a bottle neck? If so its a single homed > hosts, would adding another interface to the host help?To quote the Lustre Manual: enable_irq_affinity (1,Wc) is a boolean that determines whether to enable IRQ affinity. When set, socklnd attempts to maximize performance by handling device interrupts and data movement for particular (hardware) interfaces on particular CPUs. This option is not available on all platforms. This option requires an SMP system to exist and produces best performance with multiple NICs. Systems with multiple CPUs and a single NIC may see increase in the performance with this parameter disabled. So if you have multiple CPUs and a single NIC (or more precisely Lustre only uses a single NIC) I''d suggest to try: options ksocklnd enable_irq_affinity=0 Isaac
> Brock, > >> The servers are dual cpu systems. But I only see a single >> socknal_sd thread. >> > It can''t be so. Please provide output of ''ps aux | grep socknal'' > and ''cat /proc/cpuinfo''.You are right: root 3171 2.3 0.0 0 0 ? R Jan10 1934:02 [socknal_sd00] root 3172 0.0 0.0 0 0 ? S Jan10 0:00 [socknal_sd01] root 3173 0.0 0.0 0 0 ? S Jan10 0:00 [socknal_cd00] root 3174 0.0 0.0 0 0 ? S Jan10 0:00 [socknal_cd01] root 3175 0.0 0.0 0 0 ? S Jan10 0:00 [socknal_cd02] root 3176 0.0 0.0 0 0 ? S Jan10 0:00 [socknal_cd03] root 3177 0.0 0.0 0 0 ? S Jan10 0:00 [socknal_reaper] brockp 4749 0.0 0.0 51100 700 pts/0 S+ 12:33 0:00 grep sockn Notice the amount of cpu time given to sd00 and how sd01 has done nothing. What could cause this? brockp at nyx167 ~]$ grep Proc /proc/cpuinfo model name : AMD Opteron(tm) Processor 240 model name : AMD Opteron(tm) Processor 240> > Sincerely, > Maxim > >
Brock,> Notice the amount of cpu time given to sd00 and how sd01 has done > nothing. What could cause this?Please try Isaac''s recommendation:> So if you have multiple CPUs and a single NIC (or more precisely Lustre > only uses a single NIC) I''d suggest to try: > options ksocklnd enable_irq_affinity=0Also, please note, that you need to have several heavy-loaded tcp connections to get fair load balancing. In the case of point-to-point test the recommendation above may not help. Sincerely, Maxim
On Mar 7, 2008, at 1:23 PM, Maxim V. Patlasov wrote:> Brock, > >> Notice the amount of cpu time given to sd00 and how sd01 has done >> nothing. What could cause this? > Please try Isaac''s recommendation: >> So if you have multiple CPUs and a single NIC (or more precisely >> Lustre only uses a single NIC) I''d suggest to try: >> options ksocklnd enable_irq_affinity=0How do you do this on a live system? Taking away the filesystem is ''bad''> > Also, please note, that you need to have several heavy-loaded tcp > connections to get fair load balancing. In the case of point-to- > point test the recommendation above may not help.It was saw only when I had 150+ serial gromacs jobs running trrjecov every 60 seconds. We have gotten around the 60 seconds problem.> > Sincerely, > Maxim > >
On Mar 07, 2008 13:27 -0500, Brock Palen wrote:> On Mar 7, 2008, at 1:23 PM, Maxim V. Patlasov wrote: > > Brock, > >> Notice the amount of cpu time given to sd00 and how sd01 has done > >> nothing. What could cause this? > > Please try Isaac''s recommendation: > >> So if you have multiple CPUs and a single NIC (or more precisely > >> Lustre only uses a single NIC) I''d suggest to try: > >> options ksocklnd enable_irq_affinity=0 > > How do you do this on a live system? Taking away the filesystem is > ''bad''On relatively modern kernels you can change SOME module parameters: echo 0 > /sys/modules/ksocklnd/{parameters}/enable_irq_affinity The "{parameters}" subdir is present or not, depending on the kernel version (yay consistency!). That said, it is implementation dependent on whether changing the parameter after the module is loaded has any effect. I suspect in this case it will not have any effect because the kernel threads are already bound to their CPUs, but this isn''t my strongest area. Maxim, Isaac, what are your thoughts about disabling IRQ affinity by default? In the past this was important for maximizing performance with N CPUs and N ethernet NICs, but the CPUs have gotten much faster and more cores and I believe other customers have found better performance with irq_affinity disabled. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
> > echo 0 > /sys/modules/ksocklnd/{parameters}/enable_irq_affinityI set the option, ''cat'' confirms it was set, so far i see no effect (zero cpu time on 01) but I will keep a eye on it.> > The "{parameters}" subdir is present or not, depending on the kernel > version (yay consistency!). > > That said, it is implementation dependent on whether changing the > parameter after the module is loaded has any effect. I suspect in > this case it will not have any effect because the kernel threads > are already bound to their CPUs, but this isn''t my strongest area. > > Maxim, Isaac, what are your thoughts about disabling IRQ affinity > by default? In the past this was important for maximizing performance > with N CPUs and N ethernet NICs, but the CPUs have gotten much faster > and more cores and I believe other customers have found better > performance > with irq_affinity disabled. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > >
Hi Brock, As Andreas said, change it by proc wouldn''t affect any existed connections , but it will affect new connections. Regards Liang Brock Palen wrote:>> echo 0 > /sys/modules/ksocklnd/{parameters}/enable_irq_affinity >> > > I set the option, ''cat'' confirms it was set, so far i see no effect > (zero cpu time on 01) but I will keep a eye on it. > >> The "{parameters}" subdir is present or not, depending on the kernel >> version (yay consistency!). >> >> That said, it is implementation dependent on whether changing the >> parameter after the module is loaded has any effect. I suspect in >> this case it will not have any effect because the kernel threads >> are already bound to their CPUs, but this isn''t my strongest area. >> >> Maxim, Isaac, what are your thoughts about disabling IRQ affinity >> by default? In the past this was important for maximizing performance >> with N CPUs and N ethernet NICs, but the CPUs have gotten much faster >> and more cores and I believe other customers have found better >> performance >> with irq_affinity disabled. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Sr. Staff Engineer, Lustre Group >> Sun Microsystems of Canada, Inc. >> >> >> >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On Fri, Mar 07, 2008 at 03:26:23PM -0700, Andreas Dilger wrote:> > Maxim, Isaac, what are your thoughts about disabling IRQ affinity > by default? In the past this was important for maximizing performance > with N CPUs and N ethernet NICs, but the CPUs have gotten much faster > and more cores and I believe other customers have found better performance > with irq_affinity disabled. >Agree, and with ksocklnd bonding feature deprecated it''s now more common to configure lnet with a single NIC. I''ve committed the change and filed a documentation bug to update the manuals accordingly. Thanks, Isaac