I am experimenting with using Crossbow (2009.06) for large scale bandwidth management. In essence this means defining flows for thousands of individual remote_ip''s, each with their own ''maxbw'' parameter. The bandwidth control seems to be working fine - except for the limitation of not being able to enforce bandwidths below 1200Kb/sec. However, when defining many flows, I seem to encounter a problems at around 11,500 flows. At this point the system starts to become unresponsive before finally locking up completely, requiring a power cycle to recover. Is there a known limitation on the number of flows that can be defined on a system? -and are there any configuration parameters that can influence this? BR, Michael A ************* Example steps to reproduce the problem: bt1=1 bt2=1 flownum=1 while [ $flownum -lt 12001 ]; do flowadm add-flow -t -l bge0 -a remote_ip=10.10.$bt2.$bt1 -p maxbw=1205K flow$flownum; flownum=$[$flownum+1]; bt1=$[$bt1+1]; if [ $bt1 -eq 256 ]; then bt1=1; bt2=$[$bt2+1]; fi done -- This message posted from opensolaris.org
Michael, The limitations is typically based on system size, CPU and memory available etc. We might have some issues with the hash scaling as well. What kind of system and RAM do you have? We should try it inhouse and see what we get to. To be precise, there are not supposed to be any limitations but our typical testing has been with few hundred flows only and currently our flows are based on S/W unlike VNICs which are H/W based so there are more overheads. BTW, you do know that you can create flows on remote subnets as well (although we don''t allow subnets and IP addresses to mix right now). See if you can make work with subnets and reduce the number of flows you need while we look at this issue. Thanks, Sunay Michael Andersen wrote:> I am experimenting with using Crossbow (2009.06) for large scale bandwidth management. > > In essence this means defining flows for thousands of individual remote_ip''s, each with their own ''maxbw'' parameter. The bandwidth control seems to be working fine - except for the limitation of not being able to enforce bandwidths below 1200Kb/sec. > > However, when defining many flows, I seem to encounter a problems at around 11,500 flows. At this point the system starts to become unresponsive before finally locking up completely, requiring a power cycle to recover. > > Is there a known limitation on the number of flows that can be defined on a system? -and are there any configuration parameters that can influence this? > > > BR, > Michael A > > > ************* > > Example steps to reproduce the problem: > > bt1=1 > bt2=1 > flownum=1 > while [ $flownum -lt 12001 ]; do > flowadm add-flow -t -l bge0 -a remote_ip=10.10.$bt2.$bt1 -p maxbw=1205K flow$flownum; > flownum=$[$flownum+1]; > bt1=$[$bt1+1]; > if [ $bt1 -eq 256 ]; then > bt1=1; > bt2=$[$bt2+1]; > fi > done-- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow
HI Sunay,> The limitations is typically based on system size, > CPU and memory > available etc. We might have some issues with the > hash scaling as > well. What kind of system and RAM do you have? We > should try > it inhouse and see what we get to.The system that got in trouble at ~11,500 was an x86 with 1G memory (512M swap). I managed to repeat the test on another x86 with 1G memory - this one with 2G swap configured. This second system survived the creation of 12K flows. On this surviving system, the memory requirement for the 12K flows appears to be in the area of 500M.> BTW, you do know that you can create flows on remote > subnets as well > (although we don''t allow subnets and IP addresses to > mix right now). > See if you can make work with subnets and reduce the > number of flows > you need while we look at this issue.Yes, I realize it would be more practical with flows defined at subnet level and the commands I used to test the flow creations may indicate that each IP in a range would get the same bandwidth ;-). However, in the actual application these experiments are conducted for, the remote_ip''s would not fall cleanly within subnet ranges and more importantly: they would require individual ''maxbw'' parameter setting. A related question: which API would be most suitable for an application that would try to maintain a high number of non-permanent flows in a near real-time fashion? BR, Michael A -- This message posted from opensolaris.org
Sunay Tripathi (Distinguished Engineer)
2009-Oct-20 22:21 UTC
[crossbow-discuss] Large scale flows definition
Michael Andersen wrote:> HI Sunay, > >> The limitations is typically based on system size, >> CPU and memory >> available etc. We might have some issues with the >> hash scaling as >> well. What kind of system and RAM do you have? We >> should try >> it inhouse and see what we get to. > > The system that got in trouble at ~11,500 was an x86 with 1G memory (512M swap). I managed to repeat the test on another x86 with 1G memory - this one with 2G swap configured. This second system survived the creation of 12K flows. On this surviving system, the memory requirement for the 12K flows appears to be in the area of 500M. >OK. Thats good information. Typically we expect people to have a more server class system (with 8 to 16Gb RAM) when they are creating thousands of flow but I know thats not true all the time. We should try and get the per flow overheads down as much as we can. We were planning to revisit this as part of H/W based flows anyway.> >> BTW, you do know that you can create flows on remote >> subnets as well >> (although we don''t allow subnets and IP addresses to >> mix right now). >> See if you can make work with subnets and reduce the >> number of flows >> you need while we look at this issue. > > Yes, I realize it would be more practical with flows defined at subnet level and the commands I used to test the flow creations may indicate that each IP in a range would get the same bandwidth ;-). > > However, in the actual application these experiments are conducted for, the remote_ip''s would not fall cleanly within subnet ranges and more importantly: they would require individual ''maxbw'' parameter setting. > > A related question: which API would be most suitable for an application that would try to maintain a high number of non-permanent flows in a near real-time fashion?flowadm internally uses libdladm anyway. But its not a public API yet. Have a look and we can talk about the parts that you need that can be exposed as publically stable APIs. Cheers, Sunay -- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow