crossbow discuss - May 2007 - major, major performance regression with crossbow

I''m testing e1000g on a T2000.  Default daily build 04/30.  (Built 
against snv-61).

I''m seeing really, really dismal IP forwarding performance with 64-byte
frames.  Like ~5.8% line utilization (it seems to top out at around 
90,000 packets per second default.)  This is a default configuration (no 
tunables set whatsover), and yes, it turns out that interrupts for my 
two NICs are interrupting on CPUs 6 and 7 (and yes, I know that this is 
a bad idea, but still, it should not show such terrible, dismal 
performance out of the box.)  This performance is about half of the 
default S10 performance.

I''ve not started using er_kernel yet to profile and find out what is 
happening.  At first I thought the problem was my test setup, but when I 
lowered the test min thresholds from 10% to 1%, I get successful results.

I''m going to try assigning additional cpus to create and setup some
soft
rings, to see how that helps.

Anyone have any ideas up front about what is wrong?

This default performance is, IMO, unacceptable.

    -- Garrett

Garrett D''Amore wrote:
> I''m testing e1000g on a T2000.  Default daily build 04/30.  (Built
> against snv-61).
> 
> I''m seeing really, really dismal IP forwarding performance with
64-byte
> frames.  Like ~5.8% line utilization (it seems to top out at around 
> 90,000 packets per second default.)  This is a default configuration (no 
> tunables set whatsover), and yes, it turns out that interrupts for my 
> two NICs are interrupting on CPUs 6 and 7 (and yes, I know that this is 
> a bad idea, but still, it should not show such terrible, dismal 
> performance out of the box.)  This performance is about half of the 
> default S10 performance.
> 
> I''ve not started using er_kernel yet to profile and find out what
is
> happening.  At first I thought the problem was my test setup, but when I 
> lowered the test min thresholds from 10% to 1%, I get successful results.
> 
> I''m going to try assigning additional cpus to create and setup
some soft
> rings, to see how that helps.
> 
> Anyone have any ideas up front about what is wrong?
> 
> This default performance is, IMO, unacceptable.
Yes, you are using code which is not complete. The soft rings have moved
to mac layer (the tuning auto tuning on Niagara is disabled in
crossbow-gate due to a bug I can''t remember right now). Raj/Gopi, can
you help Garrett.

Cheers,
Sunay
> 
>    -- Garrett
> 
> 
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/crossbow-discuss

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Sunay Tripathi wrote:
> Garrett D''Amore wrote:
>> I''m testing e1000g on a T2000.  Default daily build 04/30. 
(Built
>> against snv-61).
>>
>> I''m seeing really, really dismal IP forwarding performance
with
>> 64-byte frames.  Like ~5.8% line utilization (it seems to top out at 
>> around 90,000 packets per second default.)  This is a default 
>> configuration (no tunables set whatsover), and yes, it turns out that 
>> interrupts for my two NICs are interrupting on CPUs 6 and 7 (and yes, 
>> I know that this is a bad idea, but still, it should not show such 
>> terrible, dismal performance out of the box.)  This performance is 
>> about half of the default S10 performance.
>>
>> I''ve not started using er_kernel yet to profile and find out
what is
>> happening.  At first I thought the problem was my test setup, but 
>> when I lowered the test min thresholds from 10% to 1%, I get 
>> successful results.
>>
>> I''m going to try assigning additional cpus to create and setup
some
>> soft rings, to see how that helps.
>>
>> Anyone have any ideas up front about what is wrong?
>>
>> This default performance is, IMO, unacceptable.
>
> Yes, you are using code which is not complete. The soft rings have moved
> to mac layer (the tuning auto tuning on Niagara is disabled in
> crossbow-gate due to a bug I can''t remember right now). Raj/Gopi,
can
> you help Garrett.
Yes, ip_squeue_soft_ring tunable has been disabled in the crossbow-gate 
(See 6521633). You can still create soft rings and that too on the CPUs 
you want using dladm command.

"dladm modify-dev -d <e1000g0> -c 0,1,2" will create soft rings
on the
fly for e1000g0. The worker thread would be bound to CPUs 0, 1 and 2.

I also want to note that major changes are going on in this area of the 
code at the moment and things would become clearer by the end of this 
month atleast.

Thanks,
-krgopi

-krgopi
>
> Cheers,
> Sunay
>
>>
>>    -- Garrett
>>
>>
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://opensolaris.org/mailman/listinfo/crossbow-discuss
>
>

Okay, I don''t mind waiting at this point.  The thought at one point was
that I should retest performance with crossbow gate to see if the 
numbers were any better... but clearly crossbow-gate isn''t ready for 
that yet. :-)

This will hamper my ability to do meaningful performance related work 
from crossbow, btw.

    -- Garrett

Rajagopal Kunhappan wrote:
> Sunay Tripathi wrote:
>> Garrett D''Amore wrote:
>>> I''m testing e1000g on a T2000.  Default daily build 04/30.
(Built
>>> against snv-61).
>>>
>>> I''m seeing really, really dismal IP forwarding performance
with
>>> 64-byte frames.  Like ~5.8% line utilization (it seems to top out
at
>>> around 90,000 packets per second default.)  This is a default 
>>> configuration (no tunables set whatsover), and yes, it turns out 
>>> that interrupts for my two NICs are interrupting on CPUs 6 and 7 
>>> (and yes, I know that this is a bad idea, but still, it should not 
>>> show such terrible, dismal performance out of the box.)  This 
>>> performance is about half of the default S10 performance.
>>>
>>> I''ve not started using er_kernel yet to profile and find
out what is
>>> happening.  At first I thought the problem was my test setup, but 
>>> when I lowered the test min thresholds from 10% to 1%, I get 
>>> successful results.
>>>
>>> I''m going to try assigning additional cpus to create and
setup some
>>> soft rings, to see how that helps.
>>>
>>> Anyone have any ideas up front about what is wrong?
>>>
>>> This default performance is, IMO, unacceptable.
>>
>> Yes, you are using code which is not complete. The soft rings have
moved
>> to mac layer (the tuning auto tuning on Niagara is disabled in
>> crossbow-gate due to a bug I can''t remember right now).
Raj/Gopi, can
>> you help Garrett.
>
> Yes, ip_squeue_soft_ring tunable has been disabled in the 
> crossbow-gate (See 6521633). You can still create soft rings and that 
> too on the CPUs you want using dladm command.
>
> "dladm modify-dev -d <e1000g0> -c 0,1,2" will create soft
rings on the
> fly for e1000g0. The worker thread would be bound to CPUs 0, 1 and 2.
>
> I also want to note that major changes are going on in this area of 
> the code at the moment and things would become clearer by the end of 
> this month atleast.
>
> Thanks,
> -krgopi
>
> -krgopi
>>
>> Cheers,
>> Sunay
>>
>>>
>>>    -- Garrett
>>>
>>>
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://opensolaris.org/mailman/listinfo/crossbow-discuss
>>
>>