crossbow discuss - Feb 2009 - Problem in assigning CPU to VNIC

In our product(Telecom) we use all kinds of SUN hardware. It depends on what
range of Hardware the customer requires.One of the requirement was simple, they
wanted T5140 (this was decided based on the Application requirements and cost).
But the problem we are facing is that in T5140/T5240 some PCI slots are bound to
Single CPU. Reference
http://opensolaris.org/os/community/ldoms/files/t5240_review.pdf.

Question
If we place the 10Gb network card on any of the PCI slots the effective CPU
available for us is just 1. Of course the CPU has 64 threads, but how will we
make CPU2 also available to this Network card or VNIC ?

Regards
Vikas
-- 
This message posted from opensolaris.org

On 02/23/09 03:56, VIKAS MR wrote:
> In our product(Telecom) we use all kinds of SUN hardware. It depends on
what range of Hardware the customer requires.One of the requirement was simple,
they wanted T5140 (this was decided based on the Application requirements and
cost). But the problem we are facing is that in T5140/T5240 some PCI slots are
bound to Single CPU. Reference
http://opensolaris.org/os/community/ldoms/files/t5240_review.pdf.
> 
> Question
> If we place the 10Gb network card on any of the PCI slots the effective CPU
available for us is just 1. Of course the CPU has 64 threads, but how will we
make CPU2 also available to this Network card or VNIC ?
All sockets can access all memory and I/O resources. All cores in a 
socket can access all resources. All strands in a core can access all 
resources. You don''t have to worry about where your NIC is installed 
compared to where code may be executing. In extreme performance 
situations you may want to do some balancing and isolation, however in 
the general case everything will just work.

The only situation where you have to be concerned which PCI-express bus 
I/O is attached to is when you have two LDom service domain.
> Regards
> Vikas

Thank you for the response,

Just wanted to mention that few months back we faced a problem in which
"When you disable a CPU, the related Hardware associated with it also goes
down". This is confirmed by SUN for T5140/T5240. Since these machines are
not high end machines they don''t support  reconfiguration.

Hence my question was that when we run the command "dladm" and
associate a VNIC with CPU not "Hardware connected" with it, will it
give an error ? or will it continue ?
-- 
This message posted from opensolaris.org

On 02/23/09 23:11, VIKAS MR wrote:
> Thank you for the response,
> 
> Just wanted to mention that few months back we faced a problem in which
"When you disable a CPU, the related Hardware associated with it also goes
down". This is confirmed by SUN for T5140/T5240. Since these machines are
not high end machines they don''t support  reconfiguration.
> 
> Hence my question was that when we run the command "dladm" and
associate a VNIC with CPU not "Hardware connected" with it, will it
give an error ? or will it continue ?
So the question is ''if I assign a specific CPU to a VNIC, and that CPU 
is not present, what happens?''.  Good question. Maybe it is worth
having
an option to assign a number of CPUs to a VNIC, and not specify which ones.

Then, similar to zones, if you configure a zone to have m-n CPUs, and 
less than m are available, the zone will fail to boot. In this case, the 
VNIC might not be created, and the consumer will have the deal with a 
lack of a resource.

Hello,
Thank you for the answer. 
Here by "CPU" in T5140/T5240 systems I meant one "CPU
socket". May be I did not communicate properly in the previous message.

Yes that is a good idea to have "number of CPU''s" as fallback
option in case the  "CPU socket" to which it is assigned fails.

So is there any option like that ? 
Because, the least we want is that the VNIC goes "down" when a CPU
goes down.
-- 
This message posted from opensolaris.org

VIKAS MR wrote:
> Hello,
> Thank you for the answer. 
> Here by "CPU" in T5140/T5240 systems I meant one "CPU
socket". May be I did not communicate properly in the previous message.
> 
> Yes that is a good idea to have "number of CPU''s" as
fallback option in case the  "CPU socket" to which it is assigned
fails.
> 
> So is there any option like that ? 
> Because, the least we want is that the VNIC goes "down" when a
CPU goes down.
No, that won''t happen. VNICs register callback functions with the CPU 
subsystem to get notified of events like CPU going off-line. When a CPU 
goes off-line, VNIC will bind to a different CPU. Later when the CPU 
comes on-line, it will re-bind back to the original CPU.

-krgopi

--

On 02/24/09 22:57, VIKAS MR wrote:
> Hello,
> Thank you for the answer. 
> Here by "CPU" in T5140/T5240 systems I meant one "CPU
socket". May be I did not communicate properly in the previous message.
> 
> Yes that is a good idea to have "number of CPU''s" as
fallback option in case the  "CPU socket" to which it is assigned
fails.
> 
> So is there any option like that ? 
Not that I know of. Do you want to file an RFE, or should I?
> Because, the least we want is that the VNIC goes "down" when a
CPU goes down.
That is a separate issue. Not only will the VNIC go down, chances are 
that when the whole chip/socket fails, the system will go down. It will 
typically be able to reboot in degraded mode, meaning that 1/2 of the 
CPUs and its connected I/O will not be accessible. Through support for 
redundant paths for network and disk, Solaris has been able to run like 
this for years. In the specific case of assigning fixed CPUs to a VNIC, 
that is a ''new'' administrative operation, and may need more
flexibility.
Also, since the underlying device may have not come back, it may be 
desirable to configure the VNIC on top of an aggregation that uses 
interfaces off of both CPUs. I''ll have to verify whether you can create
a VNIC on top of an aggregation. I think you should be able to.

Given the complexities of designing around such a failure, it may be 
worth looking at other forms of availability, if possible, to handle a 
full socket failure.

Steffen

PS. The T5440 with four sockets has the ability to reconfigure the PCIe 
so that I/O is available through other paths.

See http://blogs.sun.com/pyakutis/entry/t5440_pci_e_reconfiguration

Hello,
Thank you for the answers. Yes now it makes sense. 

Regards
Vikas
-- 
This message posted from opensolaris.org