I want to keep this thread on crossbow for now, in case it''s something not already in place. The recent processor affinity thread got me thinking again. A given NIC can scatter its inbound traffic to one or more CPUs, and crossbow makes this even more configurable (per the dladm(1M) changes). Beyond that, will inbound traffic scatter across multiple CPUs? Consider a node that has multiple inbound IPsec SAs. Each inbound SA will have one or more individual flows inside it, AND if all of them hammer one CPU their high-overhead decrypt/verify operations cannot be shared among multiple CPUs. Does crossbow improve this situation? Or are we already there w.r.t. sharing inbound load? This message posted from opensolaris.org
Dan,> I want to keep this thread on crossbow for now, in case it''s something not > already in place. > > The recent processor affinity thread got me thinking again. A given NIC > can scatter its inbound traffic to one or more CPUs, and crossbow makes > this even more configurable (per the dladm(1M) changes). > > Beyond that, will inbound traffic scatter across multiple CPUs? Consider a > node that has multiple inbound IPsec SAs. Each inbound SA will have one or > more individual flows inside it, AND if all of them hammer one CPU their > high-overhead decrypt/verify operations cannot be shared among multiple CPUs.Yes, you can scatter this across multiple CPUs. Again using dladm/flowadm, you will be able to specify a ''-C'' option for CPU list which can apply to both VNIC or a flow. The way ti works is that after classification, the upcall function is a fanout function which takes as an argument a set of soft rings with their own worker thread boudn to CPUs specified via ''-C'' option. I don''t think we have code to do optimal fanout based on SA but that should be easy to do.> Does crossbow improve this situation? Or are we already there w.r.t. > sharing in bound load?More interesting would be setting up a flow/VNIC and a CPU list to deal with encrypted traffic but feed it back to classifier after decryption so that policies based on IP address, ports etc can still be honoured and decryption can be done on separate set of CPUs altogether. With N2 crypto offload engines, it would be a interesting comboniation. Cheers, Sunay -- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow
On Tue, Jan 30, 2007 at 12:24:58PM -0800, Sunay Tripathi wrote:> > Beyond that, will inbound traffic scatter across multiple CPUs? Consider a > > node that has multiple inbound IPsec SAs. Each inbound SA will have one or > > more individual flows inside it, AND if all of them hammer one CPU their > > high-overhead decrypt/verify operations cannot be shared among multiple CPUs. > > Yes, you can scatter this across multiple CPUs. Again using dladm/flowadm,What you refer to as "flowadm" is called "netrcm" on your Dec. 2006 man pages. Is that what you mean?> you will be able to specify a ''-C'' option for CPU list which can apply to > both VNIC or a flow. The way ti works is that after classification, the > upcall function is a fanout function which takes as an argument a set of > soft rings with their own worker thread boudn to CPUs specified via ''-C'' > option.Yeah. That part tracks.> I don''t think we have code to do optimal fanout based on SA but that should > be easy to do.And "based on SA" is why I brought up this thread. In the absence of other data (which is actually VERY hard to get without getting key management - e.g. IKE - involved) all you really want to do is: * Assume inbound SPI generation is random. This is almost always the case apart from manually keyed IPsec SAs. * Send an inbound IPsec packet to a specific processor based on the SPI value modulo the number of available CPUs. Assuming a nice power-of-2, it''s a masking operation. Is that simple enough to shoehorn into the Crossbow way of doing things? If you already assign CPUs based on TCP/UDP ports, it should be trivial, as the IPsec SPI is at the same offset in the packet as TCP/UDP ports.> > Does crossbow improve this situation? Or are we already there w.r.t. > > sharing in bound load?To answer my own question based on what you said - crossbow might improve this situation if it can fanout based on SA per my notes above.> More interesting would be setting up a flow/VNIC and a CPU list to deal > with encrypted traffic but feed it back to classifier after decryption so > that policies based on IP address, ports etc can still be honoured and > decryption can be done on separate set of CPUs altogether. With N2 crypto > offload engines, it would be a interesting comboniation.If one''s IPsec policy (and subsequent SAs) have port/protocol selectors involved, one could use that information to look into the existing classifier and setup a post-IPsec-processing path that honors the classifier''s entries that is cached with the SA. (This is a variant on what we do today with caching *outbound* SAs with a conn_t.) Thanks for the answer, and if you can get IPsec multi-CPU fanout going... heh heh... combine that with N2 like you say, and inbound IPsec processing doesn''t become the single-CPU hog it once was. Dan
> On Tue, Jan 30, 2007 at 12:24:58PM -0800, Sunay Tripathi wrote: > > > Beyond that, will inbound traffic scatter across multiple CPUs? Consider a > > > node that has multiple inbound IPsec SAs. Each inbound SA will have one or > > > more individual flows inside it, AND if all of them hammer one CPU their > > > high-overhead decrypt/verify operations cannot be shared among multiple CPUs. > > > > Yes, you can scatter this across multiple CPUs. Again using dladm/flowadm, > > What you refer to as "flowadm" is called "netrcm" on your Dec. 2006 man > pages. Is that what you mean?Yes, netrcm is now flowadm. Just a name change.> > you will be able to specify a ''-C'' option for CPU list which can apply to > > both VNIC or a flow. The way ti works is that after classification, the > > upcall function is a fanout function which takes as an argument a set of > > soft rings with their own worker thread boudn to CPUs specified via ''-C'' > > option. > > Yeah. That part tracks. > > > I don''t think we have code to do optimal fanout based on SA but that should > > be easy to do. > > And "based on SA" is why I brought up this thread. > > In the absence of other data (which is actually VERY hard to get without > getting key management - e.g. IKE - involved) all you really want to do is: > > * Assume inbound SPI generation is random. This is almost always the > case apart from manually keyed IPsec SAs. > > * Send an inbound IPsec packet to a specific processor based on the > SPI value modulo the number of available CPUs. Assuming a nice > power-of-2, it''s a masking operation. > > Is that simple enough to shoehorn into the Crossbow way of doing things? If > you already assign CPUs based on TCP/UDP ports, it should be trivial, as the > IPsec SPI is at the same offset in the packet as TCP/UDP ports.I think thats already being done and should appear soon.> > > Does crossbow improve this situation? Or are we already there w.r.t. > > > sharing in bound load? > > To answer my own question based on what you said - crossbow might improve > this situation if it can fanout based on SA per my notes above. > > > More interesting would be setting up a flow/VNIC and a CPU list to deal > > with encrypted traffic but feed it back to classifier after decryption so > > that policies based on IP address, ports etc can still be honoured and > > decryption can be done on separate set of CPUs altogether. With N2 crypto > > offload engines, it would be a interesting comboniation. > > If one''s IPsec policy (and subsequent SAs) have port/protocol selectors > involved, one could use that information to look into the existing classifier > and setup a post-IPsec-processing path that honors the classifier''s entries > that is cached with the SA. (This is a variant on what we do today with > caching *outbound* SAs with a conn_t.)Yes. Its also for the subsequent policy enforcement as well. So we might wanna give more CPU/Bandwidth to IPsec traffic but once decrypted, the payload might get lesser or more resources which automatically get enforced by going through the classifier as if the packet came plain text.> Thanks for the answer, and if you can get IPsec multi-CPU fanout going... heh > heh... combine that with N2 like you say, and inbound IPsec processing > doesn''t become the single-CPU hog it once was.Agreed :) Cheers, Sunay> > Dan > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://opensolaris.org/mailman/listinfo/crossbow-discuss >-- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow