crossbow discuss - Feb 2009 - flowadm isn''t capturing local iSCSI traffic

flowadm isn''t capturing local iSCSI traffic.  Should it be?

Summary setup:

snv_107 w/ xVM Hypervisor on a Sun Blade X6250, using e1000g0 link
COMSTAR iSCSI target set up locally in dom0, on port 3261
flowadm add-flow -l e1000g0 -a transport=tcp,local_port=3261 -p maxbw=60 iscsi
iSCSI initiator set up locally in dom0

"flowadm show-flow -i 5 -S iscsi" shows no traffic when accessing disk
in dom0
"snoop tcp host $host port 3261" shows traffic when accessing disk in
dom0

Traffic is correctly captured in a flow when the iSCSI target is on a
remote host and the flow is set up using a remote_ip attribute.

Detailed setup:

# ifconfig e1000g0
e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500
index 2
         inet 192.168.2.6 netmask ffffff00 broadcast 192.168.2.255
         ether 0:1e:68:da:60:8c

# host=`netstat -in -I e1000g0 | tail -1 | awk ''{print $4}''`
# svcadm enable stmf
# svcadm enable iscsi/target
# svcadm enable iscsi_initiator
# iscsiadm modify discovery --static enable

# zfs create -V 64g files/test-vol
# flowadm add-flow -l e1000g0 -a transport=tcp,local_port=3261 -p maxbw=60 iscsi
# itadm create-tpg net1 $host:3261
# itadm create-target -t net1
# sbdadm create-lu /dev/zvol/rdsk/files/test-vol
# stmfadm add-view `stmfadm list-lu | awk ''{print $3}''`
# iscsiadm add static-config `stmfadm list-target | awk ''{print
$2}''`,$host:3261
# iscsiadm list target -S
# cp `iscsiadm list target -S | tail -2 | awk ''{print $4}''`
/dev/null &
# flowadm show-flow -i 5 -S iscsi

- Bob

On 02/05/09 13:06, Bob Scheifler wrote:
> flowadm isn''t capturing local iSCSI traffic.  Should it be?
>
> Summary setup:
>
> snv_107 w/ xVM Hypervisor on a Sun Blade X6250, using e1000g0 link
> COMSTAR iSCSI target set up locally in dom0, on port 3261
> flowadm add-flow -l e1000g0 -a transport=tcp,local_port=3261 -p maxbw=60
iscsi
> iSCSI initiator set up locally in dom0
>
> "flowadm show-flow -i 5 -S iscsi" shows no traffic when accessing
disk in dom0
> "snoop tcp host $host port 3261" shows traffic when accessing
disk in dom0
>
> Traffic is correctly captured in a flow when the iSCSI target is on a
> remote host and the flow is set up using a remote_ip attribute.
>
> Detailed setup:
>
> # ifconfig e1000g0
> e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu
1500 index 2
>          inet 192.168.2.6 netmask ffffff00 broadcast 192.168.2.255
>          ether 0:1e:68:da:60:8c
>
> # host=`netstat -in -I e1000g0 | tail -1 | awk ''{print
$4}''`
> # svcadm enable stmf
> # svcadm enable iscsi/target
> # svcadm enable iscsi_initiator
> # iscsiadm modify discovery --static enable
>
> # zfs create -V 64g files/test-vol
> # flowadm add-flow -l e1000g0 -a transport=tcp,local_port=3261 -p maxbw=60
iscsi
> # itadm create-tpg net1 $host:3261
> # itadm create-target -t net1
> # sbdadm create-lu /dev/zvol/rdsk/files/test-vol
> # stmfadm add-view `stmfadm list-lu | awk ''{print $3}''`
> # iscsiadm add static-config `stmfadm list-target | awk ''{print
$2}''`,$host:3261
> # iscsiadm list target -S
> # cp `iscsiadm list target -S | tail -2 | awk ''{print
$4}''` /dev/null &
> # flowadm show-flow -i 5 -S iscsi
>
>   
Try:

#kstat -c flow -n iscsi

If the stats appear to be correct, it''s a problem with "show-flow
-S".
If the stats are all 0, the traffic isn''t being properly classifed to
the iscsi flow.

-Mike

On Thu, Feb 05, 2009 at 04:06:42PM -0500, Bob Scheifler
wrote:
> flowadm isn''t capturing local iSCSI traffic.  Should it be?
Your flow s defined over e1000g0. If the traffic is local it won''t go
anywhere near e1000g0.
> Summary setup:
> 
> snv_107 w/ xVM Hypervisor on a Sun Blade X6250, using e1000g0 link
> COMSTAR iSCSI target set up locally in dom0, on port 3261
> flowadm add-flow -l e1000g0 -a transport=tcp,local_port=3261 -p maxbw=60
iscsi
> iSCSI initiator set up locally in dom0
> 
> "flowadm show-flow -i 5 -S iscsi" shows no traffic when accessing
disk in dom0
> "snoop tcp host $host port 3261" shows traffic when accessing
disk
> in dom0
What does snoop say about where it is capturing traffic?
> Traffic is correctly captured in a flow when the iSCSI target is on a
> remote host and the flow is set up using a remote_ip attribute.
> 
> Detailed setup:
> 
> # ifconfig e1000g0
> e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu
1500 index 2
>          inet 192.168.2.6 netmask ffffff00 broadcast 192.168.2.255
>          ether 0:1e:68:da:60:8c
> 
> # host=`netstat -in -I e1000g0 | tail -1 | awk ''{print
$4}''`
> # svcadm enable stmf
> # svcadm enable iscsi/target
> # svcadm enable iscsi_initiator
> # iscsiadm modify discovery --static enable
> 
> # zfs create -V 64g files/test-vol
> # flowadm add-flow -l e1000g0 -a transport=tcp,local_port=3261 -p maxbw=60
iscsi
> # itadm create-tpg net1 $host:3261
> # itadm create-target -t net1
> # sbdadm create-lu /dev/zvol/rdsk/files/test-vol
> # stmfadm add-view `stmfadm list-lu | awk ''{print $3}''`
> # iscsiadm add static-config `stmfadm list-target | awk ''{print
$2}''`,$host:3261
> # iscsiadm list target -S
> # cp `iscsiadm list target -S | tail -2 | awk ''{print
$4}''` /dev/null &
> # flowadm show-flow -i 5 -S iscsi
> 
> - Bob
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

On 02/06/09 01:16, David Edmondson wrote:
> Your flow s defined over e1000g0. If the traffic is local it won''t
go
> anywhere near e1000g0.
> What does snoop say about where it is capturing traffic?
Bingo, thanks, but I''m still confused.  I didn''t notice that
snoop was defaulting to lo0 rather than e1000g0.  But why is
192.168.2.6 traffic (which is what snoop shows) going over lo0?

# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
         inet 127.0.0.1 netmask ff000000
e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500
index 2
         inet 192.168.2.6 netmask ffffff00 broadcast 192.168.2.255
         ether 0:1e:68:da:60:8c
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252
index 1
         inet6 ::1/128

# netstat -an | grep 3261
192.168.2.6.34482    192.168.2.6.3261     269936      0 278120      0
ESTABLISHED
192.168.2.6.3261     192.168.2.6.34482    278120      0 269940      0
ESTABLISHED
       *.3261               *.*                0      0 262300      0 LISTEN
       *.3261                            *.*                             0 
   0 262300      0 LISTEN

# iscsiadm list target -S -v
Target: iqn.1986-03.com.sun:02:2e1fb325-6b20-e2c2-c57d-98490dd10971
         Alias: -
         TPGT: 2
         ISID: 4000002a0000
         Connections: 1
                 CID: 0
                   IP address (Local): 192.168.2.6:34482
                   IP address (Peer): 192.168.2.6:3261
         ...

- Bob

On Fri, Feb 06, 2009 at 01:06:30PM -0500, Bob Scheifler
wrote:
> On 02/06/09 01:16, David Edmondson wrote:
>> Your flow s defined over e1000g0. If the traffic is local it
won''t go
>> anywhere near e1000g0.
>> What does snoop say about where it is capturing traffic?
>
> Bingo, thanks, but I''m still confused.  I didn''t notice
that
> snoop was defaulting to lo0 rather than e1000g0.  But why is
> 192.168.2.6 traffic (which is what snoop shows) going over lo0?
It''s a local destination, so there''s no need for the traffic
to go
near any wires.

On Fri, 2009-02-06 at 13:06 -0500, Bob Scheifler wrote:
> On 02/06/09 01:16, David Edmondson wrote:
> > Your flow s defined over e1000g0. If the traffic is local it
won''t go
> > anywhere near e1000g0.
> > What does snoop say about where it is capturing traffic?
> 
> Bingo, thanks, but I''m still confused.  I didn''t notice
that
> snoop was defaulting to lo0 rather than e1000g0.  But why is
> 192.168.2.6 traffic (which is what snoop shows) going over lo0?
All locally delivered packets are observable through the /dev/lo0 DLPI
device, which is what snoop has opened.  Snoop''s algorithm for picking
a
default datalink to open in the absence of user input could arguably be
improved, but I''d recommend always explicitly telling snoop what to
observe.

-Seb

>> Bingo, thanks, but I''m still confused.  I didn''t
notice that
>> snoop was defaulting to lo0 rather than e1000g0.  But why is
>> 192.168.2.6 traffic (which is what snoop shows) going over lo0?
> 
> It''s a local destination, so there''s no need for the
traffic to go
> near any wires.
So there''s no way to apply flows to local traffic?
(flowadm can''t be used on lo0, since it''s not a link)

- Bob

> > On 02/06/09 01:16, David Edmondson wrote:
 > >> Your flow s defined over e1000g0. If the traffic is local it
won''t go
 > >> anywhere near e1000g0.
 > >> What does snoop say about where it is capturing traffic?
 > >
 > > Bingo, thanks, but I''m still confused.  I didn''t
notice that
 > > snoop was defaulting to lo0 rather than e1000g0.  But why is
 > > 192.168.2.6 traffic (which is what snoop shows) going over lo0?
 > 
 > It''s a local destination, so there''s no need for the
traffic to go
 > near any wires.

Further to what David said: somewhat confusingly, /dev/lo0 has established
semantics (i.e., outside of Solaris, sigh) that all loopback packets are
seen, not just those flowing over 127.0.0.1.  FWIW, snoop -I lo0 will
allow one to see just IP packets flowing over 127.0.0.1 (or whatever
addresses are configured on the lo0 IP interface).

--
meem

On 02/05/09 17:38, Michael Lim wrote:
> Try:
> 
> #kstat -c flow -n iscsi
The stats are zero.

- Bob

On 02/06/09 13:20, Peter Memishian wrote:
> FWIW, snoop -I lo0 will
> allow one to see just IP packets flowing over 127.0.0.1 (or whatever
> addresses are configured on the lo0 IP interface).
Thanks.  So,

snoop -I lo0      shows no traffic
snoop -d lo0      shows the traffic
snoop -I e1000g0  shows the traffic
snoop -d e1000g0  show no traffic

Is that what''s expected?

- Bob

> > FWIW, snoop -I lo0 will
 > > allow one to see just IP packets flowing over 127.0.0.1 (or whatever
 > > addresses are configured on the lo0 IP interface).
 > 
 > Thanks.  So,
 > 
 > snoop -I lo0      shows no traffic
 > snoop -d lo0      shows the traffic
 > snoop -I e1000g0  shows the traffic
 > snoop -d e1000g0  show no traffic
 > 
 > Is that what''s expected?

Yes, since it''s being looped-back inside IP.

-- 
meem

>  > snoop -I lo0      shows no traffic
>  > snoop -d lo0      shows the traffic
>  > snoop -I e1000g0  shows the traffic
>  > snoop -d e1000g0  show no traffic
>  > 
>  > Is that what''s expected?
> 
> Yes, since it''s being looped-back inside IP.
OK, so should I expect flowadm to work in this case, or not?

- Bob

Peter Memishian wrote:
>  > > FWIW, snoop -I lo0 will
>  > > allow one to see just IP packets flowing over 127.0.0.1 (or
whatever
>  > > addresses are configured on the lo0 IP interface).
>  > 
>  > Thanks.  So,
>  > 
>  > snoop -I lo0      shows no traffic
>  > snoop -d lo0      shows the traffic
>  > snoop -I e1000g0  shows the traffic
>  > snoop -d e1000g0  show no traffic
>  > 
>  > Is that what''s expected?
 >
> Yes, since it''s being looped-back inside IP.
FWIW I find the above confusing; explaining it by referring to the 
implementation isn''t likely to help the administrator understand it.

 From the implementation perspective, since of the cases except ''snoop 
-d e1000g0'' is looped back internally in IP, thus it seems like we
would
have had the option to make things more uniform.

    Erik

> >  > snoop -I lo0      shows no traffic
 > >  > snoop -d lo0      shows the traffic
 > >  > snoop -I e1000g0  shows the traffic
 > >  > snoop -d e1000g0  show no traffic
 > >  > 
 > >  > Is that what''s expected?
 > > 
 > > Yes, since it''s being looped-back inside IP.
 > 
 > OK, so should I expect flowadm to work in this case, or not?

It won''t work because the packets are never making it into GLDv3.

-- 
meem

> >  > Thanks.  So,
 > >  > 
 > >  > snoop -I lo0      shows no traffic
 > >  > snoop -d lo0      shows the traffic
 > >  > snoop -I e1000g0  shows the traffic
 > >  > snoop -d e1000g0  show no traffic
 > >  > 
 > >  > Is that what''s expected?
 >  >
 > > Yes, since it''s being looped-back inside IP.
 > 
 > FWIW I find the above confusing; explaining it by referring to the 
 > implementation isn''t likely to help the administrator understand
it.
 >
 >  From the implementation perspective, since of the cases except
''snoop
 > -d e1000g0'' is looped back internally in IP, thus it seems like
we would
 > have had the option to make things more uniform.

The semantics for /dev/lo0 are strange because Linux/BSD defined them in a
strange way and we wanted to be compatible with those semantics.  In a
pure Solaris model, /dev/lo0 wouldn''t exist at all because we do not
consider lo0 a datalink.

-- 
meem

On 02/06/09 13:42, Peter Memishian wrote:
>  > OK, so should I expect flowadm to work in this case, or not?
> 
> It won''t work because the packets are never making it into GLDv3.
Ummm, is this thought to be a reasonable state of affairs?

- Bob

On Fri, Feb 06, 2009 at 01:55:26PM -0500, Bob Scheifler
wrote:
> On 02/06/09 13:42, Peter Memishian wrote:
> >  > OK, so should I expect flowadm to work in this case, or not?
> > 
> > It won''t work because the packets are never making it into
GLDv3.
> 
> Ummm, is this thought to be a reasonable state of affairs?
Crossbow''s tracking of statistics (and hence resource control) is down
in the MAC layer, and hence is MAC oriented. If that''s not what you
need then it''s worth filing an RFE.

Peter Memishian wrote:
> The semantics for /dev/lo0 are strange because Linux/BSD defined them in a
> strange way and we wanted to be compatible with those semantics.  In a
> pure Solaris model, /dev/lo0 wouldn''t exist at all because we do
not
> consider lo0 a datalink.
But how does having different semantics for /dev/net/lo0 than /dev/lo0 
make observing the system any easier?

That is the most confusing part to me.

    Erik

> > The semantics for /dev/lo0 are strange because Linux/BSD defined them
in a
 > > strange way and we wanted to be compatible with those semantics.  In
a
 > > pure Solaris model, /dev/lo0 wouldn''t exist at all because
we do not
 > > consider lo0 a datalink.
 > 
 > But how does having different semantics for /dev/net/lo0 than /dev/lo0 
 > make observing the system any easier?

There is no such thing as /dev/net/lo0.  That''s exactly my point above.

There is a /dev/lo0 and a /dev/ipnet/lo0.  /dev/lo0 provide IP loopback
packet observability equivalent to BSD/Linux, using the same name in the
filesystem that they use.  /dev/ipnet/lo0 is semantically like all of the
other /dev/ipnet/* nodes -- it provides IP packet monitoring for just the
lo0 IP interface.

-- 
meem

Peter Memishian wrote:
>  > > The semantics for /dev/lo0 are strange because Linux/BSD defined
them in a
>  > > strange way and we wanted to be compatible with those semantics.
In a
>  > > pure Solaris model, /dev/lo0 wouldn''t exist at all
because we do not
>  > > consider lo0 a datalink.
>  > 
>  > But how does having different semantics for /dev/net/lo0 than
/dev/lo0
>  > make observing the system any easier?
> 
> There is no such thing as /dev/net/lo0.  That''s exactly my point
above.
Sorry, s/net/ipnet/ in my comment.
> There is a /dev/lo0 and a /dev/ipnet/lo0.  /dev/lo0 provide IP loopback
> packet observability equivalent to BSD/Linux, using the same name in the
> filesystem that they use.  /dev/ipnet/lo0 is semantically like all of the
> other /dev/ipnet/* nodes -- it provides IP packet monitoring for just the
> lo0 IP interface.
But that doesn''t explain why /dev/ipnet/lo0 and /dev/lo0 needs to show 
different sets of packets. Further, if we think familiarity is 
important, than it would make sense to base the behavior on /dev/lo0 and 
adjust ipnet accordingly. Do you have an argument that the behavior of 
/dev/ipnet/lo0 is so much better than BSD/Linux lo0 that it can 
compensate for the unfamiliarity?

    Erik

> > There is a /dev/lo0 and a /dev/ipnet/lo0.  /dev/lo0 provide IP
loopback
 > > packet observability equivalent to BSD/Linux, using the same name in
the
 > > filesystem that they use.  /dev/ipnet/lo0 is semantically like all of
the
 > > other /dev/ipnet/* nodes -- it provides IP packet monitoring for just
the
 > > lo0 IP interface.
 > 
 > But that doesn''t explain why /dev/ipnet/lo0 and /dev/lo0 needs to
show
 > different sets of packets.

We want all of the nodes in /dev/ipnet to have the same semantics -- e.g.,
/dev/ipnet/lo0 has the same semantics as /dev/ipnet/nxge0.  We also want
/dev/lo0 to mirror BSD/Linux.  The /dev/lo0 semantics make no sense for
other IP interfaces like nxge0.  Thus, /dev/ipnet/lo0 and /dev/lo0 must
have different semantics.

 > Do you have an argument that the behavior of /dev/ipnet/lo0 is so much
 > better than BSD/Linux lo0 that it can compensate for the unfamiliarity?

It''s not better or worse, it''s just a different model
that''s based on an
"IP interface view" of the packets rather than "packet type"
(loopback
vs. non-loopback) view of the packets.  The IP interface view is a natural
view for generally looking at traffic at the IP layer which both mirrors
the semantics at the datalink layer available through /dev/net and also
allows us to leverage IP interface abstractions like IPMP interfaces and
IP tunnels.

-- 
meem

Bob,

As Dave mentioned, all the virtualization/flows etc that Crossbow
implements is a MAC layer. In general the loop back is done
higher in the stack and hence the problem.

It would be useful if you can give us some details on what you
are trying to accomplish and we can try and help you. If you
are using virtual machines and trying to track traffic between
virtual machines, then setting the flows from Host OS would work
(we need to fix the CR 6778531 which already got fixed in
crossbow-gate and should be available in snv109).

Alternatively, you can use zones within the same guest/kernel
over etherstubs forcing everything to go through mac layer and
flows would work.

Cheers,
Sunay

Bob Scheifler wrote:
> On 02/06/09 13:42, Peter Memishian wrote:
>>  > OK, so should I expect flowadm to work in this case, or not?
>>
>> It won''t work because the packets are never making it into
GLDv3.
> 
> Ummm, is this thought to be a reasonable state of affairs?
> 
> - Bob
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

> As Dave mentioned, all the virtualization/flows etc that Crossbow
> implements is a MAC layer. In general the loop back is done
> higher in the stack and hence the problem.
Is there anywhere in the man pages where the consequences of this
are explained to the developer?  That is, where is it stated
which traffic can and can''t be captured with a flow?
> It would be useful if you can give us some details on what you
> are trying to accomplish and we can try and help you.
The hoped for experiment was to see how well bandwidth limits
might serve for allocating local disk I/O capacity.  The idea was,
instead of attaching ZFS volumes directly to xVM guests, to
export them as iSCSI targets in dom0, also placing the
initiators in dom0, attach those initiators to guests, and use
Crossbow to set bandwidth limits.  A further experiment down
the road might have been to try a packet rate limit (not a
current Crossbow feature) as an approximation to an IOPS limit.
> If you
> are using virtual machines and trying to track traffic between
> virtual machines, then setting the flows from Host OS would work
> (we need to fix the CR 6778531 which already got fixed in
> crossbow-gate and should be available in snv109).
I''m also interested in that, but it''s somewhat separable.
It would be good to have clarity here.  You''re saying that once
CR 6778531 is fixed, that a flow on a guest VNIC will in fact
include traffic from other guests on the same host?
> Alternatively, you can use zones within the same guest/kernel
> over etherstubs forcing everything to go through mac layer and
> flows would work.
I''m looking for something invisible to guests, and I don''t
think
I can put each xVM guest in a separate zone.  What attributes
force traffic through the MAC layer?

- Bob

On Mon, Feb 09, 2009 at 02:30:52PM -0500, Bob Scheifler
wrote:
> > If you
> > are using virtual machines and trying to track traffic between
> > virtual machines, then setting the flows from Host OS would work
> > (we need to fix the CR 6778531 which already got fixed in
> > crossbow-gate and should be available in snv109).
> 
> I''m also interested in that, but it''s somewhat separable.
It would
> be good to have clarity here.  You''re saying that once CR 6778531
is
> fixed, that a flow on a guest VNIC will in fact include traffic from
> other guests on the same host?
Yes, I believe so. Those packets go through the MAC layer.

The problem with your iSCSI experiment is that the packets are looped
around inside IP.
> > Alternatively, you can use zones within the same guest/kernel
> > over etherstubs forcing everything to go through mac layer and
> > flows would work.
> 
> I''m looking for something invisible to guests, and I
don''t think
> I can put each xVM guest in a separate zone.  What attributes
> force traffic through the MAC layer?
If IP decides that a destination is local it won''t pass the packets
down to the MAC layer.

Hmm. I wonder if you could run the initiator in one (exclusive stack)
non-global zone and the responder on another (exclusive stack)
non-global zone. Link these two zones together using either an
etherstub or VNICs. Packets between the initiator and the responder
would then flow through the MAC layer.

> Hmm. I wonder if you could run the initiator in one (exclusive stack)
> non-global zone and the responder on another (exclusive stack)
> non-global zone. Link these two zones together using either an
> etherstub or VNICs. Packets between the initiator and the responder
> would then flow through the MAC layer.
For the base networking, it appears that it would work to use an
etherstub with 2 vnics, one vnic in an exclusive stack non-global zone,
and one vnic in the global zone.

Unfortunately, iSCSI targets don''t appear to be supported in
non-global zones.  And apparently neither are iSCSI initiators,
for that matter. :(

- Bob