xen discuss - May 2008 - bge adapter, paravirtualized domU, no network traffic from/to DomU

I''ve been trying to setup a paravirtualized domUs on OS200805 and SXCE
b89. domUs work fine except for the networking. I''m pretty sure
I''m doing everything correctly. I''ve setup a number of (HVM)
domUs on different machines successfully.

I can start domU just fine. I simply can''t use the network interface
from it. I can''t see any incoming packets on the network interface in
domU with tcpdump even when I can see them on the dom0''s interface
(when pinging domU''s ip from some other physical machine on the
network).

I''m wondering whether this is a network driver problem or some
misconfiguration. I''m having the same problem on OS200805 and SXCE b89
with Ubuntu Hardy and OS200805 guests. CPU in the machine doesn''t
support HVMs (Dell Precision 380) so I can''t test them. I''ll
try with a different network adapter tomorrow.

Any help would be appreciated.

Here are some details about the SXCE setup (OS200805 is set up in a similar
way):

--- Dom0 ---
# isainfo 
amd64 i386

# uname -a
SunOS unknown 5.11 snv_89 i86pc i386 i86xpv

# syseventadm list|grep xen
class=EC_xendev /usr/lib/xen/scripts/xpvd-event action=$subclass domain=$domain
vdev=$vdev device=$device devclass=$devclass febe=$fob
class=EC_xpvsys /usr/lib/xen/scripts/xpvsys-event subclass=$subclass
shutdown=$shutdown

# virsh dumpxml vm1 #sorry, had to re-format the output for the forum
[domain type=''xen'' id=''3'']
  [name]vm1[/name]
  [uuid]501df921-f2cb-e419-5fe5-4564f681af38[/uuid]
  [bootloader]/usr/lib/xen/bin/pygrub[/bootloader]
  [os]
    [type]linux[/type]
  [/os]
  [memory]524288[/memory]
  [vcpu]1[/vcpu]
  [on_poweroff]destroy[/on_poweroff]
  [on_reboot]restart[/on_reboot]
  [on_crash]restart[/on_crash]
  [devices]
    [interface type=''ethernet'']
      [target dev=''vif3.0''/]
      [mac address=''00:16:3e:39:d3:a1''/]
      [script path=''vif-vnic''/]
    [/interface]
    [disk type=''block'' device=''disk'']
      [driver name=''phy''/]
      [source
dev=''/dev/dsk/c2t010000137252A86B00002A004768F9BDd0p0''/]
      [target dev=''xvda1''/]
    [/disk]
    [console tty=''/dev/pts/4''/]
  [/devices]
[/domain]

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1453     2     r-----    595.6
vm1                                          3   512     1     -b----    203.0

# ifconfig bge0
bge0: flags=201004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS> mtu
1500 index 2
        inet 10.20.30.123 netmask ffffff00 broadcast 10.20.30.255
        ether 0:12:3f:74:e8:37 

# /usr/lib/vna 
vnic2   bge0    0:16:3e:39:d3:a1

--- DomU ---
# xm console vm1

# uname -a
Linux ubuntu-hardy 2.6.24-16-xen #1 SMP Thu Apr 10 15:51:04 UTC 2008 i686
GNU/Linux

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:16:3e:39:d3:a1  
          inet addr:10.20.30.28  Bcast:10.20.30.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe39:d3a1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:1308 (1.2 KB)


Regards,

Andrzej
 
 
This message posted from opensolaris.org

I have noticed similar problems when the first interface that is listed 
from the command,

#/usr/sbin/dladm show-link

is not configured properly in Dom0. Xen uses this first interface by 
default. Here is how you can override the default,

# svccfg -s xvm/xend setprop \
config/default-nic = astring: <your interface>
# svcadm refresh xvm/xend
# svcadm restart xvm/xend

The command,
# svccfg -s xvm/xend listprop

will list the current properties of xend.

Augustus.

Andrzej Szeszo wrote:
> I''ve been trying to setup a paravirtualized domUs on OS200805 and
SXCE b89. domUs work fine except for the networking. I''m pretty sure
I''m doing everything correctly. I''ve setup a number of (HVM)
domUs on different machines successfully.
> 
> I can start domU just fine. I simply can''t use the network
interface from it. I can''t see any incoming packets on the network
interface in domU with tcpdump even when I can see them on the dom0''s
interface (when pinging domU''s ip from some other physical machine on
the network).
> 
> I''m wondering whether this is a network driver problem or some
misconfiguration. I''m having the same problem on OS200805 and SXCE b89
with Ubuntu Hardy and OS200805 guests. CPU in the machine doesn''t
support HVMs (Dell Precision 380) so I can''t test them. I''ll
try with a different network adapter tomorrow.
> 
> Any help would be appreciated.
> 
> Here are some details about the SXCE setup (OS200805 is set up in a similar
way):
> 
> --- Dom0 ---
> # isainfo 
> amd64 i386
> 
> # uname -a
> SunOS unknown 5.11 snv_89 i86pc i386 i86xpv
> 
> # syseventadm list|grep xen
> class=EC_xendev /usr/lib/xen/scripts/xpvd-event action=$subclass
domain=$domain vdev=$vdev device=$device devclass=$devclass febe=$fob
> class=EC_xpvsys /usr/lib/xen/scripts/xpvsys-event subclass=$subclass
shutdown=$shutdown
> 
> # virsh dumpxml vm1 #sorry, had to re-format the output for the forum
> [domain type=''xen'' id=''3'']
>   [name]vm1[/name]
>   [uuid]501df921-f2cb-e419-5fe5-4564f681af38[/uuid]
>   [bootloader]/usr/lib/xen/bin/pygrub[/bootloader]
>   [os]
>     [type]linux[/type]
>   [/os]
>   [memory]524288[/memory]
>   [vcpu]1[/vcpu]
>   [on_poweroff]destroy[/on_poweroff]
>   [on_reboot]restart[/on_reboot]
>   [on_crash]restart[/on_crash]
>   [devices]
>     [interface type=''ethernet'']
>       [target dev=''vif3.0''/]
>       [mac address=''00:16:3e:39:d3:a1''/]
>       [script path=''vif-vnic''/]
>     [/interface]
>     [disk type=''block'' device=''disk'']
>       [driver name=''phy''/]
>       [source
dev=''/dev/dsk/c2t010000137252A86B00002A004768F9BDd0p0''/]
>       [target dev=''xvda1''/]
>     [/disk]
>     [console tty=''/dev/pts/4''/]
>   [/devices]
> [/domain]
> 
> # xm list
> Name                                        ID   Mem VCPUs      State  
Time(s)
> Domain-0                                     0  1453     2     r-----   
595.6
> vm1                                          3   512     1     -b----   
203.0
> 
> # ifconfig bge0
> bge0: flags=201004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS>
mtu 1500 index 2
>         inet 10.20.30.123 netmask ffffff00 broadcast 10.20.30.255
>         ether 0:12:3f:74:e8:37 
> 
> # /usr/lib/vna 
> vnic2   bge0    0:16:3e:39:d3:a1
> 
> --- DomU ---
> # xm console vm1
> 
> # uname -a
> Linux ubuntu-hardy 2.6.24-16-xen #1 SMP Thu Apr 10 15:51:04 UTC 2008 i686
GNU/Linux
> 
> # ifconfig eth0
> eth0      Link encap:Ethernet  HWaddr 00:16:3e:39:d3:a1  
>           inet addr:10.20.30.28  Bcast:10.20.30.255  Mask:255.255.255.0
>           inet6 addr: fe80::216:3eff:fe39:d3a1/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:0 (0.0 B)  TX bytes:1308 (1.2 KB)
> 
> 
> Regards,
> 
> Andrzej
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> xen-discuss mailing list
> xen-discuss@opensolaris.org

-- 
Augustus Franklin, Solaris Cluster
http://blogs.sun.com/augustus

On Wed, May 21, 2008 at 7:06 PM, Andrzej Szeszo <aszeszo@gmail.com>
wrote:
> I''m wondering whether this is a network driver problem or some
misconfiguration. I''m having the same problem on OS200805 and SXCE b89
with Ubuntu Hardy and OS200805 guests. CPU in the machine doesn''t
support HVMs (Dell Precision 380) so I can''t test them. I''ll
try with a different network adapter tomorrow.
Sounds like the same problem I was having, also with an Ubuntu domU
running on OS200805.

I found a solution in the thread here:

https://bugs.launchpad.net/ubuntu/+source/xen-3.2/+bug/204010

There''s a bug in the Ubuntu domU kernel and the thread contains a link
to an updated kernel.  This gets the network somewhat working but
there are still issues with UDP as well as dropped TCP connections.
For that, run "ethtool -K eth0 tx off" in the domU.


Ed Plese

Hi Augustus,

Thanks for you reply. That''s not it unfortunately :|

I am very curious if a different adapter will make any difference tomorrow.

Regards,

Andrzej
 
 
This message posted from opensolaris.org

Hi Ed,

Thanks for the reply.

The problem I am having is not Ubuntu specific. I was about to install domU
OS200805 on OS200805 dom0. domU was not able to obtain an ip address over DHCP.
I could not ping the VM after setting the ip config manually either.

I was following the steps from here:
http://blogs.sun.com/dkumar/entry/how_to_create_domu_in
http://blogs.sun.com/mrj/entry/installing_opensolaris_on

and applied this fix:
http://www.opensolaris.org/jive/servlet/JiveServlet/download/53-59684-236449-6409/xen-fix.txt

I couldn''t get past the network config of the domU.

I''m betting this is some driver issue.

Will test a different adapter tomorrow.

Regards,

Andrzej
 
 
This message posted from opensolaris.org

Apologies for multiple messages. I saw few stack traces and thought that the
messages were not sent.

Andrzej
 
 
This message posted from opensolaris.org

On Wed, May 21, 2008 at 7:57 PM, Andrzej Szeszo <aszeszo@gmail.com>
wrote:
> I was following the steps from here:
> http://blogs.sun.com/dkumar/entry/how_to_create_domu_in
> http://blogs.sun.com/mrj/entry/installing_opensolaris_on
>
> and applied this fix:
>
http://www.opensolaris.org/jive/servlet/JiveServlet/download/53-59684-236449-6409/xen-fix.txt
>
> I couldn''t get past the network config of the domU.
I looked at the same resources but hadn''t tried an OS200805 domU.  I
just tested it following the second URL above and the networking works
correctly in the domU so it seems like you''re having a different
problem than I was.


Ed Plese

Hi all,

I spent some time today trying to figure out what is causing my networking
problems in domU on xVM hosts with bge adapters.

Connectivity is fine between dom0 and domU. Connectivity to other machines on
the LAN doesn''t work as expected. I''m experiencing the same
problems on Dell Latitude D410, Dell Latitude D610, Dell Precision 380.

Everything works expected in the same domU when the xVM host is using e1000g
driver (tested with the hardware Intel adapter in Precision 380 and with the
virtual adapter in VMware Fusion).



Here is the log of what I have tested:

172.30.1.1 - ''ultra'' - my workstation
172.30.1.6 - ''xvm1'' - xVM host, bge network adapter, Dell
Latitude D610 xVM
172.30.1.9 - ''xvm2'' - xVM host, e1000g network adapter, VMware
Fusion VM running on my Mac
172.30.1.10 - ''vm0'' - xVM guest, the disk image is on the
workstation and accessed over NFS

All machines are running snv_91 OpenSolaris bits.

/dev/lofi/1 is assigned to the same disk image file on both xVM hosts

The config file used to create the VM:

name = "opensolaris"
memory = "512"
disk = [''phy:/dev/lofi/1,0,w'']
vif = ['''']


[root@xvm1 ~]# ifconfig bge0
bge0: flags=201004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS> mtu
1500 index 2
        inet 172.30.1.6 netmask ffffff00 broadcast 172.30.1.255
        ether 0:12:3f:18:56:6f 
[root@xvm1 ~]# svccfg -s xvm/xend  listprop config/default-nic
config/default-nic  astring  bge0

[root@xvm2 ~]# ifconfig e1000g0
e1000g0: flags=201004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS> mtu
1500 index 2
        inet 172.30.1.9 netmask ffffff00 broadcast 172.30.1.255
        ether 0:c:29:20:a1:f1 
[root@xvm2 ~]# svccfg -s xvm/xend  listprop config/default-nic
config/default-nic  astring  e1000g0


Behavior when the domU is running on xvm2 (host is using e1000g driver):

domU actually got a different IP address from DHCP. I have changed it manually
to 172.30.1.10 from the console.

[aszeszo@ultra ~]$ pfexec arp -d 172.30.1.10
172.30.1.10 (172.30.1.10) deleted
[aszeszo@ultra ~]$ ping -s 172.30.1.10
PING 172.30.1.10: 56 data bytes
64 bytes from 172.30.1.10: icmp_seq=0. time=1.015 ms
64 bytes from 172.30.1.10: icmp_seq=1. time=0.797 ms
64 bytes from 172.30.1.10: icmp_seq=2. time=0.838 ms
^C
----172.30.1.10 PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.797/0.883/1.015/0.116
[aszeszo@ultra ~]$ ssh 172.30.1.10
Last login: Sat Jul  5 20:24:24 2008 from 172.30.1.1
Sun Microsystems Inc.   SunOS 5.11      snv_91  January 2008
[aszeszo@vm0 ~]$

ICMP and SSH works as expected.


Behavior when the domU is running on xvm1 (host is using bge driver):

The domU didn''t get the IP address from DHCP. I set it to 172.30.1.10
manually. I have removed the /etc/resolv.conf file to avoid any DNS related
delays.

[aszeszo@ultra ~]$ pfexec arp -d 172.30.1.10
172.30.1.10 (172.30.1.10) deleted
[aszeszo@ultra ~]$ ping -s 172.30.1.10
PING 172.30.1.10: 56 data bytes
64 bytes from 172.30.1.10: icmp_seq=0. time=0.847 ms
64 bytes from 172.30.1.10: icmp_seq=1. time=0.393 ms
64 bytes from 172.30.1.10: icmp_seq=2. time=0.450 ms
^C
----172.30.1.10 PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.393/0.563/0.847/0.247
[aszeszo@ultra ~]$ ssh 172.30.1.10
^C

Nothing happens... :(

As the networking works fine between dom0 and domU I have ssh''d to the
vm though the xVM host an started tcpdump. Below is the traffic log from when I
was trying to connect to the domU from my workstation. Some outgoing and
incoming SSH packets can be actually seen on the xnf0 interface but the session
doesn''t work unfortunately.

[aszeszo@ultra ~]$ ssh -t xvm1 ssh 172.30.1.10
Last login: Sat Jul  5 19:31:20 2008 from xvm1.chochlik.n
Sun Microsystems Inc.   SunOS 5.11      snv_91  January 2008
[aszeszo@vm0 ~]$ pfexec su -
Sun Microsystems Inc.   SunOS 5.11      snv_91  January 2008
[root@vm0 ~]# tcpdump -ni xnf0 host 172.30.1.1 and port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xnf0, link-type EN10MB (Ethernet), capture size 68 bytes
19:31:45.838418 IP 172.30.1.1.60128 > 172.30.1.10.22: S
372405969:372405969(0) win 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK>
19:31:45.838461 IP 172.30.1.10.22 > 172.30.1.1.60128: S
243913573:243913573(0) ack 372405970 win 49640 <mss 1460,nop,wscale
0,nop,nop,sackOK>
19:31:49.208606 IP 172.30.1.10.22 > 172.30.1.1.60128: S
243913573:243913573(0) ack 372405970 win 49640 <mss 1460,nop,wscale
0,nop,nop,sackOK>
19:31:49.212219 IP 172.30.1.1.60128 > 172.30.1.10.22: S
372405969:372405969(0) win 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK>
19:31:49.212238 IP 172.30.1.10.22 > 172.30.1.1.60128: . ack 1 win 49640
19:31:54.708596 IP 172.30.1.10.22 > 172.30.1.1.57788: FP
233294542:233294562(20) ack 361904448 win 49640
19:31:55.968644 IP 172.30.1.10.22 > 172.30.1.1.60128: S
243913573:243913573(0) ack 372405970 win 49640 <mss 1460,nop,wscale
0,nop,nop,sackOK>
19:31:55.971742 IP 172.30.1.1.60128 > 172.30.1.10.22: S
372405969:372405969(0) win 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK>
19:31:55.971759 IP 172.30.1.10.22 > 172.30.1.1.60128: . ack 1 win 49640
19:32:01.911418 IP 172.30.1.1.57788 > 172.30.1.10.22: F 0:0(0) ack 0 win
49640
19:32:01.911452 IP 172.30.1.10.22 > 172.30.1.1.57788: . ack 1 win 49640
19:32:09.479182 IP 172.30.1.10.22 > 172.30.1.1.60128: S
243913573:243913573(0) ack 372405970 win 49640 <mss 1460,nop,wscale
0,nop,nop,sackOK>
19:32:09.479806 IP 172.30.1.1.60128 > 172.30.1.10.22: S
372405969:372405969(0) win 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK>
19:32:09.479807 IP 172.30.1.1.60128 > 172.30.1.10.22: . ack 1 win 49640
19:32:09.479827 IP 172.30.1.10.22 > 172.30.1.1.60128: . ack 1 win 49640
19:32:09.503181 IP 172.30.1.10.22 > 172.30.1.1.60128: P 1:21(20) ack 1 win
49640
19:32:11.566084 IP 172.30.1.1.60128 > 172.30.1.10.22: F 1:1(0) ack 1 win
49640
19:32:11.566117 IP 172.30.1.10.22 > 172.30.1.1.60128: . ack 2 win 49640
19:32:11.566607 IP 172.30.1.10.22 > 172.30.1.1.60128: F 21:21(0) ack 2 win
49640

Thanks for reading!

Any help would be really appreciated. Please let me know what I can do to help
diagnose the problem. I can re-run the tests, capture the traffic again, etc.

Thank you,

Andrzej
 
 
This message posted from opensolaris.org

On 5 Jul 2008, at 9:06pm, Andrzej Szeszo wrote:
> Any help would be really appreciated. Please let me know what I can  
> do to help diagnose the problem. I can re-run the tests, capture the  
> traffic again, etc.
Checksum offload is probably broken. Set the kernel variable  
''dohwcksum'' to 0 in the guest domain.

Hi David,

Yep, that was it. Thanks a lot for a tip. 

I have disabled checksum offloading in the domU that is using xnf driver but is
it not the dom0 with bge driver that is causing all the trouble? Do you think
this is a bug in xnf or bge, David?

Regards,

Andrzej
 
 
This message posted from opensolaris.org

Looks like checksum offload is broken for rge driver as well.
Workaround is adding to /etc/system at SNV (up to 93) DomU
either (1) set xnf:xnf_cksum_offload = 0 or  (2) set ip:dohwcksum = 0.
I don''t know what is difference between (1) and (2), but it helps.
Through my limited experience  it''s not broken for rtls0


--- On Sat, 7/5/08, David Edmondson <dme@sun.com> wrote:
From: David Edmondson <dme@sun.com>
Subject: Re: [xen-discuss] bge adapter, paravirtualized domU, no network traffic
from/to DomU
To: "Andrzej Szeszo" <aszeszo@gmail.com>
Cc: xen-discuss@opensolaris.org
Date: Saturday, July 5, 2008, 4:42 PM

On 5 Jul 2008, at 9:06pm, Andrzej Szeszo wrote:
> Any help would be really appreciated. Please let me know what I can  
> do to help diagnose the problem. I can re-run the tests, capture the  
> traffic again, etc.
Checksum offload is probably broken. Set the kernel variable  
''dohwcksum'' to 0 in the guest domain.

_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

On 6 Jul 2008, at 12:58am, Andrzej Szeszo wrote:
> Yep, that was it. Thanks a lot for a tip.
Sorry for causing the problem in the first place :-)
> I have disabled checksum offloading in the domU that is using xnf  
> driver but is it not the dom0 with bge driver that is causing all  
> the trouble? Do you think this is a bug in xnf or bge, David?
It''s either xnf or xnb, perhaps both. bge seems fine.

Boris'' problems with rge may actually be a problem with rge itself, as
that is broken even without Xen.

David,
Would it be possible to find  ASUS, GigaByte, Abit , EliteGroup . . . (non
Intel) board with Socket 775 (C2D CPU), no matter of price,which would have
integrated NIC acceptable for Solaris xVM (build 93) ?
Thanks.
-b.

--- On Sun, 7/6/08, David Edmondson <dme@sun.com> wrote:
From: David Edmondson <dme@sun.com>
Subject: Re: [xen-discuss] bge adapter, paravirtualized domU, no network traffic
from/to DomU
To: "Andrzej Szeszo" <aszeszo@gmail.com>
Cc: xen-discuss@opensolaris.org
Date: Sunday, July 6, 2008, 5:00 AM

On 6 Jul 2008, at 12:58am, Andrzej Szeszo wrote:
> Yep, that was it. Thanks a lot for a tip.
Sorry for causing the problem in the first place :-)
> I have disabled checksum offloading in the domU that is using xnf  
> driver but is it not the dom0 with bge driver that is causing all  
> the trouble? Do you think this is a bug in xnf or bge, David?
It''s either xnf or xnb, perhaps both. bge seems fine.

Boris'' problems with rge may actually be a problem with rge itself, as
that is broken even without Xen.

_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org