Jürgen Keil
2007-Aug-30 12:01 UTC
[crossbow-discuss] no xen domU/dom0 network connectivity when external physical link is down?
Assuming I have a system with gldv3 ethernet nic, where the external link is
down,
and the build 66 xen bits: Shouldn''t it be possible to communicate
between a
Solaris domU and the host Solaris dom0?
It seems that (with build66 xen) the local network connection between domU and
dom0
only works, when the external link is up (and IPMP is disabled).
Details:
Tecra S1 laptop, iprb and ipw nics, Solaris dom0, and Solaris domU.
xen b66 bits, iprb glv3 nic driver copied from snv_71.
System is not connected to a network (at least not on iprb, ipw is unplumbed)
# dladm show-dev
ipw0 link: up speed: 0Mb duplex: unknown
iprb0 link: down speed: 0Mb duplex: half
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
inet 127.0.0.1 netmask ff000000
iprb0: flags=201000803<UP,BROADCAST,MULTICAST,IPv4,CoS> mtu 1500 index 2
inet 192.168.2.21 netmask ffffff00 broadcast 192.168.2.255
groupname mobile
ether 0:a0:d1:d5:bc:b4
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252
index 1
inet6 ::1/128
iprb0: flags=202000800<MULTICAST,IPv6,CoS> mtu 1500 index 2
inet6 fe80::2a0:d1ff:fed5:bcb4/10
groupname mobile
ether 0:a0:d1:d5:bc:b4
ipw0: flags=202100840<RUNNING,MULTICAST,ROUTER,IPv6,CoS> mtu 1500 index 3
inet6 fe80::204:23ff:fe4f:69bf/10
ether 0:4:23:4f:69:bf
ipw0:1: flags=202180841<UP,RUNNING,MULTICAST,ADDRCONF,ROUTER,IPv6,CoS> mtu
1500 index 3
inet6 fec0::2:204:23ff:fe4f:69bf/64
# arp -an
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- -------- ---------------
iprb0 192.168.2.21 255.255.255.255 SPLdA 00:a0:d1:d5:bc:b4
iprb0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00
Note: Flags ''d'' for iprb0''s 192.168.2.21 address:
d Unverified; this is a local IP address that is
currently undergoing Duplicate Address Detection.
ARP will not respond to requests for this address
until Duplicate Address Detection completes.
Problem: no network connectivity from domU to dom0. Test is an attempt to
telnet from
domU to ip address 192.168.2.21 (ip address of iprb0 in dom0).
Problem #1:
==========
arp requests from domU to dom0 are dropped by dom0. dtrace show this:
...
0 -> putnext
0 -> ar_rput
0 | ar_rput:rput_normal
0 | ar_rput:arp-physical-in-start
0 | ar_rput:arp-physical-in-end
0 -> ar_ce_resolve_all
0 -> ar_ce_hash
0 <- ar_ce_hash returns d57bc35c
0 <- ar_ce_resolve_all returns 1
0 -> ar_ce_lookup_entry
0 | ar_ce_lookup_entry:entry arl d7c05774, proto 800,
dst_paddr 192.168.2.21 plen 4
0 -> ar_ce_hash
0 <- ar_ce_hash returns d57bc450
0 <- ar_ce_lookup_entry returns e9586654
0 -> bcmp
0 <- bcmp returns 1
0 | ar_rput:rput_request_unverified
0 -> freemsg
0 -> dblk_lastfree_desb
0 -> xnb_rx_complete
0 -> xnb_rx_schedule_unmop
0 <- xnb_rx_schedule_unmop returns e0687000
0 -> xnb_rx_perform_pending_unmop
0 -> HYPERVISOR_grant_table_op
0 <- HYPERVISOR_grant_table_op returns 0
0 <- xnb_rx_perform_pending_unmop returns 0
...
usr/src/uts/common/inet/arp/arp.c:
3280 /*
3281 * If the target is unverified by DAD, then one of two things is
true:
3282 * either it''s someone else claiming this address (on a
probe or an
3283 * announcement) or it''s just a regular request. The
former is
3284 * failure, but a regular request is not.
3285 */
3286 if (dst_ace->ace_flags & ACE_F_UNVERIFIED) {
3287 /*
3288 * Check for a reflection. Some misbehaving bridges
will
3289 * reflect our own transmitted packets back to us.
3290 */
3291 if (hlen == dst_ace->ace_hw_addr_length &&
3292 bcmp(src_haddr, dst_ace->ace_hw_addr, hlen) == 0)
{
3293 DTRACE_PROBE3(rput_probe_reflected, arl_t *,
arl,
3294 arh_t *, arh, ace_t *, dst_ace);
3295 freeb(mp);
3296 freemsg(mp1);
3297 TRACE_2(TR_FAC_ARP, TR_ARP_RPUT_END,
3298 "arp_rput_end: q %p (%S)", q,
"reflection");
3299 return;
3300 }
3301 if (is_probe || op == ARP_RESPONSE) {
3302 ar_client_notify(arl, mp1, AR_CN_FAILED);
3303 ar_ce_delete(dst_ace);
3304 } else if (err == AR_CHANGED) {
3305 ar_client_notify(arl, mp1, AR_CN_ANNOUNCE);
3306 } else {
3307 DTRACE_PROBE3(rput_request_unverified, arl_t *,
arl,
3308 arh_t *, arh, ace_t *, dst_ace);
3309 freemsg(mp1);
3310 }
3311 freeb(mp);
3312 TRACE_2(TR_FAC_ARP, TR_ARP_RPUT_END,
3313 "arp_rput_end: q %p (%S)", q,
"unverified");
3314 return;
3315 }
Problem #2:
==========
ARP duplicate address detection can be disabled, but there still isn''t
"network" connectivity between domU and dom0:
# ndd -set /dev/arp arp_fastprobe_count 0
# ifconfig iprb0 inet 0
# ifconfig iprb0 inet max
# arp -an
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- -------- ---------------
iprb0 192.168.2.21 255.255.255.255 SPLA 00:a0:d1:d5:bc:b4
iprb0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00
dtrace shows that we''re now trying to xmit something, but the response
messages seem to be dropped immediately in ar_xmit():
0 -> putnext
0 -> ar_rput
0 | ar_rput:rput_normal
0 | ar_rput:arp-physical-in-start
0 | ar_rput:arp-physical-in-end
0 -> ar_ce_resolve_all
0 -> ar_ce_hash
0 <- ar_ce_hash returns d57bc35c
0 -> ar_ce_resolve
0 -> bcmp
0 <- bcmp returns 0
0 <- ar_ce_resolve returns 0
0 <- ar_ce_resolve_all returns 2
0 -> ar_ce_lookup_entry
0 | ar_ce_lookup_entry:entry arl d7c05774, proto 800,
dst_paddr 192.168.2.21 plen 4
0 -> ar_ce_hash
0 <- ar_ce_hash returns d57bc450
0 <- ar_ce_lookup_entry returns e95861d4
0 -> ddi_get_lbolt
0 <- ddi_get_lbolt returns 767f2
0 -> ar_xmit
0 | ar_xmit:entry arl d7c05774 ap d78136c0
ap->ap_link_down 1
0 <- ar_xmit returns 3
0 -> freemsg
0 -> dblk_lastfree
0 -> kmem_cache_free
0 <- kmem_cache_free returns a4a1
0 <- dblk_lastfree returns a4a1
0 -> dblk_lastfree_desb
0 -> xnb_rx_complete
0 -> xnb_rx_schedule_unmop
0 <- xnb_rx_schedule_unmop returns e0687000
It seems arp does not send replies because the iprb0 link is down:
4271 /* ar_xmit is called to transmit an ARP Request or Response. */
4272 static void
4273 ar_xmit(arl_t *arl, uint32_t operation, uint32_t proto, uint32_t plen,
4274 const uchar_t *haddr1, const uchar_t *paddr1, const uchar_t *haddr2,
4275 const uchar_t *paddr2, const uchar_t *dstaddr, arp_stack_t *as)
4276 {
4277 arh_t *arh;
4278 uint8_t *cp;
4279 uint_t hlen;
4280 mblk_t *mp;
4281 arlphy_t *ap = arl->arl_phy;
4282
4283 if (ap == NULL) {
4284 DTRACE_PROBE1(xmit_no_arl_phy, arl_t *, arl);
4285 return;
4286 }
4287
4288 /* IFF_NOARP flag is set or link down: do not send arp messages
*/
4289 if ((arl->arl_flags & ARL_F_NOARP) ||
ap->ap_link_down)
4290 return;
# mdb -k
Loading modules: [ unix genunix specfs dtrace xpv_uppc scsi_vhci zfs random ip
hook neti sctp arp usba uhci s1394 nca fctl lofs audiosup md crypto fcip fcp
logindmux ptm ufs sppp nfs ipc ]> d78136c0::print arlphy_t
{
ap_arp_hw_type = 0x1
ap_arp_addr = 0xd7d2a528
ap_hw_addr = 0xd8c96d40
ap_hw_addrlen = 0x6
ap_xmit_mp = 0xd8c9a020
ap_xmit_addroff = 0x14
ap_xmit_sapoff = 0x1a
ap_saplen = 0xfffffffe
ap_defend_start = 0
ap_defend_count = 0
ap_notifies = 0x1
ap_link_down = 0x1
}> d7c05774::print arl_t
{
arl_next = 0
arl_rq = 0xd8c98e30
arl_wq = 0xd8c98eb0
arl_ppa = 0
arl_name = [ "iprb0" ]
arl_unbind_mp = 0xd4289a00
arl_detach_mp = 0
arl_provider_style = 0x500
arl_queue = 0
arl_queue_tail = 0
arl_flags = 0
arl_dlpi_pending = 0xffff
arl_dlpi_deferred = 0
arl_state = 0x2
arl_closing = 0
arl_index = 0x1
arl_phy = 0xd78136c0
}
Problem #3:
==========
After connecting iprb0 to a switch; link is now up. interface setup still has a
groupname. Still no network connectivity between dom0 and domU.
# dladm show-dev
ipw0 Link: up speed: 0Mb Duplex: unknown
iprb0 Link: up speed: 100Mb Duplex: full
vnic0 Link: unknown speed: 0Mb Duplex: unknown
# ifconfig iprb0 group "abc"
# ifconfig iprb0
iprb0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500
index 2
inet 192.168.2.21 netmask ff000000 broadcast 192.255.255.255
groupname abc
ether 0:a0:d1:d5:bc:b4
After deleting the groupname, it''s finally possible to telnet from domU
back
into dom0...
This message posted from opensolaris.org
Nicolas Droux
2007-Aug-30 18:21 UTC
[crossbow-discuss] no xen domU/dom0 network connectivity when external physical link is down?
Jurgen, J?rgen Keil wrote:> Assuming I have a system with gldv3 ethernet nic, where the external link is down, > and the build 66 xen bits: Shouldn''t it be possible to communicate between a > Solaris domU and the host Solaris dom0?Yes, that path should be preserved when the physical link is down.> It seems that (with build66 xen) the local network connection between domU and dom0 > only works, when the external link is up (and IPMP is disabled).Thanks for the report, yes it''s a bug. The link state of the primary interface (in this case the interface plumbed by dom0) passed to DLS should be reported as being up if a VNIC is defined on the same NIC. I filed 6599495. Thanks, Nicolas.> > Details: > > Tecra S1 laptop, iprb and ipw nics, Solaris dom0, and Solaris domU. > xen b66 bits, iprb glv3 nic driver copied from snv_71. > > System is not connected to a network (at least not on iprb, ipw is unplumbed) > > # dladm show-dev > ipw0 link: up speed: 0Mb duplex: unknown > iprb0 link: down speed: 0Mb duplex: half > > # ifconfig -a > lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 > inet 127.0.0.1 netmask ff000000 > iprb0: flags=201000803<UP,BROADCAST,MULTICAST,IPv4,CoS> mtu 1500 index 2 > inet 192.168.2.21 netmask ffffff00 broadcast 192.168.2.255 > groupname mobile > ether 0:a0:d1:d5:bc:b4 > lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1 > inet6 ::1/128 > iprb0: flags=202000800<MULTICAST,IPv6,CoS> mtu 1500 index 2 > inet6 fe80::2a0:d1ff:fed5:bcb4/10 > groupname mobile > ether 0:a0:d1:d5:bc:b4 > ipw0: flags=202100840<RUNNING,MULTICAST,ROUTER,IPv6,CoS> mtu 1500 index 3 > inet6 fe80::204:23ff:fe4f:69bf/10 > ether 0:4:23:4f:69:bf > ipw0:1: flags=202180841<UP,RUNNING,MULTICAST,ADDRCONF,ROUTER,IPv6,CoS> mtu 1500 index 3 > inet6 fec0::2:204:23ff:fe4f:69bf/64 > > # arp -an > Net to Media Table: IPv4 > Device IP Address Mask Flags Phys Addr > ------ -------------------- --------------- -------- --------------- > iprb0 192.168.2.21 255.255.255.255 SPLdA 00:a0:d1:d5:bc:b4 > iprb0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00 > > Note: Flags ''d'' for iprb0''s 192.168.2.21 address: > > d Unverified; this is a local IP address that is > currently undergoing Duplicate Address Detection. > ARP will not respond to requests for this address > until Duplicate Address Detection completes. > > > Problem: no network connectivity from domU to dom0. Test is an attempt to telnet from > domU to ip address 192.168.2.21 (ip address of iprb0 in dom0). > > Problem #1: > ==========> > arp requests from domU to dom0 are dropped by dom0. dtrace show this: > > ... > 0 -> putnext > 0 -> ar_rput > 0 | ar_rput:rput_normal > 0 | ar_rput:arp-physical-in-start > 0 | ar_rput:arp-physical-in-end > 0 -> ar_ce_resolve_all > 0 -> ar_ce_hash > 0 <- ar_ce_hash returns d57bc35c > 0 <- ar_ce_resolve_all returns 1 > 0 -> ar_ce_lookup_entry > 0 | ar_ce_lookup_entry:entry arl d7c05774, proto 800, dst_paddr 192.168.2.21 plen 4 > 0 -> ar_ce_hash > 0 <- ar_ce_hash returns d57bc450 > 0 <- ar_ce_lookup_entry returns e9586654 > 0 -> bcmp > 0 <- bcmp returns 1 > 0 | ar_rput:rput_request_unverified > 0 -> freemsg > 0 -> dblk_lastfree_desb > 0 -> xnb_rx_complete > 0 -> xnb_rx_schedule_unmop > 0 <- xnb_rx_schedule_unmop returns e0687000 > 0 -> xnb_rx_perform_pending_unmop > 0 -> HYPERVISOR_grant_table_op > 0 <- HYPERVISOR_grant_table_op returns 0 > 0 <- xnb_rx_perform_pending_unmop returns 0 > ... > > > usr/src/uts/common/inet/arp/arp.c: > > 3280 /* > 3281 * If the target is unverified by DAD, then one of two things is true: > 3282 * either it''s someone else claiming this address (on a probe or an > 3283 * announcement) or it''s just a regular request. The former is > 3284 * failure, but a regular request is not. > 3285 */ > 3286 if (dst_ace->ace_flags & ACE_F_UNVERIFIED) { > 3287 /* > 3288 * Check for a reflection. Some misbehaving bridges will > 3289 * reflect our own transmitted packets back to us. > 3290 */ > 3291 if (hlen == dst_ace->ace_hw_addr_length && > 3292 bcmp(src_haddr, dst_ace->ace_hw_addr, hlen) == 0) { > 3293 DTRACE_PROBE3(rput_probe_reflected, arl_t *, arl, > 3294 arh_t *, arh, ace_t *, dst_ace); > 3295 freeb(mp); > 3296 freemsg(mp1); > 3297 TRACE_2(TR_FAC_ARP, TR_ARP_RPUT_END, > 3298 "arp_rput_end: q %p (%S)", q, "reflection"); > 3299 return; > 3300 } > 3301 if (is_probe || op == ARP_RESPONSE) { > 3302 ar_client_notify(arl, mp1, AR_CN_FAILED); > 3303 ar_ce_delete(dst_ace); > 3304 } else if (err == AR_CHANGED) { > 3305 ar_client_notify(arl, mp1, AR_CN_ANNOUNCE); > 3306 } else { > 3307 DTRACE_PROBE3(rput_request_unverified, arl_t *, arl, > 3308 arh_t *, arh, ace_t *, dst_ace); > 3309 freemsg(mp1); > 3310 } > 3311 freeb(mp); > 3312 TRACE_2(TR_FAC_ARP, TR_ARP_RPUT_END, > 3313 "arp_rput_end: q %p (%S)", q, "unverified"); > 3314 return; > 3315 } > > Problem #2: > ==========> > ARP duplicate address detection can be disabled, but there still isn''t > "network" connectivity between domU and dom0: > > # ndd -set /dev/arp arp_fastprobe_count 0 > # ifconfig iprb0 inet 0 > # ifconfig iprb0 inet max > # arp -an > Net to Media Table: IPv4 > Device IP Address Mask Flags Phys Addr > ------ -------------------- --------------- -------- --------------- > iprb0 192.168.2.21 255.255.255.255 SPLA 00:a0:d1:d5:bc:b4 > iprb0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00 > > > dtrace shows that we''re now trying to xmit something, but the response > messages seem to be dropped immediately in ar_xmit(): > > 0 -> putnext > 0 -> ar_rput > 0 | ar_rput:rput_normal > 0 | ar_rput:arp-physical-in-start > 0 | ar_rput:arp-physical-in-end > 0 -> ar_ce_resolve_all > 0 -> ar_ce_hash > 0 <- ar_ce_hash returns d57bc35c > 0 -> ar_ce_resolve > 0 -> bcmp > 0 <- bcmp returns 0 > 0 <- ar_ce_resolve returns 0 > 0 <- ar_ce_resolve_all returns 2 > 0 -> ar_ce_lookup_entry > 0 | ar_ce_lookup_entry:entry arl d7c05774, proto 800, dst_paddr 192.168.2.21 plen 4 > 0 -> ar_ce_hash > 0 <- ar_ce_hash returns d57bc450 > 0 <- ar_ce_lookup_entry returns e95861d4 > 0 -> ddi_get_lbolt > 0 <- ddi_get_lbolt returns 767f2 > 0 -> ar_xmit > 0 | ar_xmit:entry arl d7c05774 ap d78136c0 ap->ap_link_down 1 > 0 <- ar_xmit returns 3 > 0 -> freemsg > 0 -> dblk_lastfree > 0 -> kmem_cache_free > 0 <- kmem_cache_free returns a4a1 > 0 <- dblk_lastfree returns a4a1 > 0 -> dblk_lastfree_desb > 0 -> xnb_rx_complete > 0 -> xnb_rx_schedule_unmop > 0 <- xnb_rx_schedule_unmop returns e0687000 > > It seems arp does not send replies because the iprb0 link is down: > > 4271 /* ar_xmit is called to transmit an ARP Request or Response. */ > 4272 static void > 4273 ar_xmit(arl_t *arl, uint32_t operation, uint32_t proto, uint32_t plen, > 4274 const uchar_t *haddr1, const uchar_t *paddr1, const uchar_t *haddr2, > 4275 const uchar_t *paddr2, const uchar_t *dstaddr, arp_stack_t *as) > 4276 { > 4277 arh_t *arh; > 4278 uint8_t *cp; > 4279 uint_t hlen; > 4280 mblk_t *mp; > 4281 arlphy_t *ap = arl->arl_phy; > 4282 > 4283 if (ap == NULL) { > 4284 DTRACE_PROBE1(xmit_no_arl_phy, arl_t *, arl); > 4285 return; > 4286 } > 4287 > 4288 /* IFF_NOARP flag is set or link down: do not send arp messages */ > 4289 if ((arl->arl_flags & ARL_F_NOARP) || ap->ap_link_down) > 4290 return; > > > # mdb -k > Loading modules: [ unix genunix specfs dtrace xpv_uppc scsi_vhci zfs random ip hook neti sctp arp usba uhci s1394 nca fctl lofs audiosup md crypto fcip fcp logindmux ptm ufs sppp nfs ipc ] >> d78136c0::print arlphy_t > { > ap_arp_hw_type = 0x1 > ap_arp_addr = 0xd7d2a528 > ap_hw_addr = 0xd8c96d40 > ap_hw_addrlen = 0x6 > ap_xmit_mp = 0xd8c9a020 > ap_xmit_addroff = 0x14 > ap_xmit_sapoff = 0x1a > ap_saplen = 0xfffffffe > ap_defend_start = 0 > ap_defend_count = 0 > ap_notifies = 0x1 > ap_link_down = 0x1 > } >> d7c05774::print arl_t > { > arl_next = 0 > arl_rq = 0xd8c98e30 > arl_wq = 0xd8c98eb0 > arl_ppa = 0 > arl_name = [ "iprb0" ] > arl_unbind_mp = 0xd4289a00 > arl_detach_mp = 0 > arl_provider_style = 0x500 > arl_queue = 0 > arl_queue_tail = 0 > arl_flags = 0 > arl_dlpi_pending = 0xffff > arl_dlpi_deferred = 0 > arl_state = 0x2 > arl_closing = 0 > arl_index = 0x1 > arl_phy = 0xd78136c0 > } > > Problem #3: > ==========> > After connecting iprb0 to a switch; link is now up. interface setup still has a > groupname. Still no network connectivity between dom0 and domU. > > # dladm show-dev > ipw0 Link: up speed: 0Mb Duplex: unknown > iprb0 Link: up speed: 100Mb Duplex: full > vnic0 Link: unknown speed: 0Mb Duplex: unknown > # ifconfig iprb0 group "abc" > # ifconfig iprb0 > iprb0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2 > inet 192.168.2.21 netmask ff000000 broadcast 192.255.255.255 > groupname abc > ether 0:a0:d1:d5:bc:b4 > > After deleting the groupname, it''s finally possible to telnet from domU back > into dom0... > > > This message posted from opensolaris.org > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss-- Nicolas Droux - Solaris Networking - Sun Microsystems, Inc. droux at sun.com - http://blogs.sun.com/droux