Hi, with reference to RHBZ#713585: It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the gratuitous ARP packet is not forwarded to the affected "networking equipment". The netback vif is added to a routed bridge in the host(s) and external hosts are expeted to have connection to the guest at all times, no matter the current Xen host. I experimented a bit with tcpdump, and the gARP does appear on the netfront interface. It also appears on the host bridge if sufficient time passes between completing the xenbus handshake and sending the gARP. When the guest queues eg. three gARPs in rapid succession, a variable number of them gets lost. (When all such packets disappear, then the migrated guest becomes invisible to the outside world, until it initiates network traffic on its own.) When the guest waits for about half a second before sending (queueing), the very first gARP packet successfully appears on the host bridge. I suspect it''s a timing race against the netback vif being added to the host bridge. What would be a good countermeasure? - Adding two modparams to xen-netfront (gARP requeue count & number of msecs to wait between queueing the gARPs). - (Paolo''s idea:) watching the "hotplug-status" xenstore node and sending a single gARP when the watch fires with "connected". This node belongs to the backend xenstore subtree, thus watching it from the guest doesn''t please the architecture astronaut in me. - Something else. Sorry for the naivety / verbiage. Thanks, lacos _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 06/28/2011 03:01 PM, Laszlo Ersek wrote:> - (Paolo''s idea:) watching the "hotplug-status" xenstore node and > sending a single gARP when the watch fires with "connected". This node > belongs to the backend xenstore subtree, thus watching it from the guest > doesn''t please the architecture astronaut in me.Note that watching the backend and reading its information is quite common. In fact, that''s how the state of the backend is observed in the first place. Of course you cannot write to the backend tree, but you do not have to do that. Paolo _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-06-28 at 14:01 +0100, Laszlo Ersek wrote:> Hi, > > with reference to RHBZ#713585: > > It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the > gratuitous ARP packet is not forwarded to the affected "networking > equipment". The netback vif is added to a routed bridge in the host(s) > and external hosts are expeted to have connection to the guest at all > times, no matter the current Xen host. > > I experimented a bit with tcpdump, and the gARP does appear on the > netfront interface. It also appears on the host bridge if sufficient > time passes between completing the xenbus handshake and sending the gARP. > > When the guest queues eg. three gARPs in rapid succession, a variable > number of them gets lost. (When all such packets disappear, then the > migrated guest becomes invisible to the outside world, until it > initiates network traffic on its own.) > > When the guest waits for about half a second before sending (queueing), > the very first gARP packet successfully appears on the host bridge. > > I suspect it''s a timing race against the netback vif being added to the > host bridge. What would be a good countermeasure? > > - Adding two modparams to xen-netfront (gARP requeue count & number of > msecs to wait between queueing the gARPs). > - (Paolo''s idea:) watching the "hotplug-status" xenstore node and > sending a single gARP when the watch fires with "connected". This node > belongs to the backend xenstore subtree, thus watching it from the guest > doesn''t please the architecture astronaut in me.netback already waits (or should...) for hotplug-status to fire with "connected" before moving to state XenbusStateConnected. See hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need either the netback in upstream or something newer than 43223efd9bfd (C Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit fixes pretty much the issue you describe. I expected that netfront waited for the backend to hit XenbusStateConnected before sending the grat ARP but instead I find it happens when the backend hits XenbusStateInitWait. I''m not sure if that is a problem -- it appears to have been done this way since forever (even back in the classic Xen kernels) and I''ve never noticed a gARP go missing in the way you describe, but perhaps something isn''t quite matching up any more. Ian.> - Something else. > > Sorry for the naivety / verbiage. > > Thanks, > lacos > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-06-28 at 15:01 +0200, Laszlo Ersek wrote: [...]> When the guest waits for about half a second before sending (queueing), > the very first gARP packet successfully appears on the host bridge. > > I suspect it''s a timing race against the netback vif being added to the > host bridge. What would be a good countermeasure? > > - Adding two modparams to xen-netfront (gARP requeue count & number of > msecs to wait between queueing the gARPs).Note that peer notifications are indirected through netdev notifiers and now include IPv6 NAs as well as ARPs. If repeated notifications are commonly necessary then this should probably be handled in the protocol (or in the networking core). However this sounds like a workaround whereas your other option would be a proper fix:> - (Paolo''s idea:) watching the "hotplug-status" xenstore node and > sending a single gARP when the watch fires with "connected". This node > belongs to the backend xenstore subtree, thus watching it from the guest > doesn''t please the architecture astronaut in me.[...] -- Ben Hutchings, Senior Software Engineer, Solarflare Not speaking for my employer; that''s the marketing department''s job. They asked us to note that Solarflare product names are trademarked. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Laszlo Ersek
2011-Jul-11 14:22 UTC
[Xen-devel] [PATCH] xen-netfront: delay gARP until backend switches to Connected
On 06/28/11 15:33, Ian Campbell wrote:> netback already waits (or should...) for hotplug-status to fire with > "connected" before moving to state XenbusStateConnected. See > hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need > either the netback in upstream or something newer than 43223efd9bfd (C > Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit > fixes pretty much the issue you describe. > > I expected that netfront waited for the backend to hit > XenbusStateConnected before sending the grat ARP but instead I find it > happens when the backend hits XenbusStateInitWait. I''m not sure if that > is a problem -- it appears to have been done this way since forever > (even back in the classic Xen kernels) and I''ve never noticed a gARP go > missing in the way you describe, but perhaps something isn''t quite > matching up any more.In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either. Thanks for the help and for considering this one. Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c index 3f71199..214c3a4 100644 --- a/drivers/net/xen-netfront.c +++ b/drivers/net/xen-netfront.c @@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev, case XenbusStateInitialised: case XenbusStateReconfiguring: case XenbusStateReconfigured: - case XenbusStateConnected: case XenbusStateUnknown: case XenbusStateClosed: break; @@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev, if (xennet_connect(netdev) != 0) break; xenbus_switch_state(dev, XenbusStateConnected); + break; + + case XenbusStateConnected: netif_notify_peers(netdev); break; -- 1.7.4.4 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jul-12 18:19 UTC
Re: [Xen-devel] [PATCH] xen-netfront: delay gARP until backend switches to Connected
> migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest > (without the patch), and the backport to the host side alone was not sufficient > there either.I am having a hard time parsing that. It sounds like this patch is also neccessary for upstream kernel? Or do you have another one for the upstream kernel that you are going to submit soon? (If so, please CC David Miller on it).> > Thanks for the help and for considering this one. > > Signed-off-by: Laszlo Ersek <lersek@redhat.com> > --- > drivers/net/xen-netfront.c | 4 +++- > 1 files changed, 3 insertions(+), 1 deletions(-) > > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c > index 3f71199..214c3a4 100644 > --- a/drivers/net/xen-netfront.c > +++ b/drivers/net/xen-netfront.c > @@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev, > case XenbusStateInitialised: > case XenbusStateReconfiguring: > case XenbusStateReconfigured: > - case XenbusStateConnected: > case XenbusStateUnknown: > case XenbusStateClosed: > break; > @@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev, > if (xennet_connect(netdev) != 0) > break; > xenbus_switch_state(dev, XenbusStateConnected); > + break; > + > + case XenbusStateConnected: > netif_notify_peers(netdev); > break; > > -- > 1.7.4.4 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Laszlo Ersek
2011-Jul-13 11:44 UTC
[Xen-devel] [PATCH v2] xen-netfront: delay gARP until backend switches to Connected
In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the following in the RHEL-6 guest, in order to fix the network outage after live migration. I also tested a Fedora-15 guest (without the patch), and the backport to the host side alone was not sufficient there either. Therefore I think it could be useful to apply this guest-side patch to upstream as well. Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585 v1->v2: "port" to Linus''s tree; add RHBZ ref. Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c index d29365a..f033656 100644 --- a/drivers/net/xen-netfront.c +++ b/drivers/net/xen-netfront.c @@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev, case XenbusStateInitialised: case XenbusStateReconfiguring: case XenbusStateReconfigured: - case XenbusStateConnected: case XenbusStateUnknown: case XenbusStateClosed: break; @@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev, if (xennet_connect(netdev) != 0) break; xenbus_switch_state(dev, XenbusStateConnected); + break; + + case XenbusStateConnected: netif_notify_peers(netdev); break; -- 1.7.4.4 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jul-13 13:29 UTC
Re: [Xen-devel] [PATCH v2] xen-netfront: delay gARP until backend switches to Connected
On Wed, Jul 13, 2011 at 01:44:47PM +0200, Laszlo Ersek wrote:> In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the > following in the RHEL-6 guest, in order to fix the network outage after live > migration. I also tested a Fedora-15 guest (without the patch), and theLaszlo, This description is .. well, pointless for upstream patches. Just succinctly describe the problem, how to reproduce it, and what this patch does. And do mention more about 43223efd9bfd, like "this is the front counterpart of patch xen: netback: wait for hotplug scripts to complete before signalling connected to frontend (43223efd9bfd)"> backport to the host side alone was not sufficient there either. Therefore I > think it could be useful to apply this guest-side patch to upstream as well. > > Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585 > > v1->v2: "port" to Linus''s tree; add RHBZ ref. > > Signed-off-by: Laszlo Ersek <lersek@redhat.com> > --- > drivers/net/xen-netfront.c | 4 +++- > 1 files changed, 3 insertions(+), 1 deletions(-) > > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c > index d29365a..f033656 100644 > --- a/drivers/net/xen-netfront.c > +++ b/drivers/net/xen-netfront.c > @@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev, > case XenbusStateInitialised: > case XenbusStateReconfiguring: > case XenbusStateReconfigured: > - case XenbusStateConnected: > case XenbusStateUnknown: > case XenbusStateClosed: > break; > @@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev, > if (xennet_connect(netdev) != 0) > break; > xenbus_switch_state(dev, XenbusStateConnected); > + break; > + > + case XenbusStateConnected: > netif_notify_peers(netdev); > break; > > -- > 1.7.4.4 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Laszlo Ersek
2011-Jul-13 16:11 UTC
[Xen-devel] [PATCH v3] xen-netfront: delay gARP until backend switches to Connected
On 07/13/11 15:29, Konrad Rzeszutek Wilk wrote:> On Wed, Jul 13, 2011 at 01:44:47PM +0200, Laszlo Ersek wrote: >> In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed >> the following in the RHEL-6 guest, in order to fix the network outage after >> live migration. I also tested a Fedora-15 guest (without the patch), and the > > Laszlo, > > This description is .. well, pointless for upstream patches. Just succinctly > describe the problem, how to reproduce it, and what this patch does.I avoided writing up the commit message myself because in my first posting [1] I quoted two paragraphs from Ian Campbell [2]. I intended those two cited paragraphs verbatim as the commit message (along with my short remarks), because they describe the problem exactly. Anyway: After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP message, so that networking hardware on the target host''s subnet can take notice, and public routing to the guest is re-established. However, if the packet appears on the backend interface before the backend is added to the target host''s bridge, the packet is lost, and the migrated guest''s peers become unable to talk to the guest. A sufficient two-parts condition to prevent the above is: (1) ensure that the backend only moves to Connected xenbus state after its hotplug scripts completed, ie. the netback interface got added to the bridge; and (2) ensure the frontend only queues the gARP when it sees the backend move to Connected. These two together provide complete ordering. Sub-condition (1) is satisfied by pvops commit 43223efd9bfd. In general, the full condition is sufficient, not necessary, because, according to [2], live migration has been working for a long time without satisfying sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5 host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch intends to provide (2) for upstream. [1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html [2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- drivers/net/xen-netfront.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c index d29365a..f033656 100644 --- a/drivers/net/xen-netfront.c +++ b/drivers/net/xen-netfront.c @@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev, case XenbusStateInitialised: case XenbusStateReconfiguring: case XenbusStateReconfigured: - case XenbusStateConnected: case XenbusStateUnknown: case XenbusStateClosed: break; @@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev, if (xennet_connect(netdev) != 0) break; xenbus_switch_state(dev, XenbusStateConnected); + break; + + case XenbusStateConnected: netif_notify_peers(netdev); break; -- 1.7.4.4 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Jul-14 07:44 UTC
Re: [Xen-devel] [PATCH v3] xen-netfront: delay gARP until backend switches to Connected
On Wed, 2011-07-13 at 17:11 +0100, Laszlo Ersek wrote:> On 07/13/11 15:29, Konrad Rzeszutek Wilk wrote: > > On Wed, Jul 13, 2011 at 01:44:47PM +0200, Laszlo Ersek wrote: > >> In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed > >> the following in the RHEL-6 guest, in order to fix the network outage after > >> live migration. I also tested a Fedora-15 guest (without the patch), and the > > > > Laszlo, > > > > This description is .. well, pointless for upstream patches. Just succinctly > > describe the problem, how to reproduce it, and what this patch does. > > I avoided writing up the commit message myself because in my first posting [1] > I quoted two paragraphs from Ian Campbell [2]. I intended those two cited > paragraphs verbatim as the commit message (along with my short remarks), > because they describe the problem exactly.Please don''t expect/require that maintainers trawl around and construct a commit message for you, always propose the full text you would like to see committed in each patch posting.> Anyway: > > After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP > message, so that networking hardware on the target host''s subnet can take > notice, and public routing to the guest is re-established. However, if the > packet appears on the backend interface before the backend is added to the > target host''s bridge, the packet is lost, and the migrated guest''s peers become > unable to talk to the guest. > > A sufficient two-parts condition to prevent the above is: > > (1) ensure that the backend only moves to Connected xenbus state after its > hotplug scripts completed, ie. the netback interface got added to the bridge; > and > > (2) ensure the frontend only queues the gARP when it sees the backend move to > Connected. > > These two together provide complete ordering. Sub-condition (1) is satisfied > by pvops commit 43223efd9bfd. > > In general, the full condition is sufficient, not necessary, because, according > to [2], live migration has been working for a long time without satisfying > sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5 > host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch > intends to provide (2) for upstream.I expect that 43223efd9bfd just reduces the window (enough to prevent it occuring most of the time) but doesn''t actually close it. This change seems good to me.> [1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html > [2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html > > Signed-off-by: Laszlo Ersek <lersek@redhat.com>Reviewed-by: Ian Campbell <ian.campbell@citrix.com>> --- > drivers/net/xen-netfront.c | 4 +++- > 1 files changed, 3 insertions(+), 1 deletions(-) > > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c > index d29365a..f033656 100644 > --- a/drivers/net/xen-netfront.c > +++ b/drivers/net/xen-netfront.c > @@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev, > case XenbusStateInitialised: > case XenbusStateReconfiguring: > case XenbusStateReconfigured: > - case XenbusStateConnected: > case XenbusStateUnknown: > case XenbusStateClosed: > break; > @@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev, > if (xennet_connect(netdev) != 0) > break; > xenbus_switch_state(dev, XenbusStateConnected); > + break; > + > + case XenbusStateConnected: > netif_notify_peers(netdev); > break; >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel