Xen devel - Jul 2007 - Stimulating domains to send gratuitous ARPs

Currently, netfront will send a gratuitous ARP when resumed at the end
of a live migration/resume op. However, there are other situations when
it is necessary to send a gratuitous ARP; one good example is when you
bond NICs in Dom0 for availability and plug the bond into the bridge; in
this case, when a failover from primary to backup NIC occurs, you need
to send ARPs for all the domains VIFs on the backup link to get traffic
routed correctly.

 

Now, there are a number of ways of doing this (including using ebtables
in Dom0 to track IP-MAC address translations) but it seems to me that
the simplest would be to have a way to poke the netfront driver to send
an ARP on demand (especially given that fact that it already needs/has
this functionality).

 

I''ve prototyped something fairly simple & gross based on having
netfront
setup a xenstore watch on a leaf in the domains vif directory in
xenstore (that is, device/vif/send_arp); when the watch fires, all
netfront instances send the fake ARP just like they do at the end of
migration/resume. All you need to do from dom0 then is modify the leaf
with xenstore-write/xenstore-rm. 

 

I accept that this is not very nice and we should probably have a more
organized/documented way to do this but I''m not sure what would be
appropriate; any suggestions?

 

Thanks

Simon



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> I accept that this is not very nice and we should probably have a more
> organized/documented way to do this but I’m not sure what would be
> appropriate; any suggestions?
A usespace tool to send such an ARP would be a very useful thing to
have, if it doesn''t already exist.

The gratuitous arping in netfront is not very popular with the upstream
Linux folks so this functionality is not in the paravirt_ops patches
which are going upstream, so we''d need such a tool in the end anyway.

Ian.


 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

There are existing tools for sending arps -- the problem is knowing what IP
address to use in the ARP -- it's actually pretty hard to know this in Dom0
(which is where you know that the ARP needs to be sent) and if you install it in
DomU you need a way to kick the domain to actually call the utility... (plus you
have to install a service in DomU which is something I think is a bad idea to
require).

Whilst I might agree that from a purist point of view having netfront do this is
a violation of layering, it's already accepted in some other upstream
drivers -- bonding for example; this virtual network interface driver will send
gratuitous ARPs for the IP address associated with the bond and for any VLANs
layered directly on top of the bond when failing over (it just wont send ARPs
for things above the bridge if it is plugged into a bridge).

Simon
> -----Original Message-----
> From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> Sent: Thursday, July 05, 2007 3:46 PM
> To: Graham, Simon
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> 
> On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > I accept that this is not very nice and we should probably have a
> more
> > organized/documented way to do this but I’m not sure what would be
> > appropriate; any suggestions?
> 
> A usespace tool to send such an ARP would be a very useful thing to
> have, if it doesn't already exist.
> 
> The gratuitous arping in netfront is not very popular with the upstream
> Linux folks so this functionality is not in the paravirt_ops patches
> which are going upstream, so we'd need such a tool in the end anyway.
> 
> Ian.
> 
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> There are existing tools for sending arps -- the problem is knowing what
> IP address to use in the ARP -- it's actually pretty hard to know this
> in Dom0 (which is where you know that the ARP needs to be sent) and if
> you install it in DomU you need a way to kick the domain to actually
> call the utility... (plus you have to install a service in DomU which is
> something I think is a bad idea to require).
> 
> Whilst I might agree that from a purist point of view having netfront do
> this is a violation of layering, it's already accepted in some other
> upstream drivers 
It's pretty clear it's not going to be accepted in the netfront that
goes into upstream Linux, at least not without sedating various LKML folk first
:)

For bond case you described, you don't need to send ARPs, you just need to
send any packet with the right src MAC to cause the switches to reconfigure.
Since the bond driver likely already maintains a list of MAC addrs its operating
on behalf of this is easy.

Ian

> Simon
> 
> > -----Original Message-----
> > From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> > Sent: Thursday, July 05, 2007 3:46 PM
> > To: Graham, Simon
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> >
> > On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > > I accept that this is not very nice and we should probably have a
> > more
> > > organized/documented way to do this but I’m not sure what would
be
> > > appropriate; any suggestions?
> >
> > A usespace tool to send such an ARP would be a very useful thing to
> > have, if it doesn't already exist.
> >
> > The gratuitous arping in netfront is not very popular with the
> upstream
> > Linux folks so this functionality is not in the paravirt_ops patches
> > which are going upstream, so we'd need such a tool in the end
anyway.
> >
> > Ian.
> >
> >
> >
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/5/07, Graham, Simon <Simon.Graham@stratus.com>
wrote:
>
> I agree it''s not good. However, in the example I quoted with a
bond
> underneath the bridge, the netbridge code doesn''t even know there
are
> two paths or that it should send an update (in fact, there would be no
> update since the port config hasn''t changed as far as the bridge
code is
> concerned).
It''s not a problem only for Xen.  So, you may find a solution between
bond device driver and bridge code.

So, the bond device driver broadcast a netlink message and then the
usr mode helper tells bridge to broadcast the mac address in its
forwarding table. :-)

You may need help from linux net-devel.
>
> Simon
>

-- 
best regards,
hanzhu

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/5/07, Graham, Simon <Simon.Graham@stratus.com>
wrote:
> Whilst I might agree that from a purist point of view having netfront do
this is a violation of layering, it''s already accepted in some other
upstream drivers -- bonding for example; this virtual network interface driver
will send gratuitous ARPs for the IP address associated with the bond and for
any VLANs layered directly on top of the bond when failing over (it just wont
send ARPs for things above the bridge if it is plugged into a bridge).
>
I guess the user mode helper should be inside DOM0 unless you wanna
ask the user to intall some extra tools as vmware does, considering
about HVM domain.

The user mode helper in DOM0 should watch the connect status between
netback and netfront. If it becomes connected, then sends out an fake
ethernet frame, not an fake arp reply because you can''t know the IP
address.

Xentore can provide the mac address of netfront but netfront has to
register set_mac_address method to update it when the user try to
modify it.
> Simon
>
> > -----Original Message-----
> > From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> > Sent: Thursday, July 05, 2007 3:46 PM
> > To: Graham, Simon
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> >
> > On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > > I accept that this is not very nice and we should probably have a
> > more
> > > organized/documented way to do this but I''m not sure
what would be
> > > appropriate; any suggestions?
> >
> > A usespace tool to send such an ARP would be a very useful thing to
> > have, if it doesn''t already exist.
> >
> > The gratuitous arping in netfront is not very popular with the
upstream
> > Linux folks so this functionality is not in the paravirt_ops patches
> > which are going upstream, so we''d need such a tool in the end
anyway.
> >
> > Ian.
> >
> >
> >
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

-- 
best regards,
hanzhu

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/5/07, Ian Pratt <Ian.Pratt@cl.cam.ac.uk>
wrote:
> For bond case you described, you don''t need to send ARPs, you just
need to send any packet with the right src MAC to cause the switches to
reconfigure. Since the bond driver likely already maintains a list of MAC addrs
its operating on behalf of this is easy.
Ian, that''s right.

But you can''t depend on it just like you can''t depend on the
VM
continues to talk with the right mac address after the migration. :-)
>
> Ian
>
>
> > Simon
> >
> > > -----Original Message-----
> > > From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> > > Sent: Thursday, July 05, 2007 3:46 PM
> > > To: Graham, Simon
> > > Cc: xen-devel@lists.xensource.com
> > > Subject: Re: [Xen-devel] Stimulating domains to send gratuitous
ARPs
> > >
> > > On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > > > I accept that this is not very nice and we should probably
have a
> > > more
> > > > organized/documented way to do this but I''m not
sure what would be
> > > > appropriate; any suggestions?
> > >
> > > A usespace tool to send such an ARP would be a very useful thing
to
> > > have, if it doesn''t already exist.
> > >
> > > The gratuitous arping in netfront is not very popular with the
> > upstream
> > > Linux folks so this functionality is not in the paravirt_ops
patches
> > > which are going upstream, so we''d need such a tool in
the end anyway.
> > >
> > > Ian.
> > >
> > >
> > >
> > >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

-- 
best regards,
hanzhu

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Whilst I might agree that from a purist point of view having netfront
> do
> > this is a violation of layering, it's already accepted in some
other
> > upstream drivers
> 
> It's pretty clear it's not going to be accepted in the netfront
that
> goes into upstream Linux, at least not without sedating various LKML
> folk first :)
> 
Fine (well, not fine at all really, but I understand ;-).
> For bond case you described, you don't need to send ARPs, you just need
> to send any packet with the right src MAC to cause the switches to
> reconfigure. Since the bond driver likely already maintains a list of
> MAC addrs its operating on behalf of this is easy.
> 
Well, the bonding module itself doesn't track this at all actually; when a
failover is done, it sends gratuitous ARPs for the following:
1. The IP address associated with the bond itself
2. The IP addresses associated with an VLANs above the bond (suitably
encapsulated in the VLAN header).

In addition, I don’t think that 'any old frame' will work -- it needs to
be a link-level broadcast frame to ensure that all the switches update their
forwarding tables -- in the example I cited of bonding for availability,
it's entirely possible (in fact it's desirable) to plug the two NICs
that are bonded into different switches.

This is really exactly the same as the migration case (which also doesn't
need to be an ARP from the point of view of updating ARP caches, it just needs
to be a link level broadcast frame). So I think we both need a Dom0 based
mechanism!

Assuming the netfront approach is dead, we need to identify a link level
broadcast frame that can be safely sent to update the bridge forwarding tables
with no other side effects... perhaps a RARP rq since this is deprecated anyway
now, so it will probably get no answer and even if it does, the answer should be
ignored???

Simon

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Well, the bonding module itself doesn't track this at all actually;
when
> a failover is done, it sends gratuitous ARPs for the following:
> 1. The IP address associated with the bond itself
> 2. The IP addresses associated with an VLANs above the bond (suitably
> encapsulated in the VLAN header).
The Linux bond module isn't well suited to the L2 bridged case anyway. To
get the effect you want I think you'll have to add MAC addr tracking anyhow.
> In addition, I don’t think that 'any old frame' will work -- it
needs to
> be a link-level broadcast frame to ensure that all the switches update
> their forwarding tables 
Sure, it needs to be a link-layer broadcast, but it doesn't particularly
even need to be a valid IP/ARP packet. Broadcast arp reply works as well as
anything, and has the additional benefit of also helping folk who do happen to
be using a routed rather than L2 setup.

Best,
Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Pratt wrote:
>> Whilst I might agree that from a purist point of view having netfront
do
>> this is a violation of layering, it''s already accepted in some
other
>> upstream drivers 
>>     
>
> It''s pretty clear it''s not going to be accepted in the
netfront that goes into upstream Linux, at least not without sedating various
LKML folk first :)
>   
Stephen Hemminger posted a patch to add this capability generically so 
it wouldn''t be a netfront-specific piece of behaviour.  I have not 
submitted it for upstream yet, mainly because nobody has 
confirmed/tested that it really does what we need.  But it looks 
reasonable, and it should be accepted once we confirm its correct.

    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> Stephen Hemminger posted a patch to add this capability generically so
> it wouldn't be a netfront-specific piece of behaviour.  I have not
> submitted it for upstream yet, mainly because nobody has
> confirmed/tested that it really does what we need.  But it looks
> reasonable, and it should be accepted once we confirm its correct.
> 
-sigh- ... doesn't really help unfortunately for a couple of reasons:

0. It's in some as yet unreleased kernel version -- doesn't help with
backlevel DomUs
1. You still have to configure this in every DomU since the default is to not do
it.
2. There's no way to stimulate this from Dom0 as needed (e.g. at the end of
migration
   or when some other change occurs in Dom0).

Right now, I'm experimenting with a solution that sends an invalid broadcast
RARP frame... if this works for me, it'll also work for the migration/resume
case...

Simon

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel