thr3ads.net - Xen users - [Xen-users] Multiple IPs in a domU [Feb 2007]

If this information is useful, please help other people find it:
Share via:

Dan Parsons

2007-Feb-26 05:54 UTC

[Xen-users] Multiple IPs in a domU

I''ve been trying to make multiple public IP addresses work in a domU,  
and have been unable to do it. I''ve tried every method mentioned in  
the xenu-users archive as well as through a lot of stuff mentioned in  
google.

I''ve tried using eth0:1, etc, and also eth0, eth1 with multiple
vif''s
in the domU config file.

Basically, the IP assigned to eth0 always works right away, from the  
outside world, but the ip assigned to eth1 does not work until I  
force eth1 to make an outgoing connection. Then the IP works. So,  
it''s a MAC/ARP issue.

But why isn''t the MAC for eth1 being announced? I''ve tried
specifying
the IPs in the domU config various ways and that also didn''t help.

This is on Fedora Core 6, using Xen 3.0.3. domU is also FC6 using the  
same kernel. kernel is 2.6.19-1.2911.fc6xen. Below is a bunch more  
info, please tell me if I''ve forgotten to include anything.

domU config file:

name = "gadgeteer"
memory = "512"
disk = [ ''phy:vg0/VM_gadgeteer,xvda1,w'',''phy:vg0/ 
VM_gadgeteer_swap,xvdb1,w'' ]
vif = [ ''bridge=xenbr1, mac=00:16:3E:6A:24:16'',
''bridge=xenbr1,
mac=00:16:3E:19:F3:01'' ]

nographic=1
uuid = "8541a049-0168-2bae-9d8d-b47172f520ad"
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot   = ''restart''
on_crash    = ''restart''


ifconfig -a on dom0:



eth0      Link encap:Ethernet  HWaddr 00:30:48:32:53:44
           inet addr:*SENSORED*  Bcast:XXXXXXXXXX  Mask:255.255.255.192
           inet6 addr: fe80::230:48ff:fe32:5344/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:448 errors:0 dropped:0 overruns:0 frame:0
           TX packets:1738 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:100
           RX bytes:134781 (131.6 KiB)  TX bytes:125574 (122.6 KiB)
           Base address:0x2000 Memory:c8200000-c8220000

eth1      Link encap:Ethernet  HWaddr 00:30:48:32:53:45
           inet addr:*SENSORED*  Bcast:XXXXXXXXXXXX  Mask: 
255.255.255.248
           inet6 addr: fe80::230:48ff:fe32:5345/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:3741 errors:0 dropped:0 overruns:0 frame:0
           TX packets:2007 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:324481 (316.8 KiB)  TX bytes:387326 (378.2 KiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:8 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)

peth1     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:58430 errors:0 dropped:0 overruns:0 frame:0
           TX packets:42600 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:100
           RX bytes:7298901 (6.9 MiB)  TX bytes:6151916 (5.8 MiB)
           Base address:0x2020 Memory:c8220000-c8240000

sit0      Link encap:IPv6-in-IPv4
           NOARP  MTU:1480  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

veth0     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

veth2     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0

veth3     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif0.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:29682 errors:0 dropped:0 overruns:0 frame:0
           TX packets:47312 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:4244090 (4.0 MiB)  TX bytes:4865658 (4.6 MiB)

vif0.2    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif0.3    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif18.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:314 errors:0 dropped:0 overruns:0 frame:0
           TX packets:473 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:32407 (31.6 KiB)  TX bytes:36039 (35.1 KiB)

vif18.1   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:11 errors:0 dropped:0 overruns:0 frame:0
           TX packets:689 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:524 (524.0 b)  TX bytes:64868 (63.3 KiB)

vif19.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:73 errors:0 dropped:0 overruns:0 frame:0
           TX packets:97 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:5423 (5.2 KiB)  TX bytes:8461 (8.2 KiB)

vif19.1   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:11 errors:0 dropped:0 overruns:0 frame:0
           TX packets:75 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:524 (524.0 b)  TX bytes:6322 (6.1 KiB)

xenbr1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:1563 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:84662 (82.6 KiB)  TX bytes:0 (0.0 b)

brctl show output:

bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              vif19.1
                                                         vif19.0
                                                         vif18.1
                                                         vif18.0
                                                         peth1
                                                         vif0.1

ifconfig -a on domU:

eth0      Link encap:Ethernet  HWaddr 00:16:3E:5A:B3:AA
           inet addr:*SENSORED public IP1*  Bcast:XXXXXXXXXX  Mask: 
255.255.255.0
           inet6 addr: fe80::216:3eff:fe5a:b3aa/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:101 errors:0 dropped:0 overruns:0 frame:0
           TX packets:73 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:8761 (8.5 KiB)  TX bytes:6445 (6.2 KiB)

eth1      Link encap:Ethernet  HWaddr 00:16:3E:25:72:BA
           inet addr:*SENSORED public IP2*  Bcast:XXXXXXXXXXX  Mask: 
255.255.255.0
           inet6 addr: fe80::216:3eff:fe25:72ba/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:79 errors:0 dropped:0 overruns:0 frame:0
           TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:6622 (6.4 KiB)  TX bytes:678 (678.0 b)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:8 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)



Dan Parsons



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post

2007-Feb-26 10:05 UTC

head link

Re: [Xen-users] Multiple IPs in a domU

On Sun, 2007-02-25 at 21:54 -0800, Dan Parsons wrote:> I''ve been trying to make multiple public IP addresses work in a
domU,
> and have been unable to do it. I''ve tried every method mentioned
in
> the xenu-users archive as well as through a lot of stuff mentioned in  
> google.
>  Then the IP works. So,  
> it''s a MAC/ARP issue.
Most cisco routers take up to ~9 minutes to re-arp these changes. Static
MACS must be used. 

You could tri arping''ing the gateway just specifying the address and
interface (i.e. eth0:1) or pass a -U. This should trick the router into
doing it.

The best way, use static macs when bringing up the vifs within the
dom-u. Remember, you can have only 3 physical eth devices within a
guest, as far as I know this limitation has not yet increased. 

So, its much easier to bring them up as vifs from within the dom-u
itself, just specify macs for each one. 
> But why isn''t the MAC for eth1 being announced?
It is. apring is being called each time its brought to an up state to
make sure the IP does not already exist on the network. You either have
to wait 10 minutes, send traffic from it (i.e. ping -I eth0:1 -c1 -w5
4.2.2.2 > /dev/null 2>&1 in the postup init script. It really depends
on
the router.
> I''ve tried specifying  
> the IPs in the domU config various ways and that also didn''t help.
That''s not going to make much of a difference. I often just specify
bridge, vifname and mac and handle the rest inside of the guest.
> This is on Fedora Core 6, using Xen 3.0.3. domU is also FC6 using the  
> same kernel. kernel is 2.6.19-1.2911.fc6xen. Below is a bunch more  
> info, please tell me if I''ve forgotten to include anything.
Just flushing the router''s arp hehehe :)> 
> domU config file:
,,, looked just fine to me.> [ snip ]


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dan Parsons

2007-Feb-26 11:39 UTC

head link

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Thanks for the suggestions. Did something I say make you think I'm using a
HVM? I'm using a plain old paravirtualized VM here. Does this change any of
your suggestions?

My worry about the ping fix is that if there is no traffic on that IP for a long
time, the arp record created by the ping will die. And then the router issues
another arp who-has, and we have the same problem. Indeed, it looks as if Xen is
ignoring who-has requests for eth1's IP.... Also, how can it be upstream
when the eth0 ip works just fine? It's just eth1's ip that's bad.
What is xen doing to make eth0 work, that it's not doing for eth1? And why?

Thanks
Dan Parsons  

-----Original Message-----
From: Tim Post <tim.post@netkinetics.net>
Date: Mon, 26 Feb 2007 19:30:21 
To:dparsons@nyip.net
Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

On Mon, 2007-02-26 at 10:18 +0000, Dan Parsons wrote:> Thank you, but I actually am using statically defined  MACs in the domU
config..
> So I don't believe what you said applies. Please correct me if I'm
wrong.
Ah no, you're right I had a brainfart. Mac was staring right at me in
your config file. 
> Do you have further suggestions?
I haven't run into this, but only a few of my guests are HVM. You could
try a couple of things.

If you are using Xen's network-bridge script to bring up your bridges,
try letting init do it and use a network-dummy script in its place, play
with the bridge settings and see if it helps.

forward delay, helo, maxwait, should be set to 0. I am not entirely sure
that they are set to 0 by default by brctl.
 
Or a quick fix, call ping during the boot process on the guest slightly
modifying the init strings until you can rule out exactly what the issue
is.

I'm tending to lean in the direction of something upstream, though.

Best,
--Tim
> Dan Parsons 
> 
> -----Original Message-----
> From: Tim Post <tim.post@netkinetics.net>
> Date: Mon, 26 Feb 2007 18:05:53 
> To:Dan Parsons <dparsons@nyip.net>
> Cc:xen-users@lists.xensource.com
> Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU
> 
> On Sun, 2007-02-25 at 21:54 -0800, Dan Parsons wrote:
> > I've been trying to make multiple public IP addresses work in a
domU,
> > and have been unable to do it. I've tried every method mentioned
in
> > the xenu-users archive as well as through a lot of stuff mentioned in
> > google.
> 
> >  Then the IP works. So,  
> > it's a MAC/ARP issue.
> 
> Most cisco routers take up to ~9 minutes to re-arp these changes. Static
> MACS must be used. 
> 
> You could tri arping'ing the gateway just specifying the address and
> interface (i.e. eth0:1) or pass a -U. This should trick the router into
> doing it.
> 
> The best way, use static macs when bringing up the vifs within the
> dom-u. Remember, you can have only 3 physical eth devices within a
> guest, as far as I know this limitation has not yet increased. 
> 
> So, its much easier to bring them up as vifs from within the dom-u
> itself, just specify macs for each one. 
> 
> > But why isn't the MAC for eth1 being announced?
> 
> It is. apring is being called each time its brought to an up state to
> make sure the IP does not already exist on the network. You either have
> to wait 10 minutes, send traffic from it (i.e. ping -I eth0:1 -c1 -w5
> 4.2.2.2 > /dev/null 2>&1 in the postup init script. It really
depends on
> the router.
> 
> > I've tried specifying  
> > the IPs in the domU config various ways and that also didn't help.
> 
> That's not going to make much of a difference. I often just specify
> bridge, vifname and mac and handle the rest inside of the guest.
> 
> > This is on Fedora Core 6, using Xen 3.0.3. domU is also FC6 using the
> > same kernel. kernel is 2.6.19-1.2911.fc6xen. Below is a bunch more  
> > info, please tell me if I've forgotten to include anything.
> 
> Just flushing the router's arp hehehe :)
> > 
> > domU config file:
> ,,, looked just fine to me.
> > 
> [ snip ]
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post

2007-Feb-26 12:05 UTC

head link

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

On Mon, 2007-02-26 at 11:39 +0000, Dan Parsons wrote:> Thanks for the suggestions. 
> Did something I say make you think I''m using a HVM? I''m
using a plain old paravirtualized VM here.
> Does this change any of your suggestions?
No, not at all. I just suspected everything would be effected. There are
some known netdev quirks with hvm guests, was just pointing out to check
upstream before going after issues there too.
> My worry about the ping fix is that if there is no traffic on that IP for a
long time,
> the arp record created by the ping will die. 
> And then the router issues another arp who-has, and we have the same
problem.
> Indeed, it looks as if Xen is ignoring who-has requests for eth1''s
IP....
> Also, how can it be upstream when the eth0 ip works just fine?
It''s just eth1''s ip that''s bad.
THAT''s where I mis read you. I thought this was on a bridge porting
eth0.

> What is xen doing to make eth0 work, that it''s not doing for eth1?
And why?
The magic is happening in /etc/xen/scripts/network-bridge , which is run
to construct the bridges when xen starts and take them down / restore
things when xen exits.

Within xend-config, you can pass paramaters to it. I personally don''t
use it, I like to let my network init scripts handle the bridges because
I do more with bridging than Xen needs to do. 
> 
> Thanks
> Dan Parsons  
> 
Now it makes sense, and no , I don''t think its upstream anymore.
I''m
more dense than usual for some reason today.

Best,
--Tim
> -----Original Message-----
> From: Tim Post <tim.post@netkinetics.net>
> Date: Mon, 26 Feb 2007 19:30:21 
> To:dparsons@nyip.net
> Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU (I am using static
macs)
> 
> On Mon, 2007-02-26 at 10:18 +0000, Dan Parsons wrote:
> > Thank you, but I actually am using statically defined  MACs in the
domU config..
> > So I don''t believe what you said applies. Please correct me
if I''m wrong.
> 
> Ah no, you''re right I had a brainfart. Mac was staring right at me
in
> your config file. 
> 
> > Do you have further suggestions?
> 
> I haven''t run into this, but only a few of my guests are HVM. You
could
> try a couple of things.
> 
> If you are using Xen''s network-bridge script to bring up your
bridges,
> try letting init do it and use a network-dummy script in its place, play
> with the bridge settings and see if it helps.
> 
> forward delay, helo, maxwait, should be set to 0. I am not entirely sure
> that they are set to 0 by default by brctl.
>  
> Or a quick fix, call ping during the boot process on the guest slightly
> modifying the init strings until you can rule out exactly what the issue
> is.
> 
> I''m tending to lean in the direction of something upstream,
though.
> 
> Best,
> --Tim
> 
> > Dan Parsons 
> > 
> > -----Original Message-----
> > From: Tim Post <tim.post@netkinetics.net>
> > Date: Mon, 26 Feb 2007 18:05:53 
> > To:Dan Parsons <dparsons@nyip.net>
> > Cc:xen-users@lists.xensource.com
> > Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU
> > 
> > On Sun, 2007-02-25 at 21:54 -0800, Dan Parsons wrote:
> > > I''ve been trying to make multiple public IP addresses
work in a domU,
> > > and have been unable to do it. I''ve tried every method
mentioned in
> > > the xenu-users archive as well as through a lot of stuff
mentioned in
> > > google.
> > 
> > >  Then the IP works. So,  
> > > it''s a MAC/ARP issue.
> > 
> > Most cisco routers take up to ~9 minutes to re-arp these changes.
Static
> > MACS must be used. 
> > 
> > You could tri arping''ing the gateway just specifying the
address and
> > interface (i.e. eth0:1) or pass a -U. This should trick the router
into
> > doing it.
> > 
> > The best way, use static macs when bringing up the vifs within the
> > dom-u. Remember, you can have only 3 physical eth devices within a
> > guest, as far as I know this limitation has not yet increased. 
> > 
> > So, its much easier to bring them up as vifs from within the dom-u
> > itself, just specify macs for each one. 
> > 
> > > But why isn''t the MAC for eth1 being announced?
> > 
> > It is. apring is being called each time its brought to an up state to
> > make sure the IP does not already exist on the network. You either
have
> > to wait 10 minutes, send traffic from it (i.e. ping -I eth0:1 -c1 -w5
> > 4.2.2.2 > /dev/null 2>&1 in the postup init script. It
really depends on
> > the router.
> > 
> > > I''ve tried specifying  
> > > the IPs in the domU config various ways and that also
didn''t help.
> > 
> > That''s not going to make much of a difference. I often just
specify
> > bridge, vifname and mac and handle the rest inside of the guest.
> > 
> > > This is on Fedora Core 6, using Xen 3.0.3. domU is also FC6 using
the
> > > same kernel. kernel is 2.6.19-1.2911.fc6xen. Below is a bunch
more
> > > info, please tell me if I''ve forgotten to include
anything.
> > 
> > Just flushing the router''s arp hehehe :)
> > > 
> > > domU config file:
> > ,,, looked just fine to me.
> > > 
> > [ snip ]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dan Parsons

2007-Feb-26 19:06 UTC

head link

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Thanks very much for all the info Tim, but I''m still stumped. I
don''t
know what to do next.

To clarify:

dom0 has an eth0 and an eth1, but I don''t care about my domUs having  
access to dom0''s eth0 - just eth1. And this is how it''s
working
currently, in my xend-config.sxp file I have:

(network-script network-bridge netdev=eth1)

The only thing that *isn''t* working is having more than one IP inside  
a domU. I''ve tried it using eth0:1 and eth1, inside the domU. (note  
that this mention of eth1 is separate from the dom0 eth1).

Does you or anyone else have any more specific suggestions on how to  
make multiple IPs workin in a domU? How can I get xen to just do for  
domU eth1 what it does for domU eth0?

For those new to thread, xen isn''t arp announcing domU eth1''s
IP, but
it is doing that for domU''s eth0.


Dan Parsons


On Feb 26, 2007, at 4:05 AM, Tim Post wrote:
> On Mon, 2007-02-26 at 11:39 +0000, Dan Parsons wrote:
>> Thanks for the suggestions.
>> Did something I say make you think I''m using a HVM?
I''m using a
>> plain old paravirtualized VM here.
>> Does this change any of your suggestions?
>
> No, not at all. I just suspected everything would be effected.  
> There are
> some known netdev quirks with hvm guests, was just pointing out to  
> check
> upstream before going after issues there too.
>
>> My worry about the ping fix is that if there is no traffic on that  
>> IP for a long time,
>> the arp record created by the ping will die.
>> And then the router issues another arp who-has, and we have the  
>> same problem.
>> Indeed, it looks as if Xen is ignoring who-has requests for
eth1''s
>> IP....
>
>
>> Also, how can it be upstream when the eth0 ip works just fine?  
>> It''s just eth1''s ip that''s bad.
>
> THAT''s where I mis read you. I thought this was on a bridge
porting
> eth0.
>
>
>> What is xen doing to make eth0 work, that it''s not doing for
eth1?
>> And why?
>
> The magic is happening in /etc/xen/scripts/network-bridge , which  
> is run
> to construct the bridges when xen starts and take them down / restore
> things when xen exits.
>
> Within xend-config, you can pass paramaters to it. I personally
don''t
> use it, I like to let my network init scripts handle the bridges  
> because
> I do more with bridging than Xen needs to do.
>
>>
>> Thanks
>> Dan Parsons
>>
>
> Now it makes sense, and no , I don''t think its upstream anymore.
I''m
> more dense than usual for some reason today.
>
> Best,
> --Tim
>
>> -----Original Message-----
>> From: Tim Post <tim.post@netkinetics.net>
>> Date: Mon, 26 Feb 2007 19:30:21
>> To:dparsons@nyip.net
>> Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU (I am using  
>> static macs)
>>
>> On Mon, 2007-02-26 at 10:18 +0000, Dan Parsons wrote:
>>> Thank you, but I actually am using statically defined  MACs in  
>>> the domU config..
>>> So I don''t believe what you said applies. Please correct
me if
>>> I''m wrong.
>>
>> Ah no, you''re right I had a brainfart. Mac was staring right
at me in
>> your config file.
>>
>>> Do you have further suggestions?
>>
>> I haven''t run into this, but only a few of my guests are HVM.
You
>> could
>> try a couple of things.
>>
>> If you are using Xen''s network-bridge script to bring up your
>> bridges,
>> try letting init do it and use a network-dummy script in its  
>> place, play
>> with the bridge settings and see if it helps.
>>
>> forward delay, helo, maxwait, should be set to 0. I am not  
>> entirely sure
>> that they are set to 0 by default by brctl.
>>
>> Or a quick fix, call ping during the boot process on the guest  
>> slightly
>> modifying the init strings until you can rule out exactly what the  
>> issue
>> is.
>>
>> I''m tending to lean in the direction of something upstream,
though.
>>
>> Best,
>> --Tim
>>
>>> Dan Parsons
>>>
>>> -----Original Message-----
>>> From: Tim Post <tim.post@netkinetics.net>
>>> Date: Mon, 26 Feb 2007 18:05:53
>>> To:Dan Parsons <dparsons@nyip.net>
>>> Cc:xen-users@lists.xensource.com
>>> Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU
>>>
>>> On Sun, 2007-02-25 at 21:54 -0800, Dan Parsons wrote:
>>>> I''ve been trying to make multiple public IP addresses
work in a
>>>> domU,
>>>> and have been unable to do it. I''ve tried every method
mentioned in
>>>> the xenu-users archive as well as through a lot of stuff  
>>>> mentioned in
>>>> google.
>>>
>>>>  Then the IP works. So,
>>>> it''s a MAC/ARP issue.
>>>
>>> Most cisco routers take up to ~9 minutes to re-arp these changes.  
>>> Static
>>> MACS must be used.
>>>
>>> You could tri arping''ing the gateway just specifying the
address and
>>> interface (i.e. eth0:1) or pass a -U. This should trick the  
>>> router into
>>> doing it.
>>>
>>> The best way, use static macs when bringing up the vifs within the
>>> dom-u. Remember, you can have only 3 physical eth devices within a
>>> guest, as far as I know this limitation has not yet increased.
>>>
>>> So, its much easier to bring them up as vifs from within the dom-u
>>> itself, just specify macs for each one.
>>>
>>>> But why isn''t the MAC for eth1 being announced?
>>>
>>> It is. apring is being called each time its brought to an up  
>>> state to
>>> make sure the IP does not already exist on the network. You  
>>> either have
>>> to wait 10 minutes, send traffic from it (i.e. ping -I eth0:1 -c1  
>>> -w5
>>> 4.2.2.2 > /dev/null 2>&1 in the postup init script. It
really
>>> depends on
>>> the router.
>>>
>>>> I''ve tried specifying
>>>> the IPs in the domU config various ways and that also
didn''t help.
>>>
>>> That''s not going to make much of a difference. I often
just specify
>>> bridge, vifname and mac and handle the rest inside of the guest.
>>>
>>>> This is on Fedora Core 6, using Xen 3.0.3. domU is also FC6  
>>>> using the
>>>> same kernel. kernel is 2.6.19-1.2911.fc6xen. Below is a bunch
more
>>>> info, please tell me if I''ve forgotten to include
anything.
>>>
>>> Just flushing the router''s arp hehehe :)
>>>>
>>>> domU config file:
>>> ,,, looked just fine to me.
>>>>
>>> [ snip ]
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dan Parsons

2007-Feb-26 22:45 UTC

head link

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Tom, thanks for replying. I started off doing the simple eth0:1  
eth0:2 method. It didn''t work - no traffic to the aliased IPs came  
in. eth0 would work but not eth0:1 or eth0:2. The outside world  
couldn''t ping them, though the outside world could ping eth0. I would  
very much prefer to use aliases and not separate interfaces.

I haven''t done any big config changes, nothing to xend conf or bridge  
scripts or anything like that. domU eth0 works fine from outside  
world, domU eth0:1 does not.

I really appreciate the help, I''m under the wire here and have to get  
this going :(

Xen 3.0.3 on fc6; here''s my domU conf:


# Automatically generated xen config file
name = "ns"
memory = "128"
disk = [ ''phy:vg0/VM_ns,xvda1,w'' ]
vif = [ ''bridge=xenbr1, mac=00:16:3E:6A:24:16'' ]

nographic=1
uuid = "8141e049-0168-2bae-9d8d-b471220520ad"
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot   = ''restart''
on_crash    = ''restart''



Dan Parsons


On Feb 26, 2007, at 2:34 PM, Tom Brown wrote:
> On Mon, 26 Feb 2007, Dan Parsons wrote:
>
>> Thanks for the suggestions. Did something I say make you think
I''m
>> using a HVM? I''m using a plain old paravirtualized VM here.
Does
>> this change any of your suggestions?
>>
>> My worry about the ping fix is that if there is no traffic on that  
>> IP for a long time, the arp record created by the ping will die.  
>> And then the router issues another arp who-has, and we have the  
>> same problem. Indeed,
>
> what problem?
>
> I use IP aliases in domUs all the time and have no trouble.
>
>> it looks as if Xen is ignoring who-has requests for eth1''s
IP....
>> Also, how can it be upstream when the eth0 ip works just fine?  
>> It''s just eth1''s ip that''s bad. What is xen
doing to make eth0
>> work, that it''s not doing for eth1? And why?
>
> why are you creating multiple interfaces if you don''t want them on
> separate networks/bridges? Just use aliases of eth0
> e.g.
>
> ifconfig eth0:1 192.168.1.1
> ifconfig eth0:2 192.168.1.2
> ...
>
> obviously I''m joining the thread late, so my apologies if there
> is a clear issue I''ve missed.
>
> -Tom
>
> ----------------------------------------------------------------------
> tbrown@BareMetal.com   | Courage is doing what you''re afraid to
do.
> http://BareMetal.com/  | There can be no courage unless you''re
scared.
>                        | - Eddie Rickenbacker
>
>>
>> Thanks
>> Dan Parsons
>>
>> -----Original Message-----
>> From: Tim Post <tim.post@netkinetics.net>
>> Date: Mon, 26 Feb 2007 19:30:21
>> To:dparsons@nyip.net
>> Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU (I am using  
>> static macs)
>>
>> On Mon, 2007-02-26 at 10:18 +0000, Dan Parsons wrote:
>>> Thank you, but I actually am using statically defined  MACs in  
>>> the domU config..
>>> So I don''t believe what you said applies. Please correct
me if
>>> I''m wrong.
>>
>> Ah no, you''re right I had a brainfart. Mac was staring right
at me in
>> your config file.
>>
>>> Do you have further suggestions?
>>
>> I haven''t run into this, but only a few of my guests are HVM.
You
>> could
>> try a couple of things.
>>
>> If you are using Xen''s network-bridge script to bring up your
>> bridges,
>> try letting init do it and use a network-dummy script in its  
>> place, play
>> with the bridge settings and see if it helps.
>>
>> forward delay, helo, maxwait, should be set to 0. I am not  
>> entirely sure
>> that they are set to 0 by default by brctl.
>>
>> Or a quick fix, call ping during the boot process on the guest  
>> slightly
>> modifying the init strings until you can rule out exactly what the  
>> issue
>> is.
>>
>> I''m tending to lean in the direction of something upstream,
though.
>>
>> Best,
>> --Tim
>>
>>> Dan Parsons
>>>
>>> -----Original Message-----
>>> From: Tim Post <tim.post@netkinetics.net>
>>> Date: Mon, 26 Feb 2007 18:05:53
>>> To:Dan Parsons <dparsons@nyip.net>
>>> Cc:xen-users@lists.xensource.com
>>> Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU
>>>
>>> On Sun, 2007-02-25 at 21:54 -0800, Dan Parsons wrote:
>>>> I''ve been trying to make multiple public IP addresses
work in a
>>>> domU,
>>>> and have been unable to do it. I''ve tried every method
mentioned in
>>>> the xenu-users archive as well as through a lot of stuff  
>>>> mentioned in
>>>> google.
>>>
>>>>  Then the IP works. So,
>>>> it''s a MAC/ARP issue.
>>>
>>> Most cisco routers take up to ~9 minutes to re-arp these changes.  
>>> Static
>>> MACS must be used.
>>>
>>> You could tri arping''ing the gateway just specifying the
address and
>>> interface (i.e. eth0:1) or pass a -U. This should trick the  
>>> router into
>>> doing it.
>>>
>>> The best way, use static macs when bringing up the vifs within the
>>> dom-u. Remember, you can have only 3 physical eth devices within a
>>> guest, as far as I know this limitation has not yet increased.
>>>
>>> So, its much easier to bring them up as vifs from within the dom-u
>>> itself, just specify macs for each one.
>>>
>>>> But why isn''t the MAC for eth1 being announced?
>>>
>>> It is. apring is being called each time its brought to an up  
>>> state to
>>> make sure the IP does not already exist on the network. You  
>>> either have
>>> to wait 10 minutes, send traffic from it (i.e. ping -I eth0:1 -c1  
>>> -w5
>>> 4.2.2.2 > /dev/null 2>&1 in the postup init script. It
really
>>> depends on
>>> the router.
>>>
>>>> I''ve tried specifying
>>>> the IPs in the domU config various ways and that also
didn''t help.
>>>
>>> That''s not going to make much of a difference. I often
just specify
>>> bridge, vifname and mac and handle the rest inside of the guest.
>>>
>>>> This is on Fedora Core 6, using Xen 3.0.3. domU is also FC6  
>>>> using the
>>>> same kernel. kernel is 2.6.19-1.2911.fc6xen. Below is a bunch
more
>>>> info, please tell me if I''ve forgotten to include
anything.
>>>
>>> Just flushing the router''s arp hehehe :)
>>>>
>>>> domU config file:
>>> ,,, looked just fine to me.
>>>>
>>> [ snip ]
>>
>
> ----------------------------------------------------------------------
> tbrown@BareMetal.com   | Courage is doing what you''re afraid to
do.
> http://BareMetal.com/  | There can be no courage unless you''re
scared.
>                        | - Eddie Rickenbacker

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dan Parsons

2007-Feb-27 19:45 UTC

head link

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

I finally have the problem fixed. It turns out the issue was caused  
by my colo giving me the wrong gateway IP address. Which is  
interesting - because it let the VMs connect to the outside world,  
but coming back in, the gateway didn''t like it.

Obviously not a Xen problem. Sorry guys!


Dan Parsons


On Feb 26, 2007, at 2:59 PM, Tom Brown wrote:
> On Mon, 26 Feb 2007, Dan Parsons wrote:
>
>> Tom, thanks for replying. I started off doing the simple eth0:1  
>> eth0:2 method. It didn''t work - no traffic to the aliased IPs
came
>> in. eth0 would work but not eth0:1 or eth0:2. The outside world  
>> couldn''t ping them, though the outside world could ping eth0.
I
>> would very much prefer to use aliases and not separate interfaces.
>>
>> I haven''t done any big config changes, nothing to xend conf or
>> bridge scripts or anything like that. domU eth0 works fine from  
>> outside world, domU eth0:1 does not.
>>
>> I really appreciate the help, I''m under the wire here and have
to
>> get this going :(
>
> You''re going to have to sniff this one through... it''s
not
> something I can diagnose from a few links of a config file. You''ve
> got something "in the way", and you''re going to need to
find it.
> The obvious first question is can you ping the alias from dom0 ?  
> From another domU? From other physical machines on your network.  
> From outside your network (you''ve said no to this).
>
> the best test platform would be another physical linux machine on  
> the same network. What does arp -a say? Does it show incomplete?
>
> ? (192.168.99.21) at <incomplete> on eth0
>
> at least that tells you that routing is ok and your box is trying  
> to arp the requested IP address. If not, then routing is broken and  
> xen isn''t relevent (and it probably isn''t).
>
> Leaving tcpdump running in the target domU looking for arp packets  
> may help as well... you can also look in dom0 at both the virtual  
> eth0 interface and the bridge device itself which corresponds to  
> the physical hardware... I believe the names have changed between  
> xen versions.
>
> That should give you something to chew on. By the time ARP returns  
> a real mac address you should be damn close. If arp works but ping  
> doesn''t, then routing is broken somewhere (probably the return  
> direction).
>
> -Tom
>
>>
>> Xen 3.0.3 on fc6; here''s my domU conf:
>>
>>
>> # Automatically generated xen config file
>> name = "ns"
>> memory = "128"
>> disk = [ ''phy:vg0/VM_ns,xvda1,w'' ]
>> vif = [ ''bridge=xenbr1, mac=00:16:3E:6A:24:16'' ]
>>
>> nographic=1
>> uuid = "8141e049-0168-2bae-9d8d-b471220520ad"
>> bootloader="/usr/bin/pygrub"
>> vcpus=1
>> on_reboot   = ''restart''
>> on_crash    = ''restart''
>>
>>
>>
>> Dan Parsons
>>
>>
>> On Feb 26, 2007, at 2:34 PM, Tom Brown wrote:
>>
>>> On Mon, 26 Feb 2007, Dan Parsons wrote:
>>> > Thanks for the suggestions. Did something I say make you think
>>> I''m using > a HVM? I''m using a plain old
paravirtualized VM here.
>>> Does this change > any of your suggestions?
>>> > > My worry about the ping fix is that if there is no
traffic on
>>> that IP for > a long time, the arp record created by the ping  
>>> will die. And then the > router issues another arp who-has, and
>>> we have the same problem. Indeed,
>>> what problem?
>>> I use IP aliases in domUs all the time and have no trouble.
>>> > it looks as if Xen is ignoring who-has requests for
eth1''s
>>> IP.... Also, > how can it be upstream when the eth0 ip works
just
>>> fine? It''s just eth1''s > ip that''s
bad. What is xen doing to make
>>> eth0 work, that it''s not doing > for eth1? And why?
>>> why are you creating multiple interfaces if you don''t want
them
>>> on separate networks/bridges? Just use aliases of eth0
>>> e.g.
>>> ifconfig eth0:1 192.168.1.1
>>> ifconfig eth0:2 192.168.1.2
>>> ...
>>> obviously I''m joining the thread late, so my apologies if
there
>>> is a clear issue I''ve missed.
>>> -Tom
>>>
--------------------------------------------------------------------
>>> --
>>> tbrown@BareMetal.com   | Courage is doing what you''re
afraid to do.
>>> http://BareMetal.com/  | There can be no courage unless
you''re
>>> scared.
>>> |  - Eddie Rickenbacker
>>> > > Thanks
>>> > Dan Parsons
>>> > > -----Original Message-----
>>> > From: Tim Post <tim.post@netkinetics.net>
>>> > Date: Mon, 26 Feb 2007 19:30:21
>>> > To:dparsons@nyip.net
>>> > Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU (I am  
>>> using static > macs)
>>> > > On Mon, 2007-02-26 at 10:18 +0000, Dan Parsons wrote:
>>> > > Thank you, but I actually am using statically defined 
MACs
>>> in the domU > > config..
>>> > > So I don''t believe what you said applies. Please
correct me
>>> if I''m > > wrong.
>>> > > Ah no, you''re right I had a brainfart. Mac was
staring right
>>> at me in
>>> > your config file.
>>> > > > Do you have further suggestions?
>>> > > I haven''t run into this, but only a few of my
guests are HVM.
>>> You could
>>> > try a couple of things.
>>> > > If you are using Xen''s network-bridge script to
bring up your
>>> bridges,
>>> > try letting init do it and use a network-dummy script in its  
>>> place, play
>>> > with the bridge settings and see if it helps.
>>> > > forward delay, helo, maxwait, should be set to 0. I am
not
>>> entirely sure
>>> > that they are set to 0 by default by brctl.
>>> > > Or a quick fix, call ping during the boot process on the
>>> guest slightly
>>> > modifying the init strings until you can rule out exactly what
>>> the issue
>>> > is.
>>> > > I''m tending to lean in the direction of
something upstream,
>>> though.
>>> > > Best,
>>> > --Tim
>>> > > > Dan Parsons
>>> > > > > -----Original Message-----
>>> > > From: Tim Post <tim.post@netkinetics.net>
>>> > > Date: Mon, 26 Feb 2007 18:05:53
>>> > > To:Dan Parsons <dparsons@nyip.net>
>>> > > Cc:xen-users@lists.xensource.com
>>> > > Subject: [QUAR] Re: [Xen-users] Multiple IPs in a domU
>>> > > > > On Sun, 2007-02-25 at 21:54 -0800, Dan Parsons
wrote:
>>> > > > I''ve been trying to make multiple public IP
addresses work
>>> in a domU,
>>> > > > and have been unable to do it. I''ve tried
every method
>>> mentioned in
>>> > > > the xenu-users archive as well as through a lot of
stuff
>>> mentioned in
>>> > > > google.
>>> > > > > > Then the IP works. So,
>>> > > > it''s a MAC/ARP issue.
>>> > > > > Most cisco routers take up to ~9 minutes to
re-arp these
>>> changes. > > Static
>>> > > MACS must be used.
>>> > > > > You could tri arping''ing the gateway
just specifying the
>>> address and
>>> > > interface (i.e. eth0:1) or pass a -U. This should trick
the
>>> router into
>>> > > doing it.
>>> > > > > The best way, use static macs when bringing up
the vifs
>>> within the
>>> > > dom-u. Remember, you can have only 3 physical eth devices
>>> within a
>>> > > guest, as far as I know this limitation has not yet
increased.
>>> > > > > So, its much easier to bring them up as vifs
from within
>>> the dom-u
>>> > > itself, just specify macs for each one.
>>> > > > > > But why isn''t the MAC for eth1
being announced?
>>> > > > > It is. apring is being called each time its
brought to an
>>> up state to
>>> > > make sure the IP does not already exist on the network.
You
>>> either have
>>> > > to wait 10 minutes, send traffic from it (i.e. ping -I
eth0:1
>>> -c1 -w5
>>> > > 4.2.2.2 > /dev/null 2>&1 in the postup init
script. It really
>>> depends > > on
>>> > > the router.
>>> > > > > > I''ve tried specifying
>>> > > > the IPs in the domU config various ways and that
also
>>> didn''t help.
>>> > > > > That''s not going to make much of a
difference. I often
>>> just specify
>>> > > bridge, vifname and mac and handle the rest inside of the
guest.
>>> > > > > > This is on Fedora Core 6, using Xen 3.0.3.
domU is also
>>> FC6 using the
>>> > > > same kernel. kernel is 2.6.19-1.2911.fc6xen. Below
is a
>>> bunch more
>>> > > > info, please tell me if I''ve forgotten to
include anything.
>>> > > > > Just flushing the router''s arp hehehe
:)
>>> > > > > > > domU config file:
>>> > > ,,, looked just fine to me.
>>> > > > > > [ snip ]
>>> >  
>>>
--------------------------------------------------------------------
>>> --
>>> tbrown@BareMetal.com   | Courage is doing what you''re
afraid to do.
>>> http://BareMetal.com/  | There can be no courage unless
you''re
>>> scared.
>>> |  - Eddie Rickenbacker
>>
>
> ----------------------------------------------------------------------
> tbrown@BareMetal.com   | Courage is doing what you''re afraid to
do.
> http://BareMetal.com/  | There can be no courage unless you''re
scared.
>                        | - Eddie Rickenbacker

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Feb 2007 - Multiple IPs in a domU

[Xen-users] Multiple IPs in a domU

Re: [Xen-users] Multiple IPs in a domU

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)

Re: [Xen-users] Multiple IPs in a domU (I am using static macs)