thr3ads.net - Xen users - [Xen-users] Xen 3.0, setting up a virtual network with NAT [Feb 2006]

If this information is useful, please help other people find it:
Share via:

CJ van den Berg

2006-Feb-03 16:15 UTC

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Fri, Feb 03, 2006 at 04:31:47PM +0000, Richard Jones
wrote:> I''ve got a network set up as in the diagram below:
> 
>    domU               domU
>    fake eth0          fake eth0
>    192.168.99.2       192.168.99.3
>        |                   |
>        +-----------+-------+
>                    |
>                192.168.99.1
>                dummy0
>                 * dom0 *
>                real eth0
>                public IP address
> 
> In /etc/xen/xend-config.sxp I''ve got:
>  (network-script ''network-bridge netdev=dummy0'')
What you want is:
(network-script network-nat)
> This all works fine in so far as domU can ping dom0 and dom0 can ping
> domU (ie. ping 192.168.99.2 -> 192.168.99.1 and
> ping 192.168.99.1 -> 192.168.99.2 is all OK).
This will work with the network-nat script too.
> If I enable routing in the kernel on dom0, then domU can try to ping
> the outside world, but of course the ping packets don''t return
because
> their source address is wrong (192.168.99.2, not public IP addr).
> 
> But if I add the following NAT rule:
>  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
> it makes no difference!  Somehow these packets are bypassing normal
> routing and so not being masqueraded, and so their source address is
> not rewritten.
> 
> Has anyone got this working?
Yes, I use this all the time. Use the network-nat script and vif-nat. It
should do exactly what you want.
> Note that I want to have a true virtual network here (not just domU
> NAT) because I want the domU''s to be able to talk to each other.
The NAT scripts configure routing in dom0 and then they setup routes to
each of the domUs. So everyone can ping everyone else internally and
everyone can also go through NAT to the outside world.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Richard Jones

2006-Feb-03 16:31 UTC

head link

[Xen-users] Xen 3.0, setting up a virtual network with NAT

I''ve got a network set up as in the diagram below:

   domU               domU
   fake eth0          fake eth0
   192.168.99.2       192.168.99.3
       |                   |
       +-----------+-------+
                   |
               192.168.99.1
               dummy0
                * dom0 *
               real eth0
               public IP address

In /etc/xen/xend-config.sxp I''ve got:
 (network-script ''network-bridge netdev=dummy0'')

This all works fine in so far as domU can ping dom0 and dom0 can ping
domU (ie. ping 192.168.99.2 -> 192.168.99.1 and
ping 192.168.99.1 -> 192.168.99.2 is all OK).

If I enable routing in the kernel on dom0, then domU can try to ping
the outside world, but of course the ping packets don''t return because
their source address is wrong (192.168.99.2, not public IP addr).

But if I add the following NAT rule:
 iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
it makes no difference!  Somehow these packets are bypassing normal
routing and so not being masqueraded, and so their source address is
not rewritten.

Has anyone got this working?

Note that I want to have a true virtual network here (not just domU
NAT) because I want the domU''s to be able to talk to each other.

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Richard Jones

2006-Feb-03 17:25 UTC

head link

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Fri, Feb 03, 2006 at 05:15:52PM +0100, CJ van den Berg
wrote:> On Fri, Feb 03, 2006 at 04:31:47PM +0000, Richard Jones wrote:
> > I''ve got a network set up as in the diagram below:
> > 
> >    domU               domU
> >    fake eth0          fake eth0
> >    192.168.99.2       192.168.99.3
> >        |                   |
> >        +-----------+-------+
> >                    |
> >                192.168.99.1
> >                dummy0
> >                 * dom0 *
> >                real eth0
> >                public IP address
> > 
> > In /etc/xen/xend-config.sxp I''ve got:
> >  (network-script ''network-bridge netdev=dummy0'')
> 
> What you want is:
> (network-script network-nat)
I tried the above, and also:
  (network-script ''network-nat netdev=dummy0'')
and neither works.  With these I can no longer ping my domU''s even.

Can you tell us about your configuration?  It''s really unclear what IP
addresses I should give the domU''s, whether I need dummy0 at all, and
what IP addresses should go into the domU configuration files.

It''d be great if you could share the relevant lines from your
xend-config.sxp and from your domU *-config.sxp files.

Rich.

PS. I''m using the latest Xen 3.0, downloaded today.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Patrick Wolfe

2006-Feb-03 17:58 UTC

head link

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Fri, 2006-02-03 at 16:31 +0000, Richard Jones wrote:> I''ve got a network set up as in the diagram below:
> 
>    domU               domU
>    fake eth0          fake eth0
>    192.168.99.2       192.168.99.3
>        |                   |
>        +-----------+-------+
>                    |
>                192.168.99.1
>                dummy0
>                 * dom0 *
>                real eth0
>                public IP address
> 
> In /etc/xen/xend-config.sxp I''ve got:
>  (network-script ''network-bridge netdev=dummy0'')
Instead of using dummy0, why not try using veth1 and vif0.1?

in /etc/xen/xend-config.sxp, use:

	(network-script ''network-route'')
	(vif-script ''vif-bridge'')

The ''network-route'' script will leave eth0 alone, and turn on
IP
forwarding.  Because you''ll be using bridging to connect to the
domUs, we need to specify ''vif-bridge'' here.

Assuming you''re using Debian Linux in your dom0, edit the
file /etc/network/interfaces to add:

	# dom0''s interface to the xen dmz bridge
	auto veth1
	iface veth1 inet static
		address 192.168.99.1
		netmask 255.255.255.0
		broadcast 192.168.99.255
		# optionally define MAC address for dom0
		hwaddress ether 00:16:3E:B0:01:01

	# the xen dmz bridge
	auto br1
	iface br1 inet manual
		# the following attaches veth1 to this bridge
		bridge_ports vif0.1
		bridge_stp off
		bridge_fd 0

This will define veth1 (instead of dummy0), create
the xenbr0 bridge, and connect vif0.1 (the other end of the virtual
ethernet cable which is connected to veth1) to br1 bridge.

Don''t forget to turn on IP forwarding by editing /etc/network/options:

	ip_forward=yes

If your dom0 isn''t running Debian, you''ll have to figure out
how to do
all that at boot time yourself.

Then, in your domU''s config files, use something like:

	vif=[ ''bridge=br1,mac=00:16:3E:70:01:02'' ]

and when your domU''s come up, vif-bridge will attach them to your
bridge.  Note:  configuring the mac addresses is optional.

-- 

Patrick Wolfe

email:   pwolfe@employease.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

CJ van den Berg

2006-Feb-03 21:33 UTC

head link

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Fri, Feb 03, 2006 at 05:25:13PM +0000, Richard Jones
wrote:> On Fri, Feb 03, 2006 at 05:15:52PM +0100, CJ van den Berg wrote:
> > What you want is:
> > (network-script network-nat)
> 
> I tried the above, and also:
>   (network-script ''network-nat netdev=dummy0'')
> and neither works.  With these I can no longer ping my domU''s
even.
You don''t need the dummy0 interface at all. Just use
(network-script network-nat)
> Can you tell us about your configuration?  It''s really unclear
what IP
> addresses I should give the domU''s, whether I need dummy0 at all,
and
> what IP addresses should go into the domU configuration files.
With the NAT scripts Xen assigns the IP addresses. You just need to add
dhcp=1 to the domU config file, which will add IP address configuration
parameters for the Xen assigned IP address to the kernel command line, or
use a dhcp server (which I prefer and have described below).
> It''d be great if you could share the relevant lines from your
> xend-config.sxp and from your domU *-config.sxp files.
Sure. In my xend-config.sxp I have:

(network-script network-nat)
(vif-script ''vif-nat dhcp=yes'')

and in my domU configs (I''m not using .sxp files) I just have:

nics=1 (for Xen 3.0.0)

or

vif = [''''] (for Xen 3.0.1)

That''s it. Then of course you need a simple dhcp server config. For
Debian:

aptitude install dhcp3-server

and then edit /etc/dhcp3/dhcpd.conf and add an empty section for the Xen
allocated addresses like this: (just stick it at the end of the file)

section 10.0.0.0 netmask 255.255.0.0 {
}

Then let Xen do the rest. Don''t worry if your dhcp server refuses to
start.
That''s because it doesn''t have any addresses to serve. Xen
will add a
static IP assignment to the dhcpd.conf file for each domU that is created
and restart the dhcp server.

I have also made several enhancements to my local vif-nat so that a dns
server is also updated dynamically for each created domU, which allows me to
use hostnames when communicating from one domU to another. So I never have
to figure out what IP address a domU has.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Richard Jones

2006-Feb-04 14:08 UTC

head link

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Fri, Feb 03, 2006 at 12:58:12PM -0500, Patrick Wolfe
wrote:> On Fri, 2006-02-03 at 16:31 +0000, Richard Jones wrote:
> > I''ve got a network set up as in the diagram below:
> > 
> >    domU               domU
> >    fake eth0          fake eth0
> >    192.168.99.2       192.168.99.3
> >        |                   |
> >        +-----------+-------+
> >                    |
> >                192.168.99.1
> >                dummy0
> >                 * dom0 *
> >                real eth0
> >                public IP address
> 
> Instead of using dummy0, why not try using veth1 and vif0.1?[...]

I followed your instructions, and I''m still at the point where I
can''t
get NAT working.  (BTW, hwaddr is absolutely essential - the bridge
doesn''t work otherwise).

I can ping 192.168.99.2 -> 192.168.99.1 and 192.168.99.1 -> 192.168.99.2
(ie. dom0 <-> domU).

I can ping domU <-> domU.

I''ve added the NAT rule on dom0:

  iptables --table nat --append POSTROUTING -o eth0 -j MASQUERADE

However when I try to connect out of the virtual network, NAT still
isn''t working.  In the example below, I''m trying to telnet out
to port
80 on a public address from one of the domUs.

  dom0# tcpdump -i eth0 tcp port 80
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
  13:36:31.805346 IP 192.168.99.3.2093 > 80.68.91.176.www: S
511867828:511867828(0) win 5840 <mss 1460,sackOK,timestamp 4294963735
0,nop,wscale 2>

Note that the source address is wrong (192.168.99.3 - it should have
been rewritten by NAT).

So NAT is still somehow being avoided ... Help!

Rich.

These are the interfaces on dom0:

# /sbin/ifconfig
br1       Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:496 (496.0 b)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:30:48:56:62:72
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe56:6272/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1263 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1094 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:128432 (125.4 KiB)  TX bytes:162172 (158.3 KiB)
          Interrupt:17

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)

veth1     Link encap:Ethernet  HWaddr 00:16:3E:B0:99:01
          inet addr:192.168.99.1  Bcast:192.168.99.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:feb0:9901/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:131 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:9774 (9.5 KiB)  TX bytes:1728 (1.6 KiB)

vif0.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:131 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1728 (1.6 KiB)  TX bytes:9774 (9.5 KiB)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:79 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:9 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6134 (5.9 KiB)  TX bytes:2534 (2.4 KiB)

vif2.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:51 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:4 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3614 (3.5 KiB)  TX bytes:888 (888.0 b)

This is the bridge:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br1             8000.feffffffffff       no              vif0.1
                                                        vif1.0
                                                        vif2.0

This is the routing table:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.0.0.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.99.0    0.0.0.0         255.255.255.0   U         0 0          0 veth1
0.0.0.0         10.0.0.25       0.0.0.0         UG        0 0          0 eth0


-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Richard Jones

2006-Feb-04 14:10 UTC

head link

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Fri, Feb 03, 2006 at 10:33:11PM +0100, CJ van den Berg
wrote:> Then let Xen do the rest. Don''t worry if your dhcp server refuses
to start.
> That''s because it doesn''t have any addresses to serve.
Xen will add a
> static IP assignment to the dhcpd.conf file for each domU that is created
> and restart the dhcp server.
> 
> I have also made several enhancements to my local vif-nat so that a dns
> server is also updated dynamically for each created domU, which allows me
to
> use hostnames when communicating from one domU to another. So I never have
> to figure out what IP address a domU has.
Thanks for this description.  These sorts of messages should be in the
official documentation!

The problem I have which prevents me from using dynamically assigned
IP addresses is I want incoming connections to be diverted to known
domU''s.  If the domU''s IP addresses change, then that is
considerably
more difficult.

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

CJ van den Berg

2006-Feb-06 12:07 UTC

head link

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

On Sat, Feb 04, 2006 at 02:10:16PM +0000, Richard Jones
wrote:> On Fri, Feb 03, 2006 at 10:33:11PM +0100, CJ van den Berg wrote:
> > Then let Xen do the rest. Don''t worry if your dhcp server
refuses to start.
> > That''s because it doesn''t have any addresses to
serve. Xen will add a
> > static IP assignment to the dhcpd.conf file for each domU that is
created
> > and restart the dhcp server.
> > 
> > I have also made several enhancements to my local vif-nat so that a
dns
> > server is also updated dynamically for each created domU, which allows
me to
> > use hostnames when communicating from one domU to another. So I never
have
> > to figure out what IP address a domU has.
> 
> Thanks for this description.  These sorts of messages should be in the
> official documentation!
You''re welcome.
> The problem I have which prevents me from using dynamically assigned
> IP addresses is I want incoming connections to be diverted to known
> domU''s.  If the domU''s IP addresses change, then that is
considerably
> more difficult.
The vif-nat script will only generate an IP address if you don''t
specify one
in the domU config file. If you need the domU to always use the same
address just use:

vif = [ ''ip=10.0.0.1'' ]

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Feb 2006 - Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

[Xen-users] Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT

Re: [Xen-users] Xen 3.0, setting up a virtual network with NAT