Xen users - Mar 2006 - Xen and LVS Direct Routing

We have a cluster of machines providing various services using
LVS direct routing.  For those not familiar with this, a typical
server has three IP addresses:

     (1)  A normal public IP address, unique to that machine;
     (2)  A public IP address, shared with several other machines;
     (3)  A private IP address.

Normally we have address (1) accepting ssh connections for maintenance
only.  Services such as http or telnet or ftp or rsync try to connect
to address (2), and are directed by the LVS director to the appropriate
server, using (for various reasons) (3) as a routing address.  To do
this, we use a Linux kernel with the so-called ''hidden arp''
patch
applied.  On a normal machine I do something like this:

        insmod dummy
        echo 1 > /proc/sys/net/ipv4/conf/all/hidden
        ifconfig dummy0 127.0.0.2 -arp
        echo 1 > /proc/sys/net/ipv4/conf/dummy0/hidden
        ifconfig dummy0 <address (2)> netmask 255.255.255.255 -arp up
        /sbin/ifconfig dummy0:0 <address (3)> up

in order to activate addresses (2) and (3).

Now, I would like to have virtual machines running under Xen which
can do the same thing.  I was able to add the ''hidden arp''
patch
to the Linux kernel for domU without any problems, but it isn''t
any use without the dummy kernel networking module, which won''t
compile.  Of course, I can add addresses to eth0:0, but doesn''t this
interfere with the normal address (1) if I have hidden arp enabled
for eth0?

Is anyone else running servers under Xen using LVS direct routing?

     -- Owen
     Dr A V Le Blanc

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I''ve been experimenting with LVS under domU servers (my first LVS
experience
actually). I had issues using LVS-NAT (TCP checksum errors for any client
off of the Xen server, though it worked ok from dom0) so I switched to
direct routing. It seems to work ok for me.

At the moment, I have two directors (failover using ultramonkey / heartbeat)
that are both also the webservers. I''ve setup the real IP of each
server on
eth0. The VIP is also set on eth0 of whichever director is active. On the
failover node, the VIP is added to the loopback address. This seems to work
fine, no need for the dummy driver. The webserver in my case is listening on
all IPs, not just the real. Not sure if I''ll run things this way if I
setup
production servers, but it works for testing.

I set /etc/sysctl.conf entries of the following:

net.2ipv4.ip_forward = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2

The arp_ignore and arp_announce take care of making sure that the failover
server that has the VIP on it''s loopback won''t reply to arps
for that IP.

root@lvs1# ip addr
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1300 qdisc pfifo_fast qlen 1000
    link/ether 00:16:3e:41:51:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.16/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.46/24 brd 192.168.0.255 scope global secondary eth0

root@lvs2# ip addr
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.0.46/32 brd 255.255.255.255 scope global lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1300 qdisc pfifo_fast qlen 1000
    link/ether 00:16:3e:64:34:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.17/24 brd 192.168.0.255 scope global eth0


--
Jason


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users