Tom Potwin
2009-May-25 21:40 UTC
[Xen-users] Using Xen on load balancer keeps loosing connection
Hi I just joined this list after I tried looking for help with the HAProxy, and HA heartbeat people. They''re both at a loss as to what''s causing my problem. The only place I haven''t tried is here with the Xen experts. I checked xm info for the version and got this: Release : 2.6.24-23-xen version : #1 SMP Wed Apr 1 23:47:10 UTC 2009 I''m using HAProxy and Heartbeat on two Ubuntu 8.04 servers. I have two Xen nodes on each of my physical machines. One is the load balance and Heartbeat (LB1), the other is the actual LAMP web server (WEB1). Testing HAProxy/Heartbeat setup seems that it''s working fine, by that I mean that shutting off one of the web servers, it switches to the other one. My problem is I keep loosing access to the HAProxy stats page, which only should show up on the LB nodes where HAProxy is. I know that isn''t a huge problem, but I''m worried it might be a sign of a bigger problem somewhere. The stats show up fine for about 15-20 minutes, then I get a apache generic 404 error page. I also see: "File does not exist: /var/www/apache2-default/haproxy" show up in the apache error log on WEB1 node as soon as I loose it. If I go back to my LB1 node and restart Heartbeat, it all comes back for another 15-20 minutes. There''s nothing in any of the logs that I can see, other than it stops logging when it happens. I use http://192.168.31.100/haproxy?stats to get to that stats page. The .100 is the shared address between the the load balancers. If I use 192.168.31.201, which is LB1, I get the browser''s 404 notice. If I use .100, it shows my apache generic 404 page. So somehow it stops seeing LB1, and goes to port 80 on my web server on the WEB1 node. That''s where I see the apache error saying it can''t find the HAProxy stats page. When I used the "tcpdump -q -i eth0 tcp port 80 and src host 192.168.31.100" command, it showed me looking at the stats, and the test web page: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 11:23:16.106664 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 11:23:16.254209 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 11:23:16.254409 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 262 11:23:16.254501 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 11:23:17.460534 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 11:23:17.628385 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 11:23:17.628590 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 11:23:17.839448 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 11:23:17.839460 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 524 Once I couldn''t see the stats page again, the output stopped completely. I watched it on LB2 as well. It seems like it stops listing to the .100 IP address. If I use "tcpdump -q -i eth0 tcp port 80" I see LB1 checking web1 and web2, but nothing on the .100 address. The HAProxy people said they thought it might be a Heartbeat problem, because after they checked my HAProxy setup, they couldn''t find any problems there. The heartbeat people say the same thing. Sorry for the long post, I''m just getting desperate for some help. Thanks, Tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Booth
2009-May-25 22:54 UTC
Re: [Xen-users] Using Xen on load balancer keeps loosing connection
Tom, A two part response: 1. Can you step back and give some more context? Are you attempting to simulate a proposed production environment or is this environment one that you will run in production? If it''s the latter, then I''m puzzled why you''d use xen. 2. How many physical NICs does each server have? Are you using bridged or routed networking? How many PCPUs do you have? how is dom0 and each domu configured for vcpus/caps/weights? Peter Sent from my iPhone On May 25, 2009, at 5:40 PM, Tom Potwin <wxman2@gmail.com> wrote:> Hi > > I just joined this list after I tried looking for help with the > HAProxy, and > HA heartbeat people. They''re both at a loss as to what''s causing my > problem. > The only place I haven''t tried is here with the Xen experts. I > checked xm > info for the version and got this: > Release : 2.6.24-23-xen > version : #1 SMP Wed Apr 1 23:47:10 UTC 2009 > > I''m using HAProxy and Heartbeat on two Ubuntu 8.04 servers. I have > two Xen > nodes on each of my physical machines. One is the load balance and > Heartbeat > (LB1), the other is the actual LAMP web server (WEB1). Testing > HAProxy/Heartbeat setup seems that it''s working fine, by that I mean > that > shutting off one of the web servers, it switches to the other one. My > problem is I keep loosing access to the HAProxy stats page, which only > should show up on the LB nodes where HAProxy is. I know that isn''t a > huge > problem, but I''m worried it might be a sign of a bigger problem > somewhere. > > The stats show up fine for about 15-20 minutes, then I get a apache > generic > 404 error page. I also see: "File does not exist: > /var/www/apache2-default/haproxy" show up in the apache error log on > WEB1 > node as soon as I loose it. If I go back to my LB1 node and restart > Heartbeat, it all comes back for another 15-20 minutes. There''s > nothing in > any of the logs that I can see, other than it stops logging when it > happens. > I use http://192.168.31.100/haproxy?stats to get to that stats page. > The > .100 is the shared address between the the load balancers. If I use > 192.168.31.201, which is LB1, I get the browser''s 404 notice. If I > use .100, > it shows my apache generic 404 page. So somehow it stops seeing LB1, > and > goes to port 80 on my web server on the WEB1 node. That''s where I > see the > apache error saying it can''t find the HAProxy stats page. > > When I used the "tcpdump -q -i eth0 tcp port 80 and src host 192.168.31.100 > " > command, it showed me looking at the stats, and the test web page: > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes > 11:23:16.106664 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 > 11:23:16.254209 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 > 11:23:16.254409 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 262 > 11:23:16.254501 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 > 11:23:17.460534 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 > 11:23:17.628385 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 > 11:23:17.628590 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 > 11:23:17.839448 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 > 11:23:17.839460 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 524 > > Once I couldn''t see the stats page again, the output stopped > completely. I > watched it on LB2 as well. It seems like it stops listing to the . > 100 IP > address. If I use "tcpdump -q -i eth0 tcp port 80" I see LB1 > checking web1 > and web2, but nothing on the .100 address. The HAProxy people said > they > thought it might be a Heartbeat problem, because after they checked my > HAProxy setup, they couldn''t find any problems there. The heartbeat > people > say the same thing. Sorry for the long post, I''m just getting > desperate for > some help. > > Thanks, Tom > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tom Potwin
2009-May-26 16:03 UTC
RE: [Xen-users] Using Xen on load balancer keeps loosing connection
Hi Peter To answer your questions, yes, it is going to be a working server setup. Before I go any further, I guess I had better ask you first why are you puzzled that I would use xen? I needed to have two load balancers with auto failover, and of course two LAMP web servers setup''s. I''m using a router/firewall that sends traffic from my public IP to the local shared IP 192.168.31.100. Each physical machine has two physical NIC''s. One connects to the router, and the other is just for replication/mirroring between the two servers. Attached is the cfg for the LB1 node that seems to be having the problem. Please let me know if I didn''t answer something. I''m still getting used to using xen. Tom -----Original Message----- From: Peter Booth [mailto:peter_booth@mac.com] Sent: Monday, May 25, 2009 6:55 PM To: littimes@tlt.com Cc: xen-users@lists.xensource.com Subject: Re: [Xen-users] Using Xen on load balancer keeps loosing connection Tom, A two part response: 1. Can you step back and give some more context? Are you attempting to simulate a proposed production environment or is this environment one that you will run in production? If it''s the latter, then I''m puzzled why you''d use xen. 2. How many physical NICs does each server have? Are you using bridged or routed networking? How many PCPUs do you have? how is dom0 and each domu configured for vcpus/caps/weights? Peter Sent from my iPhone On May 25, 2009, at 5:40 PM, Tom Potwin <wxman2@gmail.com> wrote:> Hi > > I just joined this list after I tried looking for help with the > HAProxy, and HA heartbeat people. They''re both at a loss as to what''s > causing my problem. > The only place I haven''t tried is here with the Xen experts. I checked > xm info for the version and got this: > Release : 2.6.24-23-xen > version : #1 SMP Wed Apr 1 23:47:10 UTC 2009 > > I''m using HAProxy and Heartbeat on two Ubuntu 8.04 servers. I have two > Xen nodes on each of my physical machines. One is the load balance and > Heartbeat (LB1), the other is the actual LAMP web server (WEB1). > Testing HAProxy/Heartbeat setup seems that it''s working fine, by that > I mean that shutting off one of the web servers, it switches to the > other one. My problem is I keep loosing access to the HAProxy stats > page, which only should show up on the LB nodes where HAProxy is. I > know that isn''t a huge problem, but I''m worried it might be a sign of > a bigger problem somewhere. > > The stats show up fine for about 15-20 minutes, then I get a apache > generic > 404 error page. I also see: "File does not exist: > /var/www/apache2-default/haproxy" show up in the apache error log on > WEB1 > node as soon as I loose it. If I go back to my LB1 node and restart > Heartbeat, it all comes back for another 15-20 minutes. There''s > nothing in any of the logs that I can see, other than it stops logging > when it happens. > I use http://192.168.31.100/haproxy?stats to get to that stats page. > The > .100 is the shared address between the the load balancers. If I use > 192.168.31.201, which is LB1, I get the browser''s 404 notice. If I use > .100, it shows my apache generic 404 page. So somehow it stops seeing > LB1, and goes to port 80 on my web server on the WEB1 node. That''s > where I see the apache error saying it can''t find the HAProxy stats > page. > > When I used the "tcpdump -q -i eth0 tcp port 80 and src host > 192.168.31.100 " > command, it showed me looking at the stats, and the test web page: > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 > bytes > 11:23:16.106664 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 > 11:23:16.254209 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 > 11:23:16.254409 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 262 > 11:23:16.254501 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0 > 11:23:17.460534 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 > 11:23:17.628385 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 > 11:23:17.628590 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 > 11:23:17.839448 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 > 11:23:17.839460 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 524 > > Once I couldn''t see the stats page again, the output stopped > completely. I watched it on LB2 as well. It seems like it stops > listing to the . > 100 IP > address. If I use "tcpdump -q -i eth0 tcp port 80" I see LB1 checking > web1 and web2, but nothing on the .100 address. The HAProxy people > said they thought it might be a Heartbeat problem, because after they > checked my HAProxy setup, they couldn''t find any problems there. The > heartbeat people say the same thing. Sorry for the long post, I''m just > getting desperate for some help. > > Thanks, Tom > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users