Christopher Bailey
2007-Aug-10 23:45 UTC
[Mongrel] Problems monitoring Mongrel with F5 BigIP
If this has already been covered, please point me to that (I didn''t find anything in my searches)... We are using F5 BigIP LTM load balancers. They have many pools of Mongrels they load balance across, and I of course want the F5 to know when a Mongrel goes down or is unavailable, etc. To do that, I need to have an F5 health monitor for HTTP make a request to the Mongrel. We do this same thing with our Nginx servers, and that works fine, but with Mongrel it doesn''t. I am wondering if anyone is successfully monitoring Mongrel instances from an F5? As further info... I''ve been working with F5 tech support on this, and we''re both stumped at the moment. The same HTTP monitor works fine with Nginx. Both try to fetch the same HTML file. Also, if I am ssh''ed in to the F5, and do a telnet to one of the Mongrels and do "GET /heartbeat.html HTTP/1.1", followed by two returns, this works fine. I use that same string (and two \n''s) in my F5''s send string for it''s HTTP monitor. According to F5 who looked at our tcpdump, the Mongrel just never responds (but in the same dump, they see the direct telnet version respond fine). The F5 makes a direct TCP request to do this. Any ideas? Suggestions? Known issue? Workarounds? Anyone successful with this type of setup? As an FYI, our system info: CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor Mongrel 1.0.1 Mongrel Cluster 1.0.2 Ruby 1.8.5 (2006-08-25) [x86_64-linux] F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070810/8525dbaa/attachment.html
I''ve never tried this but if you want to ping me offline, I have a good deal of bigip experience. I figure it will be useful for me to know how to do this. I''ll need the relevant monitor in bigip.conf. On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote:> If this has already been covered, please point me to that (I didn''t find > anything in my searches)... > > We are using F5 BigIP LTM load balancers. They have many pools of Mongrels > they load balance across, and I of course want the F5 to know when a Mongrel > goes down or is unavailable, etc. To do that, I need to have an F5 health > monitor for HTTP make a request to the Mongrel. We do this same thing with > our Nginx servers, and that works fine, but with Mongrel it doesn''t. I am > wondering if anyone is successfully monitoring Mongrel instances from an > F5? > > As further info... I''ve been working with F5 tech support on this, and > we''re both stumped at the moment. The same HTTP monitor works fine with > Nginx. Both try to fetch the same HTML file. Also, if I am ssh''ed in to > the F5, and do a telnet to one of the Mongrels and do "GET /heartbeat.html > HTTP/1.1", followed by two returns, this works fine. I use that same string > (and two \n''s) in my F5''s send string for it''s HTTP monitor. According to > F5 who looked at our tcpdump, the Mongrel just never responds (but in the > same dump, they see the direct telnet version respond fine). The F5 makes a > direct TCP request to do this. > > Any ideas? Suggestions? Known issue? Workarounds? Anyone successful with > this type of setup? > > As an FYI, our system info: > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > Mongrel 1.0.1 > Mongrel Cluster 1.0.2 > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) >
On Aug 10, 2007, at 19:45 , Christopher Bailey wrote:> As further info... I''ve been working with F5 tech support on this, > and we''re both stumped at the moment. The same HTTP monitor works > fine with Nginx. Both try to fetch the same HTML file. Also, if I > am ssh''ed in to the F5, and do a telnet to one of the Mongrels and > do "GET /heartbeat.html HTTP/1.1", followed by two returns, this > works fine. I use that same string (and two \n''s) in my F5''s send > string for it''s HTTP monitor. According to F5 who looked at our > tcpdump, the Mongrel just never responds (but in the same dump, > they see the direct telnet version respond fine). The F5 makes a > direct TCP request to do this.Christopher, First thing to do is start the mongrels in debug mode and see if anything gets logged when "Mongrel just never responds". ~Wayne s///g Wayne E. Seguin Sr. Systems Architect & Systems Administrator -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070810/4b2ae965/attachment.html
I also suspect you are going to have to explicitly close the connection in your monitor, if you aren''t already doing so. On 8/10/07, Brandorr <brandorr at opensolaris.org> wrote:> I''ve never tried this but if you want to ping me offline, I have a > good deal of bigip experience. > > I figure it will be useful for me to know how to do this. > > I''ll need the relevant monitor in bigip.conf. > > On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote: > > If this has already been covered, please point me to that (I didn''t find > > anything in my searches)... > > > > We are using F5 BigIP LTM load balancers. They have many pools of > Mongrels > > they load balance across, and I of course want the F5 to know when a > Mongrel > > goes down or is unavailable, etc. To do that, I need to have an F5 health > > monitor for HTTP make a request to the Mongrel. We do this same thing > with > > our Nginx servers, and that works fine, but with Mongrel it doesn''t. I am > > wondering if anyone is successfully monitoring Mongrel instances from an > > F5? > > > > As further info... I''ve been working with F5 tech support on this, and > > we''re both stumped at the moment. The same HTTP monitor works fine with > > Nginx. Both try to fetch the same HTML file. Also, if I am ssh''ed in to > > the F5, and do a telnet to one of the Mongrels and do "GET /heartbeat.html > > HTTP/1.1", followed by two returns, this works fine. I use that same > string > > (and two \n''s) in my F5''s send string for it''s HTTP monitor. According to > > F5 who looked at our tcpdump, the Mongrel just never responds (but in the > > same dump, they see the direct telnet version respond fine). The F5 makes > a > > direct TCP request to do this. > > > > Any ideas? Suggestions? Known issue? Workarounds? Anyone successful > with > > this type of setup? > > > > As an FYI, our system info: > > > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > > Mongrel 1.0.1 > > Mongrel Cluster 1.0.2 > > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) > > >
I think you need to close the connection after each health check. Otherwise mongrel/rails is going to block. (If my understanding is correct. Try the following health check and let me know how it works out: "GET / HTTP/1.1\nHost: \nConnection: Close\n On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote:> If this has already been covered, please point me to that (I didn''t find > anything in my searches)... > > We are using F5 BigIP LTM load balancers. They have many pools of Mongrels > they load balance across, and I of course want the F5 to know when a Mongrel > goes down or is unavailable, etc. To do that, I need to have an F5 health > monitor for HTTP make a request to the Mongrel. We do this same thing with > our Nginx servers, and that works fine, but with Mongrel it doesn''t. I am > wondering if anyone is successfully monitoring Mongrel instances from an F5? > > > As further info... I''ve been working with F5 tech support on this, and > we''re both stumped at the moment. The same HTTP monitor works fine with > Nginx. Both try to fetch the same HTML file. Also, if I am ssh''ed in to > the F5, and do a telnet to one of the Mongrels and do "GET /heartbeat.html > HTTP/1.1", followed by two returns, this works fine. I use that same string > (and two \n''s) in my F5''s send string for it''s HTTP monitor. According to > F5 who looked at our tcpdump, the Mongrel just never responds (but in the > same dump, they see the direct telnet version respond fine). The F5 makes a > direct TCP request to do this. > > Any ideas? Suggestions? Known issue? Workarounds? Anyone successful with > this type of setup? > > As an FYI, our system info: > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > Mongrel 1.0.1 > Mongrel Cluster 1.0.2 > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Christopher Bailey
2007-Aug-11 02:30 UTC
[Mongrel] Problems monitoring Mongrel with F5 BigIP
That didn''t work. I don''t think you need that though. If I just telnet in to the Mongrel, and do a pure "GET /heartbeat.html HTTP/1.1\n\n" it works. I am going to try running the Mongrel in debug and see, will do that later tonight... On 8/10/07, Brandorr <brandorr at opensolaris.org> wrote:> > I think you need to close the connection after each health check. > Otherwise mongrel/rails is going to block. (If my understanding is > correct. > > Try the following health check and let me know how it works out: > > "GET / HTTP/1.1\nHost: \nConnection: Close\n > > On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote: > > If this has already been covered, please point me to that (I didn''t find > > anything in my searches)... > > > > We are using F5 BigIP LTM load balancers. They have many pools of > Mongrels > > they load balance across, and I of course want the F5 to know when a > Mongrel > > goes down or is unavailable, etc. To do that, I need to have an F5 > health > > monitor for HTTP make a request to the Mongrel. We do this same thing > with > > our Nginx servers, and that works fine, but with Mongrel it doesn''t. I > am > > wondering if anyone is successfully monitoring Mongrel instances from an > F5? > > > > > > As further info... I''ve been working with F5 tech support on this, and > > we''re both stumped at the moment. The same HTTP monitor works fine with > > Nginx. Both try to fetch the same HTML file. Also, if I am ssh''ed in > to > > the F5, and do a telnet to one of the Mongrels and do "GET > /heartbeat.html > > HTTP/1.1", followed by two returns, this works fine. I use that same > string > > (and two \n''s) in my F5''s send string for it''s HTTP monitor. According > to > > F5 who looked at our tcpdump, the Mongrel just never responds (but in > the > > same dump, they see the direct telnet version respond fine). The F5 > makes a > > direct TCP request to do this. > > > > Any ideas? Suggestions? Known issue? Workarounds? Anyone successful > with > > this type of setup? > > > > As an FYI, our system info: > > > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > > Mongrel 1.0.1 > > Mongrel Cluster 1.0.2 > > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) > > > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070810/450e311d/attachment-0001.html
Christopher Bailey
2007-Aug-11 04:37 UTC
[Mongrel] Problems monitoring Mongrel with F5 BigIP
Update... So, I''ve now run Mongrel in debug mode. I don''t get any hits from the F5. I have ensured the Mongrel/debug mode is working by just doing a telnet and then GET request from the F5 machine to the Mongrel, and those requests show up in the Mongrel debug logs (and the regular Mongrel log). So, somehow the request just isn''t even getting into Mongrel. What''s odd about this, is that these exact same requests get to say the boxes that have Nginx on them. I also tried changing Mongrel to run on port 80 in case that was some oddity of it, but no difference. As Brandorr asked, the tcp and tcp half open F5 monitors are successful, just HTTP is not. So, I will bring it all back up with F5 and have them setup a Mongrel and test. It appears to be something with the way an F5 makes the request or something. I''ll also ask Jason H over at Joyent if they do direct monitoring of Mongrels with their F5''s, thanks for reminding me about that. On 8/10/07, Brandorr <brandorr at opensolaris.org> wrote:> > Where did you run TCPDUMP? Or the BigIP or the Linux box? Basically I > am wondering if the HTTP request is leaving the BigIP. > > Sorry to bother, I haven''t had a chance to work with BigIP in aver 1 > 1/2 years, and I miss it... > > One other thing, Joyent and Twitter use BigIPs with Mongrel.. > > On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote: > > Yes, both of the standard tcp tests, and the icmp ping tests work. We > have > > a slew of these machines, all configured identically, and the only case > that > > doesn''t work is when hitting Mongrel HTTP. Nginx works fine, as do a > few > > other things we have. I will run Mongrel debug after I put the kids to > bed > > and see :) > > > > > > On 8/10/07, Brandorr <brandorr at opensolaris.org> wrote: > > > Christopher Bailey <chris at codeintensity.com> > > > > > > Do TCP open and tcp halfopen checks work > > > > > > On 8/10/07, Christopher Bailey < chris at codeintensity.com> wrote: > > > > That didn''t work. I don''t think you need that though. If I just > telnet > > in > > > > to the Mongrel, and do a pure "GET /heartbeat.html HTTP/1.1\n\n" it > > works. > > > > I am going to try running the Mongrel in debug and see, will do that > > later > > > > tonight... > > > > > > > > > > > > On 8/10/07, Brandorr <brandorr at opensolaris.org > wrote: > > > > > I think you need to close the connection after each health check. > > > > > Otherwise mongrel/rails is going to block. (If my understanding is > > > > > correct. > > > > > > > > > > Try the following health check and let me know how it works out: > > > > > > > > > > "GET / HTTP/1.1\nHost: \nConnection: Close\n > > > > > > > > > > On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote: > > > > > > If this has already been covered, please point me to that (I > didn''t > > find > > > > > > anything in my searches)... > > > > > > > > > > > > We are using F5 BigIP LTM load balancers. They have many pools > of > > > > Mongrels > > > > > > they load balance across, and I of course want the F5 to know > when a > > > > Mongrel > > > > > > goes down or is unavailable, etc. To do that, I need to have an > F5 > > > > health > > > > > > monitor for HTTP make a request to the Mongrel. We do this same > > thing > > > > with > > > > > > our Nginx servers, and that works fine, but with Mongrel it > doesn''t. > > I > > > > am > > > > > > wondering if anyone is successfully monitoring Mongrel instances > > from an > > > > F5? > > > > > > > > > > > > > > > > > > As further info... I''ve been working with F5 tech support on > this, > > and > > > > > > we''re both stumped at the moment. The same HTTP monitor works > fine > > with > > > > > > Nginx. Both try to fetch the same HTML file. Also, if I am > ssh''ed > > in > > > > to > > > > > > the F5, and do a telnet to one of the Mongrels and do "GET > > > > /heartbeat.html > > > > > > HTTP/1.1", followed by two returns, this works fine. I use that > > same > > > > string > > > > > > (and two \n''s) in my F5''s send string for it''s HTTP monitor. > > According > > > > to > > > > > > F5 who looked at our tcpdump, the Mongrel just never responds > (but > > in > > > > the > > > > > > same dump, they see the direct telnet version respond > fine). The F5 > > > > makes a > > > > > > direct TCP request to do this. > > > > > > > > > > > > Any ideas? Suggestions? Known issue? Workarounds? Anyone > > successful > > > > with > > > > > > this type of setup? > > > > > > > > > > > > As an FYI, our system info: > > > > > > > > > > > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > > > > > > Mongrel 1.0.1 > > > > > > Mongrel Cluster 1.0.2 > > > > > > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > > > > > > > > > > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Mongrel-users mailing list > > > > > > Mongrel-users at rubyforge.org > > > > > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > > > > > _______________________________________________ > > > > > Mongrel-users mailing list > > > > > Mongrel-users at rubyforge.org > > > > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Mongrel-users mailing list > > > > Mongrel-users at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070810/38f66f55/attachment.html
Jason A. Hoffman
2007-Aug-11 06:08 UTC
[Mongrel] Problems monitoring Mongrel with F5 BigIP
Hi Chris, We run a bunch of 6400s in production. You have to terminate the lines with the /r too GET /heartbeat/index HTTP/1.1 \r\n \r\n Our''s looks like http://jxh.bingodisk.com/public/mongrel-bigip.png Regards, Jason On Aug 10, 2007, at 7:30 PM, Christopher Bailey wrote:> That didn''t work. I don''t think you need that though. If I just > telnet in to the Mongrel, and do a pure "GET /heartbeat.html HTTP/ > 1.1\n\n" it works. I am going to try running the Mongrel in debug > and see, will do that later tonight... > > > On 8/10/07, Brandorr <brandorr at opensolaris.org> wrote: I think you > need to close the connection after each health check. > Otherwise mongrel/rails is going to block. (If my understanding is > correct. > > Try the following health check and let me know how it works out: > > "GET / HTTP/1.1\nHost: \nConnection: Close\n > > On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote: > > If this has already been covered, please point me to that (I > didn''t find > > anything in my searches)... > > > > We are using F5 BigIP LTM load balancers. They have many pools > of Mongrels > > they load balance across, and I of course want the F5 to know > when a Mongrel > > goes down or is unavailable, etc. To do that, I need to have an > F5 health > > monitor for HTTP make a request to the Mongrel. We do this same > thing with > > our Nginx servers, and that works fine, but with Mongrel it > doesn''t. I am > > wondering if anyone is successfully monitoring Mongrel instances > from an F5? > > > > > > As further info... I''ve been working with F5 tech support on > this, and > > we''re both stumped at the moment. The same HTTP monitor works > fine with > > Nginx. Both try to fetch the same HTML file. Also, if I am > ssh''ed in to > > the F5, and do a telnet to one of the Mongrels and do "GET / > heartbeat.html > > HTTP/1.1", followed by two returns, this works fine. I use that > same string > > (and two \n''s) in my F5''s send string for it''s HTTP monitor. > According to > > F5 who looked at our tcpdump, the Mongrel just never responds > (but in the > > same dump, they see the direct telnet version respond fine). The > F5 makes a > > direct TCP request to do this. > > > > Any ideas? Suggestions? Known issue? Workarounds? Anyone > successful with > > this type of setup? > > > > As an FYI, our system info: > > > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > > Mongrel 1.0.1 > > Mongrel Cluster 1.0.2 > > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) > > > >
Christopher Bailey
2007-Aug-11 17:18 UTC
[Mongrel] Problems monitoring Mongrel with F5 BigIP
Jason, thanks so very much, that solved it! On 8/10/07, Jason A. Hoffman <jason at joyent.com> wrote:> > Hi Chris, > > We run a bunch of 6400s in production. > > You have to terminate the lines with the /r too > > GET /heartbeat/index HTTP/1.1 > \r\n > \r\n > > Our''s looks like > > http://jxh.bingodisk.com/public/mongrel-bigip.png > > Regards, Jason > > > > > On Aug 10, 2007, at 7:30 PM, Christopher Bailey wrote: > > > That didn''t work. I don''t think you need that though. If I just > > telnet in to the Mongrel, and do a pure "GET /heartbeat.html HTTP/ > > 1.1\n\n" it works. I am going to try running the Mongrel in debug > > and see, will do that later tonight... > > > > > > On 8/10/07, Brandorr <brandorr at opensolaris.org> wrote: I think you > > need to close the connection after each health check. > > Otherwise mongrel/rails is going to block. (If my understanding is > > correct. > > > > Try the following health check and let me know how it works out: > > > > "GET / HTTP/1.1\nHost: \nConnection: Close\n > > > > On 8/10/07, Christopher Bailey <chris at codeintensity.com> wrote: > > > If this has already been covered, please point me to that (I > > didn''t find > > > anything in my searches)... > > > > > > We are using F5 BigIP LTM load balancers. They have many pools > > of Mongrels > > > they load balance across, and I of course want the F5 to know > > when a Mongrel > > > goes down or is unavailable, etc. To do that, I need to have an > > F5 health > > > monitor for HTTP make a request to the Mongrel. We do this same > > thing with > > > our Nginx servers, and that works fine, but with Mongrel it > > doesn''t. I am > > > wondering if anyone is successfully monitoring Mongrel instances > > from an F5? > > > > > > > > > As further info... I''ve been working with F5 tech support on > > this, and > > > we''re both stumped at the moment. The same HTTP monitor works > > fine with > > > Nginx. Both try to fetch the same HTML file. Also, if I am > > ssh''ed in to > > > the F5, and do a telnet to one of the Mongrels and do "GET / > > heartbeat.html > > > HTTP/1.1", followed by two returns, this works fine. I use that > > same string > > > (and two \n''s) in my F5''s send string for it''s HTTP monitor. > > According to > > > F5 who looked at our tcpdump, the Mongrel just never responds > > (but in the > > > same dump, they see the direct telnet version respond fine). The > > F5 makes a > > > direct TCP request to do this. > > > > > > Any ideas? Suggestions? Known issue? Workarounds? Anyone > > successful with > > > this type of setup? > > > > > > As an FYI, our system info: > > > > > > CentOS 5, nearly all up to date, 64bit, Core 2 Duo processor > > > Mongrel 1.0.1 > > > Mongrel Cluster 1.0.2 > > > Ruby 1.8.5 (2006-08-25) [x86_64-linux] > > > > > > F5 BigIP LTM, currently running 9.1.2 (working on moving to 9.3) > > > > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070811/129f8883/attachment-0001.html
On Aug 11, 2007, at 13:18 , Christopher Bailey wrote:> Jason, thanks so very much, that solved it! > > On 8/10/07, Jason A. Hoffman <jason at joyent.com> wrote: > Hi Chris, > > We run a bunch of 6400s in production. > > You have to terminate the lines with the /r too > > GET /heartbeat/index HTTP/1.1 > \r\n > \r\n > > Our''s looks like > > http://jxh.bingodisk.com/public/mongrel-bigip.png > > Regards, JasonChristopher, In order to help anyone else encountering your issue, can you write up the problem description and solution in detail and send it to me off-list so that we can add it to the Mongrel docs. Thanks! ~Wayne s///g Wayne E. Seguin Sr. Systems Architect & Systems Administrator -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070812/f8a86d20/attachment.html