Mongrel users - Feb 2008 - Newbie Question: Can I setup mongrel as a proxy between a browser and the web?

I am trying to create a proxy which I can route my browser requests through
in order to gather information on the web page loading.  At this time I am
particularly interested in the many GET requests associated with loading a
single web page. Note: This does not have anything to do with rails, my goal
is to run a ruby app in the background, set firefox to proxy all requests
through it, log those requests, then return the fulfilled request back to
the browser (i.e. the web page still loads in the browser).

I posted this on a ruby forum, and it was suggested to do something along
the lines of:

require ''rubygems''
require ''mongrel''
require ''logger''

class HeaderHandler < Mongrel::HttpHandler

  @@logger = Logger.new(''headers.log'')

  def process(request,response)
    response.start(200) do |head, out|
      head["Content-Type"] = "text/plain"
      @@logger.info request.params
    end
  end
end



server = Mongrel::HttpServer.new("127.0.0.1", "2222")
server.register("/", HeaderHandler.new)
server.register("/favicon.ico",
Mongrel::Error404Handler.new(""))
server.run.join

The problem is that this only records the first request (i.e. google.com,
and not any other requests that are associated with loading that page), and
the page is never loaded in the browser as the requests is never actually
proxied.

I have played around with this a bit, and tried changing some things up, but
I haven''t really come up with anything. If anyone could point me in the
right direction, I would be grateful. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20080206/1249c858/attachment.html

Hello there.  I''m the one that made that suggestion to you earlier, I
had a feeling it was more than just capturing the headers.

Do you happen to be running apache on the webserver?  You can log all
of it and not worry about a proxy.  Here are two ways.

1.  mod_dumpio

# Put this in your apache config
DumpIOInput On
LogLevel debug

2. mod_log_forensic

# Put this in your apache config
ForensicLog /some/path/to/a/logfile.log

Try MouseHole: http://code.whytheluckystiff.net/mouseHole/

Evan

On Feb 6, 2008 3:39 PM, Dusty Doris <mongrel at dusty.name>
wrote:
> Hello there.  I''m the one that made that suggestion to you
earlier, I
> had a feeling it was more than just capturing the headers.
>
> Do you happen to be running apache on the webserver?  You can log all
> of it and not worry about a proxy.  Here are two ways.
>
> 1.  mod_dumpio
>
> # Put this in your apache config
> DumpIOInput On
> LogLevel debug
>
> 2. mod_log_forensic
>
> # Put this in your apache config
> ForensicLog /some/path/to/a/logfile.log
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
Evan Weaver
Cloudburst, LLC

Hi Guys,

Thanks for your responses.

Dusty: Im not sure what the server does, but what I am developing is
intended to be a client side testing utility. i.e. all that will be
installed on the machine is a browser, and some programming environment (i.e.
ruby)

Evan: I saw mousehole yesterday, but I didn''t quite understand how I
could
edit a script to do what I want (for now I am starting off small, and I just
want to collect header information, but eventually I assume it will be a
whole variety of information I would like to log.). I will try and take
another look at it.

Thanks for your help.

On Feb 6, 2008 1:02 PM, Evan Weaver <evan at cloudbur.st> wrote:
> Try MouseHole: http://code.whytheluckystiff.net/mouseHole/
>
> Evan
>
> On Feb 6, 2008 3:39 PM, Dusty Doris <mongrel at dusty.name> wrote:
> > Hello there.  I''m the one that made that suggestion to you
earlier, I
> > had a feeling it was more than just capturing the headers.
> >
> > Do you happen to be running apache on the webserver?  You can log all
> > of it and not worry about a proxy.  Here are two ways.
> >
> > 1.  mod_dumpio
> >
> > # Put this in your apache config
> > DumpIOInput On
> > LogLevel debug
> >
> > 2. mod_log_forensic
> >
> > # Put this in your apache config
> > ForensicLog /some/path/to/a/logfile.log
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
>
> --
> Evan Weaver
> Cloudburst, LLC
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20080206/0ed040e2/attachment.html

At 12:39 PM 2/6/2008, mongrel-users-request at rubyforge.org
wrote:
>Date: Wed, 6 Feb 2008 11:32:16 -0800
>From: "Arash Nikkar" <anikkar at gmail.com>
>Subject: [Mongrel] Newbie Question: Can I setup mongrel as a proxy
>         between a       browser and the web?
>To: mongrel-users at rubyforge.org
>Message-ID:
>         <61ddf71e0802061132i59a5b05fy7df5c74e6cc1be02 at
mail.gmail.com>
>Content-Type: text/plain; charset="iso-8859-1"
>
>I am trying to create a proxy which I can route my browser requests 
>through
>in order to gather information on the web page loading.  At this time 
>I am
>particularly interested in the many GET requests associated with 
>loading a
>single web page. Note: This does not have anything to do with rails, 
>my goal
>is to run a ruby app in the background, set firefox to proxy all 
>requests
>through it, log those requests, then return the fulfilled request back 
>to
>the browser (i.e. the web page still loads in the browser).
Hi Arash,

You may have interest in coding a solution to this, for your own 
reasons. I totally support those inclinations! :) I''m going to go a 
little off-topic with an idea though:

Personally, I use "Tamper Data" 
(https://addons.mozilla.org/en-US/firefox/addon/966) within FireFox to 
handle this type of thing. It''s a most excellent resource:
it''s like a
wire sniffer at the HTTP level only. It supports decoding request data 
and tracking POST, GET and Ajax stuff.

It''s been invaluable for me in some circumstances - I hope the 
reference is helpful.

Steve

p.s. It''s been particularly helpful in trapping precisely certain live 
GET & Ajax requests which I then convert into testing requests to 
re-create the errors.

At 12:39 PM 2/6/2008, mongrel-users-request at rubyforge.org
wrote:
>Date: Wed, 6 Feb 2008 11:32:16 -0800
>From: "Arash Nikkar" <anikkar at gmail.com>
>Subject: [Mongrel] Newbie Question: Can I setup mongrel as a proxy
>         between a       browser and the web?
>To: mongrel-users at rubyforge.org
>Message-ID:
>         <61ddf71e0802061132i59a5b05fy7df5c74e6cc1be02 at
mail.gmail.com>
>Content-Type: text/plain; charset="iso-8859-1"
>
>I am trying to create a proxy which I can route my browser requests 
>through
>in order to gather information on the web page loading.  At this time 
>I am
>particularly interested in the many GET requests associated with 
>loading a
>single web page. Note: This does not have anything to do with rails, 
>my goal
>is to run a ruby app in the background, set firefox to proxy all 
>requests
>through it, log those requests, then return the fulfilled request back 
>to
>the browser (i.e. the web page still loads in the browser).
Hi Arash,

You may have interest in coding a solution to this, for your own 
reasons. I totally support those inclinations! :) I''m going to go a 
little off-topic with an idea though:

Personally, I use "Tamper Data" 
(https://addons.mozilla.org/en-US/firefox/addon/966) within FireFox to 
handle this type of thing. It''s a most excellent resource:
it''s like a
wire sniffer at the HTTP level only. It supports decoding request data 
and tracking POST, GET and Ajax stuff.

It''s been invaluable for me in some circumstances - I hope the 
reference is helpful.

Steve

p.s. It''s been particularly helpful in trapping precisely certain live 
GET & Ajax requests which I then convert into testing requests to 
re-create the errors.

Hi Steve,

I wish I could use something like that (in fact, I often use firebug to see
what is actually going on), but I am trying to build something that is
browser independent.

Ideally, what I am looking for is something which just listens to all the
traffic going into and out of a browser (using whatever protocol) and do
some analysis on that traffic (even if that means I need to duplicate all
the data elsewhere on my disk). but alas, I come from a desktop background
and I am not that well versed in web related issues. So I suppose I will
just take baby steps (with your guy''s help, if you dont mind.)

--arash

On Feb 6, 2008 3:57 PM, Steve Midgley <public at misuse.org> wrote:
> At 12:39 PM 2/6/2008, mongrel-users-request at rubyforge.org wrote:
> >Date: Wed, 6 Feb 2008 11:32:16 -0800
> >From: "Arash Nikkar" <anikkar at gmail.com>
> >Subject: [Mongrel] Newbie Question: Can I setup mongrel as a proxy
> >         between a       browser and the web?
> >To: mongrel-users at rubyforge.org
> >Message-ID:
> >         <61ddf71e0802061132i59a5b05fy7df5c74e6cc1be02 at
mail.gmail.com>
> >Content-Type: text/plain; charset="iso-8859-1"
> >
> >I am trying to create a proxy which I can route my browser requests
> >through
> >in order to gather information on the web page loading.  At this time
> >I am
> >particularly interested in the many GET requests associated with
> >loading a
> >single web page. Note: This does not have anything to do with rails,
> >my goal
> >is to run a ruby app in the background, set firefox to proxy all
> >requests
> >through it, log those requests, then return the fulfilled request back
> >to
> >the browser (i.e. the web page still loads in the browser).
>
> Hi Arash,
>
> You may have interest in coding a solution to this, for your own
> reasons. I totally support those inclinations! :) I''m going to go
a
> little off-topic with an idea though:
>
> Personally, I use "Tamper Data"
> (https://addons.mozilla.org/en-US/firefox/addon/966) within FireFox to
> handle this type of thing. It''s a most excellent resource:
it''s like a
> wire sniffer at the HTTP level only. It supports decoding request data
> and tracking POST, GET and Ajax stuff.
>
> It''s been invaluable for me in some circumstances - I hope the
> reference is helpful.
>
> Steve
>
> p.s. It''s been particularly helpful in trapping precisely certain
live
> GET & Ajax requests which I then convert into testing requests to
> re-create the errors.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20080206/e299de22/attachment.html

It might be better if you use a packet sniffer like WireShark or  
tcpdump.  Mongrel wouldn''t be able to handle non-http traffic.

http://www.wireshark.org/
http://www.tcpdump.org/tcpdump_man.html

On Feb 7, 2008, at 8:20 AM, Arash Nikkar wrote:
> Ideally, what I am looking for is something which just listens to  
> all the traffic going into and out of a browser (using whatever  
> protocol) and do some analysis on that traffic (even if that means I  
> need to duplicate all the data elsewhere on my disk). but alas, I  
> come from a desktop background and I am not that well versed in web  
> related issues. So I suppose I will just take baby steps (with your  
> guy''s help, if you dont mind.)

Arash,   mongrel-esi is a fairly simple proxy server... You could possibly
modify or extract the bits of it you needed.  The bulk of the proxy code is
in just one file:
http://code.google.com/p/mongrel-esi/source/browse/trunk/lib/esi/handler.rb

It uses Net::HTTP library, forwarding POST and GET requests... might be
helpful...

-Todd

On Feb 6, 2008 7:32 PM, Eden Li <eden at mojiti.com> wrote:
> It might be better if you use a packet sniffer like WireShark or
> tcpdump.  Mongrel wouldn''t be able to handle non-http traffic.
>
> http://www.wireshark.org/
> http://www.tcpdump.org/tcpdump_man.html
>
> On Feb 7, 2008, at 8:20 AM, Arash Nikkar wrote:
>
> > Ideally, what I am looking for is something which just listens to
> > all the traffic going into and out of a browser (using whatever
> > protocol) and do some analysis on that traffic (even if that means I
> > need to duplicate all the data elsewhere on my disk). but alas, I
> > come from a desktop background and I am not that well versed in web
> > related issues. So I suppose I will just take baby steps (with your
> > guy''s help, if you dont mind.)
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20080207/02ee14f1/attachment.html

On 7 Feb 2008, at 00:20, Arash Nikkar wrote:
> Hi Steve,
>
> I wish I could use something like that (in fact, I often use firebug  
> to see what is actually going on), but I am trying to build  
> something that is browser independent.
>
> Ideally, what I am looking for is something which just listens to  
> all the traffic going into and out of a browser (using whatever  
> protocol) and do some analysis on that traffic (even if that means I  
> need to duplicate all the data elsewhere on my disk). but alas, I  
> come from a desktop background and I am not that well versed in web  
> related issues. So I suppose I will just take baby steps (with your  
> guy''s help, if you dont mind.)
http://www.xk72.com/charles/

"Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables  
a developer to view all of the HTTP traffic between their machine and  
the Internet. This includes requests, responses and the HTTP headers  
(which contain the cookies and caching information)."

Browser independent, it intercepts, decodes and displays all browser  
<--> server communications, including decoding Flash AMF data.
It''s
invaluable for Flash --> server and "ajax" development; sounds like
it''ll do some of what you want, too.

cheers
i
>
> --arash
>
> On Feb 6, 2008 3:57 PM, Steve Midgley <public at misuse.org> wrote:
> At 12:39 PM 2/6/2008, mongrel-users-request at rubyforge.org wrote:
> >Date: Wed, 6 Feb 2008 11:32:16 -0800
> >From: "Arash Nikkar" <anikkar at gmail.com>
> >Subject: [Mongrel] Newbie Question: Can I setup mongrel as a proxy
> >         between a       browser and the web?
> >To: mongrel-users at rubyforge.org
> >Message-ID:
> >          
> <61ddf71e0802061132i59a5b05fy7df5c74e6cc1be02 at mail.gmail.com>
> >Content-Type: text/plain; charset="iso-8859-1"
> >
> >I am trying to create a proxy which I can route my browser requests
> >through
> >in order to gather information on the web page loading.  At this time
> >I am
> >particularly interested in the many GET requests associated with
> >loading a
> >single web page. Note: This does not have anything to do with rails,
> >my goal
> >is to run a ruby app in the background, set firefox to proxy all
> >requests
> >through it, log those requests, then return the fulfilled request  
> back
> >to
> >the browser (i.e. the web page still loads in the browser).
>
> Hi Arash,
>
> You may have interest in coding a solution to this, for your own
> reasons. I totally support those inclinations! :) I''m going to go
a
> little off-topic with an idea though:
>
> Personally, I use "Tamper Data"
> (https://addons.mozilla.org/en-US/firefox/addon/966) within FireFox to
> handle this type of thing. It''s a most excellent resource:
it''s like a
> wire sniffer at the HTTP level only. It supports decoding request data
> and tracking POST, GET and Ajax stuff.
>
> It''s been invaluable for me in some circumstances - I hope the
> reference is helpful.
>
> Steve
>
> p.s. It''s been particularly helpful in trapping precisely certain
live
> GET & Ajax requests which I then convert into testing requests to
> re-create the errors.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
--
Igor Clark // POKE // 10 Redchurch Street // E2 7DD // +44 (0)20 7749  
5355 // www.pokelondon.com

Hi Igor,

sounds like its exactly what I want, provided I can plug into it
programmatically. I''ll take a look today. thanks.

On Feb 7, 2008 1:40 AM, Igor Clark <igor at pokelondon.com> wrote:
> On 7 Feb 2008, at 00:20, Arash Nikkar wrote:
>
> > Hi Steve,
> >
> > I wish I could use something like that (in fact, I often use firebug
> > to see what is actually going on), but I am trying to build
> > something that is browser independent.
> >
> > Ideally, what I am looking for is something which just listens to
> > all the traffic going into and out of a browser (using whatever
> > protocol) and do some analysis on that traffic (even if that means I
> > need to duplicate all the data elsewhere on my disk). but alas, I
> > come from a desktop background and I am not that well versed in web
> > related issues. So I suppose I will just take baby steps (with your
> > guy''s help, if you dont mind.)
>
> http://www.xk72.com/charles/
>
> "Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables
> a developer to view all of the HTTP traffic between their machine and
> the Internet. This includes requests, responses and the HTTP headers
> (which contain the cookies and caching information)."
>
> Browser independent, it intercepts, decodes and displays all browser
> <--> server communications, including decoding Flash AMF data.
It''s
> invaluable for Flash --> server and "ajax" development; sounds
like
> it''ll do some of what you want, too.
>
> cheers
> i
>
> >
> > --arash
> >
> > On Feb 6, 2008 3:57 PM, Steve Midgley <public at misuse.org>
wrote:
> > At 12:39 PM 2/6/2008, mongrel-users-request at rubyforge.org wrote:
> > >Date: Wed, 6 Feb 2008 11:32:16 -0800
> > >From: "Arash Nikkar" <anikkar at gmail.com>
> > >Subject: [Mongrel] Newbie Question: Can I setup mongrel as a proxy
> > >         between a       browser and the web?
> > >To: mongrel-users at rubyforge.org
> > >Message-ID:
> > >
> > <61ddf71e0802061132i59a5b05fy7df5c74e6cc1be02 at mail.gmail.com>
> > >Content-Type: text/plain; charset="iso-8859-1"
> > >
> > >I am trying to create a proxy which I can route my browser
requests
> > >through
> > >in order to gather information on the web page loading.  At this
time
> > >I am
> > >particularly interested in the many GET requests associated with
> > >loading a
> > >single web page. Note: This does not have anything to do with
rails,
> > >my goal
> > >is to run a ruby app in the background, set firefox to proxy all
> > >requests
> > >through it, log those requests, then return the fulfilled request
> > back
> > >to
> > >the browser (i.e. the web page still loads in the browser).
> >
> > Hi Arash,
> >
> > You may have interest in coding a solution to this, for your own
> > reasons. I totally support those inclinations! :) I''m going
to go a
> > little off-topic with an idea though:
> >
> > Personally, I use "Tamper Data"
> > (https://addons.mozilla.org/en-US/firefox/addon/966) within FireFox to
> > handle this type of thing. It''s a most excellent resource:
it''s like a
> > wire sniffer at the HTTP level only. It supports decoding request data
> > and tracking POST, GET and Ajax stuff.
> >
> > It''s been invaluable for me in some circumstances - I hope
the
> > reference is helpful.
> >
> > Steve
> >
> > p.s. It''s been particularly helpful in trapping precisely
certain live
> > GET & Ajax requests which I then convert into testing requests to
> > re-create the errors.
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
>
> --
> Igor Clark // POKE // 10 Redchurch Street // E2 7DD // +44 (0)20 7749
> 5355 // www.pokelondon.com
>
>
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20080207/25b91728/attachment.html

If you are willing to test with a Windows client, there are two very useful
utilities available: http://www.pocketsoap.com/tcptrace/ and
http://www.pocketsoap.com/tcptrace/pt.aspx (Proxy trace is probably what you
are looking for. You run it on your local machine, and just configure your
browser to use localhost as your proxy. It works with any browser, as they
all support forward proxies.)

Cheers,
Brian

On Feb 6, 2008 7:20 PM, Arash Nikkar <anikkar at gmail.com> wrote:
> Hi Steve,
>
> I wish I could use something like that (in fact, I often use firebug to
> see what is actually going on), but I am trying to build something that is
> browser independent.
>
> Ideally, what I am looking for is something which just listens to all the
> traffic going into and out of a browser (using whatever protocol) and do
> some analysis on that traffic (even if that means I need to duplicate all
> the data elsewhere on my disk). but alas, I come from a desktop background
> and I am not that well versed in web related issues. So I suppose I will
> just take baby steps (with your guy''s help, if you dont mind.)
>
> --arash
>
>
> On Feb 6, 2008 3:57 PM, Steve Midgley <public at misuse.org> wrote:
>
> > At 12:39 PM 2/6/2008, mongrel-users-request at rubyforge.org wrote:
> > >Date: Wed, 6 Feb 2008 11:32:16 -0800
> > >From: "Arash Nikkar" <anikkar at gmail.com>
> > >Subject: [Mongrel] Newbie Question: Can I setup mongrel as a proxy
> > >         between a       browser and the web?
> > >To: mongrel-users at rubyforge.org
> > >Message-ID:
> > >         <61ddf71e0802061132i59a5b05fy7df5c74e6cc1be02 at
mail.gmail.com>
> > >Content-Type: text/plain; charset="iso-8859-1"
> > >
> > >I am trying to create a proxy which I can route my browser
requests
> > >through
> > >in order to gather information on the web page loading.  At this
time
> > >I am
> > >particularly interested in the many GET requests associated with
> > >loading a
> > >single web page. Note: This does not have anything to do with
rails,
> > >my goal
> > >is to run a ruby app in the background, set firefox to proxy all
> > >requests
> > >through it, log those requests, then return the fulfilled request
back
> > >to
> > >the browser (i.e. the web page still loads in the browser).
> >
> > Hi Arash,
> >
> > You may have interest in coding a solution to this, for your own
> > reasons. I totally support those inclinations! :) I''m going
to go a
> > little off-topic with an idea though:
> >
> > Personally, I use "Tamper Data"
> > (https://addons.mozilla.org/en-US/firefox/addon/966) within FireFox to
> > handle this type of thing. It''s a most excellent resource:
it''s like a
> > wire sniffer at the HTTP level only. It supports decoding request data
> > and tracking POST, GET and Ajax stuff.
> >
> > It''s been invaluable for me in some circumstances - I hope
the
> > reference is helpful.
> >
> > Steve
> >
> > p.s. It''s been particularly helpful in trapping precisely
certain live
> > GET & Ajax requests which I then convert into testing requests to
> > re-create the errors.
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
- Brian Gupta

http://opensolaris.org/os/project/nycosug/

http://www.genunix.org/wiki/index.php/OpenSolaris_New_User_FAQ
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20080209/a70f868e/attachment.html