thr3ads.net - Mongrel development - [Mongrel-development] TST is right out [Oct 2007]

If this information is useful, please help other people find it:
Share via:

Evan Weaver

2007-Oct-22 08:21 UTC

[Mongrel-development] TST is right out

Hi Zed,

I checked in a pure-Ruby URI classifier to Mongrel trunk. Ola''s Java
port of the TST had some bug, and I don''t think it''s necessary
in the
first place. The Ruby classifier is around 25 lines instead of the
400-odd lines for the C extension and the 200-odd for the Java
extension. It uses a Regexp which is perhaps shady:

     @matcher = Regexp.new(routes.map do |uri|
        Regexp.new(''^'' + Regexp.escape(uri))
      end.join(''|''))

One surprising thing is how fast it is. I wrote a benchmark which
should favor the TST because it''s relatively balanced, and has a high
number of nodes for real-world usage. Rails only installs one route by
default; Camping usually less than five (from what I''ve seen), etc.

--

Pure Ruby, MRI:

19 URIs / 170000 requests
      user     system      total        real
  1.240000   0.010000   1.250000 (  1.251764)
.
Finished in 1.255602 seconds.

C/TST, MRI:

19 URIs / 170000 requests
      user     system      total        real
  0.450000   0.000000   0.450000 (  0.463202)
.
Finished in 0.464033 seconds.

--

Pure Ruby, JRuby:

19 URIs / 170000 requests
      user     system      total        real
  3.327000   0.000000   3.327000 (  3.328000)
.
Finished in 3.403 seconds.

J/TST, JRuby:

19 URIs / 170000 requests
      user     system      total        real
  2.091000   0.000000   2.091000 (  2.091000)
.
Finished in 2.194 seconds.

--

The benchmark is at the bottom of test/test_uriclassifier.rb, if you
want to look at it. Another surprise is that the speedup is pretty
much linear around these node counts or less. It looks like the
savings were in the static typing more than the TST.

Do you know anyone who''s running a shitload of Mongrel routes in a
single server? Is there any reason not to keep this change? I know
that TST dude put some patch in his library just for Mongrel, but
still.

Evan

-- 
Evan Weaver
Cloudburst, LLC

ry dahl

2007-Oct-22 11:50 UTC

head link

[Mongrel-development] TST is right out

> It uses a Regexp which is perhaps shady:
Why not just use Ruby''s Hash function? Is it much slower?
> Is there any reason not to keep this change? I know
> that TST dude put some patch in his library just for Mongrel, but
> still.
Are you proposing dumping the C extension?
It''s faster and it works. And it''s not 400 lines,
it''s probably less
than 100. The trie library, ext/http11/tst_*, should be viewed as
external to mongrel (included in the distribution only for
convenience) .

BTW: There has been some talk of making a more general Ruby binding to
Zed''s (?) Trei library.

Perhaps having both a Ruby and Cext would be good?

ry

Evan Weaver

2007-Oct-22 14:04 UTC

head link

[Mongrel-development] TST is right out

Hash doesn''t do partial matches.

I am proposing dumping the C extension. The Trie library is not
external, because I still have to debug the JRuby port, which is
broken. There was a maintenance cost before even for the pure C
version; we just didn''t have to deal with it ourselves.

It could certainly become its own gem, but I don''t think maintaining
it in Mongrel is worth the 5 microseconds per request that it saves
(less on real hardware).

Evan

On 10/22/07, ry dahl <ry at tinyclouds.org> wrote:> > It uses a Regexp which is perhaps shady:
>
> Why not just use Ruby''s Hash function? Is it much slower?
>
> > Is there any reason not to keep this change? I know
> > that TST dude put some patch in his library just for Mongrel, but
> > still.
>
> Are you proposing dumping the C extension?
> It''s faster and it works. And it''s not 400 lines,
it''s probably less
> than 100. The trie library, ext/http11/tst_*, should be viewed as
> external to mongrel (included in the distribution only for
> convenience) .
>
> BTW: There has been some talk of making a more general Ruby binding to
> Zed''s (?) Trei library.
>
> Perhaps having both a Ruby and Cext would be good?
>
> ry
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 14:28 UTC

head link

[Mongrel-development] TST is right out

It means that pure Ruby is slower in this particular extremely fast
section of the request stack. I''ll benchmark the entire stack and
report back on that.

Yeah, I understand that''s its a big change. On the other hand, I want
to get the JRuby gem out soon. We could do a 1.0.2 release for C only
and then a 1.1 release right away with the parser changes for both;
then people could easily choose which they wanted.

Either there are no bugs in the RCs or no one is really using them
(more likely), because I haven''t heard anything.

I dunno; I''m open to suggestions.

Evan

On 10/22/07, Luis Lavena <luislavena at gmail.com>
wrote:> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > Hi Zed,
> >
> > I checked in a pure-Ruby URI classifier to Mongrel trunk.
Ola''s Java
> > port of the TST had some bug, and I don''t think it''s
necessary in the
> > first place. The Ruby classifier is around 25 lines instead of the
> > 400-odd lines for the C extension and the 200-odd for the Java
> > extension. It uses a Regexp which is perhaps shady:
> >
> >      @matcher = Regexp.new(routes.map do |uri|
> >         Regexp.new(''^'' + Regexp.escape(uri))
> >       end.join(''|''))
> >
> > One surprising thing is how fast it is. I wrote a benchmark which
> > should favor the TST because it''s relatively balanced, and
has a high
> > number of nodes for real-world usage. Rails only installs one route by
> > default; Camping usually less than five (from what I''ve
seen), etc.
> >
> > --
> >
> > Pure Ruby, MRI:
> >
> > 19 URIs / 170000 requests
> >       user     system      total        real
> >   1.240000   0.010000   1.250000 (  1.251764)
> > .
> > Finished in 1.255602 seconds.
> >
> > C/TST, MRI:
> >
> > 19 URIs / 170000 requests
> >       user     system      total        real
> >   0.450000   0.000000   0.450000 (  0.463202)
> > .
> > Finished in 0.464033 seconds.
> >
>
> Ok, that means the Pure Ruby thing is 270% slower?
>
> All the changes introduced in trunk are too radical to be performed
> between Release Candidate and Release.
>
> This is not the right time to drop something that was working, maybe
> doing the work on a branch to later be merged...
>
> JRuby could wait, maybe we could focus on it *after* 1.0.2 final release.
>
> But, as I said, maybe is not the right time, and not the right decision.
>
> --
> Luis Lavena
> Multimedia systems
> -
> Leaders are made, they are not born. They are made by hard effort,
> which is the price which all of us must pay to achieve any goal that
> is worthwhile.
> Vince Lombardi
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 14:36 UTC

head link

[Mongrel-development] TST is right out

BTW, r721 is the revision before the change, if you want to compare.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> It means that pure Ruby is slower in this particular extremely fast
> section of the request stack. I''ll benchmark the entire stack and
> report back on that.
>
> Yeah, I understand that''s its a big change. On the other hand, I
want
> to get the JRuby gem out soon. We could do a 1.0.2 release for C only
> and then a 1.1 release right away with the parser changes for both;
> then people could easily choose which they wanted.
>
> Either there are no bugs in the RCs or no one is really using them
> (more likely), because I haven''t heard anything.
>
> I dunno; I''m open to suggestions.
>
> Evan
>
> On 10/22/07, Luis Lavena <luislavena at gmail.com> wrote:
> > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > Hi Zed,
> > >
> > > I checked in a pure-Ruby URI classifier to Mongrel trunk.
Ola''s Java
> > > port of the TST had some bug, and I don''t think
it''s necessary in the
> > > first place. The Ruby classifier is around 25 lines instead of
the
> > > 400-odd lines for the C extension and the 200-odd for the Java
> > > extension. It uses a Regexp which is perhaps shady:
> > >
> > >      @matcher = Regexp.new(routes.map do |uri|
> > >         Regexp.new(''^'' + Regexp.escape(uri))
> > >       end.join(''|''))
> > >
> > > One surprising thing is how fast it is. I wrote a benchmark which
> > > should favor the TST because it''s relatively balanced,
and has a high
> > > number of nodes for real-world usage. Rails only installs one
route by
> > > default; Camping usually less than five (from what I''ve
seen), etc.
> > >
> > > --
> > >
> > > Pure Ruby, MRI:
> > >
> > > 19 URIs / 170000 requests
> > >       user     system      total        real
> > >   1.240000   0.010000   1.250000 (  1.251764)
> > > .
> > > Finished in 1.255602 seconds.
> > >
> > > C/TST, MRI:
> > >
> > > 19 URIs / 170000 requests
> > >       user     system      total        real
> > >   0.450000   0.000000   0.450000 (  0.463202)
> > > .
> > > Finished in 0.464033 seconds.
> > >
> >
> > Ok, that means the Pure Ruby thing is 270% slower?
> >
> > All the changes introduced in trunk are too radical to be performed
> > between Release Candidate and Release.
> >
> > This is not the right time to drop something that was working, maybe
> > doing the work on a branch to later be merged...
> >
> > JRuby could wait, maybe we could focus on it *after* 1.0.2 final
release.
> >
> > But, as I said, maybe is not the right time, and not the right
decision.
> >
> > --
> > Luis Lavena
> > Multimedia systems
> > -
> > Leaders are made, they are not born. They are made by hard effort,
> > which is the price which all of us must pay to achieve any goal that
> > is worthwhile.
> > Vince Lombardi
> >
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 14:54 UTC

head link

[Mongrel-development] TST is right out

The 1.0.2/1.1 split is starting to seem like a good solution to me. It
would be nice to be able to say "Mongrel gets faster, not slower", so
I''ll see if I can squeeze out some unrelated performance increases to
make up for the slower URIClassifier--although I realize that you guys
aren''t convinced it''s necessary yet.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> BTW, r721 is the revision before the change, if you want to compare.
>
> Evan
>
> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > It means that pure Ruby is slower in this particular extremely fast
> > section of the request stack. I''ll benchmark the entire stack
and
> > report back on that.
> >
> > Yeah, I understand that''s its a big change. On the other
hand, I want
> > to get the JRuby gem out soon. We could do a 1.0.2 release for C only
> > and then a 1.1 release right away with the parser changes for both;
> > then people could easily choose which they wanted.
> >
> > Either there are no bugs in the RCs or no one is really using them
> > (more likely), because I haven''t heard anything.
> >
> > I dunno; I''m open to suggestions.
> >
> > Evan
> >
> > On 10/22/07, Luis Lavena <luislavena at gmail.com> wrote:
> > > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > > Hi Zed,
> > > >
> > > > I checked in a pure-Ruby URI classifier to Mongrel trunk.
Ola''s Java
> > > > port of the TST had some bug, and I don''t think
it''s necessary in the
> > > > first place. The Ruby classifier is around 25 lines instead
of the
> > > > 400-odd lines for the C extension and the 200-odd for the
Java
> > > > extension. It uses a Regexp which is perhaps shady:
> > > >
> > > >      @matcher = Regexp.new(routes.map do |uri|
> > > >         Regexp.new(''^'' +
Regexp.escape(uri))
> > > >       end.join(''|''))
> > > >
> > > > One surprising thing is how fast it is. I wrote a benchmark
which
> > > > should favor the TST because it''s relatively
balanced, and has a high
> > > > number of nodes for real-world usage. Rails only installs
one route by
> > > > default; Camping usually less than five (from what
I''ve seen), etc.
> > > >
> > > > --
> > > >
> > > > Pure Ruby, MRI:
> > > >
> > > > 19 URIs / 170000 requests
> > > >       user     system      total        real
> > > >   1.240000   0.010000   1.250000 (  1.251764)
> > > > .
> > > > Finished in 1.255602 seconds.
> > > >
> > > > C/TST, MRI:
> > > >
> > > > 19 URIs / 170000 requests
> > > >       user     system      total        real
> > > >   0.450000   0.000000   0.450000 (  0.463202)
> > > > .
> > > > Finished in 0.464033 seconds.
> > > >
> > >
> > > Ok, that means the Pure Ruby thing is 270% slower?
> > >
> > > All the changes introduced in trunk are too radical to be
performed
> > > between Release Candidate and Release.
> > >
> > > This is not the right time to drop something that was working,
maybe
> > > doing the work on a branch to later be merged...
> > >
> > > JRuby could wait, maybe we could focus on it *after* 1.0.2 final
release.
> > >
> > > But, as I said, maybe is not the right time, and not the right
decision.
> > >
> > > --
> > > Luis Lavena
> > > Multimedia systems
> > > -
> > > Leaders are made, they are not born. They are made by hard
effort,
> > > which is the price which all of us must pay to achieve any goal
that
> > > is worthwhile.
> > > Vince Lombardi
> > >
> >
> >
> > --
> > Evan Weaver
> > Cloudburst, LLC
> >
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 15:59 UTC

head link

[Mongrel-development] TST is right out

We could, but that''s more complexity yet.

If I optimize the classifier for the pathological case of only one
route (Rails), it might be faster than the original C extension for
most people. Would that be acceptable?

Evan

On 10/22/07, Wayne E. Seguin <wayneeseguin at gmail.com>
wrote:> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > The 1.0.2/1.1 split is starting to seem like a good solution to me. It
> > would be nice to be able to say "Mongrel gets faster, not
slower", so
> > I''ll see if I can squeeze out some unrelated performance
increases to
> > make up for the slower URIClassifier--although I realize that you guys
> > aren''t convinced it''s necessary yet.
> >
> > Evan
>
>
> Not quite yet, no :) Is this really only necessary to support JRuby?  If
so,
> perhaps we can build in to the installation the ability to compile and use
> the extension if able?
>
>   ~Wayne

-- 
Evan Weaver
Cloudburst, LLC

ry dahl

2007-Oct-22 16:03 UTC

head link

[Mongrel-development] TST is right out

> I am proposing dumping the C extension. The Trie library is not
> external, because I still have to debug the JRuby port, which is
> broken. There was a maintenance cost before even for the pure C
> version; we just didn''t have to deal with it ourselves
I don''t know much about java or jruby.

Do you have to port the trie library to Java? If so, how will you
handle the http parser?

ry

Evan Weaver

2007-Oct-22 16:07 UTC

head link

[Mongrel-development] TST is right out

The parser is ported. Ola did most of that work although I integrated
the two Ragel branches and updated some accessors. Ragel can generate
Java now.

Ola ported the Trie a while back, but it''s buggy.

Evan

On 10/22/07, ry dahl <ry at tinyclouds.org> wrote:> > I am proposing dumping the C extension. The Trie library is not
> > external, because I still have to debug the JRuby port, which is
> > broken. There was a maintenance cost before even for the pure C
> > version; we just didn''t have to deal with it ourselves
>
> I don''t know much about java or jruby.
>
> Do you have to port the trie library to Java? If so, how will you
> handle the http parser?
>
> ry
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 16:25 UTC

head link

[Mongrel-development] TST is right out

Optimization is effective, but it adds 7 lines of code.

Ruby, MRI:

1 URIs / 170000 requests
      user     system      total        real
  0.300000   0.010000   0.310000 (  0.308417)
Finished in 0.309255 seconds.

--

C/TST, MRI:

1 URIs / 170000 requests
      user     system      total        real
  0.370000   0.000000   0.370000 (  0.375385)
Finished in 0.376807 seconds.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> We could, but that''s more complexity yet.
>
> If I optimize the classifier for the pathological case of only one
> route (Rails), it might be faster than the original C extension for
> most people. Would that be acceptable?
>
> Evan
>
> On 10/22/07, Wayne E. Seguin <wayneeseguin at gmail.com> wrote:
> > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > The 1.0.2/1.1 split is starting to seem like a good solution to
me. It
> > > would be nice to be able to say "Mongrel gets faster, not
slower", so
> > > I''ll see if I can squeeze out some unrelated performance
increases to
> > > make up for the slower URIClassifier--although I realize that you
guys
> > > aren''t convinced it''s necessary yet.
> > >
> > > Evan
> >
> >
> > Not quite yet, no :) Is this really only necessary to support JRuby? 
If so,
> > perhaps we can build in to the installation the ability to compile and
use
> > the extension if able?
> >
> >   ~Wayne
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 17:13 UTC

head link

[Mongrel-development] TST is right out

Ry helped come up with a good idea. Because the new Ruby classifier
builds a Regexp out of the URI strings, it''s a small step to just
supporting Regexp''s directly, which means a framework could forgo its
own router in most situations.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> Optimization is effective, but it adds 7 lines of code.
>
> Ruby, MRI:
>
> 1 URIs / 170000 requests
>       user     system      total        real
>   0.300000   0.010000   0.310000 (  0.308417)
> Finished in 0.309255 seconds.
>
> --
>
> C/TST, MRI:
>
> 1 URIs / 170000 requests
>       user     system      total        real
>   0.370000   0.000000   0.370000 (  0.375385)
> Finished in 0.376807 seconds.
>
> Evan
>
> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > We could, but that''s more complexity yet.
> >
> > If I optimize the classifier for the pathological case of only one
> > route (Rails), it might be faster than the original C extension for
> > most people. Would that be acceptable?
> >
> > Evan
> >
> > On 10/22/07, Wayne E. Seguin <wayneeseguin at gmail.com> wrote:
> > > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > > The 1.0.2/1.1 split is starting to seem like a good solution
to me. It
> > > > would be nice to be able to say "Mongrel gets faster,
not slower", so
> > > > I''ll see if I can squeeze out some unrelated
performance increases to
> > > > make up for the slower URIClassifier--although I realize
that you guys
> > > > aren''t convinced it''s necessary yet.
> > > >
> > > > Evan
> > >
> > >
> > > Not quite yet, no :) Is this really only necessary to support
JRuby?  If so,
> > > perhaps we can build in to the installation the ability to
compile and use
> > > the extension if able?
> > >
> > >   ~Wayne
> >
> >
> > --
> > Evan Weaver
> > Cloudburst, LLC
> >
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-22 17:24 UTC

head link

[Mongrel-development] TST is right out

More tests show that the benefits of the TST start to kick in around
14 registered URIs. So that would be an argument in favor of the C
version if people are often registering that many.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> Ry helped come up with a good idea. Because the new Ruby classifier
> builds a Regexp out of the URI strings, it''s a small step to just
> supporting Regexp''s directly, which means a framework could forgo
its
> own router in most situations.
>
> Evan
>
> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > Optimization is effective, but it adds 7 lines of code.
> >
> > Ruby, MRI:
> >
> > 1 URIs / 170000 requests
> >       user     system      total        real
> >   0.300000   0.010000   0.310000 (  0.308417)
> > Finished in 0.309255 seconds.
> >
> > --
> >
> > C/TST, MRI:
> >
> > 1 URIs / 170000 requests
> >       user     system      total        real
> >   0.370000   0.000000   0.370000 (  0.375385)
> > Finished in 0.376807 seconds.
> >
> > Evan
> >
> > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > We could, but that''s more complexity yet.
> > >
> > > If I optimize the classifier for the pathological case of only
one
> > > route (Rails), it might be faster than the original C extension
for
> > > most people. Would that be acceptable?
> > >
> > > Evan
> > >
> > > On 10/22/07, Wayne E. Seguin <wayneeseguin at gmail.com>
wrote:
> > > > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > > > The 1.0.2/1.1 split is starting to seem like a good
solution to me. It
> > > > > would be nice to be able to say "Mongrel gets
faster, not slower", so
> > > > > I''ll see if I can squeeze out some unrelated
performance increases to
> > > > > make up for the slower URIClassifier--although I
realize that you guys
> > > > > aren''t convinced it''s necessary yet.
> > > > >
> > > > > Evan
> > > >
> > > >
> > > > Not quite yet, no :) Is this really only necessary to
support JRuby?  If so,
> > > > perhaps we can build in to the installation the ability to
compile and use
> > > > the extension if able?
> > > >
> > > >   ~Wayne
> > >
> > >
> > > --
> > > Evan Weaver
> > > Cloudburst, LLC
> > >
> >
> >
> > --
> > Evan Weaver
> > Cloudburst, LLC
> >
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Jeremy Kemper

2007-Oct-22 17:55 UTC

head link

[Mongrel-development] TST is right out

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> More tests show that the benefits of the TST start to kick in around
> 14 registered URIs. So that would be an argument in favor of the C
> version if people are often registering that many.
Expect tens to hundreds of routes per Rails app as folks move over to resources.

jeremy

ry dahl

2007-Oct-22 18:09 UTC

head link

[Mongrel-development] TST is right out

> Expect tens to hundreds of routes per Rails app as folks move over to
resources.
Do rails resources register themselves as handlers with mongrel? or
does Rails use Mongrel::URIClassifier?

ry

Filipe

2007-Oct-22 18:33 UTC

head link

[Mongrel-development] TST is right out

On Mon, 22 Oct 2007, Evan Weaver wrote:
> Optimization is effective, but it adds 7 lines of code.
>
Hey, if you can make the ruby version faster then the c one, I see no
problems in changing this for a next version (1.1 or 1.0.3 or whatever
we can come with).
The C version is some kind of complicated, and when I needed to close a
bug in this I asked the upstream maintainer to do this. I also asked
because I wanted to keep mongrel and upstream versions in sync. Upstream
was kind and closed the bug, but if we had this under our control and
written in ruby it would be a lot easier to maintain.

filipe {
  @             icewall.org
  GPG        1024D/A6BA423E
  Jabber  lautert at jabber.ru
}

Evan Weaver

2007-Oct-22 18:40 UTC

head link

[Mongrel-development] TST is right out

Can you elaborate?

On 10/22/07, Jeremy Kemper <jeremy at bitsweat.net>
wrote:> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > More tests show that the benefits of the TST start to kick in around
> > 14 registered URIs. So that would be an argument in favor of the C
> > version if people are often registering that many.
>
> Expect tens to hundreds of routes per Rails app as folks move over to
resources.
>
> jeremy
>

-- 
Evan Weaver
Cloudburst, LLC

Luis Lavena

2007-Oct-22 18:42 UTC

head link

[Mongrel-development] Fwd: TST is right out

That post was sent just to evan, we need to correct the Reply-To
fields for the list...

---------- Forwarded message ----------
From: Luis Lavena <luislavena at gmail.com>
Date: Oct 22, 2007 2:30 PM
Subject: Re: [Mongrel-development] TST is right out
To: Evan Weaver <evan at cloudbur.st>


On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> More tests show that the benefits of the TST start to kick in around
> 14 registered URIs. So that would be an argument in favor of the C
> version if people are often registering that many.
>
Forget just a second about Rails... the other frameworks... (a) are
registering URIs in Mongrel, or (b) just their own dispatcher?

If the answer was (a), we should keep the C part, or drop it if it''s
(b).

--
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

Luis Lavena

2007-Oct-22 18:45 UTC

head link

[Mongrel-development] TST is right out

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> Can you elaborate?
>
> On 10/22/07, Jeremy Kemper <jeremy at bitsweat.net> wrote:
> > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > More tests show that the benefits of the TST start to kick in
around
> > > 14 registered URIs. So that would be an argument in favor of the
C
> > > version if people are often registering that many.
> >
> > Expect tens to hundreds of routes per Rails app as folks move over to
resources.
> >
> > jeremy
> >
We should differentiate the dispatcher rails use and the URI mongrel
uses... URIs registered with Mongrel are the handlers, and unless you
hook upload_progress, merb in parasite mode and "/" for rails handler,
there is no other hook.

Rails has it''s own URI classifier and dispatcher, the same goes for
Merb... So...

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

Evan Weaver

2007-Oct-22 18:50 UTC

head link

[Mongrel-development] TST is right out

Right... for current Rails. I thought Jeremy was saying they were
splitting up the internal dispatcher. But that wouldn''t make sense for
Webrick, Glassfish et al, I suppose.

Evan


On 10/22/07, Luis Lavena <luislavena at gmail.com>
wrote:> On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > Can you elaborate?
> >
> > On 10/22/07, Jeremy Kemper <jeremy at bitsweat.net> wrote:
> > > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > > More tests show that the benefits of the TST start to kick
in around
> > > > 14 registered URIs. So that would be an argument in favor of
the C
> > > > version if people are often registering that many.
> > >
> > > Expect tens to hundreds of routes per Rails app as folks move
over to resources.
> > >
> > > jeremy
> > >
>
> We should differentiate the dispatcher rails use and the URI mongrel
> uses... URIs registered with Mongrel are the handlers, and unless you
> hook upload_progress, merb in parasite mode and "/" for rails
handler,
> there is no other hook.
>
> Rails has it''s own URI classifier and dispatcher, the same goes
for
> Merb... So...
>
> --
> Luis Lavena
> Multimedia systems
> -
> Leaders are made, they are not born. They are made by hard effort,
> which is the price which all of us must pay to achieve any goal that
> is worthwhile.
> Vince Lombardi
> _______________________________________________
> Mongrel-development mailing list
> Mongrel-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-development
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-23 02:07 UTC

head link

[Mongrel-development] TST is right out

Current plan then as discussed on IRC and elsewhere:

Rollback trunk, make a 1.0.2 release, and then once JRuby fixes their
socket bugs, make a 1.1 release with the new classifier and a separate
''mongrel_performance'' gem that includes the C extension. Then
people
can install the C extension if they need lots of routes, but it won''t
be required for JRuby and the features in the mongrel_performance gem
won''t have to march in lockstep with the reference implementation.

Ry''s query parser will also go in mongrel_performance once it is
ready, again, with a reference pure Ruby implementation for Windows,
MRI and JRuby in core.

This lets us do things like test out features in pure Ruby first
before committing to implementing them in C and Java, as well as
include more experimental or platform-specific hacks in the
mongrel_performance gem.

If some enterprising person wants to write a mongrel_performance-jruby
gem, they can do it, but we don''t necessarily have to maintain it.

Thoughts? I make it sound more complicated than it really is.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> Right... for current Rails. I thought Jeremy was saying they were
> splitting up the internal dispatcher. But that wouldn''t make sense
for
> Webrick, Glassfish et al, I suppose.
>
> Evan
>
>
> On 10/22/07, Luis Lavena <luislavena at gmail.com> wrote:
> > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > Can you elaborate?
> > >
> > > On 10/22/07, Jeremy Kemper <jeremy at bitsweat.net> wrote:
> > > > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > > > More tests show that the benefits of the TST start to
kick in around
> > > > > 14 registered URIs. So that would be an argument in
favor of the C
> > > > > version if people are often registering that many.
> > > >
> > > > Expect tens to hundreds of routes per Rails app as folks
move over to resources.
> > > >
> > > > jeremy
> > > >
> >
> > We should differentiate the dispatcher rails use and the URI mongrel
> > uses... URIs registered with Mongrel are the handlers, and unless you
> > hook upload_progress, merb in parasite mode and "/" for
rails handler,
> > there is no other hook.
> >
> > Rails has it''s own URI classifier and dispatcher, the same
goes for
> > Merb... So...
> >
> > --
> > Luis Lavena
> > Multimedia systems
> > -
> > Leaders are made, they are not born. They are made by hard effort,
> > which is the price which all of us must pay to achieve any goal that
> > is worthwhile.
> > Vince Lombardi
> > _______________________________________________
> > Mongrel-development mailing list
> > Mongrel-development at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-development
> >
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Wayne E. Seguin

2007-Oct-23 03:14 UTC

head link

[Mongrel-development] TST is right out

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:>
> Current plan then as discussed on IRC and elsewhere:
>
> Rollback trunk, make a 1.0.2 release, and then once JRuby fixes their
> socket bugs, make a 1.1 release with the new classifier and a separate
> ''mongrel_performance'' gem that includes the C extension.
Then people
> can install the C extension if they need lots of routes, but it
won''t
> be required for JRuby and the features in the mongrel_performance gem
> won''t have to march in lockstep with the reference implementation.
>
> Ry''s query parser will also go in mongrel_performance once it is
> ready, again, with a reference pure Ruby implementation for Windows,
> MRI and JRuby in core.
>
> This lets us do things like test out features in pure Ruby first
> before committing to implementing them in C and Java, as well as
> include more experimental or platform-specific hacks in the
> mongrel_performance gem.
>
> If some enterprising person wants to write a mongrel_performance-jruby
> gem, they can do it, but we don''t necessarily have to maintain it.
>
> Thoughts? I make it sound more complicated than it really is.
>
> Evan
>
So we really want a separate gem rather than having mongrel attempt to use
the high-performance parts first and rescue to Ruby? Just asking one last
time ;)

  ~Wayne
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-development/attachments/20071022/86bcff10/attachment.html

Evan Weaver

2007-Oct-23 03:25 UTC

head link

[Mongrel-development] TST is right out

I would prefer it. Otherwise the C extension always has to exactly
match the feature set of the Ruby reference implementation, whereas
with separate gems people will understand if the C extension lags
behind or takes shortcuts for performance.

I''d like some other people''s opinions though.

Evan

On 10/22/07, Evan Weaver <evan at cloudbur.st>
wrote:> I would prefer it. Otherwise the C extension always has to exactly
> match the feature set of the Ruby reference implementation, whereas
> with separate gems people will understand if the C extension lags
> behind or takes shortcuts for performance.
>
> I''d like some other people''s opinions though.
>
> Evan
>
> On 10/22/07, Wayne E. Seguin <wayneeseguin at gmail.com> wrote:
> >
> >
> > On 10/22/07, Evan Weaver <evan at cloudbur.st> wrote:
> > > Current plan then as discussed on IRC and elsewhere:
> > >
> > > Rollback trunk, make a 1.0.2 release, and then once JRuby fixes
their
> > > socket bugs, make a 1.1 release with the new classifier and a
separate
> > > ''mongrel_performance'' gem that includes the C
extension. Then people
> > > can install the C extension if they need lots of routes, but it
won''t
> > > be required for JRuby and the features in the mongrel_performance
gem
> > > won''t have to march in lockstep with the reference
implementation.
> > >
> > > Ry''s query parser will also go in mongrel_performance
once it is
> > > ready, again, with a reference pure Ruby implementation for
Windows,
> > > MRI and JRuby in core.
> > >
> > > This lets us do things like test out features in pure Ruby first
> > > before committing to implementing them in C and Java, as well as
> > > include more experimental or platform-specific hacks in the
> > > mongrel_performance gem.
> > >
> > > If some enterprising person wants to write a
mongrel_performance-jruby
> > > gem, they can do it, but we don''t necessarily have to
maintain it.
> > >
> > > Thoughts? I make it sound more complicated than it really is.
> > >
> > > Evan
> > >
> >
> > So we really want a separate gem rather than having mongrel attempt to
use
> > the high-performance parts first and rescue to Ruby? Just asking one
last
> > time ;)
> >
> >   ~Wayne
> > _______________________________________________
> > Mongrel-development mailing list
> > Mongrel-development at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-development
> >
> >
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>

-- 
Evan Weaver
Cloudburst, LLC

Filipe

2007-Oct-23 10:51 UTC

head link

[Mongrel-development] TST is right out

On Mon, 22 Oct 2007, Evan Weaver wrote:
> I would prefer it. Otherwise the C extension always has to exactly
> match the feature set of the Ruby reference implementation, whereas
> with separate gems people will understand if the C extension lags
> behind or takes shortcuts for performance.
>
> I''d like some other people''s opinions though.
>
other person here ;)

For me it''s cool to have it in a different gem - also if we found a bug
in it we can only release this specific gem and do not need to release the
entire mongrel.


filipe {
  @             icewall.org
  GPG        1024D/A6BA423E
  Jabber  lautert at jabber.ru
}

Evan Weaver

2007-Oct-23 12:04 UTC

head link

[Mongrel-development] TST is right out

> I don''t know about you, but my experience with people tells me
that
> when they see a gem called mongrel_performance they''d go
"ooooh, my
> awsum blawg can has needz hi-perfurmance",
You''re right. What if we give it a boring or scary name instead?
''mongrel_experimental'' or something.

Evan

On 10/23/07, Filipe <filipe at icewall.org> wrote:> On Mon, 22 Oct 2007, Evan Weaver wrote:
>
> > I would prefer it. Otherwise the C extension always has to exactly
> > match the feature set of the Ruby reference implementation, whereas
> > with separate gems people will understand if the C extension lags
> > behind or takes shortcuts for performance.
> >
> > I''d like some other people''s opinions though.
> >
>
> other person here ;)
>
> For me it''s cool to have it in a different gem - also if we found
a bug
> in it we can only release this specific gem and do not need to release the
> entire mongrel.
>
>
> filipe {
>   @             icewall.org
>   GPG        1024D/A6BA423E
>   Jabber  lautert at jabber.ru
> }
>

-- 
Evan Weaver
Cloudburst, LLC

Wayne E. Seguin

2007-Oct-23 15:06 UTC

head link

[Mongrel-development] TST is right out

On 10/23/07, Evan Weaver <evan at cloudbur.st>
wrote:>
> > I don''t know about you, but my experience with people tells
me that
> > when they see a gem called mongrel_performance they''d go
"ooooh, my
> > awsum blawg can has needz hi-perfurmance",
>
> You''re right. What if we give it a boring or scary name instead?
> ''mongrel_experimental'' or something.
>
I was just making sure that''s the route we wanted to take.

Now that we''re taking that route, I''m thinking that we should
create two new
gems:

mongrel_treats - Known performance boosting C extensions

mongrel_chocolate - The dog likes it but it may not be good for him at all
(aka "experimental")

What do you all think?

  ~Wayne
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-development/attachments/20071023/9c8bb63d/attachment.html

ry dahl

2007-Oct-23 15:20 UTC

head link

[Mongrel-development] TST is right out

> mongrel_treats - Known performance boosting C extensions
>
> mongrel_chocolate - The dog likes it but it may not be good for him at all
> (aka "experimental")
I love the nomenclature - what do you have in mind for these? We don''t
have soo many performance things.

ALSO: Evan, the escape/unescape functions written in C are very simple
and fast, and probably should be in the http11 core. It shouldn''t be a
problem to port them to java, I think.

ry

Luis Lavena

2007-Oct-23 17:05 UTC

head link

[Mongrel-development] TST is right out

And again, just sent this to Evan!

Zed: you''re admin, please correct this! (I always forget to change the
To field!)

On 10/23/07, Evan Weaver <evan at cloudbur.st>
wrote:> > I don''t know about you, but my experience with people tells
me that
> > when they see a gem called mongrel_performance they''d go
"ooooh, my
> > awsum blawg can has needz hi-perfurmance",
>
> You''re right. What if we give it a boring or scary name instead?
> ''mongrel_experimental'' or something.
>
That sounds better than "cgi_multipart_eof_fix" :-P

Try type that in the terminal fast enough and your finger will feel the pain ;-)

So the "mongrel_experimental_extension" should also depend on
gem_plugin, or we could leave the whole mambo about it off the table?

# if mongrel_experimental_extension is present, use it
begin
  require ''mongrel_experimental_extension/something''
rescue LoadError
  require ''pure_ruby_classiffier''
end

Or something like that... (don''t have a ruby handy ;-)

--
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

Evan Weaver

2007-Oct-23 17:51 UTC

head link

[Mongrel-development] TST is right out

Perhaps we should call it "bone".

Evan

On 10/23/07, ry dahl <ry at tinyclouds.org> wrote:> > mongrel_treats - Known performance boosting C extensions
> >
> > mongrel_chocolate - The dog likes it but it may not be good for him at
all
> > (aka "experimental")
>
> I love the nomenclature - what do you have in mind for these? We
don''t
> have soo many performance things.
>
> ALSO: Evan, the escape/unescape functions written in C are very simple
> and fast, and probably should be in the http11 core. It shouldn''t
be a
> problem to port them to java, I think.
>
> ry
> _______________________________________________
> Mongrel-development mailing list
> Mongrel-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-development
>

-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver

2007-Oct-23 23:34 UTC

head link

[Mongrel-development] TST is right out

I''ll try both at some point (classifier into CRagel, parser into Ruby
Ragel). The Ruby URIClassifier is down in the realm of 6 dispatches
though, so that is definitely a theoretical exercise rather than a
practical one, but that''s ok.

Thanks for the admin access.

Wayne, I think we need only one extra gem. Don''t want to get carried
away.

Anyway... I think this discussion is basically resolved. I''ll try to
grind out 1.0.2 and 1.1 releases by RubyConf.

Luis, what do I have to do to package win32 builds? Then I can stop
breaking your packagin scripts. I have Parallels and all that; I can
get Visual Studio 6 if necessary.

Have fun in Rome, Zed.

Evan

On 10/23/07, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:> On Mon, 22 Oct 2007 04:21:55 -0400
> "Evan Weaver" <evan at cloudbur.st> wrote:
>
> > Hi Zed,
> >
> > I checked in a pure-Ruby URI classifier to Mongrel trunk.
Ola''s Java
> > port of the TST had some bug, and I don''t think it''s
necessary in the
> > first place. The Ruby classifier is around 25 lines instead of the
> > 400-odd lines for the C extension and the 200-odd for the Java
> > extension. It uses a Regexp which is perhaps shady:
>
> Sorry, haven''t had internet for a loooong time.  Rome blows for
internet access.
>
> If this works and you can get rid of more .c code then go for it.  It was
originally because the Rails routing sucked so bad that I wanted to make sure
mongrel could route reliably.  In the end it was overkill most likely.
>
> Only thing is, Ragel produces a regex, is verifiable, and has a Ruby
generator.  You could probably use that to parse the URL for recognition using
the existing grammar for URIs in the C parser.
>
> In addition, why not play with Ragel''s Ruby output and see if you
can kill C entirely?  That''d make a bunch of folks happy, and you can
include the .c extension version for the paranoid.
>
> --
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
>

-- 
Evan Weaver
Cloudburst, LLC

Luis Lavena

2007-Oct-23 23:42 UTC

head link

[Mongrel-development] TST is right out

On 10/23/07, Evan Weaver <evan at cloudbur.st>
wrote:> I''ll try both at some point (classifier into CRagel, parser into
Ruby
> Ragel). The Ruby URIClassifier is down in the realm of 6 dispatches
> though, so that is definitely a theoretical exercise rather than a
> practical one, but that''s ok.
>
> Thanks for the admin access.
>
> Wayne, I think we need only one extra gem. Don''t want to get
carried away.
>
> Anyway... I think this discussion is basically resolved. I''ll try
to
> grind out 1.0.2 and 1.1 releases by RubyConf.
>
> Luis, what do I have to do to package win32 builds? Then I can stop
> breaking your packagin scripts. I have Parallels and all that; I can
> get Visual Studio 6 if necessary.
>
Hehehe, I''ll mail you off the list later today.
> Have fun in Rome, Zed.
>
I didn''t know he was in Italy! :-D

Regards and have fun everybody (specially Zed).

PS: Please, someone with ML password could fix the reply-to thing?) Thanks! :-D

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

ry dahl

2007-Oct-23 23:48 UTC

head link

[Mongrel-development] TST is right out

On 10/23/07, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:> In addition, why not play with Ragel''s Ruby output and see if you
can kill C entirely?
> That''d make a bunch of folks happy, and you can include the .c
extension version for the paranoid.
That''s a great idea. I''ll try it tomorrow.
I''m interested what sort of speed increase ragel has when compiling
into Ruby.

ry

Maybe Matching Threads

Search for more maybe matching threads

Mongrel development - Oct 2007 - TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] Fwd: TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

[Mongrel-development] TST is right out

Maybe Matching Threads