Mongrel users - Jan 2007 - Anyway to "dynamically" start/stop mongrel based on web traffic?

Hey all -

I''ve got a question that I haven''t seen addressed anywhere and
was
wondering if anyone has put any thought into it or not...

Here''s my setup... I have several *small* sites running apache/mongrel.
Each has a single mongrel instance.  Most don''t get any traffic (no one
reads my blog :).

And I was thinking, I could host a couple of more sites/projects without 
worrying about ram if I could find a way have all my mongrels be off until 
a request came in... then apache (or some other server) could block until 
it could get mongrel started, then keep processing the request.

Then I could either kill off the mongrel later if no traffic was coming 
in.

Or perhaps mongrel itself could stay running, but unload (and free the 
ram) of rails until a request came in, then load it back up for awhile 
until traffic stopped and unload it...

So.. I guess I''m wondering if anyone''s done anything like this
or if it is
possible and if so what the best way to go about it might be?  I''d be 
willing to help (if I can, but not sure how likely that is :)

It would be nice to have my little apps that get little traffic not suck 
up so much ram while "idle"..

-philip

Philip Hallstrom wrote:
> Then I could either kill off the mongrel later if no traffic was coming 
> in.
> 
> Or perhaps mongrel itself could stay running, but unload (and free the 
> ram) of rails until a request came in, then load it back up for awhile 
> until traffic stopped and unload it...
What you''re talking about is inetd functionality for a rails process.
If
you''re on a sane unix, you could probably make that happen. Take a look
at /etc/inetd.conf, or your version thereof.

Whether it is a good idea or not, well, that''s another matter.
I''m
inclined to say not. :-)
-- 
GPG/PGP key ID: 0x3A410DBD | http://pgp.mit.edu
7B3F 4505 7D9A 7FDE 83C9  52C2 4909 59B9 3A41 0DBD

> Philip Hallstrom wrote:
>> Then I could either kill off the mongrel later if no traffic was coming
>> in.
>>
>> Or perhaps mongrel itself could stay running, but unload (and free the
>> ram) of rails until a request came in, then load it back up for awhile
>> until traffic stopped and unload it...
>
> What you''re talking about is inetd functionality for a rails
process. If
> you''re on a sane unix, you could probably make that happen. Take a
look
> at /etc/inetd.conf, or your version thereof.
Kind of... my understanding of inetd though is that it listens on the 
port, accepts the connection, spawns the process and the process talks via 
stdin/out till done, then exits.

What I want is a front-end webserver that realizes mongrel isn''t
running,
holds the connection while it starts it, then continues as normal.  I 
can kill the mongrel off later somehow...

Basically I want dynamically spawned FCGI processes, but I''ve used FCGI
before which is why I''m now using mongrel :)

An ugly brute force would be to simply watch apache''s logs for a 503
error
and if seen start up mongrel, but that means one user is gonna get a 
broken page...

I don''t know enough about mongrel/rails/ruby to know if mongrel can
unload
the majority of that 30mb it''s using, but still keep on listening, but
if
someone (Zed? :) wanted to give me some pointers I''d love to give it a
try
:)

-philip

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hello!

My name is Filipe and I''m from the Debian ruby-extras team.

Some time ago, Jens Kraemer created packages for mongrel and
gem plugin. Now, me and gwolf joined forces with him and we are working
to create oficial debian packages for mongrel.

So, we have some favors to ask from you (the mongrel developers), as
those favors are not big things:

* Could you please upload tar.gz files together with your gems files?
* In this tar, could the files be with the current date, not in year 
1969? :D tar complains a lot about this...
* When release candidates are released, can you append something like
"~rc1" to the tar file? This help debian auto tools to check new
releases.
* Finally,  I got this great idea for Mongrel.  All you have to do is
completely change the internal processing, add 200 more methods to the
HTTP parser, and... no. Just joking. Just the other 3 would be great!

The Debian ruby-extras team has a page[1] dedicated to upstream
developers, where those and other things that help *nix distributions
are described.

Regards,

filipe lautert

filipe { AT } icewall.org
Linux User        #279798
Jabber  lautert at jabber.ru

[1] http://pkg-ruby-extras.alioth.debian.org/upstream-devs.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFFpgscmKFbPqa6Qj4RAs5RAJ0XKjRilbzRwvo3hsRjkhtF34WrlQCfRGph
cZrSi+wlMbkCVFJvwhOJWi4=Ah68
-----END PGP SIGNATURE-----

On Thu Jan 11, 2007 at 08:01:59AM -0200, Filipe wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> Hello!
> 
> My name is Filipe and I''m from the Debian ruby-extras team.
is it possible Zed could trademark Mongrel so it could be renamed IceBeast?

LOL or IceDog.
Sounds like a good idea Filipe. I use Debian quite a bit but always
install ruby from source and use gems for all my extras like mongrel.
But for those that are afraid of compiling or want an easy way to
uninstall, apt-get is great (why would you uninstall mongrel though?)
.

On 1/11/07, carmen <_ at whats-your.name> wrote:
> On Thu Jan 11, 2007 at 08:01:59AM -0200, Filipe wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >
> > Hello!
> >
> > My name is Filipe and I''m from the Debian ruby-extras team.
>
> is it possible Zed could trademark Mongrel so it could be renamed IceBeast?
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

On Thu, Jan 11, 2007 at 04:34:36PM -0700, Kyle Kochis
wrote:
> LOL or IceDog.
> Sounds like a good idea Filipe. I use Debian quite a bit but always
> install ruby from source and use gems for all my extras like mongrel.
> But for those that are afraid of compiling or want an easy way to
> uninstall, apt-get is great (why would you uninstall mongrel though?)
not only for uninstalling, but also for people who have to maintain more
than one or two servers. 

Having the complete rails stack available via apt just makes life much
easier for them. The whole system is kept up-to-date by regular
apt-updates anyway, so not having to resort to another packaging system
for Rails on production systems is a good thing. 

Besides that, it''s just a waste of resources to compile anything on
*each* production machine in case of an upgrade. Your average web server
shouldn''t even need to have a compiler installed, imho. I''ve
never heard
of someone compiling tomcat or java on a live machine...

Jens


-- 
webit! Gesellschaft f?r neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
Schnorrstra?e 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

Jens,
Good point, at least on the first two paragraphs but I must
respectfully (yet enthusiastically) disagree on the last point:
> Besides that, it''s just a waste of resources to compile anything
on
> *each* production machine in case of an upgrade. Your average web server
> shouldn''t even need to have a compiler installed, imho.
I''ve never heard
> of someone compiling tomcat or java on a live machine...
As a user of Debian (it is my preferred Linux Distro although I am
starting to shift towards the BSD''s), it is a pain in the butt to use
Ruby1.8.2 or 1.6 and get everything like gems and rails working
properly. Take a look at
http://mongrel.rubyforge.org/docs/debian-sarge.html and
http://lists.rubyonrails.org/pipermail/rails/2006-May/037763.html. The
issue mentioned in both of these articles are VERY easy to avoid:
curl, tar, cd ./configure make and finally make install. Not only that
but since most VPS/Dedicated servers that host only one or maybe up to
5 (busy) sites that I set up at least don''t need java, tomcat, php or
anything else except what helps with their ruby needs.
Additionally, I find that Debians speed on updating Ruby''s packages is
not very good when there is a new security or bugfix version out. This
goes for every other distro i''ve seen as well so I am not bashing
Debian, I am just trying to show that a compiler can be an important
thing on a production server. Additionally, I use nginx as I have
found it to be great for running with ruby on mongrel but currently
the unstable and testing branches of debian still have version
0.4.13-2 and because nginx updates constantly it''s latest stable
version is 0.5.6. I''m not the kind of guy that likes to wait a year or
so for it to be updated in debian again. same goes with Mongrel. If
you have Ruby you should RubyGems and if you have that then it is SO
easy to get mongrel. I have nothing against debian adding mongrel to
apt but I personally will never use it.
One last point: Like any debian guy, I care about security and want to
have the latest patches and regularly do apt-get update and upgrade.
But because I manage a few high traffic sites that use ruby I also
must have plan if one (or more) of the sites get exploited because of
a new found security issue in ruby (or anything else for that matter).
Perhaps there are no new versions out that address this issue but
after some searching I find the root of the problem and make a patch.
So I use that patch to compile a secure version of that once exploited
software.

PLEASE: I do not want to start a flame war here and I really do like
debian and apt but I have to disagree with the philosophy of never
needing a compiler. thanks for listening to my rampling.

On Fri, Jan 12, 2007 at 09:09:33AM -0700, Kyle Kochis
wrote:
> Jens,
> Good point, at least on the first two paragraphs but I must
> respectfully (yet enthusiastically) disagree on the last point:
> 
> > Besides that, it''s just a waste of resources to compile
anything on
> > *each* production machine in case of an upgrade. Your average web
server
> > shouldn''t even need to have a compiler installed, imho.
I''ve never heard
> > of someone compiling tomcat or java on a live machine...
> 
> As a user of Debian (it is my preferred Linux Distro although I am
> starting to shift towards the BSD''s), it is a pain in the butt to
use
> Ruby1.8.2 or 1.6 and get everything like gems and rails working
> properly. Take a look at
> http://mongrel.rubyforge.org/docs/debian-sarge.html and
> http://lists.rubyonrails.org/pipermail/rails/2006-May/037763.html. The
> issue mentioned in both of these articles are VERY easy to avoid:
> curl, tar, cd ./configure make and finally make install. 
I don''t really want to warm up that old ''Debian+Ruby
sucks'' discussion,
however I can''t resist to comment on these ''issues''.

It''s a known fact that a stable Debian lags somewhat behind the rest of
the world in terms of software versions, so if you pick it you have to
take that into account and be prepared to backport something if you want
to live on the edge *and* stay on the Debian way of managing your
software. I know, not everybody is in the position to pick *his* distro
every time, and yes, Debian can give people a hard time in the
beginning. 

For the issues mentioned in the Mail you link to - the real bug here is
with rubygems which doesn''t fail if compiling a c extension fails, but
happily ignores the error. So people think Mongrel installed correctly
and blame Debian for it not working. If you saw the real error at gem
install time, everybody could guess he needs a compiler and ruby-dev to 
compile things.

[..]
> One last point: Like any debian guy, I care about security and want to
> have the latest patches and regularly do apt-get update and upgrade.
> But because I manage a few high traffic sites that use ruby I also
> must have plan if one (or more) of the sites get exploited because of
> a new found security issue in ruby (or anything else for that matter).
> Perhaps there are no new versions out that address this issue but
> after some searching I find the root of the problem and make a patch.
> So I use that patch to compile a secure version of that once exploited
> software.
you have some good points here and I really agree with you on the
security patch thing. 

In fact, that''s why I initially started maintaining my own set of
ruby/mongrel debian packages. The point is I do this on one machine that
is *not* a production system, and then just do an apt-get upgrade on the
live servers. 

Having Mongrel officially in debian is just one step further, and makes
it way easier to get started with maintaining your own mongrel packages.

Maybe that''s only me, but I really prefer to do the whole patch-and-
compile-part of the story only once. I take care for three and a half
servers running Rails apps on Mongrel. That''s not much but even then I
have better things to do than e.g. to manually patch and install Ruby 4
times when the next cgi.rb security leak arises.

Imho, that''s DRY applied to systems administration :-)


cheers,
Jens


-- 
webit! Gesellschaft f?r neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
Schnorrstra?e 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

On Fri, 12 Jan 2007, Jens Kraemer wrote:
> On Fri, Jan 12, 2007 at 09:09:33AM -0700, Kyle Kochis wrote:
>
> It''s a known fact that a stable Debian lags somewhat behind the
rest of
> the world in terms of software versions, so if you pick it you have to
> take that into account and be prepared to backport something if you want
> to live on the edge *and* stay on the Debian way of managing your
> software. I know, not everybody is in the position to pick *his* distro
> every time, and yes, Debian can give people a hard time in the
> beginning.
So is backports. If you want to stay using the debian''s stable  version
of an package, you just point your mirros to security and wait for
developers to make security changes. But if you want touse new version,
you can find a lot of things at debian backports.

Mongrel package will not made it into etch - just for lenny. So we want to
create backports of it to etch.
>
> For the issues mentioned in the Mail you link to - the real bug here is
> with rubygems which doesn''t fail if compiling a c extension fails,
but
> happily ignores the error. So people think Mongrel installed correctly
> and blame Debian for it not working. If you saw the real error at gem
> install time, everybody could guess he needs a compiler and ruby-dev to
> compile things.
hum.....
apt-get install gcc ruby1.8-dev
should solve it, no?
>
> [..]
>
>> One last point: Like any debian guy, I care about security and want to
>> have the latest patches and regularly do apt-get update and upgrade.
>> But because I manage a few high traffic sites that use ruby I also
>> must have plan if one (or more) of the sites get exploited because of
>> a new found security issue in ruby (or anything else for that matter).
>> Perhaps there are no new versions out that address this issue but
>> after some searching I find the root of the problem and make a patch.
>> So I use that patch to compile a secure version of that once exploited
>> software.
>
> you have some good points here and I really agree with you on the
> security patch thing.
>
> In fact, that''s why I initially started maintaining my own set of
> ruby/mongrel debian packages. The point is I do this on one machine that
> is *not* a production system, and then just do an apt-get upgrade on the
> live servers.
>
> Having Mongrel officially in debian is just one step further, and makes
> it way easier to get started with maintaining your own mongrel packages.
After etch, there are a lot of things that ruby-extras team is planning
to ruby in Debian. You can expect to see more packages and softwares and
more compatibility. For example, one package that is being worked is
ruby-full (or something like this), that installs... hum... full ruby :D
So, we just ask some help for upstream developers, like create .tar.gz 
files, not only gems.
>
> Maybe that''s only me, but I really prefer to do the whole
patch-and-
> compile-part of the story only once. I take care for three and a half
> servers running Rails apps on Mongrel. That''s not much but even
then I
> have better things to do than e.g. to manually patch and install Ruby 4
> times when the next cgi.rb security leak arises.
>
> Imho, that''s DRY applied to systems administration :-)
Me too! Do it once and let the rest of the world get it
:D
>
filipe lautert

filipe { AT } icewall.org
Linux User        #279798
Jabber  lautert at jabber.ru

On Thu, 11 Jan 2007 08:01:59 -0200 (BRST)
Filipe <filipe at icewall.org> wrote:

I finally have time to answer this...
 
> So, we have some favors to ask from you (the mongrel developers), as
> those favors are not big things:
> 
> * Could you please upload tar.gz files together with your gems files?
As an open source contributor, I know you feel like donating some time to make
this happen.  If you write the rake file task and patch that creates the tar.gz
with the proper version number and puts it into pkg/ then I''ll run that
task for you and upload the file.  That way you get control over how you like it
packaged rather than me doing it 20 times until you say it''s nice.

Otherwise, I tag each release in svn.  Maybe you cold convince the debian build
system to use that.  Should be easier as well since you don''t have to
troll a download page.
> * In this tar, could the files be with the current date, not in year 
> 1969? :D tar complains a lot about this...
I believe only rubygems does this because it uses a ruby version of tar.
> * When release candidates are released, can you append something like
> "~rc1" to the tar file? This help debian auto tools to check new
> releases.
There''s also a problem with how Rake and/or rubygems likes its version
numbers.  It won''t accept ~rc1 style endings for various design
reasons.
> * Finally,  I got this great idea for Mongrel.  All you have to do is
> completely change the internal processing, add 200 more methods to the
> HTTP parser, and... no. Just joking. Just the other 3 would be great!
Ha!
 
> The Debian ruby-extras team has a page[1] dedicated to upstream
> developers, where those and other things that help *nix distributions
> are described.
I''ll give you guys a slight clue though.  Mongrel is packaged on lots
of other operating systems.  So far OSX, SuSE, FreeBSD, Win32, Fedora and
probably more I don''t know anything about.  Not a single one of them
asked me to change the project setup to accommodate their build systems.  OSX
asked me to change the license but that''s it.

If Debian has to ask people to change their projects, then maybe
there''s something wrong with Debian''s build system.  I know
Debian folks can *never* admit there might be something wrong, but just consider
it.  There might just be some improvement you could make to be as capable as the
other OS out there.

Just a thought.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Thu, 11 Jan 2007 08:01:59 -0200 (BRST)
Filipe <filipe at icewall.org> wrote:
> 
> Hello!
> 
> My name is Filipe and I''m from the Debian ruby-extras team.
Oh! One more condition I *have* to insist on.  If you guys carve the damn
package up into a billion other little packages and then shove the files around
into a thousand locations then I''m gonna scream.  At least if you do
that, man up and update the Mongrel Debian documentation on the Mongrel site.

I''ll be really pissed if I''m supporting poor Debian users
because you decided to rip my project to shreds.

It''s fine if you put stuff in new locations, but document it.  Dammit
document it.

Feel free to contact me off list for help or questions.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Fri, 12 Jan 2007, Zed A. Shaw wrote:
> On Thu, 11 Jan 2007 08:01:59 -0200 (BRST)
> Filipe <filipe at icewall.org> wrote:
>
> I finally have time to answer this...
>
>> So, we have some favors to ask from you (the mongrel developers), as
>> those favors are not big things:
>>
>> * Could you please upload tar.gz files together with your gems files?
>
> As an open source contributor, I know you feel like donating some time to
make this happen.  If you write the rake file task and patch that creates the
tar.gz with the proper version number and puts it into pkg/ then I''ll
run that task for you and upload the file.  That way you get control over how
you like it packaged rather than me doing it 20 times until you say
it''s nice.
Ok, I will. But it is a simple tar.gz with the same content of
data.tar.gz from the gem.
>
> Otherwise, I tag each release in svn.  Maybe you cold convince the debian
build system to use that.  Should be easier as well since you don''t
have to troll a download page.
After etch I can raise this subject. Now, the core developers are going
crazy :/
>
>> * In this tar, could the files be with the current date, not in year
>> 1969? :D tar complains a lot about this...
>
> I believe only rubygems does this because it uses a ruby version of tar.
Strange. I''ll go up with this and see why rubygems or ruby do this.
Thanks for the path.
>
>> * When release candidates are released, can you append something like
>> "~rc1" to the tar file? This help debian auto tools to check
new
>> releases.
>
> There''s also a problem with how Rake and/or rubygems likes its
version numbers.  It won''t accept ~rc1 style endings for various design
reasons.
Hum... then -rc1? :D Well, I take a look at this when generating the rake
task.
>
>> * Finally,  I got this great idea for Mongrel.  All you have to do is
>> completely change the internal processing, add 200 more methods to the
>> HTTP parser, and... no. Just joking. Just the other 3 would be great!
>
> Ha!
>
>> The Debian ruby-extras team has a page[1] dedicated to upstream
>> developers, where those and other things that help *nix distributions
>> are described.
>
> I''ll give you guys a slight clue though.  Mongrel is packaged on
lots of other operating systems.  So far OSX, SuSE, FreeBSD, Win32, Fedora and
probably more I don''t know anything about.  Not a single one of them
asked me to change the project setup to accommodate their build systems.  OSX
asked me to change the license but that''s it.
>
> If Debian has to ask people to change their projects, then maybe
there''s something wrong with Debian''s build system.  I know
Debian folks can *never* admit there might be something wrong, but just consider
it.  There might just be some improvement you could make to be as capable as the
other OS out there.
Hey, I can make packages with the actual format of the files. But I need
to admit it: creating packages from gems give me a lot of extra work.
The reason is that in Debian we try not to touch the original source,
like in [1]. We just download it, then apply patches (if needed) and compile it.
But well, there can be other ways. We just need time (a lot of) and will
(lots and lots of) to propose it and get it aproved. Debian is a lot
bureaucratic.

And maybe people from  OSX, SuSE, FreeBSD, Win32, Fedora, Gentoo (they have
some packages too) can benefit from the tar file too, don ''t you think?
I don''t know how they build systems works, so I cannot speak for them.

The big problem for debian ruby-extra team is that we are trying to bind
diferrents worlds: one that is a lot conservative (debian) and one that
is fast and exciting (ruby)! It gives a lot of trouble. But still being
cool.
>
> Just a thought.
>
Thanks for listen and for the chance to write the task.


filipe lautert

filipe { AT } icewall.org
Linux User        #279798
Jabber  lautert at jabber.ru


1.
http://ftp.br.debian.org/debian/pool/main/libx/libxml-ruby/libxml-ruby_0.3.8.4.orig.tar.gz

On Fri, 12 Jan 2007, Zed A. Shaw wrote:
> On Thu, 11 Jan 2007 08:01:59 -0200 (BRST)
> Filipe <filipe at icewall.org> wrote:
>
>>
>> Hello!
>>
>> My name is Filipe and I''m from the Debian ruby-extras team.
>
> Oh! One more condition I *have* to insist on.  If you guys carve the damn
package up into a billion other little packages and then shove the files around
into a thousand locations then I''m gonna scream.  At least if you do
that, man up and update the Mongrel Debian documentation on the Mongrel site.
Well.... at the moment, we just thought of two (2!!!!) packages: one
called "mongrel" (ooooooohhh) and one called libgemsplugin-ruby . I
think we don''t need more packages, don''t you? Your patch to
cgi is
already in debian ruby package, so we don''t need to package it. And
maybe in the future we can create new packages for fastthread,
mongrel_cluster, mongrel_service, etc. Those should by different
packages because they are already distinct gems.

And I can update the page about mongrel in debian. I''ll do it when we
finish our work.
>
> I''ll be really pissed if I''m supporting poor Debian users
because you decided to rip my project to shreds.
>
> It''s fine if you put stuff in new locations, but document it. 
Dammit document it.
We are documenting it, take it easy :D I already start to write the man
page for mongrel_rails.

And when the packages are ready, any problems with then can be redirect
to our mailling-list (debian-ruby at lists.debian.org).
>
> Feel free to contact me off list for help or questions.
Thanks! Good weekend!

filipe lautert

filipe { AT } icewall.org
Linux User        #279798
Jabber  lautert at jabber.ru

> And maybe people from  OSX, SuSE, FreeBSD, Win32, Fedora, Gentoo (they have
> some packages too) can benefit from the tar file too, don ''t you
think?
> I don''t know how they build systems works, so I cannot speak for
them.
On FreeBSD I just grab the gem and install it for the user. On deinstall 
the system does a gem uninstall. This way bringing in new gem to the 
FreeBSD ports tree is quite easy in contracst to grab a tar and do a 
manual install through the package system.

> 
> filipe lautert
> 
Regards,
Jonathan

-- 
Jonathan Weiss
http://blog.innerewut.de

> Gentoo (they have
> some packages too) can benefit from the tar file too, don ''t you
think?
> I don''t know how they build systems works, so I cannot speak for
them.
the ebuild is almost totally empty, besides this line: 

inherit ruby gems

im thinking they could proably factor this line into the eclass, since
presumably rubyforge has a naming convention:
SRC_URI="http://mongrel.rubyforge.org/releases/gems/${P}.gem"


you might wnat to look into something similar. i like to think of the package
manager as a wrapper, not an un-networked custom unwrapper/rewrapper and hope
everything ends up back in the same box (and usually with competing version
numbers and optional modules it doesnt)

> you might wnat to look into something similar. i like to think of the
package manager as a wrapper, not an un-networked custom unwrapper/rewrapper and
hope everything ends up back in the same box (and usually with competing version
numbers and optional modules it doesnt)
the other important reason for hooking into gems, besides maintainer sanity, is
to allow the user to mix and match between installation methods. what happens
when the user installed mongrel bypassing gems, then installs camping? im pretty
sure it installs mongrel again, and then when they get a bug they dont even know
which version is being used..

> the other important reason for hooking into gems, besides maintainer
sanity, is to allow the user to mix and match between installation methods. what
happens when the user installed mongrel bypassing gems, then installs camping?
im pretty sure it installs mongrel again, and then when they get a bug they dont
even know which version is being used..
the Gentoo method isnt perfect - it fails in the inverse scenario where youve
installed everything via gems, then want to install something through portage
that depends on installed gems (stuff gets reinstalled). the other problem is
its tied to hardcoded version numbers in the ebuild filenames. theres no way for
it to discover that newer versions of the gems exist. does freeBSD solve either
of these problems?

Zed A. Shaw wrote:
> 
> I''ll give you guys a slight clue though.  Mongrel is packaged on
lots
> of other operating systems.  So far OSX, SuSE, FreeBSD, Win32, Fedora
> and probably more I don''t know anything about.  Not a single one
of them
> asked me to change the project setup to accommodate their build systems.
> OSX asked me to change the license but that''s it.
> 
> If Debian has to ask people to change their projects, then maybe
there''s
> something wrong with Debian''s build system.  I know Debian folks
can
> *never* admit there might be something wrong, but just consider it.
> There might just be some improvement you could make to be as capable as
> the other OS out there.
> 
> Just a thought.
Agreed. I''m the maintainer for the (unmentioned) OpenBSD port as well,
and this struck me as a little ridiculous straight away.

On 1/12/07, Filipe <filipe at icewall.org> wrote:
> On Fri, 12 Jan 2007, Zed A. Shaw wrote:
> >> * When release candidates are released, can you append something
like
> >> "~rc1" to the tar file? This help debian auto tools to
check new
> >> releases.
> >
> > There''s also a problem with how Rake and/or rubygems likes
its version numbers.  It won''t accept ~rc1 style endings for various
design reasons.
>
> Hum... then -rc1? :D Well, I take a look at this when generating the rake
> task.
I *THINK* this is not the rubygems standard.  See this:

http://rubygems.org/read/chapter/16

I also *THINK* these constraints are due to the way RubyGems allows
specification of variable gem version, for example, "> 0.1.0" or
""1.0.1".  If they allowed people to put random character strings
in
version numbers that logic couldn''t work.

That brings up a couple of questions that I''d like to know the answers
to:

1. Why does debian or apt need to distinguish between release
candidates?  Is this automatically tied to -dev packages or something?
2. How are rubygems supposed to indicate release candidates, or beta
or whatever you call something that is pre-release?

-- Chad

On Sat, 13 Jan 2007, Chad Woolley wrote:
> On 1/12/07, Filipe <filipe at icewall.org> wrote:
>> On Fri, 12 Jan 2007, Zed A. Shaw wrote:
>>>> * When release candidates are released, can you append
something like
>>>> "~rc1" to the tar file? This help debian auto tools
to check new
>>>> releases.
>>>
>>> There''s also a problem with how Rake and/or rubygems likes
its version numbers.  It won''t accept ~rc1 style endings for various
design reasons.
>>
>> Hum... then -rc1? :D Well, I take a look at this when generating the
rake
>> task.
>
> I *THINK* this is not the rubygems standard.  See this:
>
> http://rubygems.org/read/chapter/16
>
> I also *THINK* these constraints are due to the way RubyGems allows
> specification of variable gem version, for example, "> 0.1.0"
or "> "1.0.1".  If they allowed people to put random character
strings in
> version numbers that logic couldn''t work.
Well.... but rubygems doesn''t defines the way to versioning release
candidates. Actual release candidate 1 is called  mongrel-1.0.gem. So,
how will be called release candidate 2? mongrel-1.0.1.gem? Or it will
have the same name?
>
> That brings up a couple of questions that I''d like to know the
answers to:
>
> 1. Why does debian or apt need to distinguish between release
> candidates?  Is this automatically tied to -dev packages or something?
No, it''s just because we''ve a file called watch, with this
content:

http://mongrel.rubyforge.org/releases/gems/mongrel-([0-9\.]*)\.gem

And a tool that checks the site and remind me that there is a new
version.

Ow, and the package version in debian always has the same version than
upstream. So I thought it is someway stranger to have a release
candidate package called "1.0".
> 2. How are rubygems supposed to indicate release candidates, or beta
> or whatever you call something that is pre-release?
Don''t know. Rubygems policy doesn''t say anything about this.
At least
points 5 and 6... it just says that "Any public release of a gem
should have a different version. Normally that means incrementing the
build number.".

filipe lautert

filipe { AT } icewall.org
Linux User        #279798
Jabber  lautert at jabber.ru

On Fri, 12 Jan 2007, carmen wrote:
>>>the ebuild is almost totally empty, besides this line:
>>>
>>>.inherit ruby gems
>>>
>>>im thinking they could proably factor this line into the eclass,
since presumably rubyforge has a naming convention:
>>>SRC_URI="http://mongrel.rubyforge.org/releases/gems/${P}.gem"
yes, it has. Mongrel is a litle diferent from other projects, but still it has a
naming convention.
>>>
>>>you might wnat to look into something similar. i like to think of
the package manager as a wrapper, not an un-networked custom unwrapper/rewrapper
and
>>>hope everything ends up back in the same box (and usually with
competing version numbers and optional modules it doesnt)
>>>
>>>
>> the other important reason for hooking into gems, besides maintainer
sanity, is to allow the user to mix and match between installation methods. what
happens when the user installed mongrel bypassing gems, then installs camping?
im pretty sure it installs mongrel again, and then when they get a bug they dont
even know which version is being used..
Yes, it reinstalls. But the way I think is this: if I''m a user and want
to get along with debian version of mongrel, that''s fine.
But if I don''t like it (and I hate debian''s packages), I can
install everything from rubygems and be happy too. But now, mix the two methods
really means trouble for me.
>
> the Gentoo method isnt perfect - it fails in the inverse scenario where
youve installed everything via gems, then want to install something through
portage that depends on installed gems (stuff gets reinstalled). the other
problem is its tied to hardcoded version numbers in the ebuild filenames. theres
no way for it to discover that newer versions of the gems exist. does freeBSD
solve either of these problems?
Well... my watch file again:
http://mongrel.rubyforge.org/releases/gems/mongrel-([0-9\.]*)\.gem

Maybe it helps.


filipe lautert

filipe { AT } icewall.org
Linux User        #279798
Jabber  lautert at jabber.ru

On 1/15/07, Filipe <filipe at icewall.org> wrote:
> On Sat, 13 Jan 2007, Chad Woolley wrote:
>
> >> Hum... then -rc1? :D Well, I take a look at this when generating
the rake
> >> task.
> >
> > I *THINK* this is not the rubygems standard.  See this:
> >
> > http://rubygems.org/read/chapter/16
...
> > 1. Why does debian or apt need to distinguish between release
> > candidates?  Is this automatically tied to -dev packages or something?
>
> No, it''s just because we''ve a file called watch, with
this content:
>
> http://mongrel.rubyforge.org/releases/gems/mongrel-([0-9\.]*)\.gem
>
> And a tool that checks the site and remind me that there is a new
> version.
>
> Ow, and the package version in debian always has the same version than
> upstream. So I thought it is someway stranger to have a release
> candidate package called "1.0".
I do agree this is rather strange, but it''s not unheard of.  In any
case, I don''t think it''s appropriate for a packaging system to
try to
enforce constraints on the versioning standards of the the software it
packages.

See this for all the various flavors of versioning standards:

http://en.wikipedia.org/wiki/Version#Software_versioning_schemes

Some of these allow identification of pre-release or beta versions
while still conforming to the RubyGems standard.  For example, the
Linux versioning approach (odd numbers for development releases, even
for stable) would be compatible:
http://en.wikipedia.org/wiki/Version#Odd-numbered_versions_for_development_releases

HOWEVER, the linux approach WOULD cause problems when you consider
that RubyGems own version specification approach is primitive - you
can only say things like "> x.y" or "<= x.y"

I''ll point this thread out to the RubyGems list, I''m curious
what they
would have to say.  At a minimum, I think the RubyGems versioning
standard docs could use some clarification.

-- Chad

On 1/15/07, Chad Woolley <thewoolleyman at gmail.com> wrote:
>
> I''ll point this thread out to the RubyGems list, I''m
curious what they
> would have to say.  At a minimum, I think the RubyGems versioning
> standard docs could use some clarification.
I imagine that what is said will reduce down to something like this:
"Provide a patch that retains the ability to selectively require
certain versions or version ranges, yet provides the version
specification freedom that you think should exist, and we''ll consider
it."


Kirk Haines

On 1/15/07, Kirk Haines <wyhaines at gmail.com>
wrote:
> On 1/15/07, Chad Woolley <thewoolleyman at gmail.com> wrote:
>
> >
> > I''ll point this thread out to the RubyGems list, I''m
curious what they
> > would have to say.  At a minimum, I think the RubyGems versioning
> > standard docs could use some clarification.
>
> I imagine that what is said will reduce down to something like this:
> "Provide a patch that retains the ability to selectively require
> certain versions or version ranges, yet provides the version
> specification freedom that you think should exist, and we''ll
consider
> it."
You are probably right.  However, the RubyGems "Rational Versioning
Policy" ( http://rubygems.org/read/chapter/7 ) doesn''t seem to
account
for the beta/release candidate phase of the development cycle for a
post-1.0 release.  It looks like the best you can do is to assume that
any x.0.0 release is a release candidate, and should be treated as
such.  However, there''s still no standard way for a gem developer to
indicate that a given post-x.0.0 version is now REALLY finished, and
should be safe for widespread use.

-- Chad

On 1/15/07, Chad Woolley <thewoolleyman at gmail.com> wrote:
> You are probably right.  However, the RubyGems "Rational Versioning
> Policy" ( http://rubygems.org/read/chapter/7 ) doesn''t seem
to account
> for the beta/release candidate phase of the development cycle for a
> post-1.0 release.  It looks like the best you can do is to assume that
> any x.0.0 release is a release candidate, and should be treated as
> such.  However, there''s still no standard way for a gem developer
to
> indicate that a given post-x.0.0 version is now REALLY finished, and
> should be safe for widespread use.
Nope, and I doubt that there ever will be.  Everybody does versioning
differently, so all a person can ever really do is look at the project
information and then judge the version number in the context of the
other project information.

And given that, the gems versioning support is a compromise position
that provides some flexibility in choosing how to assign version
numbers while still providing a useful capability to select specific
versions or ranges of versions.  I''d welcome even more capability in
that regard, personally, but it''s hardly a show stopper.

After all, is a version like 1.0-rc9 really any more informative than 0.3.13.4?


Kirk Haines

On 1/15/07, Kirk Haines <wyhaines at gmail.com>
wrote:
> On 1/15/07, Chad Woolley <thewoolleyman at gmail.com> wrote:
>
> > You are probably right.  However, the RubyGems "Rational
Versioning
> > Policy" ( http://rubygems.org/read/chapter/7 ) doesn''t
seem to account
> > for the beta/release candidate phase of the development cycle for a
> > post-1.0 release.  It looks like the best you can do is to assume that
> > any x.0.0 release is a release candidate, and should be treated as
> > such.  However, there''s still no standard way for a gem
developer to
> > indicate that a given post-x.0.0 version is now REALLY finished, and
> > should be safe for widespread use.
>
> Nope, and I doubt that there ever will be.  Everybody does versioning
> differently, so all a person can ever really do is look at the project
> information and then judge the version number in the context of the
> other project information.
Here''s the answer I got on the RubyGems developer list:

http://rubyforge.org/pipermail/rubygems-developers/2007-January/002443.html

"The typical workaround is to not post release-candidate gems to
Rubyforge, and
instead host them on a separate gem server."

A reasonable answer, especially since it''s so easy to set up a local
gem server, and point to that on your RubyForge project page (or even
host the gem files directly on your RubyForge account).

-- Chad

Zed A. Shaw dijo [Fri, Jan 12, 2007 at 10:07:16AM
-0800]:
> I finally have time to answer this...
I don''t  ;-) I''m in the middle of a high workload cycle -
Anyway,
thanks to Filipe for finally posting this. He has been pestering me
over and over to  upload my work to the Debian SVN repository so more
people can work on the packaging until I get time to take it again :)
> > So, we have some favors to ask from you (the mongrel developers), as
> > those favors are not big things:
> > 
> > * Could you please upload tar.gz files together with your gems files?
> 
> As an open source contributor, I know you feel like donating some
> time to make this happen.  If you write the rake file task and patch
> that creates the tar.gz with the proper version number and puts it
> into pkg/ then I''ll run that task for you and upload the file. 
That
> way you get control over how you like it packaged rather than me
> doing it 20 times until you say it''s nice. 
It''s quite easy: The Gem file, as you surely know, is just an
uncompressed tar containing the metadata and a data.tar.gz, which
contains your work. What I do is to rename your .gem to .tar and
compress it, so I work with the closest I can get to the ideal
"pristine sources" .orig.tar.gz - Our build process then becomes: (of
course, this is an automated process, but still)

- Extract Mongrel tarball (and discardable metadata) from the
  .orig.tar.gz (your renamed Gem), and add a debian/ directory where
  the packaging information is stored
- Extract data.tar.gz, creating your whole structure
- Patch your files with whatever modifiations we have to apply (which
  are stored in debian/patches and applied via the dpatch framework)
- Build the package

So... Well, it works correctly, but implies a couple extra steps -
This is fine by itself, but if somebody outside our team needs to work
on the package (say, to do some urgent QA work), I expect him to have
a "WTF" look for some minutes ;-)
> Otherwise, I tag each release in svn.  Maybe you cold convince the
> debian build system to use that.  Should be easier as well since you
> don''t have to troll a download page. 
Umh... I prefer working against something as close as possible to the
"pristine sources" ideal. Taking one file from you, and having that as
our reference. The Gem file works, although suboptimally  Another of
our tools tracks upstream new versions by looking at URLs with some
regex magic on them. AFAICT, we cannot automate this step as it is
now. 
> > * In this tar, could the files be with the current date, not in year 
> > 1969? :D tar complains a lot about this...
> 
> I believe only rubygems does this because it uses a ruby version of tar.
Ok, so we should ask them for a bugfix - I''m relatively new to Ruby
culture. What''s this Ruby tar''s name?
> > * When release candidates are released, can you append something like
> > "~rc1" to the tar file? This help debian auto tools to check
new
> > releases.
> 
> There''s also a problem with how Rake and/or rubygems likes its
> version numbers.  It won''t accept ~rc1 style endings for various
> design reasons. 
Nah, I agree here with Zed - We can accomodate your version number
changes. We cannot expect everybody on the Free Software world to
agree on a versioning standard! The ~rc1 notation is _very_
Debian-centric - and very seldom used, IMHO.
> > The Debian ruby-extras team has a page[1] dedicated to upstream
> > developers, where those and other things that help *nix distributions
> > are described.
> 
> I''ll give you guys a slight clue though.  Mongrel is packaged on
> lots of other operating systems.  So far OSX, SuSE, FreeBSD, Win32,
> Fedora and probably more I don''t know anything about.  Not a
single
> one of them asked me to change the project setup to accommodate
> their build systems.  OSX asked me to change the license but
that''s
> it. 
> 
> If Debian has to ask people to change their projects, then maybe
> there''s something wrong with Debian''s build system.  I
know Debian
> folks can *never* admit there might be something wrong, but just
> consider it.  There might just be some improvement you could make to
> be as capable as the other OS out there. 
Yes, I understand your point here. Only that we know how to do things
better ;-)

No, really - I guess that every Linux distribution you quoted would
prefer you shipping an orig.tar.gz (call it whatever you want), as the
"pristine sources" religion was _started_ by the RedHat people and
all... We can all accomodate. Debian does, however, have one uniqueness
factor: Being a completely distributed and volunteer project, we put a
lot of emphasis on things being as standard and automatic as possible,
for the sake of our colleagues :)

Thanks a lot, anyway!

-- 
Gunnar Wolf - gwolf at gwolf.org - (+52-55)5623-0154 / 1451-2244
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF

carmen dijo [Fri, Jan 12, 2007 at 05:07:08PM -0500]:
>
> > the other important reason for hooking into gems, besides
> > maintainer sanity, is to allow the user to mix and match between
> > installation methods. what happens when the user installed mongrel
> > bypassing gems, then installs camping? im pretty sure it installs
> > mongrel again, and then when they get a bug they dont even know
> > which version is being used.. 
> 
> the Gentoo method isnt perfect - it fails in the inverse scenario
> where youve installed everything via gems, then want to install
> something through portage that depends on installed gems (stuff gets
> reinstalled). the other problem is its tied to hardcoded version
> numbers in the ebuild filenames. theres no way for it to discover
> that newer versions of the gems exist. does freeBSD solve either of
> these problems? 
I agree with your points... But anyway, that''s one of the reasons
Debian is so different from Gentoo/FreeBSD (with their source-based
packaging systems). Yes, it would be unfair to plainly state that we
pursue an unbreakable system while you don''t, and it would surely lead
to a flamefest ;-) Just please take a look at the two following pages,
which explain the rationale behind our requests:

http://pkg-ruby-extras.alioth.debian.org/rubygems.html
http://pkg-ruby-extras.alioth.debian.org/upstream-devs.html

Greetings,

-- 
Gunnar Wolf - gwolf at gwolf.org - (+52-55)5623-0154 / 1451-2244
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF

Filipe dijo [Mon, Jan 15, 2007 at 11:29:39AM -0200]:
>
> > the Gentoo method isnt perfect - it fails in the inverse scenario
> > where youve installed everything via gems, then want to install
> > something through portage that depends on installed gems (stuff
> > gets reinstalled). the other problem is its tied to hardcoded
> > version numbers in the ebuild filenames. theres no way for it to
> > discover that newer versions of the gems exist. does freeBSD solve
> > either of these problems? 
> 
> Well... my watch file again:
> http://mongrel.rubyforge.org/releases/gems/mongrel-([0-9\.]*)\.gem
Oh, so _there_ it is :) Ok, Zed, forget my rant about not having the
files accessible ;-)

-- 
Gunnar Wolf - gwolf at gwolf.org - (+52-55)5623-0154 / 1451-2244
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF

Chad Woolley dijo [Mon, Jan 15, 2007 at 01:15:33PM
-0700]:
> > I imagine that what is said will reduce down to something like this:
> > "Provide a patch that retains the ability to selectively require
> > certain versions or version ranges, yet provides the version
> > specification freedom that you think should exist, and we''ll
consider
> > it."
> 
> You are probably right.  However, the RubyGems "Rational Versioning
> Policy" ( http://rubygems.org/read/chapter/7 ) doesn''t seem
to account
> for the beta/release candidate phase of the development cycle for a
> post-1.0 release.  It looks like the best you can do is to assume that
> any x.0.0 release is a release candidate, and should be treated as
> such.  However, there''s still no standard way for a gem developer
to
> indicate that a given post-x.0.0 version is now REALLY finished, and
> should be safe for widespread use.
Would not make too much sense, still - What one author considers to be
1.0-ready will still be 0.1-like for another author''s standards. No,
the only thing worth requesting is IMHO to keep sanity - Debian
introduced the useful-but-for-me-still-worth-avoiding ~ syntax, made
to accomodate some upstream strange practices - The ~ character sorts
version-wise just before the exact version number preceding it. So you
have: 

0.99 < 1.0~beta1 < 1.0~beta3 < 1.0~beta3~almost < 1.0

Greetings :)

-- 
Gunnar Wolf - gwolf at gwolf.org - (+52-55)5623-0154 / 1451-2244
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF