So, I was looking at the way msysGit wraps the git command in a batch file, and I thought we could do the same thing for compiling extensions in gems. Something like this in \path_to_ruby\bin: REM make.cmd @echo off setlocal set path=%path%;c:\ruby\repo\oci\sandbox\mingw C:\ruby\repo\oci\sandbox\msys\bin\bash.exe --login -i -c "make %*" exit /B %ErrorLevel% This way we wouldn''t need to worry about adding mingw to the path, except when it''s needed (ie. when make is called). We may need to add msys to the path as well, I''m not sure. I can use this method to `gem install mongrel`. It compiles native extensions and all that. Of course when I try to run mongrel it fails ;), but I think that''s a separate issue. What do you think? Am I oversimplifying the problem again? My setup: Windows XP, SP2 ruby 1.8.6 (2008-03-03 patchlevel 114) [i386-mingw32] rubygems 1.1.1 Thanks, Gordon
On Fri, Apr 18, 2008 at 12:57 PM, Gordon Thiesfeld <gthiesfeld at gmail.com> wrote:> So, I was looking at the way msysGit wraps the git command in a batch > file, and I thought we could do the same thing for compiling > extensions in gems. > Something like this in \path_to_ruby\bin: > > REM make.cmd > @echo off > setlocal > set path=%path%;c:\ruby\repo\oci\sandbox\mingw > C:\ruby\repo\oci\sandbox\msys\bin\bash.exe --login -i -c "make %*" > exit /B %ErrorLevel% > > This way we wouldn''t need to worry about adding mingw to the path, > except when it''s needed (ie. when make is called). We may need to add > msys to the path as well, I''m not sure. I can use this method to `gem > install mongrel`. It compiles native extensions and all that. Of > course when I try to run mongrel it fails ;), but I think that''s a > separate issue. >There is no need to add mingw or msys to the path, since when bash start it parses profile and setup $PATH to include both, mingw and msys tools. Anyway, we will need wrap ''gcc'' too, since is something extconf uses to try definitions that create the makefile. The only problem is that ruby doesn''t like it, and I had problems in the past due that, and was ruby exec code. Let me explain: When you issue ''make'' in the command line, the cmd interpreter (cmd.exe) will try to find an executable of make, based on ''make'' + one of the extensions in PATHEXT environment variable, in the order they are listed: make.com, make.exe, make.bat, make.cmd, etc. The problem is ruby exec doesn''t work like that. if you supply a command without the extension, it will look for executables (com and exe), but you must explicitly indicate .bat or .cmd when executing batch files. try creating a make.bat that just echo something and do: ruby -e "system(''make --version'')" Without mingw/msys in the path, that will fail until you supply .bat/.cmd to the system command. So, that will require us patch rubygems and replace conditions for RUBY_PLATFORM and issue ''make.bat/cmd'' instead of plain ''make''. Take in consideration that right now there are conditions for mswin (nmake) and others platforms, plain make.> What do you think? Am I oversimplifying the problem again? >I like the idea and you aren''t oversimplifying it, I think Ruby is broken in that aspect. Maybe we can fix that damn thing in ruby C code and workaround that? win32/win32.c:989 (CreateChild function), call dln_find_exe dlc.c:1671 (dln_find_exe function), call dln_find_1 dlc.c:1796 (dln_find_1 function), call eaccess file.c:897 (eaccess function), call access API with ''path'' and mode X_OK (io.h) Or maybe I ended tracing it to the wrong place :-P> Thanks,Thanks to you for your interest. -- Luis Lavena Multimedia systems - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams
On Fri, Apr 18, 2008 at 8:15 PM, Luis Lavena <luislavena at gmail.com> wrote:> > There is no need to add mingw or msys to the path, since when bash > start it parses profile and setup $PATH to include both, mingw and > msys tools. > > Anyway, we will need wrap ''gcc'' too, since is something extconf uses > to try definitions that create the makefile. > > The only problem is that ruby doesn''t like it, and I had problems in > the past due that, and was ruby exec code. > > Let me explain: > > When you issue ''make'' in the command line, the cmd interpreter > (cmd.exe) will try to find an executable of make, based on ''make'' + > one of the extensions in PATHEXT environment variable, in the order > they are listed: > > make.com, make.exe, make.bat, make.cmd, etc. > > The problem is ruby exec doesn''t work like that. if you supply a > command without the extension, it will look for executables (com and > exe), but you must explicitly indicate .bat or .cmd when executing > batch files. > > try creating a make.bat that just echo something and do: > > ruby -e "system(''make --version'')":-P C:\ruby\ruby_mingw>which make.exe which: no make.exe in (c:\wix;c:\windows\system32;c:\windows\system;C:\Program Files\Subversion\bin;C:\Program Files\Bazaar;C:\Program Files\Git\cmd;C:\ruby\ruby_mingw\bin;C:\bin;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;C:\Program Files\Putty;c:\bin\console2) C:\ruby\ruby_mingw>which make.cmd C:\ruby\ruby_mingw\bin\make.cmd C:\ruby\ruby_mingw>irb>> `make --version`=> "GNU Make version 3.79.1, by Richard Stallman and Roland McGrath.\nBuilt for i686-pc-msys\nCopyright (C) 1988, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 2000\n\tFree Software Foundation, Inc.\nThis is free software; see the source for copying conditions.\nThere is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A\nPARTICULAR PURPOSE.\n\nReport bugs to <bug-make at gnu.org>.\n\n">>OK, I''ve seen this problem before too, with Rakefiles that shell out to use ''gem'' and things like that, and I''ve always made the same assumption that Ruby didn''t process bat or cmd file. But what was bugging me after I read your email was that my make.cmd worked, and there is no other make in the path. First of all, it does seem to run bat and cmd files, but they have to be in the path not in the same folder you''re in. This may be a ''feature'' ;-), similar to the way *nix won''t let you run shell commands in your current folder without adding ./ in front of them. Make a simple batch file, like this: REM test.cmd @echo off date /t Stick it in ruby\bin, or somewhere else in the path and then try to run it. It works for me. The problem with the gem command and commands created by RubyGems, is that Ruby will find the ruby file at the command first and try to execute it, and the shell doesn''t know how to run it (Windows doesn''t do shebang lines). So, if you move the ruby file with no extension out of the bin folder, and adjust the batch file to point to it, it will work. At least it works on my machine ;-).> I like the idea and you aren''t oversimplifying it, I think Ruby is > broken in that aspect. > > Maybe we can fix that damn thing in ruby C code and workaround that? > > win32/win32.c:989 (CreateChild function), call dln_find_exe > dlc.c:1671 (dln_find_exe function), call dln_find_1 > dlc.c:1796 (dln_find_1 function), call eaccess > file.c:897 (eaccess function), call access API with ''path'' and mode > X_OK (io.h) >Maybe Ruby''s not totally broken here, just undocumented? Take a look and let me know what you think. --Gordon
On Sat, Apr 19, 2008 at 11:51 AM, Gordon Thiesfeld <gthiesfeld at gmail.com> wrote:> On Fri, Apr 18, 2008 at 8:15 PM, Luis Lavena <luislavena at gmail.com> wrote: > > > > There is no need to add mingw or msys to the path, since when bash > > start it parses profile and setup $PATH to include both, mingw and > > msys tools. > > > > Anyway, we will need wrap ''gcc'' too, since is something extconf uses > > to try definitions that create the makefile. > > > > The only problem is that ruby doesn''t like it, and I had problems in > > the past due that, and was ruby exec code. > > > > Let me explain: > > > > When you issue ''make'' in the command line, the cmd interpreter > > (cmd.exe) will try to find an executable of make, based on ''make'' + > > one of the extensions in PATHEXT environment variable, in the order > > they are listed: > > > > make.com, make.exe, make.bat, make.cmd, etc. > > > > The problem is ruby exec doesn''t work like that. if you supply a > > command without the extension, it will look for executables (com and > > exe), but you must explicitly indicate .bat or .cmd when executing > > batch files. > > > > try creating a make.bat that just echo something and do: > > > > ruby -e "system(''make --version'')" > > :-P > > C:\ruby\ruby_mingw>which make.exe > which: no make.exe in > (c:\wix;c:\windows\system32;c:\windows\system;C:\Program > Files\Subversion\bin;C:\Program Files\Bazaar;C:\Program > Files\Git\cmd;C:\ruby\ruby_mingw\bin;C:\bin;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;C:\Program > Files\Putty;c:\bin\console2) > > C:\ruby\ruby_mingw>which make.cmd > C:\ruby\ruby_mingw\bin\make.cmd > > C:\ruby\ruby_mingw>irb > >> `make --version` > => "GNU Make version 3.79.1, by Richard Stallman and Roland McGrath.\nBuilt for > i686-pc-msys\nCopyright (C) 1988, 89, 90, 91, 92, 93, 94, 95, 96, 97, > 98, 99, 2000\n\tFree Software Foundation, Inc.\nThis is free software; > see the source for > copying conditions.\nThere is NO warranty; not even for > MERCHANTABILITY or FITNESS FOR A\nPARTICULAR PURPOSE.\n\nReport bugs > to <bug-make at gnu.org>.\n\n" > >> >What The Hell??? I tried that before send the mail to the list, just to confirm what I was getting for the past years... and it failed... Now that I try again, and again, and again, it works??? (make.bat) put into %HOME%/bin (which is in PATH). @ECHO OFF ECHO.Simulate Make. ECHO.parameters to be passed: %* Now, using CMD.exe, in HOME or any other drive letter: C:\>make --version Simulate Make. parameters to be passed: --version (make.bat is in D:\Users\Luis\bin\make.bat), don''t have msys tools in the path for this test. Ok, now Ruby: C:\>ruby -e "puts Dir.pwd; system(''make --version'')" C:/ Simulate Make. parameters to be passed: --version The damn thing worked! I always got mixed results from system/backticks/exec ... But!, now I know how to make it fail... 1) Create a empty ''make'' file and put in a path that is listed *before* the one that holds your make.bat (system32 is a good place). D:\Users\Luis>ruby -e "puts Dir.pwd; system(''make --version'')" D:/Users/Luis No output from system call... because it found the first occurrence of make (extension less). 2) Now move the empty make file into a directory of PATH that happens *after* the position make.bat is located (example, I moved it to c;\program files\putty) C:\>ruby -e "puts Dir.pwd; system(''make --version'')" C:/ Simulate Make. parameters to be passed: --version Bingo! so it seems is not just looking for executable files, but any file that matches the search criteria (make.*), the first found, the first served, no matter if is an executable or not.> OK, I''ve seen this problem before too, with Rakefiles that shell out > to use ''gem'' and things like that, and I''ve always made the same > assumption that Ruby didn''t process bat or cmd file. But what was > bugging me after I read your email was that my make.cmd worked, and > there is no other make in the path. First of all, it does seem to run > bat and cmd files, but they have to be in the path not in the same > folder you''re in. This may be a ''feature'' ;-), similar to the way > *nix won''t let you run shell commands in your current folder without > adding ./ in front of them. Make a simple batch file, like this: >Yes, a true feature :-P> REM test.cmd > @echo off > date /t > > Stick it in ruby\bin, or somewhere else in the path and then try to > run it. It works for me. > > The problem with the gem command and commands created by RubyGems, is > that Ruby will find the ruby file at the command first and try to > execute it, and the shell doesn''t know how to run it (Windows doesn''t > do shebang lines). So, if you move the ruby file with no extension out > of the bin folder, and adjust the batch file to point to it, it will > work. At least it works on my machine ;-).The issue with ''gem'' is that rubygems generates to files: gem and gem.bat which is a stub to execute the true gem ruby script. If we move from script+stub to scripts similar of what ruby ships (take a look at irb.bat) can workaround the issues with "cross-compatibility" raised of being using system/exec/backticks/whatever> > I like the idea and you aren''t oversimplifying it, I think Ruby is > > broken in that aspect. > > > > Maybe we can fix that damn thing in ruby C code and workaround that? > > > > win32/win32.c:989 (CreateChild function), call dln_find_exe > > dlc.c:1671 (dln_find_exe function), call dln_find_1 > > dlc.c:1796 (dln_find_1 function), call eaccess > > file.c:897 (eaccess function), call access API with ''path'' and mode > > X_OK (io.h) > > > > Maybe Ruby''s not totally broken here, just undocumented? Take a look > and let me know what you think. >Please let me congrats you Gordon, you just hit one of the big issues that blocks true cross-platform implementation. Also, these injected stubs/scripts can workaround the pipes issues for scripts using .rb and being tried to executed standalone (this is an old issue of ruby and how cmd creates the process and pass the information to the child process). With the fix for Vista the MinGW guys are working, maybe then we could add a ''gcc'' and ''make'' stubs that transparently load the mingw/msys environment for you :-D So the Developer Kit only will be the packages we already have (a bit of cleanup) and let it drop these stubs into ruby/bin :-) I love the idea! Thank you for taking the time exposing your point, It was really enlightening Regards, -- Luis Lavena Multimedia systems - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams
> The issue with ''gem'' is that rubygems generates to files: gem and > gem.bat which is a stub to execute the true gem ruby script. > > If we move from script+stub to scripts similar of what ruby ships > (take a look at irb.bat) can workaround the issues with > "cross-compatibility" raised of being using > system/exec/backticks/whatever >Ok, I''ve stubbed make, gcc, and sh, and rubygems tests pass. I''m working on a patch for rubygems bin scripts as well, but I can''t find a simple way to do it without breaking a lot of tests. I''ll keep at it. I did discover that the exit /b %errorlevel% doesn''t do the right thing when called in Ruby, so I''m just making the cmd files so that the last line is the one we care about.. This is my rake.cmd. It seems to be working fine. Let me know if you see any problems with it. @ECHO OFF goto endofruby #!c:/ruby/ruby_mingw/bin/ruby.exe # # This file was generated by RubyGems. # # The application ''rake'' is installed as part of a gem, and # this file is here to facilitate running it. # require ''rubygems'' version = ">= 0" if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then version = $1 ARGV.shift end gem ''rake'', version load ''rake'' __END__ :endofruby "%~d0%~p0ruby" -x "%~f0" %*> Please let me congrats you Gordon, you just hit one of the big issues > that blocks true cross-platform implementation. > > Also, these injected stubs/scripts can workaround the pipes issues for > scripts using .rb and being tried to executed standalone (this is an > old issue of ruby and how cmd creates the process and pass the > information to the child process).Yeah, I''ve been bitten by this one too :-). The only problem with batch files is that it makes ctrl-c cause the extremely annoying "Terminate batch job (Y/N)?". I did find a way to modify cmd.exe so that it doesn''t do this, but it would be nice if there was a cleaner solution.
On Mon, Apr 21, 2008 at 11:03 PM, Gordon Thiesfeld <gthiesfeld at gmail.com> wrote:> > The issue with ''gem'' is that rubygems generates to files: gem and > > gem.bat which is a stub to execute the true gem ruby script. > > > > If we move from script+stub to scripts similar of what ruby ships > > (take a look at irb.bat) can workaround the issues with > > "cross-compatibility" raised of being using > > system/exec/backticks/whatever > > > > Ok, I''ve stubbed make, gcc, and sh, and rubygems tests pass. I''m > working on a patch for rubygems bin scripts as well, but I can''t find > a simple way to do it without breaking a lot of tests. I''ll keep at > it. I did discover that the exit /b %errorlevel% doesn''t do the right > thing when called in Ruby, so I''m just making the cmd files so that > the last line is the one we care about.. > > This is my rake.cmd. It seems to be working fine. Let me know if you > see any problems with it. > > @ECHO OFF > goto endofruby > #!c:/ruby/ruby_mingw/bin/ruby.exe > # > # This file was generated by RubyGems. > # > # The application ''rake'' is installed as part of a gem, and > # this file is here to facilitate running it. > # > > require ''rubygems'' > > version = ">= 0" > > if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then > version = $1 > ARGV.shift > end > > gem ''rake'', version > load ''rake'' > __END__ > :endofruby > "%~d0%~p0ruby" -x "%~f0" %* >I was one of the responsible of change the .cmd files to .bat in latest rubygems releases. Why? because .bat take precedence instead of .cmd, also because, even we don''t *support* win9x, at least it will work out of the box. I''ll change to uppercase all the references to windows/batch commands (GOTO). I''ll take a look at rubygems (again) to see if we can get rid of the separate batch/script files for the next release, or at least, our implementation, since I''m thinking seriously not ship 1.8.7... (there is a heated discussion about that). An alternative will be have a series of patches that can be rolled/applied to current 1.8.6-p114 and fix particular things in ruby source or rubygems until an official release ship them. As a RubyGems contributor I can slip these changes in, but we will require wait for the next release to make them available.> > Please let me congrats you Gordon, you just hit one of the big issues > > that blocks true cross-platform implementation. > > > > Also, these injected stubs/scripts can workaround the pipes issues for > > scripts using .rb and being tried to executed standalone (this is an > > old issue of ruby and how cmd creates the process and pass the > > information to the child process). > > Yeah, I''ve been bitten by this one too :-). The only problem with > batch files is that it makes ctrl-c cause the extremely annoying > "Terminate batch job (Y/N)?". I did find a way to modify cmd.exe so > that it doesn''t do this, but it would be nice if there was a cleaner > solution. >There is no easy solution without binary patching cmd.exe, since in-memory patches will require us alter ruby.exe and we can get caught by DEP protection :-P We need to talk about the the future on this, do you feel comfortable we set a wiki page and start drawing a picture about the goals for the release? Thank you for your time Gordon. -- Luis Lavena Multimedia systems - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams
On Mon, Apr 21, 2008 at 9:55 PM, Luis Lavena <luislavena at gmail.com> wrote:> We need to talk about the the future on this, do you feel comfortable > we set a wiki page and start drawing a picture about the goals for the > release? >This sounds like a great idea. I''ve been meaning to do some work on the wiki for a few weeks now. I should be on gtalk most of the day, too. Thanks, Gordon