wxruby development - Oct 2006 - wxRuby-0.6.0, wxRuby2, SWIG and....

Hi, my first post here so a little background first...

I started out using wxRuby-0.6.0 about 4 months ago while trying to get a grip 
on Ruby as a means of writing a little budgetting app I''ve been meaning
to
write for a couple of years. As I didn''t like the way that the wxruby 
examples are coded (having to punch the individual C++ api methods that way 
isn''t nice) I started work on some ruby/wxruby integration stuff
originally
based on the rubygtk library. As I got deeper into this work a few issues 
came up that required me to look at the wxruby C++ wrapper and to have a 
cursory look at the progress of wxRuby2. wxRuby2 seems to have a work rate 
issue (compared to progress on the original) that I wouldn''t expect to
exist
when using something like SWIG - I may be wrong and you think SWIG is the 
best thing since sliced bread - but glancing (and I do mean glancing - this 
isn''t something I''ve investigated even cursorily - thats what
this email is
for) SWIG seems to be a mixed bag of really good and really really bad 
(impression gained from the doc on how to add a class to wxruby2).

So at this point I was wondering whether my time would be better spent waiting 
for wxRuby2 to progress somewhat and contribute code, fix 0.6.0 or to have a 
go at something else entirely... which is what I did.

Being a C++ programmer by trade I''ve written a slew of templates to 
essentially do what SWIG is meant to do without the automated nature of SWIG. 
As wxRuby is of immediate concern to me I''m using it as a testbed for
this
C++/Ruby wrapper replacement and I''d like to know if you think there is
any
future where you might find it more useful than SWIG for what you are trying 
to do. I''m thinking that my template wrappers will have a more linear 
application curve (ie not as good as SWIG at its best but nowhere near as bad 
as SWIG in its worse case scenario).

My apologies if I have misread what has been said regarding the ease with 
which SWIG does its job and if thats so just put me straight right away...

I''ve attached button.cpp and button.h which are my current drop-in 
replacements for the original wxRuby-0.6.0 files of the same name. I may post 
the code inline if there is an issue with attachments on this mailing list.

I haven''t made any effort yet to remove the necessity for the init0
functions.
The MapRubyObjectToCppObject is there at the moment for compatibility with 
the rest of the code base and the init/constructor functions will merge soon. 
I''m fairly sure I can make the instance method definitions disappear 
altogether when the wrapper is along the same simplistic lines of these. 

I''m not going to post the template wrappers anywhere for a few weeks
yet - the
code will be released as a new project on rubyforge barring any unobvious 
show stoppers.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: button.cpp
Type: text/x-c++src
Size: 3424 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/wxruby-development/attachments/20061016/76fb988e/attachment.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: button.h
Type: text/x-c++hdr
Size: 909 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/wxruby-development/attachments/20061016/76fb988e/attachment-0001.bin

Cabal wrote:
> Hi, my first post here so a little background first...
Hello Cabal,

Are you saying that the button.cpp and button.h files were generated 
automatically or that you used some templating mechanism you came up 
with to create them?

SWIG is a mixed bag sometimes.  The learning curve to implement a 
project with the complexity of wxWidgets is non-trivial.  However, I do 
feel that with wxRuby2 we''ve already gotten over the major hurdles 
presented.  If you look at the recent class additions you''d see that 
they (almost always) consisted of only a .i (SWIG interface) file of 
about 10 lines.  Each .i file is really just a copy of an existing .i 
file with the names changed.

Having said that, if you have a solution to wrapping libraries that''s 
specific to Ruby and can deal with:

- overriding C++ methods from Ruby
- pointer arguments that could be input, output or both
- complex argument and return types
- conversion of arguments from ruby types to C/C++ types
- collapsing/expanding of arguments (e.g.  ruby array to int n, char ** 
lines)
- memory management

Then I''d say you''re really on to something.

At this point what I think we really need is assistance with wxRuby2.  
wxRuby 0.6.0 is a dead-end project now.  We''ve officially stopped all 
support for it and closed all open issues associated with it.

Roy

Hi Roy - I''ll try and answer your questions inline...

On Monday 16 October 2006 16:36, you wrote:
> Cabal wrote:
> > Hi, my first post here so a little background first...
> Hello Cabal,
> 
> Are you saying that the button.cpp and button.h files were generated 
> automatically or that you used some templating mechanism you came up 
> with to create them?
The button.* files supplied are hand written - Button was the class I chose to 
get the wrapper up and going fairly easily. I don''t intend attempting
to
write any auto-generation script at this point, I''m still extending the
functionality to cover various use cases as I come across them. At the moment 
I''m proceeding by hijacking the the 0.6.0 files and replacing the code
for
each function to make use of the wrappers.
> 
> SWIG is a mixed bag sometimes.  The learning curve to implement a 
> project with the complexity of wxWidgets is non-trivial.  However, I do 
> feel that with wxRuby2 we''ve already gotten over the major hurdles
> presented.  If you look at the recent class additions you''d see
that
> they (almost always) consisted of only a .i (SWIG interface) file of 
> about 10 lines.  Each .i file is really just a copy of an existing .i 
> file with the names changed.
I''m glad to hear you''ve wrestled SWIG into submission :)
> 
> Having said that, if you have a solution to wrapping libraries
that''s
> specific to Ruby and can deal with:
> 
> - overriding C++ methods from Ruby
Not sure I understand this one. Do you mean just being able to call the C++ 
methods from Ruby? Once ruby knows how to call a method then
''overriding''
would be an ability handled by ruby in some derived
class...
> - pointer arguments that could be input, output or both
I don''t see any issues with handling pointers (not at this point
anyway). I''ve
got a few specialisations on wx pointer types already.
> - complex argument and return types
I presume you''re talking about non-pod datatypes and references and
pointers
to them. No issues upto now for wx types passed as arguments and return 
values.
> - conversion of arguments from ruby types to C/C++ types
One of the easier bits.
> - collapsing/expanding of arguments (e.g.  ruby array to int n, char ** 
> lines)
At the minute I''m concentrating on passing args using ''int
argc, VALUE* argv''
method signatures and upto 10 VALUE arguments. From the existing code that 
I''ve looked at I haven''t seen much cause for expanding and
collapsing ruby
arrays as a separate operation per se. 
> - memory management
I''ve specialised the templates for wx objects to link memory management
into
the wxObjectRefData class. The wrapper will require some sort of help along 
these lines whichever C++ library it is used for - but I can''t see that
being
any different for any other method of wrapping a
library.
> 
> Then I''d say you''re really on to something.
> 
> At this point what I think we really need is assistance with wxRuby2.  
> wxRuby 0.6.0 is a dead-end project now.  We''ve officially stopped
all
> support for it and closed all open issues associated with it.
Yes - I''d got that impression :) Unfortunately wxRuby2 didn''t
look like it was
going to be released in a usable form anytime soon (now rectified, 
possibly :) ) and now I''ve got my teeth into this I''m keen to
see where it
leads. I hope it will prove useful somewhere along the lines to wxRuby2 (as 
you''ve kindly provided me with a testbed and copious examples in 0.6.0)
but I
won''t go bugging you about it :)


Good luck and happy coding...

Cabal

Cabal wrote:
> Hi Roy - I''ll try and answer your questions inline...
>   
Thanks for the response.
>> - overriding C++ methods from Ruby
>>     
> Not sure I understand this one. Do you mean just being able to call the C++
> methods from Ruby? Once ruby knows how to call a method then
''overriding''
> would be an ability handled by ruby in some derived class...
>   
I meant does your wrapper handle overriding virtual C++ functions with 
Ruby functions.  As in, can I replace the default C++ behavior with Ruby 
and call the parent with super().

I also forgot to ask if your method maintains the object heirarchy.

Roy

Hi

Cabal wrote:
> Hi, my first post here so a little background first...
>   
Welcome, thanks for your interest in wxRuby.
> wxRuby2 seems to have a work rate 
> issue (compared to progress on the original) that I wouldn''t
expect to exist
> when using something like SWIG
You''re right that wxRuby2 progress has not been that even. But
that''s
not mainly because of SWIG - it''s primarily the simple reason that
there
has been a shortage of people with time and willingness to work on the 
project.

At the moment we have a core team of four developers, plust about the 
same number again submitting occasional patches. If you look at the last 
three to four months on the m.l  and commits, we''ve made sustained and 
rapid progress: around 40 new classes, squashed some tricky bugs, plus 
the additional but important work associated with a project, such as 
documentation and packaging.  SWIG plus our Kevin''s early work moulding
SWIG to our needs has been key to porting a lot of tricky functionality 
in that time.
>  - I may be wrong and you think SWIG is the 
> best thing since sliced bread - 
Hehe - I doubt any of the wxruby2 developers would think that! SWIG is a 
very mixed blessing. You aren''t the first person to question its use
for
wxruby. But the consensus probably is that its the the best available 
tool, under active development and with a future.
> but glancing (and I do mean glancing - this 
> isn''t something I''ve investigated even cursorily - thats
what this email is
> for) SWIG seems to be a mixed bag of really good and really really bad 
>   
True. The way it parses header files to automatically wrap methods is 
slick. Typemaps are very important for us - but the way they are 
declared and used is horrible. Directors are also really important - but 
completely undocumented for Ruby.

SWIG-ruby is also pretty immature, and using it on wxWidgets has tested 
its limits - the size of the library, plus the typical difficulties of 
GUI programming (highly platform specific; long-running; event-driven in 
unpredictable ways).
> I''d like to know if you think there is any 
> future where you might find it more useful than SWIG for what you are
trying
> to do.
Thanks for posting the sample code for button.cpp. To be honest, I
can''t
see us switching from SWIG for the upcoming version 2. There would have 
to be a compelling reason to switch in the future.

Roy has given you some specific feedback on C++ constructs that might be 
challenging. My feedback would be: I don''t know C, or C++. Your
solution
seems geared to people who want to simplify and reduce repetition in 
writing wrapper code. But it doesn''t help those who don''t know
or want
to know C++. It just seems more of the macro/template madness that makes 
C++, to me, such an unappealing language to learn (SWIG is like this too).

Anyway, overall, I think there is definitely still space out there for a 
very well-designed, easy to use, library wrapping tool for Ruby. It 
would be more than just us interested in such a tool. But, to be a 
compelling alternative, it would have to enable wrapping work to be done 
*in* Ruby. Kevin''s suggested before that a more useful project might be
to develop a higher-level Ruby-based abstraction on top of SWIG (or your 
own work), and I agree. Your potential user base would be much wider: 
all Ruby programmers, not just the limited set of Ruby programmers who 
are also keen C++ programmers.

If your main goal is to get a pretty complete WxRuby pretty soon, I''d 
say you''re definitely best off contributing to this project. We have 
roughly 90% of the API ported (not including C++ classes that are 
redundant in Ruby), and an active team to share the work. We''ll be
happy
to help explain how the SWIGgy mess works.

If you have an itch to scratch about C++ wrapping in general, and don''t
want to build on SWIG, then it''s probalby not for you.

Either way, thanks again for your interest, keep in touch, and good luck.

alex

> My understanding (which may be flawed!) of your code is that you have a 
> one-way wrapper that only allows Ruby to call into C++, not to allow C++ 
> to call into Ruby.
> 
Roy, 

Thnx for the clarification - you are indeed correct. I have made no attempt or 
given any thought to someone wanting to get execution going the other way. At 
this stage I''m attempting a straight forward wrapper to enable Ruby
access to
functionality in C/C++ libs. 
Do you have any examples or projects where what you are suggesting has been 
done systematically (wxRuby2 perhaps?) - I wouldn''t mind getting a
heads up
on the techniques involved. While it would be something to add after initial 
release, it would be a good idea to factor in requirements right now. Though 
having said that, if you say you get this for free from SWIG I have a feeling 
I''m going to be working on this for the next week or so...

Cabal

Cabal wrote:
> Do you have any examples or projects where what you are suggesting has been
> done systematically (wxRuby2 perhaps?)
WxRuby2 is one. We often want to inherit from the core classes but have 
them call Ruby methods.

For example TreeCtrl implements a method ''on_compare_items'' to
control
the sort order. Users can define this method in a Ruby subclass to 
customise how items in the tree are sorted. The library will pick this 
up and call the ruby subclass method from the C++
internals.
>  I wouldn''t mind getting a heads up 
> on the techniques involved. 
Read up on ''directors'' in the SWIG docs. There''s
little on Ruby
specifically, but the python page should be helpful:

http://www.swig.org/Doc1.3/Python.html (seems down at the moment)

cheers
alex

Alex Fenton wrote:
> WxRuby2 is one. We often want to inherit from the core classes but have 
> them call Ruby methods.
Yes. Generically there are two cases where C++ code needs to call into 
Ruby code:

1. Overriding, especially of pure virtual methods. In wx, these are 
mostly the on_xxx methods. If you can''t override App#on_init, you
can''t
really write a wx app.

2. Events. When something like a button is pressed, an event handler is 
called, and the user is going to want to write some Ruby code to respond 
to that event.

Last time I checked, JRuby (Ruby running in a Java JVM) does not have 
this ability, which (as far as I can tell) makes it impossible to write 
GUI apps. They are working on it.

In SWIG, directors do in fact give this to you "for free". You can
look
at the generated code for something like Wx::Button to see how directors 
are implemented. It''s mostly straightforward, except that any call 
coming in to a director might be coming from the C++ side (in which case 
it needs to be sent to the Ruby code), or from the Ruby subclass code 
(in which case it needs to be forwarded up to the C++ code).

They keep a boolean state to manage this, and we have had some problems 
where it does not work correctly, usually due to discrepancies between 
our declarations of the inheritance hierarchy vs. what wx actually has 
*on that platform* (the inheritance chain varies from platform to platform).

Kevin

Cabal wrote:
> Hi, my first post here so a little background first...
> 
> wxRuby2 seems to have a work rate 
> issue (compared to progress on the original) that I wouldn''t
expect to exist
> when using something like SWIG - I may be wrong and you think SWIG is the 
> best thing since sliced bread - but glancing (and I do mean glancing - this
> isn''t something I''ve investigated even cursorily - thats
what this email is
> for) SWIG seems to be a mixed bag of really good and really really bad 
> (impression gained from the doc on how to add a class to wxruby2).
Hi, and thanks for sharing your thoughts and ideas.

A little background on my experience: The first Ruby/C++ wrapper was 
Ruby/FLTK. I co-designed and wrote it, and we mostly used C macros. Not 
as elegant as C++ templates, but similar in concept, I think. By the 
time I left that project to switch to wx, it was clear to me that any 
practical solution would require code generation. Hand-generating code 
for 10,000 methods was not only time-consuming, but (more critically) 
error-prone.

(Side note: I still like many things about FLTK, and think it is a great 
toolkit for certain applications. I switched to wx because of its native 
widgets, because I want my apps to be accessible to blind users.)

My next project was wxruby, and as you have seen from the 0.6.0 code 
base, I created a simple wrapper-generator script. When I started, the 
script was very small and simple, and it was easy to get the first few 
classes wrapped. However, over time, the script grew and grew. Each new 
class, and each new category of method signature required changes to the 
script.

It was hard for anyone other than me to add support for new classes. 
People were not motivated to learn about my script, because that 
knowledge would not help them on any other project. Method signature 
tweaks (such as int, int* -> Array) had to be hand-coded. Callbacks and 
virtual method overrides had to be hand-coded. Progress was slow.

Around that time SWIG added Director support for Ruby, so when I 
re-evaluated SWIG it was clear to me that it could do what we needed. 
Thus, wxruby2 was born. It took far fewer person-hours to get wxruby2 to 
its current state than it took to get wxruby 0.6.0 to its state. Adding 
new classes is usually *much* easier with SWIG, and is rarely if ever 
more difficult than it would have been with 0.6.0.

Contributors to wxruby2 can bring any existing SWIG knowledge with them, 
and can take SWIG knowledge with them to their next project. From a 
personal standpoint, when a defacto standard (like SWIG) is "good 
enough", there have to be compelling reasons not to use it.

So that''s my story. Code generation was mandatory, and SWIG was the 
easiest way to get there.
> 
> As wxRuby is of immediate concern to me I''m using it as a testbed
for this
> C++/Ruby wrapper replacement and I''d like to know if you think
there is any
> future where you might find it more useful than SWIG for what you are
trying
> to do. I''m thinking that my template wrappers will have a more
linear
> application curve (ie not as good as SWIG at its best but nowhere near as
bad
> as SWIG in its worse case scenario).
Honestly, I think your approach might have value for smaller libraries, 
but I would be very surprised if you were able to match wxruby2''s 
current functionality any time soon. I remain open, however, and want to 
use the best technology for wxruby, and would be willing to switch if 
you could make a compelling case.

(Side note: A co-worker is writing an RPC-like library that allows code 
from different languages to easily talk to each other. He has posed the 
possibility of creating a "wxruby" that is implemented by calling 
wxpython code. Instead of wrapping C++, we would wrap python in ruby. I 
have considered it, and if wxruby2 had not been started, I would 
probably prototype it to see if it would be a viable option.)
> My apologies if I have misread what has been said regarding the ease with 
> which SWIG does its job and if thats so just put me straight right away...
It''s probably not as bad as you think. We have hit a few SWIG bugs, and
most of those have been fixed. Some SWIG features are really hard to 
understand how to use, but once we learn them, they are easy to apply. 
Many of our remaining problems are due more to the cross-platform 
variations of wx than to SWIG itself.
> 
> I''ve attached button.cpp and button.h which are my current drop-in
> replacements for the original wxRuby-0.6.0 files of the same name. I may
post
> the code inline if there is an issue with attachments on this mailing list.
Your code looks fine. If you can address the issues Roy mentioned, and 
at some point auto-generate everything, it could be a cool tool.

Kevin

>> Having said that, if you have a solution to wrapping libraries
that''s
>> specific to Ruby and can deal with:
>>
>> - overriding C++ methods from Ruby
> Not sure I understand this one. Do you mean just being able to call the C++
> methods from Ruby? Once ruby knows how to call a method then
''overriding''
> would be an ability handled by ruby in some derived class...
We covered this one in other emails.
>> - pointer arguments that could be input, output or both
> I don''t see any issues with handling pointers (not at this point
anyway). I''ve
> got a few specialisations on wx pointer types already.
"Output pointers" generally need to be converted to ruby return
values.
SWIG allows use to describe the pattern and it generates the code 
automatically.
>> - complex argument and return types
> I presume you''re talking about non-pod datatypes and references
and pointers
> to them. No issues upto now for wx types passed as arguments and return 
> values.
>> - conversion of arguments from ruby types to C/C++ types
> One of the easier bits.
Here again, SWIG allows us to declare once that String<->wxString, etc. 
You''ll definitely want to avoid having to hand-code each case.
>> - collapsing/expanding of arguments (e.g.  ruby array to int n, char **
>> lines)
> At the minute I''m concentrating on passing args using
''int argc, VALUE* argv''
> method signatures and upto 10 VALUE arguments. From the existing code that 
> I''ve looked at I haven''t seen much cause for expanding
and collapsing ruby
> arrays as a separate operation per se. 
I''m not sure what you mean by this. From ruby, if you have an array,
you
certainly don''t want to pass both the array length and the array
itself.
In 0.6.0 we hand-coded each case to only take the array and not also the 
length. That''s probably what you are doing as well. SWIG allows us to 
declare the pattern once, and it will fold them for us.
>> - memory management
> I''ve specialised the templates for wx objects to link memory
management into
> the wxObjectRefData class. The wrapper will require some sort of help along
> these lines whichever C++ library it is used for - but I can''t see
that being
> any different for any other method of wrapping a library.
There are some tricky issues with ownership. As you know, a new object 
might be created in response to a ruby "new" call, or might have been 
created by the C++ library and wrapped so it can be used in ruby. Later, 
the ruby code might want to dispose of the object, in which case C++ 
needs to be notified. Or C++ might dispose of the objects on its own, in 
which case the Ruby object needs to get disconnected. We have started 
addressing this in wxruby2, thanks to some SWIG features that are hard 
to understand, but which do all the hard work for us.

It''s an issue we did not really solve in wxruby 0.6.0, and was one of 
the reasons we switched to SWIG.
> Yes - I''d got that impression :) Unfortunately wxRuby2
didn''t look like it was
> going to be released in a usable form anytime soon (now rectified, 
> possibly :) ) 
The pace of development of wxruby2 has been frustrating. It has been 
"pretty close" for a very long time, but I (and others) just
haven''t had
enough time to work on it. Fortunately, Alex and Roy have stepped in and 
have done most of the work for the last few months, bringing us to our 
first alpha release.
> and now I''ve got my teeth into this I''m keen to see where
it
> leads. I hope it will prove useful somewhere along the lines to wxRuby2 (as
> you''ve kindly provided me with a testbed and copious examples in
0.6.0) but I
> won''t go bugging you about it :)
Understood. I hope we have helped you, and that your project will end up 
being useful, whether to us or to someone else.

Paraphrasing a famous quote: "SWIG is the worst wrapping tool...except 
for all the others." None of us love SWIG, but it is far better than 
anything else I have seen so far. If you can provide an easier way for 
ruby folk to wrap C++ libraries, I would be very happy.
> Good luck and happy coding...
And to you.

Kevin