Instiki devel - Aug 2005 - Instiki-ar tests pass, live instance running

Hey guys, I just wanted to let you know that I got Instiki running on
ActiveRecord.  All tests pass locally on postgresql, mysql, sqlite2,
and sqlite3.  I have an instance at
http://instiki-ar.techno-weenie.net:2533/ running on lighttpd and
sqlite.

First thing: I wasn''t able to make checkins.  My login doesn''t
seem to
work.  I have diff files at 5 milestones along the way, so hopefully I
can get a decent revision history in there.

All the non-AR code (chunks, diffs, etc) were virtually untouched. 
This allowed a portion of the unit tests to run with very little
changes.  As for the models themselves, I did my best to keep the same
interfaces.  Here are the areas I ran into difficulty:

WikiService - I changed this to a Wiki class with most of the same
functionality.  Instead of a persistent object, it gets recreated on
each request.  I had to make some slight changes to the controllers,
but nothing major.

Web boolean options - These get modeled differently in the DB, so I
had to use the ? method (published? count_pages? etc).

Model constructors - I changed the constructors to take an options
hash like other AR objects.  This wasn''t a big deal since the
controllers access methods like @wiki.read_page for everything.

Tests were a huge challenge.  I converted some of the setup methods
into fixtures, but this threw a lot of things off (counts, RSS feeds,
etc).  Most of my changes were adding pages that were nonexistent
before.  Another test issue had to do with madeleine.  Since the
objects were always in memory, modifying a page in the controller
would also modify the same object in the test.  This isn''t so with
ActiveRecord, so there are many places that I''m forcing the reload of
fixture data or has_many associations.

-- 
rick
http://techno-weenie.net

Rick Olson wrote:
>Hey guys, I just wanted to let you know that I got Instiki running on
>ActiveRecord.  
>
WOW!

Looks like in a week that I was yet again deprived of home internet 
things have moved :)
I''ll look at your password issue now.

Regards,
Alex

Rick Olson wrote:
>Hey guys, I just wanted to let you know that I got Instiki running on
>ActiveRecord.  All tests pass locally on postgresql, mysql, sqlite2,
>and sqlite3.  I have an instance at
>http://instiki-ar.techno-weenie.net:2533/ running on lighttpd and
>sqlite.
>  
>
I''ve finally had a chance to take it for a ride tonight. It works! I 
mean, wow, it works, man!!! :)))
See also the commit comment in http://dev.instiki.org/changeset/341 
(it''s actually by me, not Rick - as are a few others).

Now, before we unleash it on the unsuspecting world in a form of an 
official release, we need few things.

1. Functional testing.
  a) I''m going to update Watir test (test/watir/e2e) to work with SQL 
backend.
  b) I''ll don my tester hat on the weekend and try to break it
  c) after this, I''m going to ask people on instiki-users, rubyonrails 
and TextDrive support forum to check it out from SVN and give it a swing.

2. Performance
  It certainly looks good enough for a small wiki as it is. What we need 
is to see how it copes with a large one under load. I guess, the best 
way to handle that would be to take an export of rubyonrails.org, the 
largest public wiki out there, import it into instiki-ar and hit it with 
a lot of load. Statistically, with robots.txt in place more than 99% of 
page hits go to show/<random page> and the RSS feeds, so these are the 
two that need to be automated in vusers. All other pages are hit rarely, 
therefore can be tested manually. The only problem I have is a test box. 
Mine is a wee Windows XP laptop without lighttpd. So, I can do the 
import and write the vusers, and even test the Windows/Webrick scenario, 
but that''s not our target deployment environment, obviously. Can we do 
something better?

3. Packaging
Apart from actually coding the new backend (which seems to be over, 
kudos to Rick), that''s the most interesting issue. Baically, we need a 
build that will package it as a single download with the SQLite3 
binaries inside, for Windows, Linux and Mac. I guess, I knonw who will 
do it for Mac (Oleg, the same guy who made the 0.10.2 distribution). I 
might as well do it for the Windows. The question is: who is doing it 
for a Linux?

What else?

Alex

PS Speaking of robots.txt... would this be a useful thing to put in 
./public?
PPS Here it goes again: WOW, MAN!!!

> I''ve finally had a chance to take it for a ride tonight. It works!
I
> mean, wow, it works, man!!! :)))
> See also the commit comment in http://dev.instiki.org/changeset/341
> (it''s actually by me, not Rick - as are a few others).
> 
> Now, before we unleash it on the unsuspecting world in a form of an
> official release, we need few things.
> 
> 1. Functional testing.
>   a) I''m going to update Watir test (test/watir/e2e) to work with
SQL
> backend.
>   b) I''ll don my tester hat on the weekend and try to break it
>   c) after this, I''m going to ask people on instiki-users,
rubyonrails
> and TextDrive support forum to check it out from SVN and give it a swing.
I noticed that the PDF and export functions did not work on my instiki
installation on textdrive.  I haven''t had a chance to investigate
further though.
 
> 2. Performance
>   It certainly looks good enough for a small wiki as it is. What we need
> is to see how it copes with a large one under load. I guess, the best
> way to handle that would be to take an export of rubyonrails.org, the
> largest public wiki out there, import it into instiki-ar and hit it with
> a lot of load. Statistically, with robots.txt in place more than 99% of
> page hits go to show/<random page> and the RSS feeds, so these are
the
> two that need to be automated in vusers. All other pages are hit rarely,
> therefore can be tested manually. The only problem I have is a test box.
> Mine is a wee Windows XP laptop without lighttpd. So, I can do the
> import and write the vusers, and even test the Windows/Webrick scenario,
> but that''s not our target deployment environment, obviously. Can
we do
> something better?
I have a VC account on TextDrive so I can probably set it up with
SQLite and Lighttpd.  The RoR wiki would definitely qualify as a large
wiki too.  This brings up the issue of migration scripts.  If people
are going to upgrade to this, they''ll need a simple way to migrate
their wiki installation.  I have zero experience with madeleine, so I
have no clue how it would work.

I have a nagging feeling that performance won''t be too hot for
anything but a small site.  The "querying" is done with a PageSet
object that loops through every page and pulls out
WikiWords/Authors/etc.  This is fine since they''re in memory, but not
if it has to pull from a database.  I looked at some serialization
options (even got halfway to finishing the abandoned wrappings module
of AR), but it wasn''t working out for me.  I was thinking of more of
an SQL approach using tags.  When posts are saved they calculate the
wiki words and authors and save them as tags in some many <-> many
join.  This seems like the only efficient way to grab all pages in a
web by a certain author, for example.
 
> 3. Packaging
> Apart from actually coding the new backend (which seems to be over,
> kudos to Rick), that''s the most interesting issue. Baically, we
need a
> build that will package it as a single download with the SQLite3
> binaries inside, for Windows, Linux and Mac. I guess, I knonw who will
> do it for Mac (Oleg, the same guy who made the 0.10.2 distribution). I
> might as well do it for the Windows. The question is: who is doing it
> for a Linux?
Hmm, not my area of expertise.  I''m sure there''s someone that
can help
out with the linux side of the house.
 
> What else?
> 
> Alex
> 
> PS Speaking of robots.txt... would this be a useful thing to put in
> ./public?
> PPS Here it goes again: WOW, MAN!!!
> 
> _______________________________________________
> Instiki-devel mailing list
> Instiki-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/instiki-devel
> 

-- 
rick
http://techno-weenie.net

Rick Olson wrote:
>>I''ve finally had a chance to take it for a ride tonight. It
works! I
>>mean, wow, it works, man!!! :)))
>>See also the commit comment in http://dev.instiki.org/changeset/341
>>(it''s actually by me, not Rick - as are a few others).
>>
>>Now, before we unleash it on the unsuspecting world in a form of an
>>official release, we need few things.
>>
>>1. Functional testing.
>>  a) I''m going to update Watir test (test/watir/e2e) to work
with SQL
>>backend.
>>  b) I''ll don my tester hat on the weekend and try to break it
>>  c) after this, I''m going to ask people on instiki-users,
rubyonrails
>>and TextDrive support forum to check it out from SVN and give it a
swing.
>>    
>>
>
>I noticed that the PDF and export functions did not work on my instiki
>installation on textdrive.  I haven''t had a chance to investigate
>further though.
> 
>  
>
>>2. Performance
>>  It certainly looks good enough for a small wiki as it is. What we need
>>is to see how it copes with a large one under load. I guess, the best
>>way to handle that would be to take an export of rubyonrails.org, the
>>largest public wiki out there, import it into instiki-ar and hit it with
>>a lot of load. Statistically, with robots.txt in place more than 99% of
>>page hits go to show/<random page> and the RSS feeds, so these are
the
>>two that need to be automated in vusers. All other pages are hit rarely,
>>therefore can be tested manually. The only problem I have is a test box.
>>Mine is a wee Windows XP laptop without lighttpd. So, I can do the
>>import and write the vusers, and even test the Windows/Webrick scenario,
>>but that''s not our target deployment environment, obviously.
Can we do
>>something better?
>>    
>>
>
>I have a VC account on TextDrive so I can probably set it up with
>SQLite and Lighttpd.
>
That I have as well... not sure if it''s a good idea to hit TextDrive 
with a performance test though... :)
OK, I think we can do most of our testing/tuning on our own PCs, and 
only then do in on a TextDrive, for a short time, with realistic load 
rather than stress conditions.
>  The RoR wiki would definitely qualify as a large
>wiki too.  This brings up the issue of migration scripts.  If people
>are going to upgrade to this, they''ll need a simple way to migrate
>their wiki installation.  I have zero experience with madeleine, so I
>have no clue how it would work.
>  
>
As you may imagine, I can handle that with ease... and pleasure :)
 
>I have a nagging feeling that performance won''t be too hot for
>anything but a small site... When posts are saved they calculate the
>wiki words and authors and save them as tags in some many <-> many
>join.
>  
>
Yeah, I don''t even need a performance test to predict this much. Page 
must be rendered on save, and any local wiki hyperlinks must be stored 
in a separate table. Besides, chunks and markup rendering is quite slow 
itself (typically, hundreds of milliseconds per page). I haven''t had a 
good look at this part yet, but as far as I understand we are not 
caching the HTML representation in any way yet, are we? It might be a 
good idea to store it in DB with a revision.

So, can you make this happen, while I''m toiling away on the 
testing/migration/packaging part?

Alex

On Wed, Aug 10, 2005, Alex Verhovsky wrote:
> 2. Performance
>  It certainly looks good enough for a small wiki as it is. What we
> need is to see how it copes with a large one under load. I guess, the
> best way to handle that would be to take an export of rubyonrails.org,
> the largest public wiki out there, import it into instiki-ar and hit
> it with a lot of load. Statistically, with robots.txt in place more
> than 99% of page hits go to show/<random page> and the RSS feeds,
> so these are the two that need to be automated in vusers. All other
> pages are hit rarely, therefore can be tested manually. The only
> problem I have is a test box.  Mine is a wee Windows XP laptop without
> lighttpd. So, I can do the import and write the vusers, and even test
> the Windows/Webrick scenario, but that''s not our target deployment
> environment, obviously. Can we do something better?
I''ve got access to a few fairly beefy linux boxes (at home, a 1ghz
celeron, at work a dual 3ghz xeon :D) where I can do some load
testing.  If we can get a dump of the rails wiki, I can import it there
and use various benchmarking tools to try to get a feel for loads.

We should try to figure out a load testing strategy.  Use
script/benchmarker?  I don''t know.

That brings me to another point that I''ve noticed.  When Instiki made
the shift to be rails-based, it was sort of a stripped-down version of
rails.  There are a bunch of scripts missing, directories that didn''t
exist, and the like.  The dispatchers didn''t exist, etc.

Rick created the log directory which went a long way, but I''ve been
wondering if it would help to go the rest of the way?  Add the scripts,
add the dispatchers, and just generally railsify it the rest of the way.

Along the same lines, shebang lines where they exist are horrifically
inconsistent.  I propose we convert them all to the (relatively)
standard #!/usr/bin/env ruby, which''ll work just about everywhere but
windows.  Sorry, Alex :)

I''d be happy to do that, I don''t feel like I''ve
contributed enough yet.
> 3. Packaging
> Apart from actually coding the new backend (which seems to be over,
> kudos to Rick), that''s the most interesting issue. Baically, we
need
> a build that will package it as a single download with the SQLite3
> binaries inside, for Windows, Linux and Mac. I guess, I knonw who will
> do it for Mac (Oleg, the same guy who made the 0.10.2 distribution). I
> might as well do it for the Windows. The question is: who is doing it
> for a Linux?
Well, we''ve already got the old gem.  I can get that updated to make
sure it''s everything we need.  I could also see about building a Debian
package (and then maybe Ubuntu?).  That''s the only linux I''ve
got
around.
> What else?
Speaking of Linux, I was bitching to Rick the other night about
something I noticed in the code.  DOS line endings.  Everywhere.  I have
my vim specially set up to show me all sorts of whitespace problems and
I noticed a lot.  Would you guys be opposed to The Mother Of All
Whitespace Patches, fixing the line endings, trailing spaces, and all
that crap?

Migration script.  I want to do this :)  I''ll start looking at it
today.

I''ve heard lots of people worrying about the new AR Instiki losing its
three-step installation.  I''m not sure how feasible this is going to
be, but I''d like to attempt to write a Madeleine adapter for AR.  If it
works, we could ship with the app set up for it by default and people
could continue to drop the files into place and fire up the wiki with
minimum effort.
> PS Speaking of robots.txt... would this be a useful thing to put in
> ./public?  PPS Here it goes again: WOW, MAN!!!
I think so.  Also, I agree.  WOW :)

Ben

On Wed, Aug 10, 2005, Rick Olson wrote:
> I have a nagging feeling that performance won''t be too hot for
> anything but a small site.  The "querying" is done with a PageSet
> object that loops through every page and pulls out
> WikiWords/Authors/etc.  This is fine since they''re in memory, but
not
> if it has to pull from a database.  I looked at some serialization
> options (even got halfway to finishing the abandoned wrappings module
> of AR), but it wasn''t working out for me.  I was thinking of more
of
> an SQL approach using tags.  When posts are saved they calculate the
> wiki words and authors and save them as tags in some many <-> many
> join.  This seems like the only efficient way to grab all pages in a
> web by a certain author, for example.
My own nagging feeling is that performance improvements are going to
require some significant refactoring.  Right now it seems that you''ve
successfully converted the Madeleine-based logic to the database, in my
mind the task is now to convert that to database logic.

Typo does exactly what Alex suggests, when you save a page it renders
it into another column in the database.  This seems like a good
optimization to me... you only save a page once, but it''s viewed a lot.
We could either keep rendered versions of all revisions, or just accept
a performance hit when looking at old versions.  It all depends on
whether or not database size is a concern.  I''m really curious how big
the database for the Rails wiki (for instance) is.

I agree with your tags idea, too.  Best to have a highly indexed and
optimized table with pointers to linked pages.

Ben

Ben Bleything wrote:
>I''ve got access to a few fairly beefy linux boxes (at home, a 1ghz
>celeron, at work a dual 3ghz xeon :D) where I can do some load
>testing.  If we can get a dump of the rails wiki, I can import it there
>and use various benchmarking tools to try to get a feel for loads.
>
>We should try to figure out a load testing strategy.  Use
>script/benchmarker?  I don''t know.
>  
>
I''ll try to write a little something that simulates load over weekend. 
It will be in Ruby, so everybody will be able to run it.
>That brings me to another point that I''ve noticed.  When Instiki
made
>the shift to be rails-based, it was sort of a stripped-down version of
>rails.  There are a bunch of scripts missing, directories that
didn''t
>exist, and the like.  The dispatchers didn''t exist, etc.
>
>Rick created the log directory which went a long way, but I''ve been
>wondering if it would help to go the rest of the way?  Add the scripts,
>add the dispatchers, and just generally railsify it the rest of the way.
>  
>
Yes, please do that.
>Along the same lines, shebang lines where they exist are horrifically
>inconsistent.  I propose we convert them all to the (relatively)
>standard #!/usr/bin/env ruby, which''ll work just about everywhere
but
>windows.  Sorry, Alex :)
>  
>
I don''t care, because I don''t need shebang on Windows - I
simply type
''instiki'', and it executes ''ruby
instiki.rb''.

Re shebangs and end of lines, please do that. EOLs should be dealt with 
through SVN properties, although I don''t remember the exact way.
>Well, we''ve already got the old gem.  I can get that updated to
make
>sure it''s everything we need.
>
I don''t think there is currently a way to get SQLite via a gem (you can
get the driver, but not the database engine, i.e. the shared library, 
itself). So, I''ve been thinking in terms of placing SQLite binaries
into

lib/sqlite/win
lib/sqlite/linux
lib/sqlite/osx

or something like that.
>  I could also see about building a Debian
>package (and then maybe Ubuntu?).  That''s the only linux
I''ve got
>around.
>  
>
If you are willing to maintain it for the next couple of years. Which 
means not just building it for new versions, but also responding to 
people wth problems. Personally, I wouldn''t bother because any Debian 
user is tech-savvy enough to use other packaging options.
>Migration script.  I want to do this :)  I''ll start looking at it
today.
>  
>
Aargh... ok, do that :)
>I''ve heard lots of people worrying about the new AR Instiki losing
its
>three-step installation.  I''m not sure how feasible this is going
to
>be, but I''d like to attempt to write a Madeleine adapter for AR.
>
The biggest problem with Madeleine is (a) version compatibility and (b) 
it stiffles new development. It just takes away too much time. I like to 
write software, not crawl for hours around ObjectSpace. It''s not funny.
So, that''s a negative. Unless SQLite somehow fails to fill the "No
Step
Three" ticket, but why would it?

Alex

Ben Bleything wrote:
>I''m really curious how big
>the database for the Rails wiki (for instance) is.
>  
>
Oh, it is just a few hundred pages of plain text, how big could that be? 
We are certainly not talking terrabytes here.
To be more specific, it is in tens of megabytes.

Alex

On Wed, Aug 10, 2005, Alex Verhovsky wrote:
> >Rick created the log directory which went a long way, but I''ve
been
> >wondering if it would help to go the rest of the way?  Add the
> >scripts, add the dispatchers, and just generally railsify it the rest
> >of the way.
>
> Yes, please do that.
Will do.
> >Along the same lines, shebang lines where they exist are horrifically
> >inconsistent.  I propose we convert them all to the (relatively)
> >standard #!/usr/bin/env ruby, which''ll work just about
everywhere but
> >windows.  Sorry, Alex :)
>
> I don''t care, because I don''t need shebang on Windows - I
simply type
> ''instiki'', and it executes ''ruby
instiki.rb''.
>
> Re shebangs and end of lines, please do that. EOLs should be dealt
> with through SVN properties, although I don''t remember the exact
way.
I''ll figure it out.  I''m not sure whether svn will actually
convert
line-endings on checkout/checkin, but I''ll check it out.
> I don''t think there is currently a way to get SQLite via a gem
(you
> can get the driver, but not the database engine, i.e. the shared
> library, itself). So, I''ve been thinking in terms of placing
SQLite
> binaries into
>
> lib/sqlite/win
> lib/sqlite/linux
> lib/sqlite/osx
>
> or something like that.
You mean like shipping the statically-linked sqlite binaries with
Instiki packages?  That sounds pretty good to me, I didn''t even think
of
it as an option.

I''m sure we can find a way to include that binary in the gem.  gems are
just fancy zipfiles with metadata, aren''t they?
> > I could also see about building a Debian package (and then maybe
> >Ubuntu?).  That''s the only linux I''ve got around.
>
> If you are willing to maintain it for the next couple of years. Which
> means not just building it for new versions, but also responding to
> people wth problems. Personally, I wouldn''t bother because any
Debian
> user is tech-savvy enough to use other packaging options.
Alright.  I might do this later anyway, just to say I maintain a debian
package ;)  But for now, cool.
> >Migration script.  I want to do this :) I''ll start looking at
it
> >today.
>
> Aargh... ok, do that :)
You''re welcome to if you want!  I just wanted to step up a little bit
more :)  You''d get through it faster than I would for sure.  Either way
is fine, just let me know.
> The biggest problem with Madeleine is (a) version compatibility and
> (b) it stiffles new development. It just takes away too much time. I
> like to write software, not crawl for hours around ObjectSpace.
It''s
> not funny.  So, that''s a negative. Unless SQLite somehow fails to
fill
> the "No Step Three" ticket, but why would it?
Well, I was thinking that it would require people to install SQLite, but
then (with prompting) I remembered they ship binaries.  So that''s a
plus.  Maybe won''t be needed.

Ben

Ben Bleything wrote:
>>>Migration script.  I want to do this :) I''ll start looking
at it
>>>today.
>>>      
>>>
>>Aargh... ok, do that :)
>>    
>>
>
>You''re welcome to if you want!  I just wanted to step up a little
bit
>more :)  You''d get through it faster than I would for sure.  Either
way
>is fine, just let me know.
>  
>
You do it. I will just focus on QA then. And maintaining the 
instiki.org, too. Warning: if we evr meet in person, never say anything 
about Texas, poker and holdemup to me, please. It would be dangerous for 
your life.

SQLite does ship precompiled binaries for Linux and Win. Not for Mac though.

Alex.