webgen users - Mar 2007 - Webgen and incremental generation (nukumi2 did it)

> There is already a plugin in the feature request tracker (created by
> Fritz Heinrichmeyer) which provides blogging and rss support
> (although for webgen 0.3.8). Based on this and comments of other
> people I have taken notes so as to what one would expect from such a
> plugin and when I have enough time I will post them to this mailing
> list for further discussion!
I add one feature request:

How about incremental generation?
On my Powerbook G4, webgen takes 4 seconds for generating a site with
32 pages. I don''t really care about the speed (it''s ruby after
all),
but it should have the right complexity -- i.e. if I write a post a
day, after 100 days, it should not take 100 times more than the first
day.

The program I used before webgen is nukumi2:
http://chneukirchen.org/blog/static/projects/nukumi2.html
which I left because it''s not actively developed.

One cool thing about nukumi2 is that each regeneration only
regenerates the minimum set of files, and it seemed very smart.

So I CCed this to Chris, nukumi2''s author, maybe he can share its
experience about how to implement incremental generation: myself, I
can only think of naive ways, when there probably is a very good
solution.

Cheers,
-- 
Andrea Censi
      "Life is too important to be taken seriously" (Oscar Wilde)
Web: http://www.dis.uniroma1.it/~censi

"Andrea Censi" <andrea.censi at dis.uniroma1.it> writes:
> One cool thing about nukumi2 is that each regeneration only
> regenerates the minimum set of files, and it seemed very smart.
>
> So I CCed this to Chris, nukumi2''s author, maybe he can share its
> experience about how to implement incremental generation: myself, I
> can only think of naive ways, when there probably is a very good
> solution.
Hehe.  It''s not that clever, but let me explain it.

Nukumi2 constructs a tree that describes which source files map to
which target files (blog entries -> front page, archive page, etc.)  I
look at the mtimes of all target files, and the mtimes of all source
files, and then regenerate the targets which contain newer parts.

Due to the general aspect of Nukumi2, and the rather complex data
structure below it, this turns a bit hairy, but I think for webgen
this approach should be easy to do.  It works a lot like make.

I currently run into scaling problems regarding the set of data to be
published, since it needs to read all entries at the moment, and
Anarchaia has about 700 entries by now that are read from disk
everytime.  I''ll probably move to dynamic publishing in the future.

-- 
Christian Neukirchen  <chneukirchen at gmail.com>  http://chneukirchen.org