dovecot - Dec 2007 - "pipe" plugin - is anyone interested (or using it)?

Hi,

A few months ago, I sent a message to this list asking if people would
be interested by a "pipe" plugin (see
http://dovecot.org/pipermail/dovecot/2007-May/023005.html ). I received
a few answers of people who told they were intersted.

A few months later, I sent a new message announcing the availability of
my work (see
http://dovecot.org/pipermail/dovecot/2007-August/024805.html ).

Since then, I have received no feedback...

So, is anyone using this plugin? Is anyone using it?

Timo, on my first message, you seemed to be interested to merge such a
feature (see http://dovecot.org/pipermail/dovecot/2007-May/023055.html
). Are you still interested?

All comments are welcome. I think the feature (or a more generic one)
would be a worthy addition for dovecot, be it for ham/spam learning, to
implement an "outbox" or anything else I can't think about.

All comments about my work are welcome.
(Please note that I have not (yet) tried to port it to dovecot 1.1...)


Cheers,

Nicolas Boullis
Ecole Centrale Paris

On Dec 5, 2007 7:29 AM, Nicolas Boullis <nicolas.boullis at ecp.fr>
wrote:
> A few months ago, I sent a message to this list asking if people would
> be interested by a "pipe" plugin (see
> http://dovecot.org/pipermail/dovecot/2007-May/023005.html ). I received
> a few answers of people who told they were intersted.
>
> A few months later, I sent a new message announcing the availability of
> my work (see
> http://dovecot.org/pipermail/dovecot/2007-August/024805.html ).
>
> Since then, I have received no feedback...
>
> So, is anyone using this plugin? Is anyone using it?
Hi Nicolas-

I don't remember seeing either of your earlier messages (although I
admit to not always being able to closely follow the list despite
being a subscriber), but I do like your plugin. I haven't had a chance
to install and try it, but I'm adding it to my list of things to do.
I've struggled to come up with an easy-to-use solution for manual
classification of messages for end users, and ultimately sort of
pushed it aside, but this seems like an elegant and easy-to-use
solution.

Could you maybe explain how the configuration works in a little more
detail? For some reason I'm having a bit of trouble wrapping my head
around how this works:

(...)
namespace private {
   separator = .
   location = maildir:/var/mail/%u
   inbox = yes
   hidden = no
}

namespace public {
   separator = .
   prefix = learn.
   location = maildir:/var/learn/%u
   inbox = no
   hidden = no
}
(...)
plugin {
(...)
  pipe = /var/learn/%u/.spam:spamc -d some.host -L spam
  pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham
(...)
}

Thanks,
Ben

On 05.12.2007 17:29, Nicolas Boullis wrote:
> Hi,
>
> A few months ago, I sent a message to this list asking if people would
> be interested by a "pipe" plugin (see
> http://dovecot.org/pipermail/dovecot/2007-May/023005.html ). I received
> a few answers of people who told they were intersted.
>
> A few months later, I sent a new message announcing the availability of
> my work (see
> http://dovecot.org/pipermail/dovecot/2007-August/024805.html ).
>
> Since then, I have received no feedback...
>
> So, is anyone using this plugin? Is anyone using it?
>
> Timo, on my first message, you seemed to be interested to merge such a
> feature (see http://dovecot.org/pipermail/dovecot/2007-May/023055.html
> ). Are you still interested?
>
> All comments are welcome. I think the feature (or a more generic one)
> would be a worthy addition for dovecot, be it for ham/spam learning, to
> implement an "outbox" or anything else I can't think about.
>
> All comments about my work are welcome.
> (Please note that I have not (yet) tried to port it to dovecot 1.1...)
>
>
> Cheers,
>
> Nicolas Boullis
> Ecole Centrale Paris
>   
Hi Nicolas,

Very promising for me too, not tried yet, but will add to my todo!

On Wed, 2007-12-05 at 15:29 +0100, Nicolas Boullis
wrote:
> Timo, on my first message, you seemed to be interested to merge such a
> feature (see http://dovecot.org/pipermail/dovecot/2007-May/023055.html
> ). Are you still interested?
I downloaded it when you first mentioned it, but looks like I never got
around to actually looking at it. Yes, something like this would be
nice..

One thing at least that should be changed is to configure it using
virtual mailbox names instead of full mailbox paths. This isn't really
possible with v1.0, but v1.1 should make it pretty easy.

A few things about the code:

	save_dest_mail = mail_alloc(ctx->transaction,
				    MAIL_FETCH_PHYSICAL_SIZE, NULL);

Quota plugin wanted to know the message's physical size, but your plugin
doesn't need it. So you could just use 0 instead of
MAIL_FETCH_PHYSICAL_SIZE.

I think write() can return partially written data if the other side
isn't reading it fast enough. Using write_full() instead would anyway be
better/safer.

Do you really need to wait for the executed process to finish? Since
this is the only plugin currently creating child processes, I'd setup a
SIGCHLD handler and waitpid() there to get rid of the zombie:
lib_signals_set_handler(SIGCHLD, TRUE, chld_handler, NULL);

Multiple commands are now processed sequentially. I don't know if
there's real need for multiple commands, but it would be faster to read
the message input just once and send it to all pipes in parallel.

"return 0 * WEXITSTATUS(status);" returns always 0 :)

I'd also leave stderr as-is for the child process so it could log
errors, and for handling syscall failures use:
i_fatal("dup2() failed: %m");

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20071205/b7f14326/attachment-0002.bin>

On Wed, December 5, 2007 3:29 pm, Nicolas Boullis wrote:
...
> A few months ago, I sent a message to this list asking if people would
> be interested by a "pipe" plugin
...
> Since then, I have received no feedback...
...
> So, is anyone using this plugin? Is anyone using it?
>
Haven't tried it so I can add no useful comments on it?s use.

But a pipe plugin is in my opinion a very important feature.

On my postfix/dovecot system I'm currently forwarding mails that need
piping to a qmail server, since that?s the only way to do it easily on a
per user basis.
It seems irrational that dovecot itself doesn?t support this considering
dovecot being much ahead of qmail in every other aspect.

By the way I use it for much the same tasks as you do; for spam reporting
but also for certain e-mails that need be put into another database
besides it?s mailbox.

Regards, Mikkel

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 5 Dec 2007, Nicolas Boullis wrote:

It's on my todo list in order to support user-site SPAM training.

BTW: Could you please add the example of your first message into the 
plugin tgz? So the useage info is with the package.

Bye,

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBR1eswi9SORjhbDpvAQJLzwf+O9rHYeOiZsllYd/kfM/VV2UTy3ZFL+Wi
cJRq3JKYSq3lYeGVKntXqHfp7HqeB6hAr4qHhGAmcStA6mVM4OgjQ5NQZMT7veyI
exweEGKQDpAMvEXZ4wL09Jydf8f+TYZMOyLMmZUzdnKVJyk9tUUy8JR+2PeUbubq
RgL/NGsygmuEIY8fOVLQRdZwjZ5uO5OoN1vvGFN2TUFsePo7c/ro/bq1S4dhst4f
mjsKa/aqxPNlQPK+gKrnKfbX0V84e3u4XsgYYaPyCwnD3yMqnVe4nE0nIYeoGDFq
DP8C8kVGtMVz300229lK/HcuxSWbUh2Z9l/XJWdIT2R9HfOT2fvE7Q==ccab
-----END PGP SIGNATURE-----

Timo Sirainen wrote:
> 
> I downloaded it when you first mentioned it, but looks like I never got
> around to actually looking at it. Yes, something like this would be
> nice..
> 
> One thing at least that should be changed is to configure it using
> virtual mailbox names instead of full mailbox paths. This isn't really
> possible with v1.0, but v1.1 should make it pretty easy.
You already told me about this last time, but I haven't yet come to read
the source code of dovecot 1.1...

> A few things about the code:
> 
> 	save_dest_mail = mail_alloc(ctx->transaction,
> 				    MAIL_FETCH_PHYSICAL_SIZE, NULL);
> 
> Quota plugin wanted to know the message's physical size, but your
plugin
> doesn't need it. So you could just use 0 instead of
> MAIL_FETCH_PHYSICAL_SIZE.
Thanks for pointing this.

> I think write() can return partially written data if the other side
> isn't reading it fast enough. Using write_full() instead would anyway
be
> better/safer.
I just reread my code, and I think my use of write looks safe, since
only the amount that was correctly written is skipped with
i_stream_skip. Do you think I'm missing something?

> Do you really need to wait for the executed process to finish? Since
> this is the only plugin currently creating child processes, I'd setup a
> SIGCHLD handler and waitpid() there to get rid of the zombie:
> lib_signals_set_handler(SIGCHLD, TRUE, chld_handler, NULL);
Waiting is required if we want the append to fail if the command fails.
I guess it should better be a configurable option, don't you think so?

> Multiple commands are now processed sequentially. I don't know if
> there's real need for multiple commands, but it would be faster to read
> the message input just once and send it to all pipes in parallel.
I don't think there is real need for multiple commands, but I think it
was easier for me to program things like this... ;-)

> "return 0 * WEXITSTATUS(status);" returns always 0 :)
Hummm... I guess I should avoid drinking alcohol before I program... ;-)
It probably should be "return WEXITSTATUS(status);"; I have no idea
why
I changed this in such a strange way...

> I'd also leave stderr as-is for the child process so it could log
> errors, and for handling syscall failures use:
> i_fatal("dup2() failed: %m");
OK, point taken.


One question still: would you consider merging my plugin in dovecot if I
ported it to 1.1?


Cheers,

Nicolas