dovecot - Jan 2010 - Sieve > Pigeonhole > external storage with LDAP or other data source available to dovecot

I am looking at porting and generalizing an old in-house patch that I'm 
using for the CMU Sieve plugin.  It allows sieve script to test & lookup 
arguments from LDAP.

Before getting too involved in this, I'd like to discuss my goals here 
in the hopes that someone else also thinks this would be useful.  Many 
heads may make better design?

Goal:
------
Allows scripts which are still stored on the filesystem to lookup 
parameters stored in an LDAP directory.

Useful if you provide a global sieve script to implement functions such 
as "vacation", forwarding to small groups (not a MLM), wish to toggle 
whether a message is kept in the mailbox after being forwarded, etc.

<example>
# determine if we always auto-respond, or if we respond with a more 
vacation-ish setting of every 30d
#
# sample LDAP entry:
# uid: testuser at domain.com
# mailResponderMode: vacation
# mailResponderText: "I'm on vacation, far far from the Internet" 
(base64 encoded string...)

# "always" reply mode, sanitized to once a day
if ldap "mailResponderMode" "reply" {
         vacation :days 1 "ldap at mailResponderText";
}

# traditional "vacation" mode, say reply once every 30d
if ldap "mailResponderMode" "vacation" {
         vacation :days 30 "ldap at mailResponderText";
}
</example>

This introduces:

   1. a new command "ldap", to lookup attributes and their values in
an
      LDAP directory using the user's context
   2. an extention to the vacation command that looks up the vacation
      reply text that is triggered by supplying "ldap at
ldapAttribute" as
      the text.
   3. other actions not shown above which extends redirect and a few
      other commands to lookup data from LDAP

Related work:

   1. Pigeonhole low priority TODO would like to implement alternate
      script storage, eg: LDAP & SQL.  I'm not immediately interested in
      alternate types of script storage, but for what I want to acheive,
      I need to sanely access at least an LDAP directory.
   2. draft-ietf-sieve-external-lists
      (http://tools.ietf.org/html/draft-ietf-sieve-external-lists-01)
      proposes a mechanism to pull mailing list addresses from external
      storage mechanisms such as LDAP, ACAP or relational databases. 
      I'm interested in this, but would like to extend this
      functionality beyond just lists as the example above demonstrates.
   3. there are security and resource consumption issues with doing
      this, suggesting that limits should be set, or the use may have to
      be restricted to global scripts eg: sieve_before or sieve_after

The question:
- do others on this list see value in the functionality described above?

If there's some consensus that this would be useful, I'll flesh out howI
think a generalized version of this old patch may behave, then (!) work 
on an implementation as a plugin to dovecot's sieve.

cheers,

-Martin Foster
martin_foster at pacific.net.au
martin.foster at pacnet.com

On 5.1.2010, at 10.02, Martin F. Foster wrote:
> I am looking at porting and generalizing an old in-house patch that I'm
using for the CMU Sieve plugin.  It allows sieve script to test & lookup
arguments from LDAP.
..
> Related work:
> 
>  1. Pigeonhole low priority TODO would like to implement alternate
>     script storage, eg: LDAP & SQL.  I'm not immediately interested
in
>     alternate types of script storage, but for what I want to acheive,
>     I need to sanely access at least an LDAP directory.
>  2. draft-ietf-sieve-external-lists
>     (http://tools.ietf.org/html/draft-ietf-sieve-external-lists-01)
>     proposes a mechanism to pull mailing list addresses from external
>     storage mechanisms such as LDAP, ACAP or relational databases.     
I'm interested in this, but would like to extend this
>     functionality beyond just lists as the example above demonstrates.
I haven't looked into that draft or thought much about this, but would doing
the lookups via Dovecot's lib-dict be ok? That would of course need a
dict-ldap backend implemented, but it would be a generic way to solve this, if
its API is good enough.

> On 5.1.2010, at 10.02, Martin F. Foster wrote:
>
>    
>> I am looking at porting and generalizing an old in-house patch that
I'm using for the CMU Sieve plugin.  It allows sieve script to test& 
lookup arguments from LDAP.
>>      
> ..
>    
>> Related work:
>>
>>   1. Pigeonhole low priority TODO would like to implement alternate
>>      script storage, eg: LDAP&  SQL.  I'm not immediately
interested in
>>      alternate types of script storage, but for what I want to acheive,
>>      I need to sanely access at least an LDAP directory.
>>   2. draft-ietf-sieve-external-lists
>>      (http://tools.ietf.org/html/draft-ietf-sieve-external-lists-01)
>>      proposes a mechanism to pull mailing list addresses from external
>>      storage mechanisms such as LDAP, ACAP or relational databases.    
I'm interested in this, but would like to extend this
>>      functionality beyond just lists as the example above demonstrates.
>>      
> I haven't looked into that draft or thought much about this, but would
doing the lookups via Dovecot's lib-dict be ok? That would of course need a
dict-ldap backend implemented, but it would be a generic way to solve this, if
its API is good enough.
>    
After another day of looking into this, I believe that there are two 
concerns to be addressed:
1. dovecot implementation: how dovecot/pigeonhole provides access to 
these external sources in a sane way. I do like the dict interface for 
this, which could keep LDAP url's (or SQL queries, etc) out of sieve 
scripts, and into nicely compartementalized configuration files. Being 
very ldap focussed at my site, I hadn't looked at dict before today.
2. sieve language: an extension which would allow scripts to get 
information from an external source. I'm asking the 
draft-ietf-sieve-external-lists authors a few questions about this.


The end result would be something like what follows:

Dovecot implementation (proposed):
-------------------------------------

<dovecot.conf>
dict {
dict_sieve = ldap:/etc/dovecot/dovecot-sieve-ldap.conf
}

plugin {
# TODO: proxy this dict?
sieve_extsrc = dict
}
</dovecot.conf>

then I'm not sure how you specify multiple hosts with the dict-sql 
format, so I'm using the postfix convention. there's probably a better 
way for dovecot.
<dovecot-sieve-ldap.conf>
map {
name = responder_mode
server_host = ldap://server1.mycorp.com, ldap://server2.mycorp.com
search_base = o=mailstuff
query_filter = (&(objectclass=mailschema)(uid=%u))
result_attribute = responder_mode
result_scope = sub
bind = no
version = 3
}
map {
name = responder_text
server_host = ldap://server1.mycorp.com, ldap://server2.mycorp.com
search_base = o=mailstuff
query_filter = (&(objectclass=mailschema)(uid=%u))
result_attribute = responder_text
result_scope = sub
bind = no
version = 3
}
map {
name = forward_keep
server_host = ldap://server1.mycorp.com, ldap://server2.mycorp.com
search_base = o=mailstuff
query_filter = (&(objectclass=mailschema)(uid=%u))
result_attribute = mailForwardKeep
result_scope = sub
bind = no
version = 3
}
map {
# in this case we can have many responses
name = forward_addresses
server_host = ldap://server1.mycorp.com, ldap://server2.mycorp.com
search_base = o=mailstuff
query_filter = (&(objectclass=mailschema)(uid=%u))
result_attribute = mailForwardAddress
result_scope = sub
bind = no
version = 3
}
</dovecot-sieve-ldap.conf>

Sieve language:
---------------------
then with proposed new functions of the sieve language (all names 
plucked out of my head 2hrs ago, would have to think about the semantics 
some more):

1. a ?:list? argument that expects a string-list

2. a ?:extsrc? argument that expects an <ext-name: string> and returns a 
string-list

3. a ?:extsrc1? argument that expects an <ext-name: string> and returns 
a string

4. a new test that uses information that is not in the message, say 
?arbitrary [comparator] string string? (syntax needs work)


sample uses in scripts are then:

1. redirect to a mailing list sourced from ldap (or wheverver):

# mailForwardAddress attribute can be present >=0 times

redirect :list :extsrc ?dict:forward_addresses?

2. decide whether to keep or discard forwarded messages (eg: after the 
above rule)

# mailForwardKeep attribute should only be present once, if at all

if arbitrary :is :extsrc1 ?dict:forward_keep? ?TRUE? {

keep;

}

3. decide vacation action and text based in information in an ldap dir:

# mailResponderMode & mailResponderText attrs should only be present 
once, if at all

if arbitrary :is :extsrc1 ?dict:responder_mode? ?alwaysreply? {

vacation :days 1 :extsrc1 ?dict:responder_text";

}

if arbitrary :is :extsrc1 ?dict:responder_mode? ?holiday? {

vacation :days 30 :extsrc1 ?dict:responder_text?;

}


... where the "dict:mapname" syntax would be dovecot implementation 
specific. could also support the URL & TAG-URI schemes as proposed by 
draft-ietf-sieve-external-lists. This would allow query URLs as a more 
general case, using whatever's configured in /etc/ldap/ldap.conf (or 
whatever the platform's ldap client lib uses). Doing this does raise 
some resource consumption concerns, hence there should probably be a 
means of restricting external queries to global scripts only (or 
sieve_before, sieve_after), and a means of providing timeout values, etc.

eg:

if arbitrary :is :extsrc1 ?ldap:///o=myorg?mailForwardKeep? ?TRUE? {

keep;

}



I'll keep on thinking about the design, staring intently at the dovecot 
sources and http://wiki.dovecot.org/Design for another couple days 
before attempting to write some code to test this. Feedback on the 
design of both the sieve language extension, & dovecot interface very 
appreciated.

Cheers,

-Martin Foster

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 5 Jan 2010, Martin F. Foster wrote:
> Allows scripts which are still stored on the filesystem to lookup
parameters
> stored in an LDAP directory.
> # traditional "vacation" mode, say reply once every 30d
> if ldap "mailResponderMode" "vacation" {
>        vacation :days 30 "ldap at mailResponderText";
> }
How about to mimick the variable extension in such way that the values are 
first put into variables? I refer to ldap at .. syntex especially.

e.g.

require ["variables", "variables_ldap"];

set :ldap "LDAP query" [ attibute_name1 attribute_name2 ... ]

If "LDAP query" is missing, a default query like
"(|(mail=%1)(uid=%1))" is
performed.

attribute_name is the variable name, the values is placed in.

((Well, the actual syntax is just a hint))

Dunno, how you can deal with multi-value LDAP attributes.

Regards,

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBS0Xd4L+Vh58GPL/cAQLdcAf/d+C/XJiHofsAXqdbXCvNMDUWEl5UcGTf
PVzu1OFpTvTFIVXAomL5WIYeTHQsk3P7TuLhloQUK/+ublh9dzg9S35Hg+j7h3VN
AA+/1j8UhrukDFCI/7/LcKPiJQEtrcWsuaCtX0+QpUFTRo/AQMKNIcVFwhJw60IZ
ziBF9RhpqDC4n2F7dfJ6EB3zlKPP6GTVPZGlYFeLPGFeHijQ+LZvdZ3Fg7yipuSr
4fW41SCBKy0ihoHzHUahEJKHIwiU7IIpnNn8v+OJ0drwks0V5mKRd9N+LcJ/yWJZ
jC/NETAI2Hxk2j4lyM/iKiydSPU2IIYGglSk4BIa8ojDy2q8r9XQGA==yGmu
-----END PGP SIGNATURE-----

Martin F. Foster wrote:
> I am looking at porting and generalizing an old in-house patch that I'm
> using for the CMU Sieve plugin.  It allows sieve script to test &
lookup
> arguments from LDAP.
> 
> Before getting too involved in this, I'd like to discuss my goals here 
> in the hopes that someone else also thinks this would be useful.  Many 
> heads may make better design?
> 
I thought this would be an excellent opportunity to test the new 
Pigeonhole plugin support and also to do something interesting with the 
namespace functionality of the variables extension. That is why I gave 
this a short look, which promptly resulted in a preliminary implementation.

If you want to play around with it, you can download it from:

http://hg.rename-it.nl/pigeonhole-sieve-extdata/

It will compile and run with both Pigeonhole for Dovecot v1.2 and 
Pigeonhole for Dovecot v2.0, but only with the tip Mercurial revisions 
(since only those have plugin support).

I swiped together many of the ideas posted here on the mailing list and 
from those I concocted the following Sieve language extension:

http://hg.rename-it.nl/pigeonhole-sieve-extdata/raw-file/tip/doc/rfc/spec-bosch-sieve-external-data.txt

Note that - despite its looks - this is not intended as a IETF draft 
submission, but merely as a internal memo for the specification of this 
vendor-specific extension (in a familiar format). It could be further 
extended with an 'extdata_open' command to dynamically link a resource 
URI to a data store identifier, but thus far I did not bother to write 
that idea down.

All the necessary installation information should be found in the 
INSTALL and README files. Note that I have only tested this against a 
flat file dict and not something like SQL. So that may still be broken 
somehow.

Tell me what you think.

Regards,

-- 
Stephan Bosch
stephan at rename-it.nl

Le 09-Jan-10 20:29, Stephan Bosch a ?crit :
>
> I thought this would be an excellent opportunity to test the new
> Pigeonhole plugin support and also to do something interesting with the
> namespace functionality of the variables extension. That is why I gave
> this a short look, which promptly resulted in a preliminary implementation.
>
> If you want to play around with it, you can download it from:
>
> http://hg.rename-it.nl/pigeonhole-sieve-extdata/
>
> It will compile and run with both Pigeonhole for Dovecot v1.2 and
> Pigeonhole for Dovecot v2.0, but only with the tip Mercurial revisions
> (since only those have plugin support).
Thanks Stephan,

I have the extdata plugin compiled, and am now trying to make a global 
sieve script to test it.  I'm running into problems when using sievec to 
compile the script - I'm not getting the plugin's variables that are 
provided in dovecot.conf's plugin {}.  I added a few sieve_sys_warning 
messages to track the calls to sieve_setting_get()

<output>
/usr/local/bin/sievec -d -P sieve_extdata 
/usr/local/etc/dovecot/sieve/before.sv
Warning: sieve: sieve_setting_get: req: 'sieve_max_script_size' res: 
'(null)'
Warning: sieve: sieve_setting_get: req: 'sieve_max_actions' res:
'(null)'
Warning: sieve: sieve_setting_get: req: 'sieve_max_redirects' res:
'(null)'
Warning: sieve: sieve_setting_get: req: 'sieve_subaddress_sep' res:
'(null)'
Warning: sieve: sieve_setting_get: req: 'recipient_delimiter' res:
'(null)'
Warning: sieve: sieve_setting_get: req: 'sieve_plugin_dir' res:
'(null)'
Warning: sieve: sieve_setting_get: req: 'sieve_plugins' res:
'(null)'
Warning: sieve: sieve_setting_get: req: 'sieve_extdata_dict_uri' res: 
'(null)'
Warning: sieve: extdata: no dict uri specified, extension is 
unconfigured (sieve_extdata_dict_uri is not set).

* Required extensions:

   0: vacation (9)
   1: variables (17)
   2: vnd.dovecot.extdata (29)

* Main program (block: 1):

00000000: EXTENSIONS [3]:
00000001:   vacation
00000002:   variables
00000003:     SCOPE [0] (end: 00000008)
00000008:   vnd.dovecot.extdata
00000009: VACATION
0000000a:   (source line: 3)
0000000d:   subject: VAR ${extdata.vacation_subject}
00000023:   reason: VAR ${extdata.vacation_message}
00000038:   handle: STR[77] 
"${extdata.vacation_message}${extdata.vacation_subject}<default-from><NO-MIME>"
00000088: [End of code]
</output>

I'm new to dovecot's internals, it looks like it just needs some 
environment variables provided, but I'm puzzled as to how this is to be 
done.
I've traced it through sievec.c -> sieve_tool_sieve_env -> 
sieve_tool_get_setting -> getenv() call.

More info:
1. I've traced the process, there's no attempt to open or stat any 
dovecot.conf
2. using tip dovecot-1.2, dovecot-1.2-sieve, pigeonhole-sieve-extdata.  
I pulled+updated & rebuilt and confirmed the above behaviour just prior 
to sending this message.
3. The test scripts run because the test framework uses the test_config 
mechanism to provide sieve_extdata_dict_uri.

Any hints on how to get the dovecot.conf data to sievec?

Thanks,

-Martin Foster

got it, I have extdata working fine now for simple commands like 
redirect.  vacation doesn't seem to be working properly - in that a 
message is not returned to the sender.  I'll continue to test this with 
and without extdata and report back if I still can't get this to work by 
the end of my day (~18:00 @ UTC+11)

I have also updated the wiki with sieve plugin testing information 
http://wiki.dovecot.org/LDA/Sieve/Dovecot "Script Compling with Sieve 
Plugins"

Cheers,

-Martin Foster

Stephan Bosch wrote :
> Martin F. Foster wrote:
>> oh wow - ignore my last message about sievec&  environment.  As the
>> output says, it's just a warning... it is not blocking the
compilation
>> of the sieve script as I originally thought.
>>
>> <compile script>
>> #  /usr/local/bin/sievec -P sieve_extdata
>> /usr/local/etc/dovecot/sieve/before.sv
>> /usr/local/etc/dovecot/sieve/before.sieve
>> Warning: sieve: extdata: no dict uri specified, extension is
>> unconfigured (sieve_extdata_dict_uri is not set).
>>
>> # echo $?
>> 0
>> </compile script>
>>      
> This will compile the before.sv into a binary before.sieve, which is
> wrong. First of all, Pigeonhole expects scripts to have the .sieve
> extension. Second, binaries are expected to have the .svbin extension.
> The errors you get are caused by the deliver plugin's attempts at
> compiling a binary (since the .sieve is the binary and a .svbin file
> does not exist) :)
>
> Regards,
>
> Stephan.
>
>