thr3ads.net - dovecot - How to access mailbox metadata in Lua push driver [Aug 2020]

If this information is useful, please help other people find it:
Share via:

Ralf Becker

2020-Aug-03 19:58 UTC

How to access mailbox metadata in Lua push driver

Some answers to my questions, a first version of my script and more
questions ;)

Am 03.08.20 um 18:15 schrieb Ralf Becker:
> Currently looking into the following questions:
>
> - can I get the rfc 5423 type of event somehow (obviously I can set it
> on the event myself depending of the function called)
event.name> - looking at the example code, it looks like it can be called for
> multiple messages, when does that happen (LMTP send more then one)
still no idea, maybe Ake?

I noticed that some events have the same uid-validity, are the from a
single transaction, eg. I delete my Trash?
> - why is the mailbox status put into an other structure and send with
> a different notifiction
> - does anyone have a code snippet to send a JSON encoded message
> (probably easy to figure out looking at Lua docu)
these two I managed to solve im my current version of the script, which
also support now all message event types:

-- To use -- -- plugin { -- push_notification_driver
lua:file=/etc/dovecot/dovecot-push.lua -- push_lua_url
http://push.notification.server/handler -- } -- -- server is sent a PUT
message with JSON body like push_notification_driver ox:url=<push_lua_url>
user_from_metadata -- local http = require "socket.http" local ltn12 =
require "ltn12" -- luarocks install json-lua local json = require
"JSON" function table_get(t, k, d)
  return t[k] or d end function script_init()
  return 0 end function dovecot_lua_notify_begin_txn(user)
    local meta =
user:metadata_get("/private/vendor/vendor.dovecot/http-notify")
    return {user=user, event=dovecot.event(),
ep=user:plugin_getenv("push_lua_url"), messages={}, meta=meta}
end function dovecot_lua_notify_event_message_new(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil or ctx.meta
== '') then return end -- get mailbox status local mbox =
ctx.user:mailbox(event.mailbox)
    mbox:sync()
    local status = mbox:status(dovecot.storage.STATUS_RECENT,
dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES)
    mbox:free()
    table.insert(ctx.messages, {
      user = ctx.meta,
      ["imap-uidvalidity"] = event.uid_validity,
      ["imap-uid"] = event.uid,
      folder = event.mailbox,
      event = event.name,
      from = event.from,
      subject = event.subject,
      snippet = event.snippet,
      unseen = status.unseen
    })
end function dovecot_lua_notify_event_message_append(ctx, event)
  dovecot_lua_notify_event_message_new(ctx, event)
end function dovecot_lua_notify_event_message_read(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil or ctx.meta
== '') then return end -- get mailbox status local mbox =
ctx.user:mailbox(event.mailbox)
    mbox:sync()
    local status = mbox:status(dovecot.storage.STATUS_RECENT,
dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES)
    mbox:free()
    table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        unseen = status.unseen
    })
end function dovecot_lua_notify_event_message_trash(ctx, event)
    dovecot_lua_notify_event_message_read(ctx, event)
end function dovecot_lua_notify_event_message_expunge(ctx, event)
    dovecot_lua_notify_event_message_read(ctx, event)
end function dovecot_lua_notify_event_flags_set(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil or ctx.meta
== '') then return end table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        flags = event.flags,
        ["keywords-set"] = event.keywords_set
    })
end function dovecot_lua_notify_event_flags_clear(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil or ctx.meta
== '') then return end table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        flags = event.flags,
        ["keywords-clear"] = event.keywords_clear,
        ["keywords-old"] = event.keywords_old
    })
end function dovecot_lua_notify_end_txn(ctx)
    -- report all states for i,msg in ipairs(ctx.messages) do local e =
dovecot.event(ctx.event)
    e:set_name("lua_notify_mail_finished")
    reqbody = json:encode(msg)
    e:log_debug(ctx.ep .. " - sending " .. reqbody)
    res, code = http.request({
        method = "PUT",
        url = ctx.ep,
        source = ltn12.source.string(reqbody),
        headers={
            ["content-type"] = "application/json;
charset=utf-8",
            ["content-length"] = tostring(#reqbody)
        }
    })
    e:add_int("result_code", code)
    e:log_info("Mail notify status " .. tostring(code))
    end end


This leads to a couple more questions ;)

- is there a way (eg. return value) to stop event processing already in
dovecot_lua_notify_begin_txn

- sometimes multiple events are generated, eg. when I read an email:

{"event":"FlagsClear","flags":[],"folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"keywords-old":[],"user":"user=5::42;***"}
{"event":"FlagsSet","flags":["\\Seen"],"folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"user":"user=5::42;***"}
{"event":"MessageRead","folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"unseen":0,"user":"user=5::42;***"}

Ralf

-- 
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Gesch?ftsf?hrer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20200803/581220a1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20200803/581220a1/attachment-0001.sig>

Ralf Becker

2020-Aug-04 09:42 UTC

head link

Errors in Lua push docu

https://doc.dovecot.org/configuration_manual/push_notification/#message-events


  *

    dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid,
    uid_validity, flags, keywords_set})

--> dovecot_lua_notify_event_flags_set(context, {name, mailbox, uid,
uid_validity, flags, keywords})

Called when message flags or keywords are set. flags is a bitmask.
keywords_set is a table of strings of the keywords set by the event.

  *

    dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid,
    uid_validity, flags, keywords_clear, keywords_old})

--> dovecot_lua_notify_event_flags_clear(context, {name, mailbox, uid,
uid_validity, flags, flags_old, keywords, keywords_old})

Ralf

-- 
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Gesch?ftsf?hrer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20200804/f7e09d65/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20200804/f7e09d65/attachment.sig>

Aki Tuomi

2020-Aug-04 09:55 UTC

head link

How to access mailbox metadata in Lua push driver

> On 03/08/2020 22:58 Ralf Becker <rb at egroupware.org> wrote:
> 
> 
> Some answers to my questions, a first version of my script and more
questions ;)
> 
> Am 03.08.20 um 18:15 schrieb Ralf Becker:
> > Currently looking into the following questions:
> > 
> > 
> > - can I get the rfc 5423 type of event somehow (obviously I can set it
on the event myself depending of the function called)
> event.name
> 
> > - looking at the example code, it looks like it can be called for
multiple messages, when does that happen (LMTP send more then one)
> > 
> still no idea, maybe Ake?
> I noticed that some events have the same uid-validity, are the from a
single transaction, eg. I delete my Trash?
> 
Because these events also apply for more than just LMTP. You could be storing
bunch of mails with IMAP APPEND.
> > - why is the mailbox status put into an other structure and send with
a different notifiction
The LUA code very faitfully implements the C API for push notifications in
Dovecot.
> > - does anyone have a code snippet to send a JSON encoded message
(probably easy to figure out looking at Lua docu)
> these two I managed to solve im my current version of the script, which
also support now all message event types:
<snip7>
> 
> This leads to a couple more questions ;)
> - is there a way (eg. return value) to stop event processing already in
dovecot_lua_notify_begin_txn
There is no really good way to do this. You can error out in begin_txn, but that
will emit error.

The only doable way, right now, is to maybe add key {disabled=True} to your
context, and then add

if ctx.disabled:
  return 

elsewhere.

> - sometimes multiple events are generated, eg. when I read an email:
>
{"event":"FlagsClear","flags":[],"folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"keywords-old":[],"user":"user=5::42;***"}
> 
{"event":"FlagsSet","flags":["\\Seen"],"folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"user":"user=5::42;***"}
> 
{"event":"MessageRead","folder":"INBOX/eGroupWare/calconnect","imap-uid":2275,"imap-uidvalidity":1499767470,"unseen":0,"user":"user=5::42;***"}
> 
This is what happens when you read unseen mail.
> Ralf
>  
> 
> -- 
> Ralf Becker
> EGroupware GmbH [www.egroupware.org (http://www.egroupware.org)]
> Handelsregister HRB Kaiserslautern 3587
> Gesch?ftsf?hrer Birgit und Ralf Becker
> Leibnizstr. 17, 67663 Kaiserslautern, Germany
> Telefon +49 631 31657-0
>
Aki

Ralf Becker

2020-Aug-04 15:11 UTC

head link

How to access mailbox metadata in Lua push driver

In case someone is interested this is the final version of the script:

-- To use -- -- plugin { -- push_notification_driver
lua:file=/etc/dovecot/dovecot-push.lua -- push_lua_url
https://Bearer:<push-token>@<egroupware-domain>/egroupware/push -- }
--
-- server is sent a PUT message with JSON body like
push_notification_driver = ox:url=<push_lua_url> user_from_metadata --
plus additionally the events MessageAppend, MessageExpunge, FlagsSet and
FlagsClear -- MessageTrash and MessageRead are ignored, so are empty
FlagSet/Clear or FlagSet NonJunk (TB) -- local http = require
"socket.http" local ltn12 = require "ltn12" -- luarocks
install json-lua local json = require "JSON" function table_get(t, k,
d)
  return t[k] or d end function script_init()
  return 0 end function dovecot_lua_notify_begin_txn(user)
    local meta =
user:metadata_get("/private/vendor/vendor.dovecot/http-notify")
    if (meta == nil or meta:sub(1,5) ~= "user=")
    then meta = nil;
    else meta = meta:sub(6)
    end return {user=user, event=dovecot.event(),
ep=user:plugin_getenv("push_lua_url"), messages={}, meta=meta}
end function dovecot_lua_notify_event_message_new(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil) then
return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox)
    mbox:sync()
    local status = mbox:status(dovecot.storage.STATUS_RECENT,
dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES)
    mbox:free()
    table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        from = event.from,
        subject = event.subject,
        snippet = event.snippet,
        unseen = status.unseen,
        messages = status.messages
    })
end function dovecot_lua_notify_event_message_append(ctx, event)
  dovecot_lua_notify_event_message_new(ctx, event)
end -- ignored, as FlagSet flags=[\Seen] is sent anyway too -- function
dovecot_lua_notify_event_message_read(ctx, event) --
dovecot_lua_notify_event_message_expunge(ctx, event) -- end -- ignored,
as most MUA nowadays expunge immediatly -- function
dovecot_lua_notify_event_message_trash(ctx, event) --
dovecot_lua_notify_event_message_expunge(ctx, event) -- end function
dovecot_lua_notify_event_message_expunge(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil) then
return end -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox)
    mbox:sync()
    local status = mbox:status(dovecot.storage.STATUS_RECENT,
dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES)
    mbox:free()
    -- agregate multiple Expunge (or Trash or Read) if (#ctx.messages == 1 and
ctx.messages[1].user == ctx.meta and ctx.messages[1].folder == event.mailbox and
ctx.messages[1]["imap-uidvalidity"] == event.uid_validity and
ctx.messages[1].event == event.name)
    then if (type(ctx.messages[1]["imap-uid"]) ~= 'table')
then ctx.messages[1]["imap-uid"] =
{ctx.messages[1]["imap-uid"]}
        end table.insert(ctx.messages[1]["imap-uid"], event.uid)
        ctx.messages[1].unseen = status.unseen
        ctx.messages[1].messages = status.messages
        return;
    end table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        unseen = status.unseen,
        messages = status.messages
    })
end function dovecot_lua_notify_event_flags_set(ctx, event)
    -- check if there is a push token registered if (ctx.meta == nil or
(#event.flags == 0 and #event.keywords == 0) or -- ignore TB sends it empty
(#event.keywords == 1 and event.keywords[1] == "NonJunk")) -- ignore
TB NonJunk then return end local status = nil;
    if (#event.flags == 1 and event.flags[1] == "\\Seen")
    then -- get mailbox status local mbox = ctx.user:mailbox(event.mailbox)
        mbox:sync()
        status = mbox:status(dovecot.storage.STATUS_RECENT,
dovecot.storage.STATUS_UNSEEN, dovecot.storage.STATUS_MESSAGES)
        mbox:free()
    end -- agregate multiple FlagSet if (#ctx.messages == 1 and
ctx.messages[1].user == ctx.meta and ctx.messages[1].folder == event.mailbox and
ctx.messages[1]["imap-uidvalidity"] == event.uid_validity and
ctx.messages[1].event == event.name and arrayEqual(ctx.messages[1].flags,
event.flags) and arrayEqual(ctx.messages[1].keywords, event.keywords))
    then if (type(ctx.messages[1]["imap-uid"]) ~= 'table')
then ctx.messages[1]["imap-uid"] =
{ctx.messages[1]["imap-uid"]}
        end table.insert(ctx.messages[1]["imap-uid"], event.uid)
        if (status ~= nil)
        then ctx.messages[1].unseen = status.unseen
        end return;
    end table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        flags = event.flags,
        keywords = event.keywords,
        unseen = status.unseen
    })
end function arrayEqual(t1, t2)
    if (#t1 ~= #t2)
    then return false end if (#t1 == 1 and t1[1] == t2[1])
    then return true end return json:encode(t1) == json:encode(t2)
end function dovecot_lua_notify_event_flags_clear(ctx, event)
    -- check if there is a push token registered AND something to clear (TB
sends it empty) if (ctx.meta == nil or (#event.flags == 0 and #event.keywords ==
0)) then return end table.insert(ctx.messages, {
        user = ctx.meta,
        ["imap-uidvalidity"] = event.uid_validity,
        ["imap-uid"] = event.uid,
        folder = event.mailbox,
        event = event.name,
        flags = event.flags,
        ["flags-old"] = event.flags_old,
        keywords = event.keywords,
        ["keywords-old"] = event.keywords_old
    })
end function dovecot_lua_notify_end_txn(ctx)
    -- report all states for i,msg in ipairs(ctx.messages) do local e =
dovecot.event(ctx.event)
        e:set_name("lua_notify_mail_finished")
        reqbody = json:encode(msg)
        e:log_debug(ctx.ep .. " - sending " .. reqbody)
        res, code = http.request({
            method = "PUT",
            url = ctx.ep,
            source = ltn12.source.string(reqbody),
            headers={
                ["content-type"] = "application/json;
charset=utf-8",
                ["content-length"] = tostring(#reqbody)
            }
        })
        e:add_int("result_code", code)
        e:log_info("Mail notify status " .. tostring(code))
    end end

It's also in our Github repo:
https://raw.githubusercontent.com/EGroupware/swoolepush/master/doc/dovecot-push.lua

Ralf

-- 
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Gesch?ftsf?hrer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20200804/03ac958e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20200804/03ac958e/attachment-0001.sig>

Possibly Parallel Threads

Search for more possibly parallel threads

dovecot - Aug 2020 - How to access mailbox metadata in Lua push driver

How to access mailbox metadata in Lua push driver

Errors in Lua push docu

How to access mailbox metadata in Lua push driver

How to access mailbox metadata in Lua push driver

Possibly Parallel Threads