dovecot - Aug 2022 - rawlog data in a lua script

Hi,   as far as I know I cannot configure Dovecot to pipe the rawlog into
rsyslog. Or can I, how?   The rawlog feature in Dovecot writes multiple files
(two for each connection, one for raw requests and one for raw responses) into a
predefined directory for the user. This generates dozens or hundreds of files
per user per day, each file with a timestamp in it, so the filename is not
predictable.   Even if it works, I'm not sure if syslog (rsyslog or
syslog-ng) should be (ab)used to collect the rawlog file contents, which might
be hundreds of MB per minute if someone FETCHes all his emails while setting up
a new account in Thunderbird or so. That sounds like a suboptional idea. Syslog
cannot handle binary text I guess, and it might have limits like "line
length limits" or similar. It sounds like the wrong tool for the job.  
Michael

Am 28-Jul-2022 15:28:16 +0200 schrieb dovecot at ptld.com:
> I'm searching for a possibility to have the rawlog feature in lua,
which would be much easier for processing.
> 
> It would be much easier to hook to the "raw request and response
events" inside Dovecot and have the rawlog-data in a lua script,
> where I can prepare it and send it to another maschine for
monitoring/collection/analysis/statistics or similar, for example via HTTP.
rsyslog has this feature (omprog) allowing you to setup any script/program for
it to pipe logs to in real time.

https://www.rsyslog.com/doc/master/configuration/modules/omprog.html
https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md

Works similar in concept to postfix policy servers if you are familiar with
them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20220808/cf645307/attachment.htm>

Hi, Michael, did you consider my suggestion to use raw events instead of rawlogs
for this?

Aki
> On 08/08/2022 14:52 EEST michael.zork at feierfighter.de wrote:
> 
> 
> Hi,
> 
> as far as I know I cannot configure Dovecot to pipe the rawlog into
rsyslog. Or can I, how?
> 
> The rawlog feature in Dovecot writes multiple files (two for each
connection, one for raw requests and one for raw responses) into a predefined
directory for the user. This generates dozens or hundreds of files per user per
day, each file with a timestamp in it, so the filename is not predictable.
> 
> Even if it works, I'm not sure if syslog (rsyslog or syslog-ng) should
be (ab)used to collect the rawlog file contents, which might be hundreds of MB
per minute if someone FETCHes all his emails while setting up a new account in
Thunderbird or so. That sounds like a suboptional idea. Syslog cannot handle
binary text I guess, and it might have limits like "line length
limits" or similar. It sounds like the wrong tool for the job.
> 
> Michael
> 
> 
> Am 28-Jul-2022 15:28:16 +0200 schrieb dovecot at ptld.com:
> > > I'm searching for a possibility to have the rawlog feature in
lua, which would be much easier for processing.
> > > 
> > > It would be much easier to hook to the "raw request and
response events" inside Dovecot and have the rawlog-data in a lua script,
> > > where I can prepare it and send it to another maschine for
monitoring/collection/analysis/statistics or similar, for example via HTTP.
> > 
> > 
> > rsyslog has this feature (omprog) allowing you to setup any
script/program for it to pipe logs to in real time.
> > 
> > https://www.rsyslog.com/doc/master/configuration/modules/omprog.html
> >
https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md
> > 
> > Works similar in concept to postfix policy servers if you are familiar
with them.