dovecot - Aug 2018 - sdbox filesystem backup potential excludes

On 8/14/2018 12:55 AM, Aki Tuomi wrote:
>
> On 13.08.2018 19:51, Daniel Miller wrote:
>> When doing a filesystem backup of an moderate sdbox mailstore (300GB)
>> - are there any files that can be safely excluded from the backup?
>> Like *.log or *.backup?? Or are they all "vital" for
recovery?
>>
>> I'm already excluding the sdbox/virtual folders as it looks like
they
>> get created and updated as needed.
>>
> Hi, can you provide 'doveconf -n'. sdbox/virtual folders are not
part of
> sdbox mail format. In general, if you want to avoid data loss, you can
> only omit dovecot.index.cache files, but omitting these can come with
> high impact when they are regenerated. Omitting dovecot.index.log or
> dovecot.index will cause loss of flags.
>
> Aki
I have virtual folders enabled & defined.

What does "high impact" mean?? A few seconds to a few minutes on
initial
mailbox opening for regeneration?? I can live with that as this is for 
emergency backup/restore purposes.

doveconf -n
# 2.2.31 (65cde28): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.17 (e179378)
# OS: Linux 4.4.0-131-generic x86_64 Ubuntu 16.04.5 LTS
auth_cache_size = 4 k
auth_master_user_separator = *
auth_mechanisms = plain login
default_login_user = nobody
default_vsz_limit = 1 G
dict {
 ? acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
imap_client_workarounds = tb-extra-mailbox-sep
imap_idle_notify_interval = 29 mins
listen = *
mail_attachment_hash = %{sha512}
mail_plugins = fts fts_solr acl zlib virtual
mail_prefetch_count = 10
mail_shared_explicit_inbox = yes
mailbox_list_index = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext
mmap_disable = yes
namespace inbox {
 ? hidden = no
 ? inbox = yes
 ? list = yes
 ? location  ? mailbox "Deleted Messages" {
 ??? auto = no
 ??? autoexpunge = 30 days
 ??? special_use = \Trash
 ? }
 ? mailbox Drafts {
 ??? auto = subscribe
 ??? special_use = \Drafts
 ? }
 ? mailbox INBOX/Archives {
 ??? auto = no
 ??? special_use = \Archive
 ? }
 ? mailbox Sent {
 ??? auto = subscribe
 ??? special_use = \Sent
 ? }
 ? mailbox "Sent Items" {
 ??? auto = no
 ??? special_use = \Sent
 ? }
 ? mailbox "Sent Messages" {
 ??? auto = no
 ??? special_use = \Sent
 ? }
 ? mailbox Trash {
 ??? auto = subscribe
 ??? autoexpunge = 30 days
 ??? special_use = \Trash
 ? }
 ? mailbox virtual/Flagged {
 ??? comment = All my flagged messages
 ??? special_use = \Flagged
 ? }
 ? prefix  ? separator = /
 ? subscriptions = yes
 ? type = private
}
namespace usershares {
 ? list = children
 ? location = sdbox:/var/mail/%%d/%%n/sdbox
 ? prefix = shared/%%n/
 ? separator = /
 ? subscriptions = no
 ? type = shared
}
namespace virtual {
 ? list = children
 ? location = virtual:/var/mail/%d/%n/sdbox/virtual
 ? prefix = virtual/
 ? separator = /
 ? subscriptions = no
}
passdb {
 ? args = /usr/local/etc/dovecot/master-users
 ? driver = passwd-file
 ? master = yes
}
passdb {
 ? args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
 ? driver = ldap
}
plugin {
 ? acl = vfile:/usr/local/etc/dovecot/global-acls:cache_secs=300
 ? acl_shared_dict = proxy::acl
 ? fts = solr
 ? fts_autoindex = yes
 ? fts_solr = break-imap-search url=http://127.0.0.1:8983/solr/dovecot/
 ? mailbox_alias_new = Sent Messages
 ? mailbox_alias_new2 = Sent Items
 ? mailbox_alias_new3 = Deleted Messages
 ? mailbox_alias_old = Sent
 ? mailbox_alias_old2 = Sent
 ? mailbox_alias_old3 = Trash
 ? sieve = file:~/sieve;active=~/.dovecot.sieve
}
protocols = imap lmtp sieve
service auth {
 ? unix_listener /var/spool/postfix/private/auth {
 ??? group = postfix
 ??? mode = 0660
 ??? user = postfix
 ? }
 ? unix_listener auth-userdb {
 ??? group = mail
 ??? mode = 0600
 ??? user = vmail
 ? }
}
service config {
 ? unix_listener config {
 ??? user = vmail
 ? }
}
service dict {
 ? unix_listener dict {
 ??? group = mail
 ??? mode = 0660
 ??? user = vmail
 ? }
}
service doveadm {
 ? user = vmail
}
service imap-login {
 ? process_min_avail = 10
 ? service_count = 1
}
service imap-postlogin {
 ? executable = script-login /usr/local/etc/dovecot/post-login.sh
 ? user = $default_internal_user
}
service imap {
 ? executable = imap imap-postlogin
}
service lmtp {
 ? process_min_avail = 5
 ? unix_listener /var/spool/postfix/private/dovecot-lmtp {
 ??? group = mail
 ??? mode = 0666
 ??? user = vmail
 ? }
}
service managesieve-login {
 ? inet_listener sieve {
 ??? port = 4190
 ? }
 ? inet_listener sieve_deprecated {
 ??? port = 2000
 ? }
 ? process_min_avail = 0
 ? service_count = 1
}
ssl_cert = </usr/local/etc/dovecot/ssl/certs/mail.amfes.com.crt
ssl_key =? # hidden, use -P to show it
userdb {
 ? driver = prefetch
}
userdb {
 ? args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
 ? driver = ldap
}
verbose_proctitle = yes
protocol lmtp {
 ? mail_plugins = fts fts_solr acl zlib virtual sieve
 ? postmaster_address = postmaster at amfes.com
}
protocol lda {
 ? mail_plugins = fts fts_solr acl zlib virtual sieve
}
protocol imap {
 ? mail_max_userip_connections = 50
 ? mail_plugins = fts fts_solr acl zlib virtual imap_acl imap_zlib 
mailbox_alias
}
local 192.168.0.4 {
 ? protocol imap {
 ??? ssl_cert = </etc/letsencrypt/live/mail.amfes.com/fullchain.pem
 ??? ssl_key =? # hidden, use -P to show it
 ? }
}

Daniel

On 14.08.2018 21:23, Daniel Miller wrote:
> On 8/14/2018 12:55 AM, Aki Tuomi wrote:
>>
>> On 13.08.2018 19:51, Daniel Miller wrote:
>>> When doing a filesystem backup of an moderate sdbox mailstore
(300GB)
>>> - are there any files that can be safely excluded from the backup?
>>> Like *.log or *.backup?? Or are they all "vital" for
recovery?
>>>
>>> I'm already excluding the sdbox/virtual folders as it looks
like they
>>> get created and updated as needed.
>>>
>> Hi, can you provide 'doveconf -n'. sdbox/virtual folders are
not part of
>> sdbox mail format. In general, if you want to avoid data loss, you can
>> only omit dovecot.index.cache files, but omitting these can come with
>> high impact when they are regenerated. Omitting dovecot.index.log or
>> dovecot.index will cause loss of flags.
>>
>> Aki
>
> I have virtual folders enabled & defined.
>
> What does "high impact" mean?? A few seconds to a few minutes on
> initial mailbox opening for regeneration?? I can live with that as
> this is for emergency backup/restore purposes.
It means there will be nothing cached, it depends on your users and what
they do. If they only open first 20 mails it will not be that bad.
>
> mmap_disable = yes
If you are not using NFS, don't disable mmap.
> ? mailbox virtual/Flagged {
> ??? comment = All my flagged messages
> ??? special_use = \Flagged
> ? }
You can't "alias" folders like this.
> namespace virtual {
> ? list = children
> ? location = virtual:/var/mail/%d/%n/sdbox/virtual
> ? prefix = virtual/
> ? separator = /
> ? subscriptions = no
> }
?
You should really not put the virtual indexes inside sdbox directory,
this can confuse sdbox. You should put this under
/var/mail/%d/%n/virtual instead.

Aki

On 8/14/2018 10:41 PM, Aki Tuomi wrote:
>
> On 14.08.2018 21:23, Daniel Miller wrote:
>> On 8/14/2018 12:55 AM, Aki Tuomi wrote:
>>> On 13.08.2018 19:51, Daniel Miller wrote:
>>>> When doing a filesystem backup of an moderate sdbox mailstore
(300GB)
>>>> - are there any files that can be safely excluded from the
backup?
>>>> Like *.log or *.backup?? Or are they all "vital" for
recovery?
>>>>
>>>> I'm already excluding the sdbox/virtual folders as it looks
like they
>>>> get created and updated as needed.
>>>>
>>> Hi, can you provide 'doveconf -n'. sdbox/virtual folders
are not part of
>>> sdbox mail format. In general, if you want to avoid data loss, you
can
>>> only omit dovecot.index.cache files, but omitting these can come
with
>>> high impact when they are regenerated. Omitting dovecot.index.log
or
>>> dovecot.index will cause loss of flags.
>>>
>>> Aki
>> I have virtual folders enabled & defined.
>>
>> What does "high impact" mean?? A few seconds to a few minutes
on
>> initial mailbox opening for regeneration?? I can live with that as
>> this is for emergency backup/restore purposes.
> It means there will be nothing cached, it depends on your users and what
> they do. If they only open first 20 mails it will not be that bad.
Ok...so I'm *probably* ok excluding cache files then...
>> mmap_disable = yes
> If you are not using NFS, don't disable mmap.
I'm using NFS.? Mail server is running in VirtualBox guest, mounting the 
host's native storage via NFS.
>>  ? mailbox virtual/Flagged {
>>  ??? comment = All my flagged messages
>>  ??? special_use = \Flagged
>>  ? }
> You can't "alias" folders like this.
Maybe not...but it works <grin>.
>
>> namespace virtual {
>>  ? list = children
>>  ? location = virtual:/var/mail/%d/%n/sdbox/virtual
>>  ? prefix = virtual/
>>  ? separator = /
>>  ? subscriptions = no
>> }
>   
> You should really not put the virtual indexes inside sdbox directory,
> this can confuse sdbox. You should put this under
> /var/mail/%d/%n/virtual instead.
>
I haven't had any errors show up...but if it's incorrect then I'll
try
changing it.

Daniel