Dovecot version: 2.2.34 doveadm import tries to call open() on the source indexes/logs of mdbox data, but even if it should work as it is just a read-related call, it fails on read-only filesystems. The main use case for read-only filesystem is restoring from backup: the massive deduplicated backup data is exposed using FUSE. For deduplication to work efficiently, we have to backup the whole mail storage and thus the only way to handle the restore to mount the whole data. I would like to ask for some proper solution for this problem. Suggestions: first, the doveadm import should never modify the indexes, and second: the open() calls should respect the read-only filesystem and should fall back to "r" flag when using open. The problem goes further when using SIS (which goes beyond multiple terrabytes). The only workaround currently is to export the whole mdbox data to a writeable storage, and parametrize the import command to use that and the mounted SIS data separately for import...which is just problematic, ugly and error prune. Thank you for the continous work on dovecot, I hope you get this feedback in a good way. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/967594b5/attachment.html>
Hi, you can use INDEX=/writable/path/%u in your mail_location setting to define location for the required index data when importing. Also possibly you would need to define writable location for CONTROL and VOLATILEDIR. see https://wiki.dovecot.org/MailLocation <https://wiki.dovecot.org/MailLocation> Sami> On 14 Jan 2019, at 11.23, hby <hby at hby.hu> wrote: > > Dovecot version: 2.2.34 > > doveadm import tries to call open() on the source indexes/logs of mdbox data, but even if it should work as it is just a read-related call, it fails on read-only filesystems. > > The main use case for read-only filesystem is restoring from backup: the massive deduplicated backup data is exposed using FUSE. For deduplication to work efficiently, we have to backup the whole mail storage and thus the only way to handle the restore to mount the whole data. > > I would like to ask for some proper solution for this problem. > > Suggestions: first, the doveadm import should never modify the indexes, and second: the open() calls should respect the read-only filesystem and should fall back to "r" flag when using open. > The problem goes further when using SIS (which goes beyond multiple terrabytes). > > The only workaround currently is to export the whole mdbox data to a writeable storage, and parametrize the import command to use that and the mounted SIS data separately for import...which is just problematic, ugly and error prune. > > Thank you for the continous work on dovecot, I hope you get this feedback in a good way.-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/2ec96d6e/attachment-0001.html>
Hi, Thanks for the fast answer! Let me clear the situation: 1. I want to import from a read-only filesystem where the mdbox files are 2. The indexes are necessary because of the mdbox file layout, but the import process cannot open them 3. If I specify random writable dir for indexes the import does not do anything, no message will be imported as obviously the indexes are necessary for mdbox 4. The CONTROL parameter is only usable for maildir format To make it more specific, I'm using the following command to restore the messages right now: doveadm -o mail_location=mdbox:/backup/restore/XXX -o mail_attachment_dir=/mnt/_attachments backup -u XXX doveadm dsync-server -u XXX For this to work, I have to copy the full mdbox data to the writeable /backup/restore/XXX. If I only change the INDEX location, the process still tries to open the log files and fails with open() error on them. Sami Ketola <sami.ketola at dovecot.fi> ezt ?rta (id?pont: 2019. jan. 14., H, 10:46):> Hi, > > you can use INDEX=/writable/path/%u in your mail_location setting to > define location for the required index data when importing. > Also possibly you would need to define writable location for CONTROL and > VOLATILEDIR. > > see https://wiki.dovecot.org/MailLocation > > Sami > > On 14 Jan 2019, at 11.23, hby <hby at hby.hu> wrote: > > Dovecot version: 2.2.34 > > doveadm import tries to call open() on the source indexes/logs of mdbox > data, but even if it should work as it is just a read-related call, it > fails on read-only filesystems. > > The main use case for read-only filesystem is restoring from backup: the > massive deduplicated backup data is exposed using FUSE. For deduplication > to work efficiently, we have to backup the whole mail storage and thus the > only way to handle the restore to mount the whole data. > > I would like to ask for some proper solution for this problem. > > Suggestions: first, the doveadm import should never modify the indexes, > and second: the open() calls should respect the read-only filesystem and > should fall back to "r" flag when using open. > The problem goes further when using SIS (which goes beyond multiple > terrabytes). > > The only workaround currently is to export the whole mdbox data to a > writeable storage, and parametrize the import command to use that and the > mounted SIS data separately for import...which is just problematic, ugly > and error prune. > > Thank you for the continous work on dovecot, I hope you get this feedback > in a good way. > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/a1a4126e/attachment-0001.html>