dovecot - May 2015 - Dovecot Replication - Architecture Endianness?

Hi all,

I've had an interesting use case come up which - to cut the story short 
- one way to solve the problem I am looking at may be to replicate a 
small number of mailboxes to a third remote server.

I've currently had replication running between my main dovecot machine 
and another remote VM for some time and working well (so I'm not new to 
replication and I've got a good working config), but I've a need to add 
a third to the mix for a select number of mailboxes.  The arch on both 
of those is Gentoo x86_64 and with latest 2.1.16 -hg.

I have attempted this so far by rsync'ing the initial Maildirs and then 
once the bulk of the data has been transferred rely on dovecot's 
replication to keep things in sync.  I figure that this should in theory 
mean that the subsequent updates in both directions are incremental and 
the bulk of the data gets moved while the device is here on my desk 
using rsync.

I've attempted to do this using a Raspberry Pi as a remote device, but 
when I set it up the dovecot replication process seems to need to start 
the replication over from scratch even after the rsync is done.  I know 
this is happening as the disk utilisation on the Pi skyrockets once the 
replication starts and I end up with thousands of double ups of all the 
mails ...  which defeats the entire point of the process.

If I do an identical configuration but on a third Gentoo x86_64 VM 
locally it all works as expected.  No double ups of mails and the 
"catchup" between the two devices is practically instant.  Same 
filesystem even.  The only difference appears to be the system architecture.

So main my question is this.  Is there a known architecture/endian 
limitation on replication?   I guess cross-arch replication is not 
something many people try but is it supposed to work anyway?

Has anyone else got replication working across different arch's?

Also is there a way to restrict replication users aside from a crude 
hack around system first and last UIDs?

Thanks,
Reuben

On 05/03/2015 01:48 PM, Reuben Farrelly wrote:
> Hi all,
> 
> I've had an interesting use case come up which - to cut the story short
- one way to solve the problem I am looking at may be to
> replicate a small number of mailboxes to a third remote server.
> 
> I've currently had replication running between my main dovecot machine
and another remote VM for some time and working well (so
> I'm not new to replication and I've got a good working config), but
I've a need to add a third to the mix for a select number of
> mailboxes.  The arch on both of those is Gentoo x86_64 and with latest
2.1.16 -hg.
> 
> I have attempted this so far by rsync'ing the initial Maildirs and then
once the bulk of the data has been transferred rely on
> dovecot's replication to keep things in sync.  I figure that this
should in theory mean that the subsequent updates in both
> directions are incremental and the bulk of the data gets moved while the
device is here on my desk using rsync.
> 
> I've attempted to do this using a Raspberry Pi as a remote device, but
when I set it up the dovecot replication process seems to
> need to start the replication over from scratch even after the rsync is
done.  I know this is happening as the disk utilisation
> on the Pi skyrockets once the replication starts and I end up with
thousands of double ups of all the mails ...  which defeats
> the entire point of the process.
> 
> If I do an identical configuration but on a third Gentoo x86_64 VM locally
it all works as expected.  No double ups of mails and
> the "catchup" between the two devices is practically instant. 
Same filesystem even.  The only difference appears to be the
> system architecture.
> 
> So main my question is this.  Is there a known architecture/endian
limitation on replication?   I guess cross-arch replication
> is not something many people try but is it supposed to work anyway?
I think you are bumping against Dovecot index endianess restrictions. I dont
think cross-arch dsync can currently work very
efficiently.
http://wiki2.dovecot.org/Design/Indexes/MainIndex?highlight=%28endian%29

> Also is there a way to restrict replication users aside from a crude hack
around system first and last UIDs?
You can set the userdb to return an empty mail_replica variable for users you
want to exclude from replication.
http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752

br,
Teemu Huovila

On 4/05/2015 11:06 PM, Teemu Huovila wrote:
> On 05/03/2015 01:48 PM, Reuben Farrelly wrote:
>> Hi all,
>>
>> I've had an interesting use case come up which - to cut the story
short - one way to solve the problem I am looking at may be to
>> replicate a small number of mailboxes to a third remote server.
>>
>> I've currently had replication running between my main dovecot
machine and another remote VM for some time and working well (so
>> I'm not new to replication and I've got a good working config),
but I've a need to add a third to the mix for a select number of
>> mailboxes.  The arch on both of those is Gentoo x86_64 and with latest
2.1.16 -hg.
>>
>> I have attempted this so far by rsync'ing the initial Maildirs and
then once the bulk of the data has been transferred rely on
>> dovecot's replication to keep things in sync.  I figure that this
should in theory mean that the subsequent updates in both
>> directions are incremental and the bulk of the data gets moved while
the device is here on my desk using rsync.
>>
>> I've attempted to do this using a Raspberry Pi as a remote device,
but when I set it up the dovecot replication process seems to
>> need to start the replication over from scratch even after the rsync is
done.  I know this is happening as the disk utilisation
>> on the Pi skyrockets once the replication starts and I end up with
thousands of double ups of all the mails ...  which defeats
>> the entire point of the process.
>>
>> If I do an identical configuration but on a third Gentoo x86_64 VM
locally it all works as expected.  No double ups of mails and
>> the "catchup" between the two devices is practically instant.
Same filesystem even.  The only difference appears to be the
>> system architecture.
>>
>> So main my question is this.  Is there a known architecture/endian
limitation on replication?   I guess cross-arch replication
>> is not something many people try but is it supposed to work anyway?
> I think you are bumping against Dovecot index endianess restrictions. I
dont think cross-arch dsync can currently work very
> efficiently.
> http://wiki2.dovecot.org/Design/Indexes/MainIndex?highlight=%28endian%29
Ok.  That explains why the rsync won't work.  But if I kick off a 
dovecot to dovecot replication (without doing the rsync first) will this 
work any better once the system catches up?  This assumes (possibly 
incorrectly - please correct me if I am wrong) that the index files 
themselves aren't dsync'd byte-by-byte but instead the metadata/content 
from them is sent instead, and then the indexes are written to remote 
disk by the remote dovecot in the right arch and format that the remote 
machine can read and understand.  Because if that's the case then I can 
probably make that work - just taking a hit on the initial sync which 
could take longer.

Even if this doesn't end up working I figure I'll get to learn a little 
more about the indexes themselves in the process.

Thanks for any advice,
Reuben

On 4/05/2015 11:06 PM, Teemu Huovila wrote:
>> Also is there a way to restrict replication users aside from a crude
hack around system first and last UIDs?
> You can set the userdb to return an empty mail_replica variable for users
you want to exclude from replication.
> http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752
>
> br,
> Teemu Huovila
One last question.  Is it possible to achieve this with system users and 
PAM or do I need to basically create a new static userdb for system users?

Could this be done via a per-user LDA setting or sieve?

Thanks,
Reuben