dovecot - Nov 2015 - fail: doveadm backup -s "<state>" with a huuge number of mailboxes

Hello,

I'm using 2.2.13.

For syncing I'm using repetitions of about the
following (simplified for readability):

    state_file=.s
    doveadm \
        -o imapc_master_user=<master> \
        -o imapc_password=<pw>      \
        -o imapc_host=<host>        \
        -o mail_fsync=never         \
        backup -s "$(<.s)" -u "<user>" -R imapc:
>.s_new
    mv .s_new .s

The user has about 4501 IMAP folders. This makes a huge
state output of about 128 kB. This in turn leads to an overly long
command line in the next iteration. (excuse: command line too long)

Is there any chance to solve this? I think, the only possibility would
be to have doveadm-backup -s read the state from a *file*, wouldn't it?

(Apart from the fact that I do not see a manpage for this, except
entries in the Wiki (but this may be due to poor packaging) of the
Debian dovecot package.)

    Best regards from Dresden/Germany
    Viele Gr??e aus Dresden
    Heiko Schlittermann
-- 
 SCHLITTERMANN.de ---------------------------- internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --------------- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20151112/f3485470/attachment.sig>

On 12 Nov 2015, at 12:56, Heiko Schlittermann <hs at schlittermann.de>
wrote:
> 
> Hello,
> 
> I'm using 2.2.13.
> 
> For syncing I'm using repetitions of about the
> following (simplified for readability):
> 
>    state_file=.s
>    doveadm \
>        -o imapc_master_user=<master> \
>        -o imapc_password=<pw>      \
>        -o imapc_host=<host>        \
>        -o mail_fsync=never         \
>        backup -s "$(<.s)" -u "<user>" -R
imapc: >.s_new
>    mv .s_new .s
> 
> The user has about 4501 IMAP folders. This makes a huge
> state output of about 128 kB. This in turn leads to an overly long
> command line in the next iteration. (excuse: command line too long)
> 
> Is there any chance to solve this? I think, the only possibility would
> be to have doveadm-backup -s read the state from a *file*, wouldn't it?
Right now no way to read it from a file. But is this Linux? I thought Linux had
unlimited (or at least very large) command line length nowadays.

Timo Sirainen <tss at iki.fi> (Do 12 Nov 2015 23:12:21 CET):
?
> > The user has about 4501 IMAP folders. This makes a huge
> > state output of about 128 kB. This in turn leads to an overly long
> > command line in the next iteration. (excuse: command line too long)
> > 
> > Is there any chance to solve this? I think, the only possibility would
> > be to have doveadm-backup -s read the state from a *file*,
wouldn't it?
> 
> Right now no way to read it from a file. But is this Linux? I thought Linux
had unlimited (or at least very large) command line length nowadays.

Yes, the product of size and number of the arguments is limited:

    *** 189961 args (    10 chars each) with total length 1899610
    ***  99503 args (    20 chars each) with total length 1990060
    ***   2087 args (  1000 chars each) with total length 2087000
    ***     16 args (128000 chars each) with total length 2048000

The upper limit for the sum of parameters seems to be ~2 MB

But,

     x127=$(perl -e 'print "x" x (127 * 2**10)')    # 127 KB 
     x128=$(perl -e 'print "x" x (128 * 2**10)')    # 128 KB

     /bin/echo $x127                                # works
     /bin/echo $x127 $x127                          # works

     /bin/echo $x128                                # breaks
     /bin/echo $x128 $x128                          # breaks

It seems that a *single* command line parameter is limited to somewhat
near 128 kB. The complete amount of command line data is near 2 MB.

Conclusion: doveadm sync should be able to read a state *file*. IMHO

    Best regards from Dresden/Germany
    Viele Gr??e aus Dresden
    Heiko Schlittermann
-- 
 SCHLITTERMANN.de ---------------------------- internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --------------- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20151112/9f4d3918/attachment-0001.sig>