> On Dec 10, 2022, at 11:09 AM, John Tulp <johntulp at tulpholdings.com> wrote: > > > > On Fri, 2022-12-09 at 20:03 -0800, Doug Hardie wrote: >>> On Dec 4, 2022, at 1:42 PM, Doug Hardie <bc979 at lafn.org> wrote: >>> >>>> On Dec 3, 2022, at 11:50 PM, Doug Hardie <bc979 at lafn.org> wrote: >>>> >>>> I started to investigate using doveadm backup to backup my mail >>>> system. I have a small number of users and the mail store is not >>>> large. It uses maildir format. I setup a test system that is not >>>> connected to the internet and started up dovecot. I used the >>>> following command to backup one user: >>>> >>>> doveadm backup -u ben remote:test >>>> >>>> ben is the user is in the mail store. Test is the actual server >>>> name. That worked just fine. The maildir was copied completely >>>> (as best as I can tell with ls). Then I tried the second user: >>>> >>>> doveadm backup -u jean remote:test >>>> >>>> This gives 2 error messages: >>>> >>>> doveadm(jean)[]<McRYOFI0jGN1MwAAmC+70w>: Error: Mailbox INBOX: >>>> Failed to get attribute >>>> vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox >>>> attributes not enabled >>>> >>>> doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command >>>> returned error 65: ssh test doveadm dsync-server -ujean -U >>>> >>>> In addition, the maildir directories are created, but there are no >>>> emails in any of them (e.g., cur). What is the problem with the >>>> 2nd and why does it behave differently from the first? >>> >>> I managed to resolve most of the issue. I use pigeonhole on the >>> primary server. Apparently not having pigeonhole installed on the >>> test machine caused the errors above. The test machine was never >>> intended to receive mail hence, no need to install pigeonhole as the >>> LDA would never be used. However, when the backup was running, it >>> choked on transferring the sieve file. I have no idea where the >>> mentioned file resides as I couldn't find it anywhere on the primary >>> server. Installing pigeonhole resolved the issue for all but one >>> user. With that user, I get the following error messages: >>> >>> doveadm(doug)[]<NAfSLTASjWNWTAAAmC+70w>: Panic: file >>> istream-seekable.c: line 238 (read_from_buffer): assertion failed: >>> (*ret_r > 0) >>> Abort >>> >>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: read(test) failed: >>> EOF (last sent=mail, last recv=mail_request (EOL)) >>> >>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: Remote command >>> returned error 134: ssh test doveadm dsync-server -udoug -U >>> >>> In addition, only a few cur files are transferred in INBOX. >>> Repeating the backup generates the same errors and no additional >>> emails are transferred. I am wondering if the problem is something >>> in the INBOX. >> >> >> >> >> I have pretty much reached a dead end. I am unable to determine the >> cause of this abnormal termination. The logged messages don't give >> much help. I can't tell if it is the primary server or test machine >> that is terminating ssh. I have setup a second test machine. Both of >> the test machines are raspberry pi 4s. The real backup machine is >> intel. On both the test machines, 2 of the three users are backedup >> properly with no errors. Only one use has the issue. It builds the >> directory structure correctly then starts transferring the INBOX cur >> files. Eight files are transferred correctly before it stop. It is >> the same set of 8 on both test machines. That leads me to believe >> there is something funny in one of the messages, but which one. There >> are over 100 messages in that directory. The order of file transfer >> is not by date. >> >> >> How do I get doveadm to tell me which file is being transferred when >> the problem occurs? > > if the problem is in one of the messages, perhaps change the test set. > for convenience, can temporarily rename/move things to get them out of > the way. divide test set in half, then in half again, etc., to zero in > on the one causing the issue, that'll show if a particular message is > problem or not. > > once you find a problem message, if the why isn't obvious, i'd try > looking at it in hex, check permissions, etc.I have found the cause for the errors: Somehow I had overlooked the -D argument to doveadm. That is really helpful in diagnosing backup issues. I discovered that there was one email that was 130+ MB. Doveadm cannot handle that. It appears there is a message size limit somewhere. I don't know if that is changeable or not. Anyway, removing that email from the mailbox, deleting it from the Trash, and purging the account enabled backup to work properly. I don't get too many emails that large, but it does happen occasionally. How do I go about telling dovecot to handle those? -- Doug
-- John Tulp <johntulp at tulpex.com> tulpex On Sun, 2022-12-11 at 00:05 -0800, Doug Hardie wrote:> > On Dec 10, 2022, at 11:09 AM, John Tulp <johntulp at tulpholdings.com> wrote: > > > > > > > > On Fri, 2022-12-09 at 20:03 -0800, Doug Hardie wrote: > >>> On Dec 4, 2022, at 1:42 PM, Doug Hardie <bc979 at lafn.org> wrote: > >>> > >>>> On Dec 3, 2022, at 11:50 PM, Doug Hardie <bc979 at lafn.org> wrote: > >>>> > >>>> I started to investigate using doveadm backup to backup my mail > >>>> system. I have a small number of users and the mail store is not > >>>> large. It uses maildir format. I setup a test system that is not > >>>> connected to the internet and started up dovecot. I used the > >>>> following command to backup one user: > >>>> > >>>> doveadm backup -u ben remote:test > >>>> > >>>> ben is the user is in the mail store. Test is the actual server > >>>> name. That worked just fine. The maildir was copied completely > >>>> (as best as I can tell with ls). Then I tried the second user: > >>>> > >>>> doveadm backup -u jean remote:test > >>>> > >>>> This gives 2 error messages: > >>>> > >>>> doveadm(jean)[]<McRYOFI0jGN1MwAAmC+70w>: Error: Mailbox INBOX: > >>>> Failed to get attribute > >>>> vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox > >>>> attributes not enabled > >>>> > >>>> doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command > >>>> returned error 65: ssh test doveadm dsync-server -ujean -U > >>>> > >>>> In addition, the maildir directories are created, but there are no > >>>> emails in any of them (e.g., cur). What is the problem with the > >>>> 2nd and why does it behave differently from the first? > >>> > >>> I managed to resolve most of the issue. I use pigeonhole on the > >>> primary server. Apparently not having pigeonhole installed on the > >>> test machine caused the errors above. The test machine was never > >>> intended to receive mail hence, no need to install pigeonhole as the > >>> LDA would never be used. However, when the backup was running, it > >>> choked on transferring the sieve file. I have no idea where the > >>> mentioned file resides as I couldn't find it anywhere on the primary > >>> server. Installing pigeonhole resolved the issue for all but one > >>> user. With that user, I get the following error messages: > >>> > >>> doveadm(doug)[]<NAfSLTASjWNWTAAAmC+70w>: Panic: file > >>> istream-seekable.c: line 238 (read_from_buffer): assertion failed: > >>> (*ret_r > 0) > >>> Abort > >>> > >>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: read(test) failed: > >>> EOF (last sent=mail, last recv=mail_request (EOL)) > >>> > >>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: Remote command > >>> returned error 134: ssh test doveadm dsync-server -udoug -U > >>> > >>> In addition, only a few cur files are transferred in INBOX. > >>> Repeating the backup generates the same errors and no additional > >>> emails are transferred. I am wondering if the problem is something > >>> in the INBOX. > >> > >> > >> > >> > >> I have pretty much reached a dead end. I am unable to determine the > >> cause of this abnormal termination. The logged messages don't give > >> much help. I can't tell if it is the primary server or test machine > >> that is terminating ssh. I have setup a second test machine. Both of > >> the test machines are raspberry pi 4s. The real backup machine is > >> intel. On both the test machines, 2 of the three users are backedup > >> properly with no errors. Only one use has the issue. It builds the > >> directory structure correctly then starts transferring the INBOX cur > >> files. Eight files are transferred correctly before it stop. It is > >> the same set of 8 on both test machines. That leads me to believe > >> there is something funny in one of the messages, but which one. There > >> are over 100 messages in that directory. The order of file transfer > >> is not by date. > >> > >> > >> How do I get doveadm to tell me which file is being transferred when > >> the problem occurs? > > > > if the problem is in one of the messages, perhaps change the test set. > > for convenience, can temporarily rename/move things to get them out of > > the way. divide test set in half, then in half again, etc., to zero in > > on the one causing the issue, that'll show if a particular message is > > problem or not. > > > > once you find a problem message, if the why isn't obvious, i'd try > > looking at it in hex, check permissions, etc. > > I have found the cause for the errors: Somehow I had overlooked the -D argument to doveadm. That is really helpful in diagnosing backup issues. I discovered that there was one email that was 130+ MB. Doveadm cannot handle that. It appears there is a message size limit somewhere. I don't know if that is changeable or not. Anyway, removing that email from the mailbox, deleting it from the Trash, and purging the account enabled backup to work properly. I don't get too many emails that large, but it does happen occasionally. How do I go about telling dovecot to handle those? > > -- Doug > >sorry, no clue. i did find this link quickly... https://doc.dovecot.org/configuration_manual/quota/ looks like the following section near the bottom of page may apply ? "Maximum Saved Mail Size" again, sorry, email is not "my thing" (chuckle inserted here).
> On Dec 11, 2022, at 12:42 AM, John Tulp <johntulp at tulpex.com> wrote: > > > -- > John Tulp <johntulp at tulpex.com> > tulpex > > On Sun, 2022-12-11 at 00:05 -0800, Doug Hardie wrote: >>> On Dec 10, 2022, at 11:09 AM, John Tulp <johntulp at tulpholdings.com> wrote: >>> >>> >>> >>> On Fri, 2022-12-09 at 20:03 -0800, Doug Hardie wrote: >>>>> On Dec 4, 2022, at 1:42 PM, Doug Hardie <bc979 at lafn.org> wrote: >>>>> >>>>>> On Dec 3, 2022, at 11:50 PM, Doug Hardie <bc979 at lafn.org> wrote: >>>>>> >>>>>> I started to investigate using doveadm backup to backup my mail >>>>>> system. I have a small number of users and the mail store is not >>>>>> large. It uses maildir format. I setup a test system that is not >>>>>> connected to the internet and started up dovecot. I used the >>>>>> following command to backup one user: >>>>>> >>>>>> doveadm backup -u ben remote:test >>>>>> >>>>>> ben is the user is in the mail store. Test is the actual server >>>>>> name. That worked just fine. The maildir was copied completely >>>>>> (as best as I can tell with ls). Then I tried the second user: >>>>>> >>>>>> doveadm backup -u jean remote:test >>>>>> >>>>>> This gives 2 error messages: >>>>>> >>>>>> doveadm(jean)[]<McRYOFI0jGN1MwAAmC+70w>: Error: Mailbox INBOX: >>>>>> Failed to get attribute >>>>>> vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox >>>>>> attributes not enabled >>>>>> >>>>>> doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command >>>>>> returned error 65: ssh test doveadm dsync-server -ujean -U >>>>>> >>>>>> In addition, the maildir directories are created, but there are no >>>>>> emails in any of them (e.g., cur). What is the problem with the >>>>>> 2nd and why does it behave differently from the first? >>>>> >>>>> I managed to resolve most of the issue. I use pigeonhole on the >>>>> primary server. Apparently not having pigeonhole installed on the >>>>> test machine caused the errors above. The test machine was never >>>>> intended to receive mail hence, no need to install pigeonhole as the >>>>> LDA would never be used. However, when the backup was running, it >>>>> choked on transferring the sieve file. I have no idea where the >>>>> mentioned file resides as I couldn't find it anywhere on the primary >>>>> server. Installing pigeonhole resolved the issue for all but one >>>>> user. With that user, I get the following error messages: >>>>> >>>>> doveadm(doug)[]<NAfSLTASjWNWTAAAmC+70w>: Panic: file >>>>> istream-seekable.c: line 238 (read_from_buffer): assertion failed: >>>>> (*ret_r > 0) >>>>> Abort >>>>> >>>>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: read(test) failed: >>>>> EOF (last sent=mail, last recv=mail_request (EOL)) >>>>> >>>>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: Remote command >>>>> returned error 134: ssh test doveadm dsync-server -udoug -U >>>>> >>>>> In addition, only a few cur files are transferred in INBOX. >>>>> Repeating the backup generates the same errors and no additional >>>>> emails are transferred. I am wondering if the problem is something >>>>> in the INBOX. >>>> >>>> >>>> >>>> >>>> I have pretty much reached a dead end. I am unable to determine the >>>> cause of this abnormal termination. The logged messages don't give >>>> much help. I can't tell if it is the primary server or test machine >>>> that is terminating ssh. I have setup a second test machine. Both of >>>> the test machines are raspberry pi 4s. The real backup machine is >>>> intel. On both the test machines, 2 of the three users are backedup >>>> properly with no errors. Only one use has the issue. It builds the >>>> directory structure correctly then starts transferring the INBOX cur >>>> files. Eight files are transferred correctly before it stop. It is >>>> the same set of 8 on both test machines. That leads me to believe >>>> there is something funny in one of the messages, but which one. There >>>> are over 100 messages in that directory. The order of file transfer >>>> is not by date. >>>> >>>> >>>> How do I get doveadm to tell me which file is being transferred when >>>> the problem occurs? >>> >>> if the problem is in one of the messages, perhaps change the test set. >>> for convenience, can temporarily rename/move things to get them out of >>> the way. divide test set in half, then in half again, etc., to zero in >>> on the one causing the issue, that'll show if a particular message is >>> problem or not. >>> >>> once you find a problem message, if the why isn't obvious, i'd try >>> looking at it in hex, check permissions, etc. >> >> I have found the cause for the errors: Somehow I had overlooked the -D argument to doveadm. That is really helpful in diagnosing backup issues. I discovered that there was one email that was 130+ MB. Doveadm cannot handle that. It appears there is a message size limit somewhere. I don't know if that is changeable or not. Anyway, removing that email from the mailbox, deleting it from the Trash, and purging the account enabled backup to work properly. I don't get too many emails that large, but it does happen occasionally. How do I go about telling dovecot to handle those? >> >> -- Doug >> >> > > sorry, no clue. i did find this link quickly... > > https://doc.dovecot.org/configuration_manual/quota/ > > looks like the following section near the bottom of page may apply ? > > "Maximum Saved Mail Size" > > again, sorry, email is not "my thing" (chuckle inserted here).As best as I can tell, that requires quota to be setup as plugin which I do not. Also, it indicates that the default value is 0 which is unlimited. -- Doug