dovecot - Feb 2016 - Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder

Hello,

I have an el-cheapo shared hosting account on Dreamhost, and have had it
for a very long time.

For the most part everything usually works fairly well, considering I do
keep a lot of folders, and mail, on some of my accounts.

They are running dovecot, but still don't have a response as to the
version, or doveconf -n output yet.

My problem is, one of my most used folders, which was working fine up
until a week or so ago, stopped loading the messages, and after some
frustrating troubleshooting via email with people who don't listen very
well, I finally got a tarball of this folder, and they are using maildir.

There are about 24,000 messages in there (non-zero-byte files). This
number sounds about right. All other folders (including INBOX, Sent,
etc) are still working fine.

The problem, though, is there are over 815,000 zero-byte-files in the
cur directory, all showing as hardlinks (looks like maybe a whole bunch
of duplicates for each of the real message files in the cur directory).

There are also 43 non-zero-byte message files in the new directory, and 
1,515 of these zero-byte hardlinks (to message files in the new directory).

There are also no non-zero-byte message files in the tmp directory, but
there are 52 of the hardlinks, linked to something in the new directory.

I've never seen any of these kinds of zero-byte files before on the one
server I managed for a long time (not shared, just used for a single
domain).

Anyone ever seen this before?

Would running:

doveadm index -u myuser *

or

doveadm force-resync -u myuser *

be appropriate commands to try to repair the damage (whatever it is)?

Any other commands I could suggest running?

Thanks. I know I haven't given much to go on.

Charles
/*




*/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 8 Feb 2016, Charles Marcus wrote:
> My problem is, one of my most used folders, which was working fine up
> until a week or so ago, stopped loading the messages, and after some
> frustrating troubleshooting via email with people who don't listen very
> well, I finally got a tarball of this folder, and they are using maildir.
>
> There are about 24,000 messages in there (non-zero-byte files). This
> number sounds about right. All other folders (including INBOX, Sent,
> etc) are still working fine.
>
> The problem, though, is there are over 815,000 zero-byte-files in the
> cur directory, all showing as hardlinks (looks like maybe a whole bunch
> of duplicates for each of the real message files in the cur directory).
"zero-byte-files ... showing as hardlinks"

You mean this:

hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1

?

This is a pseudo-notation of tar to indicate hardlinks. This is no 
zero-byte file. yes, these entries are duplicates of other messages.

Note, https://en.wikipedia.org/wiki/Hard_link
if two files are hardlinked together, there is no "to" or
"from". You
cannot tell, which existed before. They just indicate that those directory 
entries point to the same physical file with the same access rights and 
times and data.
Extract the tar file to a Unix-like, inode-based filesystem supporting 
hardlinks to see.
> There are also 43 non-zero-byte message files in the new directory, and
> 1,515 of these zero-byte hardlinks (to message files in the new directory).
> There are also no non-zero-byte message files in the tmp directory, but
> there are 52 of the hardlinks, linked to something in the new directory.
if there is such entry in the tmp directory, it indicates a failed 
delivery attempt. If one entry in "tmp" is hardlinked to one entry in 
"new" of the same mailbox, it may indicate that the message was to
spool
into another mailbox (via hardlink, too), which failed fatally.

Is it possible that those messages are messages from your hoster and the 
message was to spool to many user mailboxes?
> I've never seen any of these kinds of zero-byte files before on the one
> server I managed for a long time (not shared, just used for a single
> domain).
See above.
> Anyone ever seen this before?
What does "stopped loading messages" mean?

The MUA cannot download messages?

Check if the server returns OK selecting the mailbox and if the numbers 
match, see 
http://wiki2.dovecot.org/TestInstallation

You could use

a select INBOX
b copy 1 "mailbox-name"

to copy a new message there and re-select the broken mailbox and compare 
the numbers.

Also you could test, if the server crashes on a message in the mailbox, 
try

c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)]
c FETCH 1:* FLAGS
c FETCH 1:* BODY[TEXT]
> Would running:
>
> doveadm index -u myuser *
only, if the index is corrupt.
> or
>
> doveadm force-resync -u myuser *
you can run doveadm, but cannot doveconf on the server?
>
> be appropriate commands to try to repair the damage (whatever it is)?
>
> Any other commands I could suggest running?
>
> Thanks. I know I haven't given much to go on.
- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb
hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco
ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY
Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm
xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL
4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg==oCQf
-----END PGP SIGNATURE-----

Just to close this out... as I said, this is on a hosted (cheapo)
dreamhost account, and their support was - well, crap.

I was trying to provide suggestions for commands they could run, but I
finally gave up.

They were able to get me a 'tarbell' (thought it was a typo when I first
saw it, but they used the exact same word many times over multiple
emails, so... ) of the folder, so I was able to restore it to my own
dovecot server, and I discovered it was actually crashing dovecot after
a few minutes after a select (clicked on folder).

Interestingly, I accidentally fixed it by simply decompressing it on my
Windows box - all of those hardlinks gave errors, and when it was
finished, they were not in the resulting uncompressed folder. Restoring
that to my dovecot server and deleting the index files resulted in a
successful restoration of the mails.


On 2/9/2016 2:02 AM, Steffen Kaiser <skdovecot at smail.inf.fh-brs.de>
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Mon, 8 Feb 2016, Charles Marcus wrote:
>
>> My problem is, one of my most used folders, which was working fine up
>> until a week or so ago, stopped loading the messages, and after some
>> frustrating troubleshooting via email with people who don't listen
very
>> well, I finally got a tarball of this folder, and they are using
maildir.
>>
>> There are about 24,000 messages in there (non-zero-byte files). This
>> number sounds about right. All other folders (including INBOX, Sent,
>> etc) are still working fine.
>>
>> The problem, though, is there are over 815,000 zero-byte-files in the
>> cur directory, all showing as hardlinks (looks like maybe a whole bunch
>> of duplicates for each of the real message files in the cur directory).
> "zero-byte-files ... showing as hardlinks"
>
> You mean this:
>
> hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1
>
> ?
>
> This is a pseudo-notation of tar to indicate hardlinks. This is no 
> zero-byte file. yes, these entries are duplicates of other messages.
>
> Note, https://en.wikipedia.org/wiki/Hard_link
> if two files are hardlinked together, there is no "to" or
"from". You
> cannot tell, which existed before. They just indicate that those directory 
> entries point to the same physical file with the same access rights and 
> times and data.
> Extract the tar file to a Unix-like, inode-based filesystem supporting 
> hardlinks to see.
>
>> There are also 43 non-zero-byte message files in the new directory, and
>> 1,515 of these zero-byte hardlinks (to message files in the new
directory).
>> There are also no non-zero-byte message files in the tmp directory, but
>> there are 52 of the hardlinks, linked to something in the new
directory.
> if there is such entry in the tmp directory, it indicates a failed 
> delivery attempt. If one entry in "tmp" is hardlinked to one
entry in
> "new" of the same mailbox, it may indicate that the message was
to spool
> into another mailbox (via hardlink, too), which failed fatally.
>
> Is it possible that those messages are messages from your hoster and the 
> message was to spool to many user mailboxes?
>
>> I've never seen any of these kinds of zero-byte files before on the
one
>> server I managed for a long time (not shared, just used for a single
>> domain).
> See above.
>
>> Anyone ever seen this before?
> What does "stopped loading messages" mean?
>
> The MUA cannot download messages?
>
> Check if the server returns OK selecting the mailbox and if the numbers 
> match, see 
> http://wiki2.dovecot.org/TestInstallation
>
> You could use
>
> a select INBOX
> b copy 1 "mailbox-name"
>
> to copy a new message there and re-select the broken mailbox and compare 
> the numbers.
>
> Also you could test, if the server crashes on a message in the mailbox, 
> try
>
> c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)]
> c FETCH 1:* FLAGS
> c FETCH 1:* BODY[TEXT]
>
>> Would running:
>>
>> doveadm index -u myuser *
> only, if the index is corrupt.
>
>> or
>>
>> doveadm force-resync -u myuser *
> you can run doveadm, but cannot doveconf on the server?
>
>> be appropriate commands to try to repair the damage (whatever it is)?
>>
>> Any other commands I could suggest running?
>>
>> Thanks. I know I haven't given much to go on.
> - -- 
> Steffen Kaiser
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb
> hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco
> ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY
> Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm
> xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL
> 4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg=> =oCQf
> -----END PGP SIGNATURE-----