dovecot - Jul 2007 - Small problem with src/lib/mountpoint.c [now with patch attached!]

Greetings -

[SIGH: I hit the "Send" button instead of "Attach". 
Here's Take 2...]

Whilst playing with getting quotas from NFS-mounted filestores I've  
just discovered a slight problem with src/lib/mountpoint.c

I had things working on a Solaris 10 test machine which had the  
mailstore mounted 'normally' using an entry in /etc/vfstab.

However when I changed to use the automounter for the mailstore  
obtaining the quota broke.

The problem lies with mountpoint_get().  On a 'normally' mounted  
filestore this obtains the device_path of the remote filestore in the  
form

     remoteservername.some.domain:/remote/path

However when the mailstore was automounted I was instead getting back

     auto_direct

which is the name of the automounter map we're using.

The /etc/mnttab file has TWO entries for the auto-mounted filestore,  
like this:

auto_direct     /mailstore/messages/p   autofs   
direct,ignore,dev=4740014       1184648400
crypt2.york.ac.uk:/vol/vol9/p   /mailstore/messages/p   nfs      
proto=tcp,xattr,dev=4700386     1184668792

Although there are two entries they have different device numbers.   
The mountpoint_get() function attempts to guard against this by  
checking the device number obtained matches that of the path being  
located.

However for some reason I don't understand when getmntent() returns  
the FIRST of the above two entries it supplies the device number of  
the SECOND.  Thus the device numbers match in the mountpoint_get()  
test and it thinks it has found the correct entry.  This leads to the  
device_path being set to the map name -- auto_direct -- instead of  
the host:/path

My fix is to add an additional fstype for the test to ignore:  
"autofs".  The loop skips over any such mounts (along with the  
existing tests for "swap" and "ignore").

I'm attaching my minimal patch in case it's of help to anyone else.

(Timo: would it be possible to get this added into the distribution  
code if you agree it's right?)

Cheers,
Mike Brudenell

-- 
The Computing Service, University of York, Heslington, York Yo10 5DD, UK
Tel:+44-1904-433811  FAX:+44-1904-433740

* Unsolicited commercial e-mail is NOT welcome at this e-mail address. *

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mountpoint.c.patch
Type: application/octet-stream
Size: 617 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070717/a6631a1e/attachment-0002.obj>
-------------- next part --------------

On 17.7.2007, at 13.55, Mike Brudenell wrote:
> auto_direct     /mailstore/messages/p   autofs   
> direct,ignore,dev=4740014       1184648400
> crypt2.york.ac.uk:/vol/vol9/p   /mailstore/messages/p   nfs      
> proto=tcp,xattr,dev=4700386     1184668792
>
> Although there are two entries they have different device numbers.   
> The mountpoint_get() function attempts to guard against this by  
> checking the device number obtained matches that of the path being  
> located.
What do you mean different device numbers? Both have the same mount  
path, so how can they be different?
> (Timo: would it be possible to get this added into the distribution  
> code if you agree it's right?)
I suppose..

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070717/e0b1242e/attachment-0002.bin>

Hi Timo, et al -

On 17 Jul 2007, at 18:12, Timo Sirainen wrote:
> On 17.7.2007, at 13.55, Mike Brudenell wrote:
>
>> auto_direct     /mailstore/messages/p   autofs   
>> direct,ignore,dev=4740014       1184648400
>> crypt2.york.ac.uk:/vol/vol9/p   /mailstore/messages/p   nfs      
>> proto=tcp,xattr,dev=4700386     1184668792
>>
>> Although there are two entries they have different device  
>> numbers.  The mountpoint_get() function attempts to guard against  
>> this by checking the device number obtained matches that of the  
>> path being located.
>
> What do you mean different device numbers? Both have the same mount  
> path, so how can they be different?
I'm afraid I don't understand the innards of the automounter.  All I  
can say is what I see...

We use the automounter extensively as it gives central management for  
shares and saves having to edit /etc/vfstab files on umpteen machines  
all over the place.

A host making use of the automounter has entries in the /etc/mnttab  
for all the filestores available to it TO BE mounted, even when they  
AREN'T ACTUALLY mounted at the moment.

When something access a file or directory in the filestore the  
automounter leaps in and furtively mounts the filestore before the  
granting the access.  This results in a SECOND entry for that  
filestore in the /etc/mnttab file, and has the same mount point as  
the first entry.

The two entries have different "dev=NNNNN" values in their options  
fields to distinguish them.  The thing I find confusing/surprising is  
that:

1.  When I visually inspect the contents of /etc/mnttab the two  
entries for
     /mailstore/messages/p have DIFFERENT "dev=NNNNN" entries, but

2.  When I put debug logging into Dovecot's loop that iterates  
through the
     /etc/mnttab entries using getmntent() both entries come through  
with the
     SAME "dev=NNNNN" value ... the first (autofs) entry is returned  
with the
     device number from the second (real) mount's entry in the file.

     If the different (unique) device numbers were being yielded by  
getmntent()
     then there wouldn't be a problem: Dovecot is checking these and  
would
     skip the first (autofs) entry as the number was wrong.

I'll do some testing using a minimal test program to check this  
happens then.

Cheers,
Mike B-)

-- 
The Computing Service, University of York, Heslington, York Yo10 5DD, UK
Tel:+44-1904-433811  FAX:+44-1904-433740

* Unsolicited commercial e-mail is NOT welcome at this e-mail address. *

Ah!  After a bit of test programming I understand what's going on now.

If I'm right then there are a few incorrect assumptions in Dovecot's  
logic.  (Sorry! :-)


On 17 Jul 2007, at 18:12, Timo Sirainen wrote:
>> auto_direct     /mailstore/messages/p   autofs   
>> direct,ignore,dev=4740014       1184648400
>> crypt2.york.ac.uk:/vol/vol9/p   /mailstore/messages/p   nfs      
>> proto=tcp,xattr,dev=4700386     1184668792
>>
>> Although there are two entries they have different device  
>> numbers.  The mountpoint_get() function attempts to guard against  
>> this by checking the device number obtained matches that of the  
>> path being located.
>
> What do you mean different device numbers? Both have the same mount  
> path, so how can they be different?
Looking again at the above two example lines from our /etc/mnttab I  
see that the first entry includes "ignore" as an option.  As I  
recollected that Dovecot's code in mountpoint_get() checks for  
filesystems marked "ignore" and should, umm, ignore them(!) when  
searching for the correct entry in the mount tab ... yet it wasn't  
working.

So I've just written a minimal program to display all the entries of / 
etc/mnttab and their fields using getmntent() and now think Dovecot's  
checks are incorrect, at least on Solaris...


PROBLEM 1 :: Skipping swap and ignored filestores is incorrect
--------------------------------------------------------------
The mountpoint_get() function in lib/mountpoint.c uses getmntent() to  
fetch each entry from /etc/mnttab in turn.  It then checks the  
ent.mnt_fstype field to see if it is MNTTYPE_SWAP ("swap") or  
MNTTYPE_IGNORE ("ignore") and, if it is, the entry is skipped.

However I this appears to be incorrect for Solaris 10 (and probably  
earlier versions too).  For example:
   * swap does NOT return "swap" as the value of ent.mnt_fstype
   * ignored filestores do NOT return "ignore" as the value of  
ent.mnt_fstype

Instead you get:

   * swap returns "swap" has the value of ent.mnt_special and
"tmpfs"
for
     ent.mnt_fstype

   * ignored filesystems return "ignore" as one of the options within
the
     ent.mnt_mntopts field.  (NB: this might be one of several comma- 
separated
     options and could appear anywhere in the list.)

So I now believe that:

1.  My fix of also skipping filestores whose ent.mnt_fstype is  
"autofs" is not
     fully correct -- the problem should really be dealt with by  
properly
     checking for the "ignore" option instead;

2.  The "ignore" option should be being checked within  
ent.mnt_mntopts instead
     of ent.mnt_fstype; this would deal with the problem of the  
template autofs
     entry being picked up and used as it has the "ignore" option.

3.  Swap should be detected by checking either ent.mnt_special being  
"swap" or
     ent.mnt_fstype being "tmpfs" -- I'm not sure which is
correct,
or if both
     tests are needed.


PROBLEM 2 :: Matching an entry in /etc/mnttab using getmntent() is  
incorrect
------------------------------------------------------------------------ 
----
As for the oddity with device numbers I mentioned previously, I now  
understand this and think it shows up another incorrect assumption...

When using the automounter you get a permanent entry in /etc/mnttab  
that I will refer to as a 'template' entry.  In its options field it  
includes a device number of the form "dev=XXXXX" where XXXXX is a  
hexadecimal number (at least, it is hex on Solaris 10!).

When the automounter REALLY mounts the filestore as it is needed it  
ADDS a second entry for the mounted filesystem.  This has its own  
(different) "dev=XXXXX" number.  Don't ask me why this is:
presumably
there are Good Reasons.

When locating the entry for a path in /etc/mnttab Dovecot tries to do  
this by comparing the device number for the entry with the device  
number of the path being located.  It does this by using stat() on  
the mount point given in the entry.

Whilst this works for manually-listed mounts it does not work when  
the auto-mounter is involved.  This is how Dovecot is being fooled...

1.  We start off with just the automounter's 'template' entry in
/etc/
mnttab:

     auto_direct  /mailstore/messages/p  autofs   
direct,ignore,dev=4740014  1184648400

2.  Dovecot starts by performing a stat on the path to be located.  This
     causes the automounter to secretly mount the real filestore --  
adding the
     second entry to /etc/mnttab with its different device number:

     crypt2.york.ac.uk:/vol/vol9/p  /mailstore/messages/p  nfs   
proto=tcp,xattr,dev=4700386  1184668792

     The number stat() returns is THIS device number -- 4700386 --  
and NOT that
     from the 'template' (4740014).  This gets stored away in the  
"st" structure
     ready for later comparisons.

3.  Dovecot now uses getmntent() to fetch each entry from /etc/mnttab  
in turn.
     It calls stat() on the mount point in the entry to gets its  
device number
     into the "st2" structure.

     In the case of using the automounter the first entry reached is the
     'template' entry, with mount point /mailstore/messages/p.  Using  
stat()
     on this returns its device number ... and the number you get  
back is that
     of the real mounted filesystem.

     Dovecot then compares the deivce numbers in the "st" and
"st2"
structures,
     finds they match and so assumes it has found the matching entry in
     /etc/mnttab

     Unfortunately it hasn't: it has only got the template, which  
doesn't have
     the remote host and fspath of the server from which the  
filestore is
     mounted.  This entry even has a different device number  
(4740014) which
     would have shown up the problem ... but unfortunately Dovecot isn't
     checking the device number from the entry itself.

I believe the correct solution is to change step (3) so that it does  
NOT stat() the mount point of each entry: you can't rely on getting  
the right device number for that entry when the automounter is used.

Instead the proper solution is to parse the mount options in  
ent.mnt_mntopts and extract the device number from the "dev=" option,
then compare THAT with the device number of the path being located  
(from step (1)).

Of course this is fraught with problems because the mount options is  
a comma-separated text string and the "dev=" option could appear  
anywhere within it.  Also, the value of "dev=" is in hexadecimal (at  
least it is on Solaris 10), making comparison with the decimal  
integer from the start() in step (1) non-trivial.

Given this complexity it may be sufficient to instead properly check  
and skip filestores marked "ignore".  The automounter's
'template'
entries are so marked; thus skipping them would allow Dovecot to pass  
over them in the list and locate the proper entry for the real,  
mounted, filestore instead.

Unfortunately this check still involves parsing the ent.mnt_mntopts  
string, as the "ignore" option could appear as any one of:

     ignore
     ignore,
     ,ignore
     ,ignore,

So some clever coding is still needed or a regular expression such as
     ^(.*,)?ignore(,.*)?$
Alternatively there's the dreaded strtok() function -- dreaded  
because it actually changes the source string by over-writing  
separator characters with '\0' as it tokenises it.  Lovely!  :-(


Cheers,
Mike B-)

-- 
The Computing Service, University of York, Heslington, York Yo10 5DD, UK
Tel:+44-1904-433811  FAX:+44-1904-433740

* Unsolicited commercial e-mail is NOT welcome at this e-mail address. *