I have noticed this in snv_114 now at 117. I have the following filesystems. fs was created using zfs create pool/fs movies created using zfs create pool/fs/movies pool/fs/movies pool/fs/music pool/fs/photos pool/fs/archives at boot /lib/svc/method/fs-local fails where zfs mount -a is called. failed to mount pool/fs/movies, directory not empty. This send filesystem/local into maintenance. zfs mount -a cannot successful mount this zpool/zfs structure from the command line either. Any one knows what''s wrong or have any guidance. -- This message posted from opensolaris.org
On Tue 21/07/09 03:13 , "Andre Lue" no-reply at opensolaris.org sent:> I have noticed this in snv_114 now at 117. > > I have the following filesystems. > fs was created using zfs create pool/fs > movies created using zfs create pool/fs/movies > pool/fs/movies > pool/fs/music > pool/fs/photos > pool/fs/archives > > at boot /lib/svc/method/fs-local fails where zfs mount -a is called. > failed to mount pool/fs/movies, directory not empty. This send > filesystem/local into maintenance. > zfs mount -a cannot successful mount this zpool/zfs structure from the > command line either. Any one knows what''s wrong or have any guidance.What happens if you mount pool/fs and ls the mountpoint? It looks like you already have a movies directory. You should be able to check this from the console. -- Ian.
Hi Ian, Thanks for the the reply. I will check your recommendation when I get a chance. However this happens on any zfs that have hierarchical zfs filesystems. I noticed it started this problem since snv_114. This same filesystem structure last worked fine with snv_110. -- This message posted from opensolaris.org
On Jul 21, 2009, at 10:14 AM, Andre Lue wrote:> Hi Ian, > > Thanks for the the reply. I will check your recommendation when I > get a chance. However this happens on any zfs that have hierarchical > zfs filesystems. I noticed it started this problem since snv_114. > This same filesystem structure last worked fine with snv_110.You''re experiencing a pool import race. Either flatten your heirarchy or use legacy mounts. -- richard
You might be running into 6827199 - zfs mount -a performs mounts in the wrong order. There is a workaround contained in the bug (http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6827199) -- This message posted from opensolaris.org
We are also running into this bug. Our system is a Solaris 10u4 SunOS sunsystem9 5.10 Generic_127112-10 i86pc i386 i86pc ZFS version 4 We opened a Support Case (Case ID 71912304) which after some discussion came to the "conclusion" that we should not use /etc/reboot for rebooting. This leads me to the conclusion that /etc/reboot is not supported on ZFS? I cannot believe that Is there better solution to this problem, what if the machine crashes? Unfortunately, since the pool contains so many datasets (around 5000) that correcting the mount order by hand would incur serious downtime. (importing and therefore also sharing this pool takes around 45min. - which is also why we are using /etc/reboot to avoid additional downtime for unsharing, umounting, exporting) Is there a backport of this fix for S10 in progress. Why does ZFS confuse the mount order? After all, the datasets are ordered hierarchically. -- This message posted from opensolaris.org
Hi, Martin Uhl wrote:>We opened a Support Case (Case ID 71912304) which after some discussion came to the "conclusion" that we should not use /etc/reboot for rebooting. > > >Yes. You are using "/etc/reboot" which is the same as calling "/usr/sbin/halt": % ls -l /etc/reboot lrwxrwxrwx 1 root root 16 Mar 9 2009 /etc/reboot -> ../usr/sbin/halt* From the halt man page: NOTES The halt and poweroff utilities do not cleanly shutdown smf(5) services. Execute the scripts in /etc/rcnum.d or exe- cute shutdown actions in inittab(4). To ensure a complete shutdown of system services, use shutdown(1M) or init(1M) to reboot a Solaris system. You should do a proper shutdown to avoid having problems with smf services.>This leads me to the conclusion that /etc/reboot is not supported on ZFS? I cannot believe that > >If you umount a ZFS FS that has some other FS''s underneath it, then the mount points for the "child" FS needs to be created to have those mounted; that way if you don''t export the pool the dirs won''t be deleted and next time you import the pool the FS will fail to mount because your mount point is not empty. IMHO this is not a bug. For example: # uname -a SunOS va64-v20zc-gmp03 5.11 onnv-clone i86pc i386 i86pc # zpool import -d /zfs_tests tank # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui yes on tank/gongui/test /gongui/test yes on # zfs umount tank/gongui/test # zfs umount tank/gongui # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui no on tank/gongui/test /gongui/test no on # zfs mount tank/gongui/test # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui no on tank/gongui/test /gongui/test yes on If we export the pool in a clean way & reimport it there are no problems: # zpool export tank # zpool import -d /zfs_tests tank # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui yes on tank/gongui/test /gongui/test yes on But my cu''s is calling "/etc/reboot" ( i.e. halt)because the export takes a long time... that wont export the pool so the dirs won''t be clearer... that leads the problem... # zfs umount tank/gongui/test # zfs umount tank/gongui # zfs mount tank/gongui/test # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui no on tank/gongui/test /gongui/test yes on # /etc/reboot syncing file systems... done SunOS Release 5.11 Version onnv-clone 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. [...] Reading ZFS config: done. Mounting ZFS filesystems: (1/16)cannot mount ''/gongui'': directory is not empty (16/16) svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: exit status 1 Nov 20 11:56:54 svc.startd[100004]: svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95. Nov 20 11:56:55 svc.startd[100004]: system/filesystem/local:default failed fatally: transitioned to maintenance (see ''svcs -xv'' for details) va64-v20zc-gmp03 console login: root Password: Last login: Fri Nov 20 11:40:11 on console Sun Microsystems Inc. SunOS 5.11 onnv-clone Oct. 22, 2009 SunOS Internal Development: gs112050 2009-Oct-22 [onnv-clone] bfu''ed from /net/ardbeg.sfbay//builds/gs112050/onnv-clone/archives/i386/nightly on 2009-10-22 Sun Microsystems Inc. SunOS 5.11 snv_125 November 2008 # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui no on tank/gongui/test /gongui/test yes on>Is there better solution to this problem, what if the machine crashes? > >Crashes are abnormal conditions. If it crashes you should fix the problem to avoid future crashes and probably you will need to clear the pool dir hierarchy prior to import the pool.>Unfortunately, since the pool contains so many datasets (around 5000) that correcting the mount order by hand would incur serious downtime. (importing and therefore also sharing this pool takes around 45min. - which is also why we are using /etc/reboot to avoid additional downtime for unsharing, umounting, exporting >Is there a backport of this fix for S10 in progress. > >That''s what should be investigated in your case; i know there are some fixes in progress specially for the sharing part but more data is needed to see what''s going on here.>Why does ZFS confuse the mount order? After all, the datasets are ordered hierarchically. > >In your particular case you''ve the explanation above. Thanks, Gonzalo.
> If you umount a ZFS FS that has some other FS''s underneath it, then the > mount points for the "child" FS needs to be created to have those > mounted; that way if you don''t export the pool the dirs won''t be deleted > and next time you import the pool the FS will fail to mount because your > mount point is not empty. IMHO this is not a bug.As far as I see it, the dirs will be created when the filesystem is mounted in the wrong order. That has nothing to do with removing dirs on export. e.g. sunsystem9:[private] > zfs list -r -o name,mountpoint,mounted,canmount private/vmware NAME MOUNTPOINT MOUNTED CANMOUNT private/vmware //private/vmware no on private/vmware/datastores //private/vmware/datastores no on private/vmware/datastores/cdimages //private/vmware/datastores/cdimages yes on sunsystem9:[private] > sudo zfs umount private/vmware/datastores/cdimages sunsystem9:[private] > cd vmware sunsystem9:[vmware] > find . . ./datastores ./datastores/cdimages sunsystem9:[vmware] > cd .. sunsystem9:[private] > sudo zfs mount private/vmware cannot mount ''//private/vmware'': directory is not empty ---> /private/vmware contains subdirs which prevent mounting of private/vmware sunsystem9:[vmware] > sudo rm -rf datastores sunsystem9:[private] > sudo zfs mount private/vmware ----> now I removed the dirs, therefore it is working sunsystem9:[private] > sudo zfs umount private/vmware sunsystem9:[private] > sudo zfs mount private/vmware/datastores/cdimages ----> that recreates the dir hierarchy /private/vmware/datastores/cdimages sunsystem9:[private] > sudo zfs umount private/vmware/datastores/cdimages ----> that does not remove these aforementioned dirs sunsystem9:[private] > sudo zfs mount private/vmware cannot mount ''//private/vmware'': directory is not empty ----> obviously that will fail. So AFAIK those directories will be created on mount but not removed on unmount The problem is not that exporting will not remove dirs (which I doubt it should) but mounting datasets in the wrong order does create to many dirs. Which then inhibits mounting of some datasets. The dirs blocking the mount are created at import/mount time.> That''s what should be investigated in your case; i know there are some > fixes in progress specially for the sharing part but more data is needed > to see what''s going on here.what data do you need? Greetings, Martin -- This message posted from opensolaris.org
Hi, Martin Uhl wrote:>----> obviously that will fail. > >So AFAIK those directories will be created on mount but not removed on unmount > >Good point. I was not aware of this. Will check with engineering.>The problem is not that exporting will not remove dirs (which I doubt it should) but mounting datasets in the wrong order does create to many dirs. Which then inhibits mounting of some datasets. >The dirs blocking the mount are created at import/mount time. > > >how you know that?? If you''re correct you should been able to reproduce the problem by doing a "clean" shutdown (or an export/import), can you reproduce it this way?? Thanks, Gonzalo.>>That''s what should be investigated in your case; i know there are some >>fixes in progress specially for the sharing part but more data is needed >>to see what''s going on here. >> >> > >what data do you need? > >Greetings, Martin > >
>> Is there better solution to this problem, what if the machine crashes? >> > > Crashes are abnormal conditions. If it crashes you should fix the problem to > avoid future crashes and probably you will need to clear the pool dir > hierarchy prior to import the pool.Are you serious? I really hope that you have nothing to do with OS development, or database development for that matter. A crash can be something as simple as the battery in a portable computer running out of power. A user should _not_ have to do anything manual to get the system going again. We got rid off manual fsck many years ago. Let''s not move back to the stone age!
>>The dirs blocking the mount are created at import/mount time. >how you know that??In the previous example I could reconstruct that using zfs mount. Just look at the last post. I doubt ZFS removes mount directories.>If you''re correct you should been able to reproduce >the problem by doing a "clean" shutdown (or an export/import), can you >reproduce it this way??The server is in a production environment and we cannot afford the necessary downtime for that. Unfortunately the server has lots of datasets which cause import/export times of 45 mins. We import the pool with the -R parameter, might that contribute to the problem? Perhaps a zfs mount -a bug in correspondence with the -R parameter? Greetings, Martin -- This message posted from opensolaris.org
Martin, i think we should continue offline. Anyway see my comments/answers inline. Thanks, Gonzalo. --> Martin Uhl wrote:>>>The dirs blocking the mount are created at import/mount time. >>> >>> >>how you know that?? >> >> > >In the previous example I could reconstruct that using zfs mount. Just look at the last post. > >You said "...the dirs blocking the mount are created at import/mount time.." and your previous post suggest a different scenario: mount points created prior to the import/not cleared when doing a umount. Fixing the umount "problem" is expensive and will not resolve the problem you''ve. I''ve tested on my lab system setting a breakpoint in zfs_umount() which is called for each FS of a pool when you export it but not called for the FS''s of an imported pool (other than rootpool) when you stop your system via "/etc/reboot".>I doubt ZFS removes mount directories. > >It does. With a simple dtrace script i saw that is done in "zpool_disable_datasets()->remove_mountpoint()" when you export the pool. # zfs list -o name,mountpoint,mounted,canmount -r tank NAME MOUNTPOINT MOUNTED CANMOUNT tank /tank yes on tank/gongui /gongui no on tank/gongui/test /gongui/test yes on # zfs mount tank/gongui cannot mount ''/gongui'': directory is not empty # dtrace -q -n ''syscall::rmdir:entry{printf("Mountpoint deleted: %s\n",stringof(copyinstr(arg0)));ustack();}'' -c "zpool export tank" Mountpoint deleted: /tank libc.so.1`rmdir+0x7 libzfs.so.1`zpool_disable_datasets+0x319 zpool`zpool_do_export+0x10f zpool`main+0x158 zpool`_start+0x7d Mountpoint deleted: /gongui/test libc.so.1`rmdir+0x7 libzfs.so.1`zpool_disable_datasets+0x32c zpool`zpool_do_export+0x10f zpool`main+0x158 zpool`_start+0x7d>If you''re correct you should been able to reproduce >the problem by doing a "clean" shutdown (or an export/import), can you >reproduce it this way?? > > > >The server is in a production environment and we cannot afford the necessary downtime for that. >Unfortunately the server has lots of datasets which cause import/export times of 45 mins. > >We import the pool with the -R parameter, might that contribute to the problem? Perhaps a zfs mount -a bug in correspondence with the -R parameter? > > >See above. If you export the pool you shouldn''t have problems. We should study if we can lower the time to import/export the pools but my recommendation is to do a proper shutdown. Thanks, Gonzalo.>Greetings, Martin > >
> We import the pool with the -R parameter, might that contribute to the problem? Perhaps a zfs mount -a bug in correspondence with the -R parameter?This Bugreport seems to confirm this: http://bugs.opensolaris.org/view_bug.do?bug_id=6612218 Note that the /zz directory mentioned in the bugreport does not exist before the zfs set mountpoint command. Greetings, Martin -- This message posted from opensolaris.org