zfs discuss - Jun 2009 - cannot mount ''/tank/home'': directory is not empty

same history happened to me on build 111 and i still don''t know what to
do
-- 
This message posted from opensolaris.org

Arthur Bundo wrote:
> same history happened to me on build 111 and i still don''t know
what to do
>   
What do you see in /tank/home?

Can you do an overlay mount, zfs mount -O tank/home?

-- 
Ian.

I have a "/export/home/x" directory (my home directory) which can not
be mounted during boot time and i have a lot of stuff there, too critical for me
to experiment there,  i can login at failsafe session and i open nautilus as
root and on the snapshots on / i see no export/home/x but on export/home/x
itself i still have the good stuff, tried to open new user and copy the stuff
there, but still can not boot normally.

I have no experience with zfs, though the zfs snapshots would work flawlessly,
any help very much appreciated.
-- 
This message posted from opensolaris.org

>>>>> "ab" == Arthur Bundo <no-reply at
opensolaris.org> writes:
    ab> I have a "/export/home/x" directory

something like:

umount /export/home/x
rmdir /export/home/x
rmdir /export/home     <-- not needed.  but it should give no error if 
                           /export/home is really empty, so it''s a good
                           check
zfs mount -a           <-- should work now that /export/home/x is gone.

I have this problem too, intermittently.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090609/2f7b2dff/attachment.bin>

>>> rmdir /export/home/x 
but i need the /export/home/x folder, in fact from the whole opensolaris folders
this is the only one i need really, i just cant get it mounted normally at boot
and as user x it drops me to a root account folder with no contents and in
failsafe terminal only. i cant login as root either on gnome so i can make some
repairs, the only way i can login is by login as x at terminal without gnome
then i pfexec gdm, then i login as x in failsafe mode, then i open pfexec
nautilus where i see the snapshots, on the snapshots list i have /export/home/x
but that''s it, when i reboot later the system cant get it mounted,
saying it is not empty, i cant remove the /export/home/x folder it is my home
folder and i have sensitive info there.

thanks for your time
-- 
This message posted from opensolaris.org

On Tue, 09 Jun 2009 01:37:53 -0700 (PDT)
Arthur Bundo <no-reply at opensolaris.org> wrote:
> >>> rmdir /export/home/x 
> 
> but i need the /export/home/x folder, in fact from the whole opensolaris
folders this is the only one i need really, i just cant get it mounted normally
at boot and as user x it drops me to a root account folder with no contents and
in failsafe terminal only. i cant login as root either on gnome so i can make
some repairs, the only way i can login is by login as x at terminal without
gnome then i pfexec gdm, then i login as x in failsafe mode, then i open pfexec
nautilus where i see the snapshots, on the snapshots list i have /export/home/x
but that''s it, when i reboot later the system cant get it mounted,
saying it is not empty, i cant remove the /export/home/x folder it is my home
folder and i have sensitive info there.
> 
> thanks for your time
You could try setting the mountpoint property to a 
different value after logging in as root:

# zfs set mountpoint=/var/tmp/newmountpoint rpool/export/home/x


James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog
Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel

>>>>> "ab" == Arthur Bundo <no-reply at
opensolaris.org> writes:
    ab> the only way i can login is by login as x
    ab> at terminal without gnome then i pfexec gdm

login to the console.  Do not use X11 at all.  nautilus is completely
out of the question I''m afraid.

After you login can you do something like:

 exec pfexec su -

so that your home directory is ''/''?  you might be unable to
unmount
something that you need to unmount, if you login as the user
''x''.

    ab> but i need the /tank/home/x folder, in fact from the whole
    ab> opensolaris folders this is the only one i need really, i just
    ab> cant get it mounted normally at boot and as user x it drops me
    ab> to a root account folder with no contents and in failsafe
    ab> terminal 

You are making me dizzy and hurting my brain.

Question: Is /tank/home/x a separate filesystem, or is it part of
          the /tank filesystem?  You can answer by doing:
          
          cd /tank/home/x
          df -k .

          If you can''t interpret the result, give up now.

     Answer ``a separate filesystem'''':

        read my instructions.  I told you to unmount it in the first
        step.  The last step will remount it.  rmdir will delete at
        most one directory, not files or subdirectoreis, so it won''t
        hurt your files---that''s why I told you to use rmdir for the
        deleting.

     Answer ``part of /tank filesystem'''':

        zfs create poolname/tank/x              (or zfs create tank/x or
whatever is working to create /tank/x)
        cd /tank/home/x
        pax -rwpe . /tank/x
        cd ..
        [check that your homedir is backed up in /tank/x]
        rm -rf /tank/home/x      <-- blow away the old copy.  this is not
safe, not like rmdir.

        [follow my earlier instructions]

        zfs rename poolname/tank/x poolname/tank/home/x

     Answer ``neither one.  a long rambling paragraph with no 72-char
              line wrapping or sentence boundaries'''':

        I can''t help you.  Good luck.

HTH.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090609/12c0cf18/attachment.bin>

i reread my post, it makes dizzy even me, right.

the problem i have is this, i had a user x with home directory /export/home/x .
some days before i upgraded from 101 to 111, and on nautilus i tried to make
snapshots of / and /export/home/x and when i reboot some time after i
couldn''t login normally, and all i get is "can not mount directory
/export/home not empty" so i have no X, i just log as user x and tried to
open X with pfexec gdm where is select failsafe mode and there i open nautilus
as root in order to try to manage the snapshots, on the snapshots of the
"/export/home/x" directory on earlier dates is empty, on the last
snapshot is there. my problem is simple . i just need the stuff in
/export/home/x and a working opensolaris.

i will try anything you guys here say but just don''t want to rm the
/export/home/x .
thanks guys i know it is my fault and my ignorance.
-- 
This message posted from opensolaris.org

On Wed, Jun 10, 2009 at 2:14 AM, Arthur Bundo<no-reply at opensolaris.org>
wrote:
> i reread my post, it makes dizzy even me, right.
>
> the problem i have is this, i had a user x with home directory
/export/home/x .
> some days before i upgraded from 101 to 111, and on nautilus i tried to
make snapshots of / and /export/home/x and when i reboot some time after i
couldn''t login normally, and all i get is "can not mount directory
/export/home not empty" so i have no X, i just log as user x and tried to
open X with pfexec gdm where is select failsafe mode and there i open nautilus
as root in order to try to manage the snapshots, on the snapshots of the
"/export/home/x" directory on earlier dates is empty, on the last
snapshot is there. my problem is simple . i just need the stuff in
/export/home/x and a working opensolaris.
>
> i will try anything you guys here say but just don''t want to rm
the /export/home/x .
> thanks guys i know it is my fault and my ignorance.
Some tips:
- Try breaking your message into smaller sentences with full stops.
- Do not try to use X, it''s useless for you.

Since you see the command line environment:
zfs list | grep home

Do you see /export/home/x as a separate entry ?

rpool/export/home                           10.7G  52.1G  10.6G  /export/home
rpool/export/home at install                   88.5K      -   154K  -

That''s what I see on my system - I have /export/home which belongs to
the pool called rpool. I do not have a separate home directory for my
user "sriram". That is, /export/home contains "sriram",
"abcd",
"demouser", etc. These are separate home directories, but they are
folders and not individual file systems.

Is that what you have ?

Please reply, and hopefully, someone else will respond to your reply.
It''s 2:30 am in my time zone, and I''m going to sleep now.

Keep those tips in mind !

-- Sriram

I cant login as root anymore with su , as x user i cant execute almost anything
as sys to do some maintenance , only in single mode at boot, name of system is
"unknown" now, just got tired of this thing now, i don''t want
to learn solaris, dont have time for that , just wanted to use it since the
broadband connection and the java environment seemed much more responsive than
linux and gnome too behaves much better in OpenSolaris, on previous releases i
did have sound via OSS now not anymore etc etc and a lot of small things i see
changed from release 59 and dont have the time to go deeply. this snapshot thing
is a killer, but i immediately run into problems with it.

thank you guys for your time and advice since it was all for free.
good luck with this monster, see you all in 6 months 

Arthur
-- 
This message posted from opensolaris.org

On Wed, Jun 10, 2009 at 4:35 AM, Arthur Bundo<no-reply at opensolaris.org>
wrote:
> I cant login as root anymore with su , as x user i cant execute almost
anything as sys to do some maintenance , only in single mode at boot,
After you log on as the use x (let''s call this user
"arthur"), see if
you can run "pfexec su -" and if you get a # in the command prompt. If
you do, then you can still rescue your data.

Alternatively, boot from the Belenix live CD, run zpool import -f and
see if the pool gets imported.
> name of system is "unknown" now, just got tired of this thing
now, i don''t want to learn solaris, dont have time for that , just
wanted to use it since the broadband connection and the java environment seemed
much more responsive than linux and gnome too behaves much better in
OpenSolaris, on previous releases i did have sound via OSS now not anymore etc
etc and a lot of small things i see changed from release 59 and dont have the
time to go deeply. this snapshot thing is a killer, but i immediately run into
problems with it.
>
Snapshots are an awesome feature indeed.

In case you are interested, try booting from Belenix and see if it
detects your sound device. You can take this discussion to
belenix-discuss, since zfs-discuss is for the zfs filesystem.
> thank you guys for your time and advice since it was all for free.
> good luck with this monster, see you all in 6 months
>
I wouldn''t be so quick to dismiss opensolaris as a monster, much less
ZFS :) There''s sometimes a learning curve (no matter low close to zero
it may be).

All you need to do, is to calmly read the instructions give earlier on
this thread, and also try to use shorter sentences and a fullstop.
> Arthur
> --
-- Sriram

Something is bothering me about this thread.  It seems to me that
if the system provides an error message such as "cannot mount
''/tank/home'': directory is not empty" then the first plan
of action
should be to look and see what is there, no? 

The issue of overlaying mounts has existed for about 30 years and
invariably one discovers that events which lead to different data in
overlapping directories is the result of some sort of procedural issue.

Perhaps once again, ZFS is a singing canary?
 -- richard

I want to thank you for your quick response.

Regarding the learning curve, i really don''t have enough time to go in
deep of the things anymore, i just like the stability of the Solaris platform in
general, and i used it at home years from now.

All i need now is some out of the box installation OS.
I see a lot of things have changed in OSolaris, regarding Belenix, i appreciate
very much your efforts on that distro, but i don''t know their internals
and simply put, it is about trust  and Opensolaris is supported under the Sun
name, and that is enough for me.

I am not a developer or sys-admin or anything connected to OS in general, i just
want stability and backward compatibility.

I am a old solaris lover (esp in x86) and i am waiting for it to thrive.
Wish all the best to the community.
Arthur

PS the snapshot idea is really a killer, and monster is mentioned in a very
appreciative sense.
-- 
This message posted from opensolaris.org

On Thu, Jun 11, 2009 at 1:45 AM, Arthur Bundo<no-reply at opensolaris.org>
wrote:
> I want to thank you for your quick response.
>
> Regarding the learning curve, i really don''t have enough time to
go in deep of the things anymore, i just like the stability of the Solaris
platform in general, and i used it at home years from now.
>
> All i need now is some out of the box installation OS.
> I see a lot of things have changed in OSolaris, regarding Belenix, i
appreciate very much your efforts on that distro, but i don''t know
their internals and simply put, it is about trust ?and Opensolaris is supported
under the Sun name, and that is enough for me.
>
Heh.. most of the OpenSolaris 2008.n distro is based on work done on
Belenix. The same person who created Belenix worked closely with the
OpenSolaris distro team to create that distro. We continue to help out
as and when we get time out from work.
> I am not a developer or sys-admin or anything connected to OS in general, i
just want stability and backward compatibility.
>
I think Richard Elling pointed out something worth investigating -
what does the present /export/home folder contain ?

-- Sriram

i''ve seen a problem where periodically a ''zfs mount
-a'' and sometimes
a ''zpool import <pool>'' can create what appears to be a
race condition
on nested mounts .. that is .. let''s say that i have:

FS				mountpoint
pool			/export
pool/fs1			/export/home
pool/fs2			/export/home/bob
pool/fs3			/export/home/bob/stuff

if "pool" is imported (or a mount -a is done) and somehow pool/fs3  
mounts first - then it will create /export/home and /export/home/bob  
and pool/fs1 and pool/fs2 will fail to mount .. this seems to be  
happening on more recent builds, but not predictably - so i''m still  
trying to track down what''s going on

On Jun 10, 2009, at 1:01 PM, Richard Elling wrote:
> Something is bothering me about this thread.  It seems to me that
> if the system provides an error message such as "cannot mount
> ''/tank/home'': directory is not empty" then the first
plan of action
> should be to look and see what is there, no?
> The issue of overlaying mounts has existed for about 30 years and
> invariably one discovers that events which lead to different data in
> overlapping directories is the result of some sort of procedural  
> issue.
>
> Perhaps once again, ZFS is a singing canary?
> -- richard
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> what does the present /export/home folder contain ?
contains nothing, it is empty
-- 
This message posted from opensolaris.org

Arthur Bundo wrote:
>> what does the present /export/home folder contain ?
>>     
>
> contains nothing, it is empty
>   
I''m not convinced.  If the directory is empty then the link count will
be 2 (. and ..).  For example,

    $ mkdir 
/tmp/empty

    $ ls -ld 
/tmp/empty

    drwxr-xr-x   2 relling  other        117 Jun 12 07:23 /tmp/empty/

The link count is the 2nd field.
If the link count is 2, and it still complains that the directory is not
empty, then there may be a bug.  OTOH, if the link count is 2 and it
complains about the directory being busy, then there would be no bug.
 -- richard

I am sorry but it says it is empty 2, it doesn''t say it is busy.

This is so unreal, I  have no idea how to reproduce this bug, i know i just
upgraded to 111b and i am sorry i am so ignorant and don''t know how to
help you guys to solve this bug.

Best regards on your wonderful work and efforts.
Arthur
-- 
This message posted from opensolaris.org

On Fri, Jun 12, 2009 at 07:04:35PM +0200, Arthur Bundo
wrote:
> I am sorry but it says it is empty 2, it doesn''t say it is busy.
> 
Hm, maybe autofs is in the play somewhere, having put his hands on /home?
Sorry, coming late to this thread, so this may be irrelevant or ruled out
before...
But IIRC, I got the same error message some time ago while trying to go
from an automounted remote to a local (zfs) file system on one of my
systems.

Kurt

Sorry, I don''t know what happened, but it seems like I was not
subscribed to receive replies for this thread so I never saw people''s
replies to my original post... user error probably :)

jone, I think you hit the nail on the head: I *do* seem to remember issuing a
''zfs mount -a'' at some point, but I unfortunately I
don''t remember why -- presumably as some panacea for file systems that
failed to mount due to the problem mentioned in this thread.

For Richard Elling, this info might be useful:
- I originally had ZFS file systems as follows:
/tank/home
/tank/home/simon
/tank/home/simon/projects
/tank/home/simon/photo
/tank/home/simon/video

As part of the boot disk of SXCE (not OpenSolaris with its rpool zpool), I had
the following:
/export/home/simon/...

Therefore when I tried to change the mountpoint of /tank/home/simon/... to
/export/home/simon/... it failed as /export/home/simon etc already existed.

It was at that point that the trouble started, but I don''t remember if
the problem occurred immediately at this point, or immediately afterwards when I
tried to restore the mountpoint back to /tank/home/simon/...

I think ''jone'' is correct in stating that there is some kind
of ''race'' condition occurring when ZFS tries to remount the
file systems after this kind of error occurs (can''t mount, blah blah
directory not empty).

If I was a betting man, I would bet that ZFS is trying to mount child file
systems before parent file systems. I think I came to this conclusion by
scanning various log files as root when I was in the single user mode after boot
errors, and then I saw that it was trying to mount e.g.
/tank/home/simon/projects before /tank/home/simon . So it looks like some table
containing the file system mount order got screwed up, if that''s
possible.

If you need any more help, I''ll try to help, but I can''t scan
back through log files for clues, as I broke the pin on that IDE boot drive
whilst re-inserting the cable... oh... *%!?$$!  :)

Cheers,
Simon

http://breden.org.uk/2009/05/01/home-fileserver-a-year-in-zfs/
-- 
This message posted from opensolaris.org

No probs, glad it worked for you too. It gave me quite a fright too when it
happened :)
-- 
This message posted from opensolaris.org