CentOS - Aug 2016 - CentOS 6: files now owned by nobody:nobody

Hi,

We are running a cluster under CentOS 6.6.  We recently attached a new 
NAS device, running CentOS 6.8 and rsync'd our user file system to it.  
We noticed that all the files were owned by nobody (with nobody as the 
group).  We copied over the /etc/passwd and /etc/group files from our 
front-end server to our NAS server.  If we log in to the NAS server we 
see the files owned by their correct owners. However, doing an ls from 
the front-end server or any of the compute nodes still shows the files 
owned by "nobody".  We rebooted one of the compute nodes but it still 
sees the files owned by nobody.

Any advice would be greatly appreciated.

Thanks

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

On 8/29/2016 3:59 PM, Pat Haley wrote:
> We are running a cluster under CentOS 6.6.  We recently attached a new 
> NAS device, running CentOS 6.8 and rsync'd our user file system to 
> it.  We noticed that all the files were owned by nobody (with nobody 
> as the group).  We copied over the /etc/passwd and /etc/group files 
> from our front-end server to our NAS server.  If we log in to the NAS 
> server we see the files owned by their correct owners. However, doing 
> an ls from the front-end server or any of the compute nodes still 
> shows the files owned by "nobody".  We rebooted one of the
compute
> nodes but it still sees the files owned by nobody.
a CentOS server isn't really a 'NAS device', as NAS implies an
appliance
storage device.

this is NFS?   NFSv3, or NFSv4?   what NFS options are on the server and 
on the NFS mount?    quite often NFS servers force root to nobody.


-- 
john r pierce, recycling bits in santa cruz

On Mon, 29 Aug 2016 18:59:31 -0400
Pat Haley wrote:
> We noticed that all the files were owned by nobody
Here are my notes for dealing with this issue:

If all users come up as nobody on a nfs mount:

Add nfs server name to the Domain = line in /etc/idmapd.conf on both the server
and the clients, i.e. Domain = nameof.server

/sbin/service rpcidmapd restart
/sbin/service nfslock restart
/sbin/service nfs restart

Also, the complete hostname as specified (nameof.server) must be in /etc/hosts
on the nfs clients as well as the server

---------------
try
/usr/sbin/nfsidmap -c
on the client.
Since I put this into my /etc/rc.local , I don't have the problem any 
longer.
---------------
1) /etc/idmapd.conf
# Set domain to the domain name shared by your NFS servers.
Domain: mycompany.com
Set local-realms to the name of the nfs servers you'll be using. THIS 
WASN'T MENTIONED ELSEWHERE.
Local-Realms: nfs1.mycompany.com,nfs2.mycompany.com
# make the above changes on all the servers in question.

2) /etc/hosts: list with all the NFS servers you specified in local-realms
above. This way DNS errors don't make your servers get hung
1.2.3.4    nfs1.mycompany.com
1.2.3.5    nfs2.mycompany.com

3) Make sure you synchronize your /etc/passwd files so that the account 
IDs match up or you'll get very strange results.

4) Reboot EVERYTHING. Restarting services was not enough. 
For documentation's sake, I restarted rpcidmapd, nfslock, and nfs, but
didn't get the correct permissions until reboot. It doesn't seem
important to run the nfs service on the
clients.

5) Client mount:
# CLI
/bin/mount -t nfs servername:/path/to/share /local/mount/point

# /etc/fstab
servername.com:/path/to/share    /local/mount/mount nfs ro,nolock    0 0

# mount -a


-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com

I believe NFSv4.  On the machine that contains the physical disks (is 
that the "server" or the "NSF mount"?) the relevant line
from /etc/fstab
seems to be

UUID=bde58f42-4ac4-4763-b0a8-f83723f0e2a0 /home ext4    defaults      1 2

while on my front-end machine its

mseas-data2:/home       /home      nfs     defaults        0 0

where mseas-data2 is the name of the machine that contains the physical 
disks.  Note that it isn't just root that's becoming "nobody"
but all
the users

Thanks


On 08/29/2016 07:14 PM, John R Pierce wrote:
> On 8/29/2016 3:59 PM, Pat Haley wrote:
>> We are running a cluster under CentOS 6.6.  We recently attached a 
>> new NAS device, running CentOS 6.8 and rsync'd our user file system
>> to it.  We noticed that all the files were owned by nobody (with 
>> nobody as the group).  We copied over the /etc/passwd and /etc/group 
>> files from our front-end server to our NAS server.  If we log in to 
>> the NAS server we see the files owned by their correct owners. 
>> However, doing an ls from the front-end server or any of the compute 
>> nodes still shows the files owned by "nobody".  We rebooted
one of
>> the compute nodes but it still sees the files owned by nobody.
>
> a CentOS server isn't really a 'NAS device', as NAS implies an 
> appliance storage device.
>
> this is NFS?   NFSv3, or NFSv4?   what NFS options are on the server 
> and on the NFS mount?    quite often NFS servers force root to nobody.
>
>
-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

On Mon, Aug 29, 2016 at 06:59:31PM -0400, Pat Haley
wrote:
> ... We
> noticed that all the files were owned by nobody (with nobody as the group).
If its NFSv4, then its most likely a problem with your idmapper.  Make
sure that the rpc.idmapd is running on your client, and that your
server has appropriate ID mapping enabled.

If its NFSv4, are you using sec=krb5*?

-- 
Jonathan Billings <billings at negate.org>

Jonathan Billings wrote:
> On Mon, Aug 29, 2016 at 06:59:31PM -0400, Pat Haley wrote:
>> ... We
>> noticed that all the files were owned by nobody (with nobody as the
>> group).
>
> If its NFSv4, then its most likely a problem with your idmapper.  Make
> sure that the rpc.idmapd is running on your client, and that your
> server has appropriate ID mapping enabled.
>
> If its NFSv4, are you using sec=krb5*?
I gather other folks have been saying to fix Domain =, and make sure
Method = nsswitch is not commented out, in /etc/idmapd.conf. Other things:
   1. *also* in /etc/idmapd.conf, go further down, and comment out or
           delete the *other* schema - for example, the UMICH schema is
           by default enabled, for some reason. *Definitely* disable that.
           Scroll to the bottom of the file, to make sure nothing else is
           on by default.
   2. nsswitch - check /etc/nsswitch.conf

Then restart idmapd.

        mark

The setting the Domain = line in /etc/idmapd.conf option solved our problem

Thanks


On 08/29/2016 07:23 PM, Frank Cox wrote:
> On Mon, 29 Aug 2016 18:59:31 -0400
> Pat Haley wrote:
>
>> We noticed that all the files were owned by nobody
> Here are my notes for dealing with this issue:
>
> If all users come up as nobody on a nfs mount:
>
> Add nfs server name to the Domain = line in /etc/idmapd.conf on both the
server and the clients, i.e. Domain = nameof.server
>
> /sbin/service rpcidmapd restart
> /sbin/service nfslock restart
> /sbin/service nfs restart
>
> Also, the complete hostname as specified (nameof.server) must be in
/etc/hosts on the nfs clients as well as the server
>
> ---------------
> try
> /usr/sbin/nfsidmap -c
> on the client.
> Since I put this into my /etc/rc.local , I don't have the problem any
> longer.
> ---------------
> 1) /etc/idmapd.conf
> # Set domain to the domain name shared by your NFS servers.
> Domain: mycompany.com
> Set local-realms to the name of the nfs servers you'll be using. THIS
> WASN'T MENTIONED ELSEWHERE.
> Local-Realms: nfs1.mycompany.com,nfs2.mycompany.com
> # make the above changes on all the servers in question.
>
> 2) /etc/hosts: list with all the NFS servers you specified in local-realms
above. This way DNS errors don't make your servers get hung
> 1.2.3.4    nfs1.mycompany.com
> 1.2.3.5    nfs2.mycompany.com
>
> 3) Make sure you synchronize your /etc/passwd files so that the account
> IDs match up or you'll get very strange results.
>
> 4) Reboot EVERYTHING. Restarting services was not enough.
> For documentation's sake, I restarted rpcidmapd, nfslock, and nfs, but
didn't get the correct permissions until reboot. It doesn't seem
important to run the nfs service on the
> clients.
>
> 5) Client mount:
> # CLI
> /bin/mount -t nfs servername:/path/to/share /local/mount/point
>
> # /etc/fstab
> servername.com:/path/to/share    /local/mount/mount nfs ro,nolock    0 0
>
> # mount -a
>
>
-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301