CentOS - May 2015 - CentOS 7, NFS 4, and a non ext4 fs (like zfs)

Hello,

we want to migrate to CentOS 7(.1) , Server, Client etc ...

Right now we need to suspend that because we have some serious issues 
with NFS4 shared ZFS ( kernel module - zfs on linux project ) Volumes 
and CentOS 7 clients.

Our current server are CentOS6.5, if we share a zfs volume and do the 
following on this share: client A reads/access file "z.txt". Now
client
B replaces (copy, move, unlink+link ) files "z.txt" with a new
different
version and now client A reads/access the file "z.txt" ( and only the 
file - do not do anything which does a "stat" on the file e.g.
"cat
z.txt") the old version/content is read. As long as you do something 
that issues a "stat" for the file/dir.

This is really a problem because a soon you are not browsing through the 
dirs e.g. updating some scripts or anything automated the changes are 
not recognized by the client.

This does not happen with a ext4 share. This also happen with a 
centos7.0 NFS Server and ZFS/XFS. We permute pretty much every nfs 
mount/share option - it always happens. Nothing does help. This does not 
happen with a Centos7.1 server and a centos7.1 client and xfs (NFS4.1). 
Of course it works with our current client server Version: Ubuntu 12.04 
/ CentOS 6.5.

It really does not seem to be a zfs issue - it does look more like a nfs 
problem with certain filessystems. I asked this already on the zfs 
mailing list.

So is there something I can do?
Is the problem known?
Is there a workaround , patch or fix?

I am still unsure if I missed something completely because the problem 
is just so serious...

With kind regards

Timo Ballin




____ ESET 11570 (20150504) ____
The message was checked by ESET Mail Security.

On 5/4/2015 1:09 AM, tballin wrote:
> Hello,
>
> we want to migrate to CentOS 7(.1) , Server, Client etc ...
>
> Right now we need to suspend that because we have some serious issues 
> with NFS4 shared ZFS ( kernel module - zfs on linux project ) Volumes 
> and CentOS 7 clients.
>
> Our current server are CentOS6.5, if we share a zfs volume and do the 
> following on this share: client A reads/access file "z.txt". Now 
> client B replaces (copy, move, unlink+link ) files "z.txt" with a
new
> different version and now client A reads/access the file "z.txt"
( and
> only the file - do not do anything which does a "stat" on the
file
> e.g. "cat z.txt") the old version/content is read. As long as you
do
> something that issues a "stat" for the file/dir.
I've used ZFS file systems on a NFS Server extensively on Solaris, and a 
fair bit with FreeBSD 9.x, 10.1, and its never had any such issues.   
AFAIK, ZFS on Linux is still considered experimental.


-- 
john r pierce, recycling bits in santa cruz

Hi

Perhaps. But they (ZoL) say its production ready. It worked since 
CentosOS 6.1 to 6.6 and still working. Problem occurs if we change the 
NFS version. Not the ZFS. And the problem also exsist with XFS on CentOS 
7. Also it works with all old Ubuntu Clients here. The main problem 
seems to be the client which seems to use NFS features which are not 
(yet) working! Also with XFS you need to use NFS 4.1 or you will have 
the old links/nodes. problem too.

As you say the Solaris/BSD NFS Server Version seems to work. Thats what 
the ZFS mailing list said. If it works under FreeBSD its the NFS Server...

I could try some other file systems... But I would also like to know 
what the issue is. I am still wondering what the problem is. because on 
the server everything looks fine.

Timo


On 05/04/2015 10:22 AM, John R Pierce wrote:
> On 5/4/2015 1:09 AM, tballin wrote:
>> Hello,
>>
>> we want to migrate to CentOS 7(.1) , Server, Client etc ...
>>
>> Right now we need to suspend that because we have some serious issues 
>> with NFS4 shared ZFS ( kernel module - zfs on linux project ) Volumes 
>> and CentOS 7 clients.
>>
>> Our current server are CentOS6.5, if we share a zfs volume and do the 
>> following on this share: client A reads/access file "z.txt".
Now
>> client B replaces (copy, move, unlink+link ) files "z.txt"
with a new
>> different version and now client A reads/access the file
"z.txt" (
>> and only the file - do not do anything which does a "stat" on
the
>> file e.g. "cat z.txt") the old version/content is read. As
long as
>> you do something that issues a "stat" for the file/dir.
>
> I've used ZFS file systems on a NFS Server extensively on Solaris, and 
> a fair bit with FreeBSD 9.x, 10.1, and its never had any such 
> issues.   AFAIK, ZFS on Linux is still considered experimental.
>
>



____ ESET 11570 (20150504) ____
The message was checked by ESET Mail Security.

On Mon, May 04, 2015 at 10:09:04AM +0200, tballin wrote:
> It really does not seem to be a zfs issue - it does look more like a nfs
> problem with certain filessystems. I asked this already on the zfs mailing
> list.
> 
> So is there something I can do?
> Is the problem known?
> Is there a workaround , patch or fix?
There's probably a bug somewhere between the Linux Kernel NFS service
and the ZFS code.  I suspect that it'll be up to the ZFS on Linux
folks to debug it, since the Linux Kernel developers will probably be
more concerned with making supported filesystems work with NFS.

I know I used to have problems back in the RHEL4 days with exporting
XFS filesystems via NFS on Linux, and we had to migrate back to ext3.
XFS support is much better now, but it's also something that both
upstream (RHEL) and the kernel developers maintain.

-- 
Jonathan Billings <billings at negate.org>

I'd like to try to help, but your question is very confusing, and 
appears to present conflicting information.

On 05/04/2015 01:09 AM, tballin wrote:
> Right now we need to suspend that because we have some serious issues
> with NFS4 shared ZFS ( kernel module - zfs on linux project ) Volumes
> and CentOS 7 clients.
>
> Our current server are CentOS6.5, if we share a zfs volume and do the
> following on this share:
Are you, here, describing the behavior of a CentOS 6.5 server with a ZFS 
volume?  As written, it sounds that way, but the context sounds like 
you're describing a CentOS 7 system.
> client A reads/access file "z.txt". Now client
> B replaces (copy, move, unlink+link ) files "z.txt" with a new
different
> version and now client A reads/access the file "z.txt" ( and only
the
> file - do not do anything which does a "stat" on the file e.g.
"cat
> z.txt") the old version/content is read.
I can't tell if you're using "cat" as an example of something
that does
or does not stat the file.

It would be helpful if you could describe the specific steps or commands 
that you are using to test this condition, what outcome you actually 
see, and what outcome you expected to see instead.
> This does not happen with a ext4 share. This also happen with a
> centos7.0 NFS Server and ZFS/XFS.
In a later message, you said that the problem affects ZFS and XFS both, 
and appears to be dependent on the NFS version.

Please describe the behavior of the following:

CentOS 7 server, ZFS volume, NFS 4 export to CentOS 7 client
CentOS 7 server, ext4 volume, NFS 4 export to CentOS 7 client
CentOS 7 server, ZFS volume, NFS 3 export to CentOS 7 client
CentOS 7 server, ext4 volume, NFS 3 export to CentOS 7 client

Script the test on the client side if possible to reduce any other 
variables.
> mount/share option - it always happens. Nothing does help. This does not
> happen with a Centos7.1 server and a centos7.1 client and xfs (NFS4.1).
Earlier you said that it DOES happen with xfs.  This is why your request 
is confusing.