thr3ads.net - Gluster users - [Gluster-users] NFS ganesha client not showing files after crash [May 2016]

If this information is useful, please help other people find it:
Share via:

Alan Hartless

2016-May-28 02:37 UTC

[Gluster-users] NFS ganesha client not showing files after crash

I had everything working well when I had a complete melt down :-) Well got
all that sorted and everything back up and running or so I thought. Now NFS
ganesha is not showing any existing files but the root level of the brick.
It's empty for all subdirectories. New files or directories added show up
as well. Everything shows up when using the fuse client.

I've tried self healing, editing files, etc but the issue persists. If I
move the folders and back, they show up. But I have a live setup and can't
afford the time to move GBs of data to a new location and back. Is there
anything I can do to trigger something for the files to show up in NFS
again without having to move directories?

Thanks,
Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160527/5b124778/attachment.html>

Jiffin Tony Thottan

2016-May-29 17:46 UTC

head link

[Gluster-users] NFS ganesha client not showing files after crash

On 28/05/16 08:07, Alan Hartless wrote:> I had everything working well when I had a complete melt down :-) Well 
> got all that sorted and everything back up and running or so I 
> thought. Now NFS ganesha is not showing any existing files but the 
> root level of the brick. It's empty for all subdirectories. New files 
> or directories added show up as well. Everything shows up when using 
> the fuse client.
>
If I understand your issue correctly
* You have created a volume using brick which contains pre existing file 
and directories
* When you tried to access  the files via ganesha, it does not show up. 
But with fuse it is visible.

Can please try to perform force lookup on the directories/files(ls <path 
to directory/file>) from the ganesha mount?
Also check the ganesha logs (/var/log/ganesha.log and 
/var/log/ganesha-gfapi.log) for clues.
IMO there was similar issue exists for older version of ganesha(v2.1 I 
guess). if possible can you also share
the ganesha configuration for that volume
> I've tried self healing, editing files, etc but the issue persists. If 
> I move the folders and back, they show up. But I have a live setup and 
> can't afford the time to move GBs of data to a new location and back. 
> Is there anything I can do to trigger something for the files to show 
> up in NFS again without having to move directories?
>
> Thanks,
> Alan
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160529/76811496/attachment.html>

Alan Hartless

2016-Jun-01 18:46 UTC

head link

[Gluster-users] NFS ganesha client not showing files after crash

Yes, I had a brick that I restored and so it had existing files. After the
crash, it wouldn't let me re-add it because it said the files were already
part of a gluster. So I followed
https://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/
to
reset it.

Also correct that I can access all files through fuse but only the root
directory via ganesha NFS4 or any directories/files that have since been
created.

Using a forced lookup on a specific file, I found that I can reach it and
even edit it. But a ls or dir will not list it or any of it's parent
directories. Even after editing the file, it does not list with ls.

I'm using gluster 3.7 and ganesha 2.3 from Gluster's Ubuntu
repositories.

I don't have a /var/log/ganesha.log but I do /var/log/ganesha-gfapi.log. I
tailed it while restarting ganesha and got this for the specific volume:

[2016-06-01 18:44:44.876385] I [MSGID: 114020] [client.c:2106:notify]
0-letsencrypt-client-0: parent translators are ready, attempting connect on
transport
[2016-06-01 18:44:44.876903] I [MSGID: 114020] [client.c:2106:notify]
0-letsencrypt-client-1: parent translators are ready, attempting connect on
transport
[2016-06-01 18:44:44.877193] I [rpc-clnt.c:1868:rpc_clnt_reconfig]
0-letsencrypt-client-0: changing port to 49154 (from 0)
[2016-06-01 18:44:44.877837] I [MSGID: 114057]
[client-handshake.c:1437:select_server_supported_programs]
0-letsencrypt-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2016-06-01 18:44:44.878234] I [MSGID: 114046]
[client-handshake.c:1213:client_setvolume_cbk] 0-letsencrypt-client-0:
Connected to letsencrypt-client-0, attached to remote volume
'/gluster_volume/letsencrypt'.
[2016-06-01 18:44:44.878253] I [MSGID: 114047]
[client-handshake.c:1224:client_setvolume_cbk] 0-letsencrypt-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2016-06-01 18:44:44.878338] I [MSGID: 108005]
[afr-common.c:4007:afr_notify] 0-letsencrypt-replicate-0: Subvolume
'letsencrypt-client-0' came back up; going online.
[2016-06-01 18:44:44.878390] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-letsencrypt-client-0:
Server lk version = 1
[2016-06-01 18:44:44.878505] I [rpc-clnt.c:1868:rpc_clnt_reconfig]
0-letsencrypt-client-1: changing port to 49154 (from 0)
[2016-06-01 18:44:44.879568] I [MSGID: 114057]
[client-handshake.c:1437:select_server_supported_programs]
0-letsencrypt-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2016-06-01 18:44:44.880155] I [MSGID: 114046]
[client-handshake.c:1213:client_setvolume_cbk] 0-letsencrypt-client-1:
Connected to letsencrypt-client-1, attached to remote volume
'/gluster_volume/letsencrypt'.
[2016-06-01 18:44:44.880175] I [MSGID: 114047]
[client-handshake.c:1224:client_setvolume_cbk] 0-letsencrypt-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2016-06-01 18:44:44.896801] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-letsencrypt-client-1:
Server lk version = 1
[2016-06-01 18:44:44.898290] I [MSGID: 108031]
[afr-common.c:1900:afr_local_discovery_cbk] 0-letsencrypt-replicate-0:
selecting local read_child letsencrypt-client-0
[2016-06-01 18:44:44.898798] I [MSGID: 104041]
[glfs-resolve.c:869:__glfs_active_subvol] 0-letsencrypt: switched to graph
676c7573-7465-7266-732d-6e6f64652d63 (0)
[2016-06-01 18:44:45.913545] I [MSGID: 104045] [glfs-master.c:95:notify]
0-gfapi: New graph 676c7573-7465-7266-732d-6e6f64652d63 (0) coming up

I also tailed it while accessing files through a mount point but nothing
was logged.

This is the ganesha config for the specific volume I'm testing with. I have
others but they are the same except for export ID and the paths.

EXPORT
{
    Export_Id = 3;
    Path = "/letsencrypt";
    Pseudo = "/letsencrypt";
    FSAL {
        name = GLUSTER;
        hostname = "localhost";
        volume = "letsencrypt";
    }
    Access_type = RW;
    Squash = No_root_squash;
    Disable_ACL = TRUE;
}

Many thanks!

On Sun, May 29, 2016 at 12:46 PM Jiffin Tony Thottan <jthottan at
redhat.com>
wrote:
>
>
> On 28/05/16 08:07, Alan Hartless wrote:
>
> I had everything working well when I had a complete melt down :-) Well got
> all that sorted and everything back up and running or so I thought. Now NFS
> ganesha is not showing any existing files but the root level of the brick.
> It's empty for all subdirectories. New files or directories added show
up
> as well. Everything shows up when using the fuse client.
>
>
> If I understand your issue correctly
> * You have created a volume using brick which contains pre existing file
> and directories
> * When you tried to access  the files via ganesha, it does not show up.
> But with fuse it is visible.
>
> Can please try to perform force lookup on the directories/files(ls <path
> to directory/file>) from the ganesha mount?
> Also check the ganesha logs (/var/log/ganesha.log and
> /var/log/ganesha-gfapi.log) for clues.
> IMO there was similar issue exists for older version of ganesha(v2.1 I
> guess). if possible can you also share
> the ganesha configuration for that volume
>
> I've tried self healing, editing files, etc but the issue persists. If
I
> move the folders and back, they show up. But I have a live setup and
can't
> afford the time to move GBs of data to a new location and back. Is there
> anything I can do to trigger something for the files to show up in NFS
> again without having to move directories?
>
> Thanks,
> Alan
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at
gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160601/7b176790/attachment.html>

Gluster users - May 2016 - NFS ganesha client not showing files after crash

[Gluster-users] NFS ganesha client not showing files after crash

[Gluster-users] NFS ganesha client not showing files after crash

[Gluster-users] NFS ganesha client not showing files after crash