Bill Mair
2013-Dec-05 19:05 UTC
[Gluster-users] Errors from PHP stat() on files and directories in a glusterfs mount
Hi, I'm trying to use glusterfs to mirror the ownCloud "data" area between 2 servers. They are using debian jessie due to some dependancies that I have for other components. This is where my issue rears it's ugly head. This is failing because I can't stat the files and directories on my glusterfs mount. /var/www/owncloud/data is where I am mounting the volume and I can reproduce the error using a simple php test application, so I don't think that it is apache or owncloud related. I'd be grateful for any pointers on how to resolve this problem. Thanks, Bill Attached is "simple.php" test and the results of executing "strace php5 simple.php" twice, once with the glusterfs mounted (simple.php.strace-glusterfs) and once against the file system when unmounted (simple.php.strace-unmounted). ------------------------------------------------------------------------ Here is what I get in the gluster log when I run the test (as root): /var/log/glusterfs/var-www-owncloud-data.log [2013-12-05 18:33:50.802250] D [client-handshake.c:185:client_start_ping] 0-gv-ocdata-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) [2013-12-05 18:33:50.825132] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] [2013-12-05 18:33:50.825322] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] [2013-12-05 18:33:50.825393] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv-ocdata-replicate-0: Number of sources: 0 [2013-12-05 18:33:50.825456] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-gv-ocdata-replicate-0: returning read_child: 0 [2013-12-05 18:33:50.825511] D [afr-common.c:1380:afr_lookup_select_read_child] 0-gv-ocdata-replicate-0: Source selected as 0 for / [2013-12-05 18:33:50.825579] D [afr-common.c:1117:afr_lookup_build_response_params] 0-gv-ocdata-replicate-0: Building lookup response from 0 [2013-12-05 18:33:50.827069] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv-ocdata-replicate-0: /check.txt: failed to get the gfid from dict [2013-12-05 18:33:50.829409] D [client-handshake.c:185:client_start_ping] 0-gv-ocdata-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) [2013-12-05 18:33:50.836719] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] [2013-12-05 18:33:50.836870] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] [2013-12-05 18:33:50.836941] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv-ocdata-replicate-0: Number of sources: 0 [2013-12-05 18:33:50.837002] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-gv-ocdata-replicate-0: returning read_child: 0 [2013-12-05 18:33:50.837058] D [afr-common.c:1380:afr_lookup_select_read_child] 0-gv-ocdata-replicate-0: Source selected as 0 for /check.txt [2013-12-05 18:33:50.837129] D [afr-common.c:1117:afr_lookup_build_response_params] 0-gv-ocdata-replicate-0: Building lookup response from 0 Other bits of information root at bbb-1:/var/www/owncloud# uname -a Linux bbb-1 3.8.13-bone30 #1 SMP Thu Nov 14 02:59:07 UTC 2013 armv7l GNU/Linux root at bbb-1:/var/www/owncloud# dpkg -l glusterfs-* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-============================================-===========================-===========================-=============================================================================================ii glusterfs-client 3.4.1-1 armhf clustered file-system (client package) ii glusterfs-common 3.4.1-1 armhf GlusterFS common libraries and translator modules ii glusterfs-server 3.4.1-1 armhf clustered file-system (server package) mount bbb-1:gv-ocdata on /var/www/owncloud/data type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) /etc/fstab UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /sdhc ext4 defaults 0 0 bbb-1:gv-ocdata /var/www/owncloud/data glusterfs defaults,_netdev,log-level=DEBUG 0 0 ls -al on the various paths root at bbb-1:/var/log/glusterfs# ll -d /sdhc/ drwxrwxr-x 7 root root 4096 Nov 28 19:15 /sdhc/ root at bbb-1:/var/log/glusterfs# ll -d /sdhc/gv-ocdata/ drwxrwx--- 5 www-data www-data 4096 Dec 5 00:50 /sdhc/gv-ocdata/ root at bbb-1:/var/log/glusterfs# ll -d /sdhc/gv-ocdata/check.txt -rw-r--r-- 2 root root 10 Dec 5 00:50 /sdhc/gv-ocdata/check.txt root at bbb-1:/var/www/owncloud# ll -d /var/www/owncloud/data/ drwxrwx--- 5 www-data www-data 4096 Dec 5 00:50 /var/www/owncloud/data/ root at bbb-1:/var/www/owncloud# ll -d /var/www/owncloud/data/check.txt -rw-r--r-- 1 root root 10 Dec 5 00:50 /var/www/owncloud/data/check.txt file & dir attr information: root at bbb-1:/var/www/owncloud# attr -l /var/www/owncloud/data Attribute "glusterfs.volume-id" has a 16 byte value for /var/www/owncloud/data root at bbb-1:/var/www/owncloud# attr -l /var/www/owncloud/data/check.txt root at bbb-1:/var/www/owncloud# root at bbb-1:/var/www/owncloud# attr -l /sdhc/gv-ocdata/ Attribute "glusterfs.volume-id" has a 16 byte value for /sdhc/gv-ocdata/ Attribute "gfid" has a 16 byte value for /sdhc/gv-ocdata/ Attribute "glusterfs.dht" has a 16 byte value for /sdhc/gv-ocdata/ Attribute "afr.gv-ocdata-client-0" has a 12 byte value for /sdhc/gv-ocdata/ Attribute "afr.gv-ocdata-client-1" has a 12 byte value for /sdhc/gv-ocdata/ root at bbb-1:/var/www/owncloud# attr -l /sdhc/gv-ocdata/check.txt Attribute "gfid" has a 16 byte value for /sdhc/gv-ocdata/check.txt Attribute "afr.gv-ocdata-client-0" has a 12 byte value for /sdhc/gv-ocdata/check.txt Attribute "afr.gv-ocdata-client-1" has a 12 byte value for /sdhc/gv-ocdata/check.txt root at bbb-1:/var/www/owncloud# -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131205/a11c93d9/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: simple-php-example.zip Type: application/zip Size: 21249 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131205/a11c93d9/attachment.zip>
Johan Huysmans
2013-Dec-10 09:42 UTC
[Gluster-users] Errors from PHP stat() on files and directories in a glusterfs mount
Hi, It seems I have a related problem (just posted this on the mailing list). Do you already have a solution for this problem? gr. Johan Huysmans On 05-12-13 20:05, Bill Mair wrote:> Hi, > > I'm trying to use glusterfs to mirror the ownCloud "data" area between > 2 servers. > > They are using debian jessie due to some dependancies that I have for > other components. > > This is where my issue rears it's ugly head. This is failing because I > can't stat the files and directories on my glusterfs mount. > > /var/www/owncloud/data is where I am mounting the volume and I can > reproduce the error using a simple php test application, so I don't > think that it is apache or owncloud related. > > I'd be grateful for any pointers on how to resolve this problem. > > Thanks, > > Bill > > Attached is "simple.php" test and the results of executing "strace > php5 simple.php" twice, once with the glusterfs mounted > (simple.php.strace-glusterfs) and once against the file system when > unmounted (simple.php.strace-unmounted). > > ------------------------------------------------------------------------ > > Here is what I get in the gluster log when I run the test (as root): > > /var/log/glusterfs/var-www-owncloud-data.log > > [2013-12-05 18:33:50.802250] D > [client-handshake.c:185:client_start_ping] 0-gv-ocdata-client-0: > returning as transport is already disconnected OR there are no frames > (0 || 0) > [2013-12-05 18:33:50.825132] D > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] > 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] > [2013-12-05 18:33:50.825322] D > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] > 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] > [2013-12-05 18:33:50.825393] D > [afr-self-heal-common.c:887:afr_mark_sources] 0-gv-ocdata-replicate-0: > Number of sources: 0 > [2013-12-05 18:33:50.825456] D > [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] > 0-gv-ocdata-replicate-0: returning read_child: 0 > [2013-12-05 18:33:50.825511] D > [afr-common.c:1380:afr_lookup_select_read_child] > 0-gv-ocdata-replicate-0: Source selected as 0 for / > [2013-12-05 18:33:50.825579] D > [afr-common.c:1117:afr_lookup_build_response_params] > 0-gv-ocdata-replicate-0: Building lookup response from 0 > [2013-12-05 18:33:50.827069] D > [afr-common.c:131:afr_lookup_xattr_req_prepare] > 0-gv-ocdata-replicate-0: /check.txt: failed to get the gfid from dict > [2013-12-05 18:33:50.829409] D > [client-handshake.c:185:client_start_ping] 0-gv-ocdata-client-0: > returning as transport is already disconnected OR there are no frames > (0 || 0) > [2013-12-05 18:33:50.836719] D > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] > 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] > [2013-12-05 18:33:50.836870] D > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] > 0-gv-ocdata-replicate-0: pending_matrix: [ 0 0 ] > [2013-12-05 18:33:50.836941] D > [afr-self-heal-common.c:887:afr_mark_sources] 0-gv-ocdata-replicate-0: > Number of sources: 0 > [2013-12-05 18:33:50.837002] D > [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] > 0-gv-ocdata-replicate-0: returning read_child: 0 > [2013-12-05 18:33:50.837058] D > [afr-common.c:1380:afr_lookup_select_read_child] > 0-gv-ocdata-replicate-0: Source selected as 0 for /check.txt > [2013-12-05 18:33:50.837129] D > [afr-common.c:1117:afr_lookup_build_response_params] > 0-gv-ocdata-replicate-0: Building lookup response from 0 > > Other bits of information > > root at bbb-1:/var/www/owncloud# uname -a > Linux bbb-1 3.8.13-bone30 #1 SMP Thu Nov 14 02:59:07 UTC 2013 armv7l > GNU/Linux > > root at bbb-1:/var/www/owncloud# dpkg -l glusterfs-* > Desired=Unknown/Install/Remove/Purge/Hold > | > Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend > |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) > ||/ Name Version Architecture Description > +++-============================================-===========================-===========================-=============================================================================================> ii glusterfs-client 3.4.1-1 armhf clustered > file-system (client package) > ii glusterfs-common 3.4.1-1 armhf GlusterFS > common libraries and translator modules > ii glusterfs-server 3.4.1-1 armhf clustered > file-system (server package) > > mount > > bbb-1:gv-ocdata on /var/www/owncloud/data type fuse.glusterfs > (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > /etc/fstab > > UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /sdhc ext4 defaults 0 0 > bbb-1:gv-ocdata /var/www/owncloud/data glusterfs > defaults,_netdev,log-level=DEBUG 0 0 > > ls -al on the various paths > > root at bbb-1:/var/log/glusterfs# ll -d /sdhc/ > drwxrwxr-x 7 root root 4096 Nov 28 19:15 /sdhc/ > > root at bbb-1:/var/log/glusterfs# ll -d /sdhc/gv-ocdata/ > drwxrwx--- 5 www-data www-data 4096 Dec 5 00:50 /sdhc/gv-ocdata/ > > root at bbb-1:/var/log/glusterfs# ll -d /sdhc/gv-ocdata/check.txt > -rw-r--r-- 2 root root 10 Dec 5 00:50 /sdhc/gv-ocdata/check.txt > > root at bbb-1:/var/www/owncloud# ll -d /var/www/owncloud/data/ > drwxrwx--- 5 www-data www-data 4096 Dec 5 00:50 /var/www/owncloud/data/ > > root at bbb-1:/var/www/owncloud# ll -d /var/www/owncloud/data/check.txt > -rw-r--r-- 1 root root 10 Dec 5 00:50 /var/www/owncloud/data/check.txt > > file & dir attr information: > > root at bbb-1:/var/www/owncloud# attr -l /var/www/owncloud/data > Attribute "glusterfs.volume-id" has a 16 byte value for > /var/www/owncloud/data > > root at bbb-1:/var/www/owncloud# attr -l /var/www/owncloud/data/check.txt > root at bbb-1:/var/www/owncloud# > > root at bbb-1:/var/www/owncloud# attr -l /sdhc/gv-ocdata/ > Attribute "glusterfs.volume-id" has a 16 byte value for /sdhc/gv-ocdata/ > Attribute "gfid" has a 16 byte value for /sdhc/gv-ocdata/ > Attribute "glusterfs.dht" has a 16 byte value for /sdhc/gv-ocdata/ > Attribute "afr.gv-ocdata-client-0" has a 12 byte value for > /sdhc/gv-ocdata/ > Attribute "afr.gv-ocdata-client-1" has a 12 byte value for > /sdhc/gv-ocdata/ > > root at bbb-1:/var/www/owncloud# attr -l /sdhc/gv-ocdata/check.txt > Attribute "gfid" has a 16 byte value for /sdhc/gv-ocdata/check.txt > Attribute "afr.gv-ocdata-client-0" has a 12 byte value for > /sdhc/gv-ocdata/check.txt > Attribute "afr.gv-ocdata-client-1" has a 12 byte value for > /sdhc/gv-ocdata/check.txt > root at bbb-1:/var/www/owncloud# > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131210/d77e25bb/attachment.html>