Hi gluster-users!
I?ve got an issue currently with my 2x brick replicate volume. The
volume is fine, and both bricks are replicating fine, but there is 1
file which refuses to be deleted or moved (and now can?t be viewed via a
glusterfs mount)
We just upgraded from |3.7.13-ubuntu1~trusty1| to
|3.7.16-ubuntu1~trusty1|, whether this has anything to do with it, I am
not sure.
The following tests have been performed on the mount points.
* Doing a stat on the file works
* Moving(renaming) the folder above works
* Opening the file |vi file| and doing a |cat file| use to work before
I did:
o |mv /var/www/site/file /home/ubuntu/file|
This mv command never completed and I had to kill -9 it after an hour
waiting.
It seems whenever the file is opened now, it results in the process
halting on a stat command. The following is a stack trace when trying to
run the code (which calls the file, its php)
|<snip> 16:17:34.358528
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
{st_mode=S_IFREG|0664, st_size=11878, ...}) = 0 16:17:34.358613
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.358699
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.358810
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
{st_mode=S_IFREG|0664, st_size=11878, ...}) = 0 16:17:34.358870
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.358966
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.359046
open("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
O_RDONLY) = 5 16:17:34.362692 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.362769 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.362848 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.362910 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.363008 mmap(NULL, 11878, PROT_READ,
MAP_SHARED, 5, 0) = 0x7eff135fd000 16:17:34.363102
stat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
{st_mode=S_IFREG|0664, st_size=11878, ...}) = 0 |
This is where the stack trace fails everytime.
------------------------------------------------------------------------
sudo gluster volume heal gv-www info
Brick demo-gluster01:/gluster
Status: Connected
Number of entries: 0
Brick demo-gluster02:/gluster
Status: Connected
Number of entries: 0
------------------------------------------------------------------------
sudo gluster volume info
Volume Name: gv-www
Type: Replicate
Volume ID: 6163f185-d58b-45f2-8bf0-eeb3a18142d9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: demo-gluster01:/gluster
Brick2: demo-gluster02:/gluster
Options Reconfigured:
cluster.self-heal-daemon: enable
features.scrub: Active
features.bitrot: on
diagnostics.client-log-level: INFO
diagnostics.brick-log-level: DEBUG
performance.readdir-ahead: on
------------------------------------------------------------------------
I just enabled bitrot and scrubbing. (To help with these possible issues)
The only real error I can see in this timeframe is the following:
Brick1:
|/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.160099] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.174907] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.176708] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.178831] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.181356] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.183602] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.186835] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.189262] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.191969] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain) |
Brick2:
|/var/log/glusterfs/var-www.log.1:[2016-10-15 00:16:48.163353] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
6b0b2d6a-f6ea-4ab8-89fc-d776afc17c35. (Possible split-brain) |
But I cannot find any log relating to this on the gluster servers.
Also I can cat the file no problem if I login to the bricks. What could
be a reason of it not working on the mount point?
Any help would be greatly appreciated.
Cheers, Tim
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161018/ec38aefe/attachment-0001.html>
Ravishankar N
2016-Oct-19 16:01 UTC
[Gluster-users] Can't delete or move file off mounted brick
On 10/18/2016 11:31 AM, Tim wrote:> > But I cannot find any log relating to this on the gluster servers. > > Also I can cat the file no problem if I login to the bricks. What > could be a reason of it not working on the mount point? >Can you provide the output of `stat(1) </brick/path-to-file>` and the output of `getfattr -d -m . -e hex </brick/path-to-file> ` from both bricks? -Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161019/2fefe786/attachment.html>