Dietmar Putz
2015-Dec-03 11:06 UTC
[Gluster-users] after upgrade to 3.6.7 : Internal error xfs_attr3_leaf_write_verify
Hello all,
on 1st december i upgraded two 6 node cluster from glusterfs 3.5.6 to 3.6.7.
all of them are equal in hw, os and patchlevel, currently running ubuntu
14.04 lts by an do-release-upgrade from 12.04 lts (this was done before
gfs upgrade to 3.5.6, not directly before upgrading to 3.6.7).
because of a geo-replication issue all of the nodes have rsync 3.1.1.3
installed instead 3.1.0 which comes by the repositories. this is the
only deviation from ubuntu repositories for 14.04 lts.
since upgrade to gfs 3.6.7 the glusterd on two nodes of the same cluster
are going offline after getting an xfs_attr3_leaf_write_verify error for
the underlying bricks as shown below.
this happens about every 4-5 hours after the problem was solved by an
umount / remount of the brick. it makes no difference to run a xfs_check
/ xfs_repair before remount.
xfs_check / xfs_repair did not show any faults. the underlying hw is a
raid 5 vol on lsi-9271 8i. megacli does not show any errors.
the syslog does not show more than the dmesg output below.
every time the same two nodes of the same cluster are affected.
as shown in dmesg and syslog, the system recognizes the
xfs_attr_leaf_write_verify error about 38 min. before finally giving up.
for both events i can not found corresponding events in gluster logs.
this is strange...the gluster is historical grown from 3.2.5, 3.3, to
3.4.6/7 which was running well for month, gfs 3.5.6 was running for
about two weeks and upgrade to 3.6.7 was done because of a geo-repl
log-flood.
even when i have no hint/evidence that this is caused by gfs 3.6.7
somehow i believe that this is the case...
does anybody experienced such an error or have some hints to getting out
of this big problem...?
unfortunately the affected cluster is the master of a geo-replication
which is not well running since update from gfs 3.4.7...fortunately both
affected gluster-nodes are not of the same sub-volume.
any help is appreciated...
best regards
dietmar
[ 09:32:29 ] - root at gluster-ger-ber-10 /var/log $gluster volume info
Volume Name: ger-ber-01
Type: Distributed-Replicate
Volume ID: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: gluster-ger-ber-11-int:/gluster-export
Brick2: gluster-ger-ber-12-int:/gluster-export
Brick3: gluster-ger-ber-09-int:/gluster-export
Brick4: gluster-ger-ber-10-int:/gluster-export
Brick5: gluster-ger-ber-07-int:/gluster-export
Brick6: gluster-ger-ber-08-int:/gluster-export
Options Reconfigured:
changelog.changelog: on
geo-replication.ignore-pid-check: on
cluster.min-free-disk: 200GB
geo-replication.indexing: on
auth.allow:
10.0.1.*,188.138.82.*,188.138.123.*,82.193.249.198,82.193.249.200,31.7.178.137,31.7.178.135,31.7.180.109,31.7.180.98,82.199.147.*,104.155.22.202,104.155.30.201,104.155.5.117,104.155.11.253,104.155.15.34,104.155.25.145,146.148.120.255,31.7.180.148
nfs.disable: off
performance.cache-refresh-timeout: 2
performance.io-thread-count: 32
performance.cache-size: 1024MB
performance.read-ahead: on
performance.cache-min-file-size: 0
network.ping-timeout: 10
[ 09:32:52 ] - root at gluster-ger-ber-10 /var/log $
[ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $gluster volume status
Status of volume: ger-ber-01
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-ger-ber-11-int:/gluster-export 49152 Y 15994
Brick gluster-ger-ber-12-int:/gluster-export N/A N N/A
Brick gluster-ger-ber-09-int:/gluster-export 49152 Y 10965
Brick gluster-ger-ber-10-int:/gluster-export N/A N N/A
Brick gluster-ger-ber-07-int:/gluster-export 49152 Y 18542
Brick gluster-ger-ber-08-int:/gluster-export 49152 Y 20275
NFS Server on localhost 2049 Y 13658
Self-heal Daemon on localhost N/A Y 13666
NFS Server on gluster-ger-ber-09-int 2049 Y 13503
Self-heal Daemon on gluster-ger-ber-09-int N/A Y 13511
NFS Server on gluster-ger-ber-07-int 2049 Y 21526
Self-heal Daemon on gluster-ger-ber-07-int N/A Y 21534
NFS Server on gluster-ger-ber-08-int 2049 Y 24004
Self-heal Daemon on gluster-ger-ber-08-int N/A Y 24011
NFS Server on gluster-ger-ber-11-int 2049 Y 18944
Self-heal Daemon on gluster-ger-ber-11-int N/A Y 18952
NFS Server on gluster-ger-ber-12-int 2049 Y 19138
Self-heal Daemon on gluster-ger-ber-12-int N/A Y 19146
Task Status of Volume ger-ber-01
------------------------------------------------------------------------------
There are no active volume tasks
- root at gluster-ger-ber-10 /var/log $
- root at gluster-ger-ber-10 /var/log $dmesg -T
...
[Wed Dec 2 12:43:47 2015] XFS (sdc1): xfs_log_force: error 5 returned.
[Wed Dec 2 12:43:48 2015] XFS (sdc1): xfs_log_force: error 5 returned.
[Wed Dec 2 12:45:58 2015] XFS (sdc1): Mounting Filesystem
[Wed Dec 2 12:45:58 2015] XFS (sdc1): Starting recovery (logdev: internal)
[Wed Dec 2 12:45:59 2015] XFS (sdc1): Ending recovery (logdev: internal)
[Wed Dec 2 13:11:53 2015] XFS (sdc1): Mounting Filesystem
[Wed Dec 2 13:11:54 2015] XFS (sdc1): Ending clean mount
[Wed Dec 2 13:12:29 2015] init: statd main process (25924) killed by
KILL signal
[Wed Dec 2 13:12:29 2015] init: statd main process ended, respawning
[Wed Dec 2 13:13:24 2015] init: statd main process (13433) killed by
KILL signal
[Wed Dec 2 13:13:24 2015] init: statd main process ended, respawning
[Wed Dec 2 17:22:28 2015] ffff8807076b1000: 00 00 00 00 00 00 00 00 fb
ee 00 00 00 00 00 00 ................
[Wed Dec 2 17:22:28 2015] ffff8807076b1010: 10 00 00 00 00 20 0f e0 00
00 00 00 00 00 00 00 ..... ..........
[Wed Dec 2 17:22:28 2015] ffff8807076b1020: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[Wed Dec 2 17:22:28 2015] ffff8807076b1030: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[Wed Dec 2 17:22:28 2015] XFS (sdc1): Internal error
xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
[Wed Dec 2 17:22:28 2015] CPU: 4 PID: 13162 Comm: xfsaild/sdc1 Not
tainted 3.13.0-67-generic #110-Ubuntu
[Wed Dec 2 17:22:28 2015] Hardware name: Supermicro X10SLL-F/X10SLL-F,
BIOS 1.1b 11/01/2013
[Wed Dec 2 17:22:28 2015] 0000000000000001 ffff8801c5691bd0
ffffffff817240e0 ffff8801b15c3800
[Wed Dec 2 17:22:28 2015] ffff8801c5691be8 ffffffffa01aa6fb
ffffffffa01a66f0 ffff8801c5691c20
[Wed Dec 2 17:22:28 2015] ffffffffa01aa755 000000d800200200
ffff8804a59ac780 ffff8800d917e658
[Wed Dec 2 17:22:28 2015] Call Trace:
[Wed Dec 2 17:22:28 2015] [<ffffffff817240e0>] dump_stack+0x45/0x56
[Wed Dec 2 17:22:28 2015] [<ffffffffa01aa6fb>]
xfs_error_report+0x3b/0x40 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ?
_xfs_buf_ioapply+0x70/0x3a0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01aa755>]
xfs_corruption_error+0x55/0x80 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01c7b70>]
xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ?
_xfs_buf_ioapply+0x70/0x3a0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ?
xfs_bdstrat_cb+0x55/0xb0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>]
_xfs_buf_ioapply+0x70/0x3a0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffff8109ac90>] ? wake_up_state+0x20/0x20
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ?
xfs_bdstrat_cb+0x55/0xb0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a8336>]
xfs_buf_iorequest+0x46/0x90 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>]
xfs_bdstrat_cb+0x55/0xb0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a856b>]
__xfs_buf_delwri_submit+0x13b/0x210 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>] ?
xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ?
xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>]
xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa0207d27>] xfsaild+0x237/0x5c0 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ?
xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[Wed Dec 2 17:22:28 2015] [<ffffffff8108b7d2>] kthread+0xd2/0xf0
[Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ?
kthread_create_on_node+0x1c0/0x1c0
[Wed Dec 2 17:22:28 2015] [<ffffffff81734c28>] ret_from_fork+0x58/0x90
[Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ?
kthread_create_on_node+0x1c0/0x1c0
[Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption detected. Unmount and
run xfs_repair
[Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_do_force_shutdown(0x8) called
from line 1320 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_buf.c. Return address =
0xffffffffa01a671c
[Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption of in-memory data
detected. Shutting down filesystem
[Wed Dec 2 17:22:28 2015] XFS (sdc1): Please umount the filesystem and
rectify the problem(s)
[Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_log_force: error 5 returned.
[Wed Dec 2 17:22:49 2015] XFS (sdc1): xfs_log_force: error 5 returned.
...
[ 19:10:49 ] - root at gluster-ger-ber-10 /var/log $xfs_info /gluster-export
meta-data=/dev/sdc1 isize=256 agcount=32,
agsize=152596472 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=4883087099, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $
[ 09:36:37 ] - root at gluster-ger-ber-10 /var/log $stat /gluster-export
stat: cannot stat ?/gluster-export?: Input/output error
[ 09:36:45 ] - root at gluster-ger-ber-10 /var/log $
[ 08:50:43 ] - root at gluster-ger-ber-10 ~/tmp/syslog $dmesg -T | grep
xfs_attr3_leaf_write_verify
[Di Dez 1 23:24:53 2015] XFS (sdc1): Internal error
xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
[Di Dez 1 23:24:53 2015] [<ffffffffa01c7b70>]
xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[Mi Dez 2 12:19:16 2015] XFS (sdc1): Internal error
xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
[Mi Dez 2 12:19:16 2015] [<ffffffffa01c7b70>]
xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[Mi Dez 2 17:22:28 2015] XFS (sdc1): Internal error
xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
[Mi Dez 2 17:22:28 2015] [<ffffffffa01c7b70>]
xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[Mi Dez 2 23:06:32 2015] XFS (sdc1): Internal error
xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
[Mi Dez 2 23:06:32 2015] [<ffffffffa01c7b70>]
xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[ 08:06:28 ] - root at gluster-ger-ber-10
/var/log/glusterfs/geo-replication $grep xfs_attr3_leaf_write_verify
/root/tmp/syslog/syslog*
Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906268] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906448]
[<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.509891] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.510414]
[<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
xfs_check
xfs_repair -> no fault
Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298098] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298259]
[<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.969849] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01a66f0
Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.970217]
[<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[ 08:06:37 ] - root at gluster-ger-ber-10 /var/log/glusterfs/geo-replication $
[ 08:04:51 ] - root at gluster-ger-ber-12 ~/tmp/syslog $grep
xfs_attr3_leaf_write_verify syslog*
Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772229] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa019a6f0
Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772504]
[<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.198659] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa019a6f0
Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.199085]
[<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
xfs_check
xfs_repair -> no fault
Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342473] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa019a6f0
Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342850]
[<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493190] XFS (sdc1):
Internal error xfs_attr3_leaf_write_verify at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01936f0
Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493550]
[<ffffffffa01b4b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[ 08:05:02 ] - root at gluster-ger-ber-12 ~/tmp/syslog $
gluster-ger-ber-10-int:
glustershd.log :
[2015-12-02 23:45:33.160852] W [socket.c:620:__socket_rwv]
0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data available)
[2015-12-02 23:45:33.170590] I [client.c:2203:client_rpc_notify]
0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-12-02 23:45:43.784388] E
[client-handshake.c:1496:client_query_portmap_cbk]
0-ger-ber-01-client-3: failed to get the port number for remote
subvolume. Please run 'gluster volume status' on server to see if brick
process is running.
[2015-12-02 23:45:43.784543] I [client.c:2203:client_rpc_notify]
0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-12-02 23:45:50.000203] W
[client-rpc-fops.c:1090:client3_3_getxattr_cbk] 0-ger-ber-01-client-3:
remote operation failed: Transport endpoint is not connected. Path: /
(00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo
[2015-12-02 23:49:33.524740] W [socket.c:620:__socket_rwv]
0-ger-ber-01-client-1: readv on 10.0.1.107:49152 failed (No data available)
[2015-12-02 23:49:33.524934] I [client.c:2203:client_rpc_notify]
0-ger-ber-01-client-1: disconnected from ger-ber-01-client-1. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-12-02 23:49:43.882976] E
[client-handshake.c:1496:client_query_portmap_cbk]
0-ger-ber-01-client-1: failed to get the port number for remote
subvolume. Please run 'gluster volume status' on server to see if
brick process is running.
sdn.log :
[2015-12-02 23:45:33.160963] W [socket.c:620:__socket_rwv]
0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data available)
[2015-12-02 23:45:33.168504] I [client.c:2203:client_rpc_notify]
0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-12-02 23:45:43.395787] E
[client-handshake.c:1496:client_query_portmap_cbk]
0-ger-ber-01-client-3: failed to get the port number for remote
subvolume. Please run 'gluster volume status' on server to see if brick
process is running.
nfs.log :
[2015-12-02 23:45:33.160856] W [socket.c:620:__socket_rwv]
0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data available)
[2015-12-02 23:45:33.180366] I [client.c:2203:client_rpc_notify]
0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-12-02 23:45:43.780186] E
[client-handshake.c:1496:client_query_portmap_cbk]
0-ger-ber-01-client-3: failed to get the port number for remote
subvolume. Please run 'gluster volume status' on server to see if brick
process is running.
[2015-12-02 23:45:43.780340] I [client.c:2203:client_rpc_notify]
0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
process will keep trying to connect to glusterd until brick's port is
available
geo-replication log :
[2015-12-02 23:44:34.624957] I [master(/gluster-export):514:crawlwrap]
_GMaster: 0 crawls, 0 turns
[2015-12-02 23:44:54.798414] E
[syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
line
164, in main main_i()
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
line
643, in main_i local.service_loop(*[r for r in [remote] if r])
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
line 1325, in service_loop g3.crawlwrap(oneshot=True)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
line
527, in crawlwrap brick_stime = self.xtime('.', self.slave)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
line
362, in xtime return self.xtime_low(rsc, path, **opts)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
line
132, in xtime_low xt = rsc.server.stime(path, self.uuid)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
line 1259, in <lambda> uuid + '.' + gconf.slave_id)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
line 322, in ff return f(*a)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
line 510, in stime 8)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py",
line 55, in lgetxattr return cls._query_xattr(path, siz, 'lgetxattr',
attr)
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py",
line 47, in _query_xattr cls.raise_oserr()
File
"/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py",
line 37, in raise_oserr raise OSError(errn, os.strerror(errn))
OSError: [Errno 5] Input/output error
[2015-12-02 23:44:54.845763] I
[syncdutils(/gluster-export):214:finalize] <top>: exiting.
[2015-12-02 23:44:54.847527] I [repce(agent):92:service_loop]
RepceServer: terminating on reaching EOF.
[2015-12-02 23:44:54.847784] I [syncdutils(agent):214:finalize] <top>:
exiting.
[2015-12-02 23:44:54.849092] I [monitor(monitor):141:set_state] Monitor:
new state: faulty
Vijay Bellur
2015-Dec-03 20:16 UTC
[Gluster-users] after upgrade to 3.6.7 : Internal error xfs_attr3_leaf_write_verify
Looks like an issue with xfs. Adding Brian to check if it is a familiar problem. Regards, Vijay ----- Original Message -----> From: "Dietmar Putz" <putz at 3qmedien.net> > To: gluster-users at gluster.org > Sent: Thursday, December 3, 2015 6:06:11 AM > Subject: [Gluster-users] after upgrade to 3.6.7 : Internal error xfs_attr3_leaf_write_verify > > Hello all, > > on 1st december i upgraded two 6 node cluster from glusterfs 3.5.6 to 3.6.7. > all of them are equal in hw, os and patchlevel, currently running ubuntu > 14.04 lts by an do-release-upgrade from 12.04 lts (this was done before > gfs upgrade to 3.5.6, not directly before upgrading to 3.6.7). > because of a geo-replication issue all of the nodes have rsync 3.1.1.3 > installed instead 3.1.0 which comes by the repositories. this is the > only deviation from ubuntu repositories for 14.04 lts. > since upgrade to gfs 3.6.7 the glusterd on two nodes of the same cluster > are going offline after getting an xfs_attr3_leaf_write_verify error for > the underlying bricks as shown below. > this happens about every 4-5 hours after the problem was solved by an > umount / remount of the brick. it makes no difference to run a xfs_check > / xfs_repair before remount. > xfs_check / xfs_repair did not show any faults. the underlying hw is a > raid 5 vol on lsi-9271 8i. megacli does not show any errors. > the syslog does not show more than the dmesg output below. > every time the same two nodes of the same cluster are affected. > as shown in dmesg and syslog, the system recognizes the > xfs_attr_leaf_write_verify error about 38 min. before finally giving up. > for both events i can not found corresponding events in gluster logs. > this is strange...the gluster is historical grown from 3.2.5, 3.3, to > 3.4.6/7 which was running well for month, gfs 3.5.6 was running for > about two weeks and upgrade to 3.6.7 was done because of a geo-repl > log-flood. > even when i have no hint/evidence that this is caused by gfs 3.6.7 > somehow i believe that this is the case... > does anybody experienced such an error or have some hints to getting out > of this big problem...? > unfortunately the affected cluster is the master of a geo-replication > which is not well running since update from gfs 3.4.7...fortunately both > affected gluster-nodes are not of the same sub-volume. > > any help is appreciated... > > best regards > dietmar > > > > > [ 09:32:29 ] - root at gluster-ger-ber-10 /var/log $gluster volume info > > Volume Name: ger-ber-01 > Type: Distributed-Replicate > Volume ID: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 > Status: Started > Number of Bricks: 3 x 2 = 6 > Transport-type: tcp > Bricks: > Brick1: gluster-ger-ber-11-int:/gluster-export > Brick2: gluster-ger-ber-12-int:/gluster-export > Brick3: gluster-ger-ber-09-int:/gluster-export > Brick4: gluster-ger-ber-10-int:/gluster-export > Brick5: gluster-ger-ber-07-int:/gluster-export > Brick6: gluster-ger-ber-08-int:/gluster-export > Options Reconfigured: > changelog.changelog: on > geo-replication.ignore-pid-check: on > cluster.min-free-disk: 200GB > geo-replication.indexing: on > auth.allow: > 10.0.1.*,188.138.82.*,188.138.123.*,82.193.249.198,82.193.249.200,31.7.178.137,31.7.178.135,31.7.180.109,31.7.180.98,82.199.147.*,104.155.22.202,104.155.30.201,104.155.5.117,104.155.11.253,104.155.15.34,104.155.25.145,146.148.120.255,31.7.180.148 > nfs.disable: off > performance.cache-refresh-timeout: 2 > performance.io-thread-count: 32 > performance.cache-size: 1024MB > performance.read-ahead: on > performance.cache-min-file-size: 0 > network.ping-timeout: 10 > [ 09:32:52 ] - root at gluster-ger-ber-10 /var/log $ > > > > > [ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $gluster volume status > Status of volume: ger-ber-01 > Gluster process Port Online Pid > ------------------------------------------------------------------------------ > > Brick gluster-ger-ber-11-int:/gluster-export 49152 Y 15994 > Brick gluster-ger-ber-12-int:/gluster-export N/A N N/A > Brick gluster-ger-ber-09-int:/gluster-export 49152 Y 10965 > Brick gluster-ger-ber-10-int:/gluster-export N/A N N/A > Brick gluster-ger-ber-07-int:/gluster-export 49152 Y 18542 > Brick gluster-ger-ber-08-int:/gluster-export 49152 Y 20275 > NFS Server on localhost 2049 Y 13658 > Self-heal Daemon on localhost N/A Y 13666 > NFS Server on gluster-ger-ber-09-int 2049 Y 13503 > Self-heal Daemon on gluster-ger-ber-09-int N/A Y 13511 > NFS Server on gluster-ger-ber-07-int 2049 Y 21526 > Self-heal Daemon on gluster-ger-ber-07-int N/A Y 21534 > NFS Server on gluster-ger-ber-08-int 2049 Y 24004 > Self-heal Daemon on gluster-ger-ber-08-int N/A Y 24011 > NFS Server on gluster-ger-ber-11-int 2049 Y 18944 > Self-heal Daemon on gluster-ger-ber-11-int N/A Y 18952 > NFS Server on gluster-ger-ber-12-int 2049 Y 19138 > Self-heal Daemon on gluster-ger-ber-12-int N/A Y 19146 > > Task Status of Volume ger-ber-01 > ------------------------------------------------------------------------------ > > There are no active volume tasks > > - root at gluster-ger-ber-10 /var/log $ > > - root at gluster-ger-ber-10 /var/log $dmesg -T > ... > [Wed Dec 2 12:43:47 2015] XFS (sdc1): xfs_log_force: error 5 returned. > [Wed Dec 2 12:43:48 2015] XFS (sdc1): xfs_log_force: error 5 returned. > [Wed Dec 2 12:45:58 2015] XFS (sdc1): Mounting Filesystem > [Wed Dec 2 12:45:58 2015] XFS (sdc1): Starting recovery (logdev: internal) > [Wed Dec 2 12:45:59 2015] XFS (sdc1): Ending recovery (logdev: internal) > [Wed Dec 2 13:11:53 2015] XFS (sdc1): Mounting Filesystem > [Wed Dec 2 13:11:54 2015] XFS (sdc1): Ending clean mount > [Wed Dec 2 13:12:29 2015] init: statd main process (25924) killed by > KILL signal > [Wed Dec 2 13:12:29 2015] init: statd main process ended, respawning > [Wed Dec 2 13:13:24 2015] init: statd main process (13433) killed by > KILL signal > [Wed Dec 2 13:13:24 2015] init: statd main process ended, respawning > [Wed Dec 2 17:22:28 2015] ffff8807076b1000: 00 00 00 00 00 00 00 00 fb > ee 00 00 00 00 00 00 ................ > [Wed Dec 2 17:22:28 2015] ffff8807076b1010: 10 00 00 00 00 20 0f e0 00 > 00 00 00 00 00 00 00 ..... .......... > [Wed Dec 2 17:22:28 2015] ffff8807076b1020: 00 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 ................ > [Wed Dec 2 17:22:28 2015] ffff8807076b1030: 00 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 ................ > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Wed Dec 2 17:22:28 2015] CPU: 4 PID: 13162 Comm: xfsaild/sdc1 Not > tainted 3.13.0-67-generic #110-Ubuntu > [Wed Dec 2 17:22:28 2015] Hardware name: Supermicro X10SLL-F/X10SLL-F, > BIOS 1.1b 11/01/2013 > [Wed Dec 2 17:22:28 2015] 0000000000000001 ffff8801c5691bd0 > ffffffff817240e0 ffff8801b15c3800 > [Wed Dec 2 17:22:28 2015] ffff8801c5691be8 ffffffffa01aa6fb > ffffffffa01a66f0 ffff8801c5691c20 > [Wed Dec 2 17:22:28 2015] ffffffffa01aa755 000000d800200200 > ffff8804a59ac780 ffff8800d917e658 > [Wed Dec 2 17:22:28 2015] Call Trace: > [Wed Dec 2 17:22:28 2015] [<ffffffff817240e0>] dump_stack+0x45/0x56 > [Wed Dec 2 17:22:28 2015] [<ffffffffa01aa6fb>] > xfs_error_report+0x3b/0x40 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ? > _xfs_buf_ioapply+0x70/0x3a0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01aa755>] > xfs_corruption_error+0x55/0x80 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ? > _xfs_buf_ioapply+0x70/0x3a0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ? > xfs_bdstrat_cb+0x55/0xb0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] > _xfs_buf_ioapply+0x70/0x3a0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffff8109ac90>] ? wake_up_state+0x20/0x20 > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ? > xfs_bdstrat_cb+0x55/0xb0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a8336>] > xfs_buf_iorequest+0x46/0x90 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] > xfs_bdstrat_cb+0x55/0xb0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a856b>] > __xfs_buf_delwri_submit+0x13b/0x210 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>] ? > xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ? > xfs_trans_ail_cursor_first+0x90/0x90 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>] > xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa0207d27>] xfsaild+0x237/0x5c0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ? > xfs_trans_ail_cursor_first+0x90/0x90 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffff8108b7d2>] kthread+0xd2/0xf0 > [Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ? > kthread_create_on_node+0x1c0/0x1c0 > [Wed Dec 2 17:22:28 2015] [<ffffffff81734c28>] ret_from_fork+0x58/0x90 > [Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ? > kthread_create_on_node+0x1c0/0x1c0 > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption detected. Unmount and > run xfs_repair > [Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_do_force_shutdown(0x8) called > from line 1320 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_buf.c. Return address > 0xffffffffa01a671c > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption of in-memory data > detected. Shutting down filesystem > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Please umount the filesystem and > rectify the problem(s) > [Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_log_force: error 5 returned. > [Wed Dec 2 17:22:49 2015] XFS (sdc1): xfs_log_force: error 5 returned. > ... > > [ 19:10:49 ] - root at gluster-ger-ber-10 /var/log $xfs_info /gluster-export > meta-data=/dev/sdc1 isize=256 agcount=32, > agsize=152596472 blks > = sectsz=512 attr=2 > data = bsize=4096 blocks=4883087099, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > [ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $ > > [ 09:36:37 ] - root at gluster-ger-ber-10 /var/log $stat /gluster-export > stat: cannot stat ?/gluster-export?: Input/output error > [ 09:36:45 ] - root at gluster-ger-ber-10 /var/log $ > > > [ 08:50:43 ] - root at gluster-ger-ber-10 ~/tmp/syslog $dmesg -T | grep > xfs_attr3_leaf_write_verify > [Di Dez 1 23:24:53 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Di Dez 1 23:24:53 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Mi Dez 2 12:19:16 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Mi Dez 2 12:19:16 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Mi Dez 2 17:22:28 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Mi Dez 2 17:22:28 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Mi Dez 2 23:06:32 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Mi Dez 2 23:06:32 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > > [ 08:06:28 ] - root at gluster-ger-ber-10 > /var/log/glusterfs/geo-replication $grep xfs_attr3_leaf_write_verify > /root/tmp/syslog/syslog* > Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906268] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906448] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.509891] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.510414] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > xfs_check > xfs_repair -> no fault > Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298098] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298259] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.969849] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.970217] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [ 08:06:37 ] - root at gluster-ger-ber-10 /var/log/glusterfs/geo-replication $ > > [ 08:04:51 ] - root at gluster-ger-ber-12 ~/tmp/syslog $grep > xfs_attr3_leaf_write_verify syslog* > Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772229] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa019a6f0 > Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772504] > [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.198659] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa019a6f0 > Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.199085] > [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > xfs_check > xfs_repair -> no fault > Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342473] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa019a6f0 > Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342850] > [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493190] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01936f0 > Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493550] > [<ffffffffa01b4b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [ 08:05:02 ] - root at gluster-ger-ber-12 ~/tmp/syslog $ > > gluster-ger-ber-10-int: > glustershd.log : > [2015-12-02 23:45:33.160852] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data available) > [2015-12-02 23:45:33.170590] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:43.784388] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-3: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if brick > process is running. > [2015-12-02 23:45:43.784543] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:50.000203] W > [client-rpc-fops.c:1090:client3_3_getxattr_cbk] 0-ger-ber-01-client-3: > remote operation failed: Transport endpoint is not connected. Path: / > (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo > [2015-12-02 23:49:33.524740] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-1: readv on 10.0.1.107:49152 failed (No data available) > [2015-12-02 23:49:33.524934] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-1: disconnected from ger-ber-01-client-1. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:49:43.882976] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-1: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if > brick process is running. > > sdn.log : > [2015-12-02 23:45:33.160963] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data available) > [2015-12-02 23:45:33.168504] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:43.395787] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-3: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if brick > process is running. > > nfs.log : > [2015-12-02 23:45:33.160856] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data available) > [2015-12-02 23:45:33.180366] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:43.780186] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-3: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if brick > process is running. > [2015-12-02 23:45:43.780340] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > > geo-replication log : > [2015-12-02 23:44:34.624957] I [master(/gluster-export):514:crawlwrap] > _GMaster: 0 crawls, 0 turns > [2015-12-02 23:44:54.798414] E > [syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL: > Traceback (most recent call last): > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", line > 164, in main main_i() > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", line > 643, in main_i local.service_loop(*[r for r in [remote] if r]) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 1325, in service_loop g3.crawlwrap(oneshot=True) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line > 527, in crawlwrap brick_stime = self.xtime('.', self.slave) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line > 362, in xtime return self.xtime_low(rsc, path, **opts) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line > 132, in xtime_low xt = rsc.server.stime(path, self.uuid) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 1259, in <lambda> uuid + '.' + gconf.slave_id) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 322, in ff return f(*a) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 510, in stime 8) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", > line 55, in lgetxattr return cls._query_xattr(path, siz, 'lgetxattr', > attr) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", > line 47, in _query_xattr cls.raise_oserr() > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", > line 37, in raise_oserr raise OSError(errn, os.strerror(errn)) > OSError: [Errno 5] Input/output error > [2015-12-02 23:44:54.845763] I > [syncdutils(/gluster-export):214:finalize] <top>: exiting. > [2015-12-02 23:44:54.847527] I [repce(agent):92:service_loop] > RepceServer: terminating on reaching EOF. > [2015-12-02 23:44:54.847784] I [syncdutils(agent):214:finalize] <top>: > exiting. > [2015-12-02 23:44:54.849092] I [monitor(monitor):141:set_state] Monitor: > new state: faulty > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Julius Thomas
2015-Dec-05 23:53 UTC
[Gluster-users] after upgrade to 3.6.7 : Internal error xfs_attr3_leaf_write_verify
Dear Gluster Users, after fixing the problem in the last mail from my colleague by upgrading to kernel 3.19.0-39-generic in case of changes with this bug in the xfs tree, the xfs filesystem crashes again after 4 - 5 hours on several peers. Has anyone recommendations for fixing this problems? Are there known issues with xfs and ubuntu 14.04? What is the latest stable release of gluster3, v3.6.3? Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018838] XFS (sdc1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x44458e670 Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018879] XFS (sdc1): Unmount and run xfs_repair Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018895] XFS (sdc1): First 64 bytes of corrupted metadata buffer: Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018916] ffff880417ff3000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018956] ffff880417ff3010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... .......... Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018984] ffff880417ff3020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019011] ffff880417ff3030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019041] XFS (sdc1): xfs_do_force_shutdown(0x8) called from line 1249 of file /build/linux-lts-vivid-1jarlV/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffc02bbd22 Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019044] XFS (sdc1): Corruption of in-memory data detected. Shutting down filesystem Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019069] XFS (sdc1): Please umount the filesystem and rectify the problem(s) Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.069906] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:15:08 gluster-ger-ber-11 gluster-export[4447]: [2015-12-05 21:15:08.797327] M [posix-helpers.c:1559:posix_health_check_thread_proc] 0-ger-ber-01-posix: health-check failed, going down Dec 5 21:15:18 gluster-ger-ber-11 kernel: [16594.068660] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:15:38 gluster-ger-ber-11 gluster-export[4447]: [2015-12-05 21:15:38.797422] M [posix-helpers.c:1564:posix_health_check_thread_proc] 0-ger-ber-01-posix: still alive! -> SIGTERM Dec 5 21:15:48 gluster-ger-ber-11 kernel: [16624.119428] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:16:18 gluster-ger-ber-11 kernel: [16654.170134] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:16:48 gluster-ger-ber-11 kernel: [16684.220834] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:17:01 gluster-ger-ber-11 CRON[17656]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Dec 5 21:17:18 gluster-ger-ber-11 kernel: [16714.271507] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:17:48 gluster-ger-ber-11 kernel: [16744.322244] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:18:18 gluster-ger-ber-11 kernel: [16774.372948] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:18:48 gluster-ger-ber-11 kernel: [16804.423650] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:19:18 gluster-ger-ber-11 kernel: [16834.474365] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:19:48 gluster-ger-ber-11 kernel: [16864.525082] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:20:18 gluster-ger-ber-11 kernel: [16894.575778] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:20:49 gluster-ger-ber-11 kernel: [16924.626464] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:21:19 gluster-ger-ber-11 kernel: [16954.677161] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:21:49 gluster-ger-ber-11 kernel: [16984.727791] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:22:19 gluster-ger-ber-11 kernel: [17014.778570] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:22:49 gluster-ger-ber-11 kernel: [17044.829240] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:23:19 gluster-ger-ber-11 kernel: [17074.880003] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:23:49 gluster-ger-ber-11 kernel: [17104.930643] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:24:19 gluster-ger-ber-11 kernel: [17134.981336] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:24:49 gluster-ger-ber-11 kernel: [17165.032049] XFS (sdc1): xfs_log_force: error -5 returned. Dec 5 21:25:19 gluster-ger-ber-11 kernel: [17195.082689] XFS (sdc1): xfs_log_force: error -5 returned. On 03.12.2015 12:06, Dietmar Putz wrote:> Hello all, > > on 1st december i upgraded two 6 node cluster from glusterfs 3.5.6 to > 3.6.7. > all of them are equal in hw, os and patchlevel, currently running > ubuntu 14.04 lts by an do-release-upgrade from 12.04 lts (this was > done before gfs upgrade to 3.5.6, not directly before upgrading to > 3.6.7). > because of a geo-replication issue all of the nodes have rsync 3.1.1.3 > installed instead 3.1.0 which comes by the repositories. this is the > only deviation from ubuntu repositories for 14.04 lts. > since upgrade to gfs 3.6.7 the glusterd on two nodes of the same > cluster are going offline after getting an xfs_attr3_leaf_write_verify > error for the underlying bricks as shown below. > this happens about every 4-5 hours after the problem was solved by an > umount / remount of the brick. it makes no difference to run a > xfs_check / xfs_repair before remount. > xfs_check / xfs_repair did not show any faults. the underlying hw is a > raid 5 vol on lsi-9271 8i. megacli does not show any errors. > the syslog does not show more than the dmesg output below. > every time the same two nodes of the same cluster are affected. > as shown in dmesg and syslog, the system recognizes the > xfs_attr_leaf_write_verify error about 38 min. before finally giving > up. for both events i can not found corresponding events in gluster logs. > this is strange...the gluster is historical grown from 3.2.5, 3.3, to > 3.4.6/7 which was running well for month, gfs 3.5.6 was running for > about two weeks and upgrade to 3.6.7 was done because of a geo-repl > log-flood. > even when i have no hint/evidence that this is caused by gfs 3.6.7 > somehow i believe that this is the case... > does anybody experienced such an error or have some hints to getting > out of this big problem...? > unfortunately the affected cluster is the master of a geo-replication > which is not well running since update from gfs 3.4.7...fortunately > both affected gluster-nodes are not of the same sub-volume. > > any help is appreciated... > > best regards > dietmar > > > > > [ 09:32:29 ] - root at gluster-ger-ber-10 /var/log $gluster volume info > > Volume Name: ger-ber-01 > Type: Distributed-Replicate > Volume ID: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 > Status: Started > Number of Bricks: 3 x 2 = 6 > Transport-type: tcp > Bricks: > Brick1: gluster-ger-ber-11-int:/gluster-export > Brick2: gluster-ger-ber-12-int:/gluster-export > Brick3: gluster-ger-ber-09-int:/gluster-export > Brick4: gluster-ger-ber-10-int:/gluster-export > Brick5: gluster-ger-ber-07-int:/gluster-export > Brick6: gluster-ger-ber-08-int:/gluster-export > Options Reconfigured: > changelog.changelog: on > geo-replication.ignore-pid-check: on > cluster.min-free-disk: 200GB > geo-replication.indexing: on > auth.allow: > 10.0.1.*,188.138.82.*,188.138.123.*,82.193.249.198,82.193.249.200,31.7.178.137,31.7.178.135,31.7.180.109,31.7.180.98,82.199.147.*,104.155.22.202,104.155.30.201,104.155.5.117,104.155.11.253,104.155.15.34,104.155.25.145,146.148.120.255,31.7.180.148 > nfs.disable: off > performance.cache-refresh-timeout: 2 > performance.io-thread-count: 32 > performance.cache-size: 1024MB > performance.read-ahead: on > performance.cache-min-file-size: 0 > network.ping-timeout: 10 > [ 09:32:52 ] - root at gluster-ger-ber-10 /var/log $ > > > > > [ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $gluster volume status > Status of volume: ger-ber-01 > Gluster process Port Online Pid > ------------------------------------------------------------------------------ > > Brick gluster-ger-ber-11-int:/gluster-export 49152 Y 15994 > Brick gluster-ger-ber-12-int:/gluster-export N/A N N/A > Brick gluster-ger-ber-09-int:/gluster-export 49152 Y 10965 > Brick gluster-ger-ber-10-int:/gluster-export N/A N N/A > Brick gluster-ger-ber-07-int:/gluster-export 49152 Y 18542 > Brick gluster-ger-ber-08-int:/gluster-export 49152 Y 20275 > NFS Server on localhost 2049 Y 13658 > Self-heal Daemon on localhost N/A Y 13666 > NFS Server on gluster-ger-ber-09-int 2049 Y 13503 > Self-heal Daemon on gluster-ger-ber-09-int N/A Y 13511 > NFS Server on gluster-ger-ber-07-int 2049 Y 21526 > Self-heal Daemon on gluster-ger-ber-07-int N/A Y 21534 > NFS Server on gluster-ger-ber-08-int 2049 Y 24004 > Self-heal Daemon on gluster-ger-ber-08-int N/A Y 24011 > NFS Server on gluster-ger-ber-11-int 2049 Y 18944 > Self-heal Daemon on gluster-ger-ber-11-int N/A Y 18952 > NFS Server on gluster-ger-ber-12-int 2049 Y 19138 > Self-heal Daemon on gluster-ger-ber-12-int N/A Y 19146 > > Task Status of Volume ger-ber-01 > ------------------------------------------------------------------------------ > > There are no active volume tasks > > - root at gluster-ger-ber-10 /var/log $ > > - root at gluster-ger-ber-10 /var/log $dmesg -T > ... > [Wed Dec 2 12:43:47 2015] XFS (sdc1): xfs_log_force: error 5 returned. > [Wed Dec 2 12:43:48 2015] XFS (sdc1): xfs_log_force: error 5 returned. > [Wed Dec 2 12:45:58 2015] XFS (sdc1): Mounting Filesystem > [Wed Dec 2 12:45:58 2015] XFS (sdc1): Starting recovery (logdev: > internal) > [Wed Dec 2 12:45:59 2015] XFS (sdc1): Ending recovery (logdev: internal) > [Wed Dec 2 13:11:53 2015] XFS (sdc1): Mounting Filesystem > [Wed Dec 2 13:11:54 2015] XFS (sdc1): Ending clean mount > [Wed Dec 2 13:12:29 2015] init: statd main process (25924) killed by > KILL signal > [Wed Dec 2 13:12:29 2015] init: statd main process ended, respawning > [Wed Dec 2 13:13:24 2015] init: statd main process (13433) killed by > KILL signal > [Wed Dec 2 13:13:24 2015] init: statd main process ended, respawning > [Wed Dec 2 17:22:28 2015] ffff8807076b1000: 00 00 00 00 00 00 00 00 > fb ee 00 00 00 00 00 00 ................ > [Wed Dec 2 17:22:28 2015] ffff8807076b1010: 10 00 00 00 00 20 0f e0 > 00 00 00 00 00 00 00 00 ..... .......... > [Wed Dec 2 17:22:28 2015] ffff8807076b1020: 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 ................ > [Wed Dec 2 17:22:28 2015] ffff8807076b1030: 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 ................ > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Wed Dec 2 17:22:28 2015] CPU: 4 PID: 13162 Comm: xfsaild/sdc1 Not > tainted 3.13.0-67-generic #110-Ubuntu > [Wed Dec 2 17:22:28 2015] Hardware name: Supermicro > X10SLL-F/X10SLL-F, BIOS 1.1b 11/01/2013 > [Wed Dec 2 17:22:28 2015] 0000000000000001 ffff8801c5691bd0 > ffffffff817240e0 ffff8801b15c3800 > [Wed Dec 2 17:22:28 2015] ffff8801c5691be8 ffffffffa01aa6fb > ffffffffa01a66f0 ffff8801c5691c20 > [Wed Dec 2 17:22:28 2015] ffffffffa01aa755 000000d800200200 > ffff8804a59ac780 ffff8800d917e658 > [Wed Dec 2 17:22:28 2015] Call Trace: > [Wed Dec 2 17:22:28 2015] [<ffffffff817240e0>] dump_stack+0x45/0x56 > [Wed Dec 2 17:22:28 2015] [<ffffffffa01aa6fb>] > xfs_error_report+0x3b/0x40 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ? > _xfs_buf_ioapply+0x70/0x3a0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01aa755>] > xfs_corruption_error+0x55/0x80 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ? > _xfs_buf_ioapply+0x70/0x3a0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ? > xfs_bdstrat_cb+0x55/0xb0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] > _xfs_buf_ioapply+0x70/0x3a0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffff8109ac90>] ? > wake_up_state+0x20/0x20 > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ? > xfs_bdstrat_cb+0x55/0xb0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a8336>] > xfs_buf_iorequest+0x46/0x90 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] > xfs_bdstrat_cb+0x55/0xb0 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a856b>] > __xfs_buf_delwri_submit+0x13b/0x210 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>] ? > xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ? > xfs_trans_ail_cursor_first+0x90/0x90 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>] > xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa0207d27>] xfsaild+0x237/0x5c0 > [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ? > xfs_trans_ail_cursor_first+0x90/0x90 [xfs] > [Wed Dec 2 17:22:28 2015] [<ffffffff8108b7d2>] kthread+0xd2/0xf0 > [Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ? > kthread_create_on_node+0x1c0/0x1c0 > [Wed Dec 2 17:22:28 2015] [<ffffffff81734c28>] ret_from_fork+0x58/0x90 > [Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ? > kthread_create_on_node+0x1c0/0x1c0 > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption detected. Unmount > and run xfs_repair > [Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_do_force_shutdown(0x8) > called from line 1320 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_buf.c. Return address = > 0xffffffffa01a671c > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption of in-memory data > detected. Shutting down filesystem > [Wed Dec 2 17:22:28 2015] XFS (sdc1): Please umount the filesystem > and rectify the problem(s) > [Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_log_force: error 5 returned. > [Wed Dec 2 17:22:49 2015] XFS (sdc1): xfs_log_force: error 5 returned. > ... > > [ 19:10:49 ] - root at gluster-ger-ber-10 /var/log $xfs_info > /gluster-export > meta-data=/dev/sdc1 isize=256 agcount=32, > agsize=152596472 blks > = sectsz=512 attr=2 > data = bsize=4096 blocks=4883087099, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > [ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $ > > [ 09:36:37 ] - root at gluster-ger-ber-10 /var/log $stat /gluster-export > stat: cannot stat ?/gluster-export?: Input/output error > [ 09:36:45 ] - root at gluster-ger-ber-10 /var/log $ > > > [ 08:50:43 ] - root at gluster-ger-ber-10 ~/tmp/syslog $dmesg -T | grep > xfs_attr3_leaf_write_verify > [Di Dez 1 23:24:53 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Di Dez 1 23:24:53 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Mi Dez 2 12:19:16 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Mi Dez 2 12:19:16 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Mi Dez 2 17:22:28 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Mi Dez 2 17:22:28 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [Mi Dez 2 23:06:32 2015] XFS (sdc1): Internal error > xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > [Mi Dez 2 23:06:32 2015] [<ffffffffa01c7b70>] > xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > > [ 08:06:28 ] - root at gluster-ger-ber-10 > /var/log/glusterfs/geo-replication $grep xfs_attr3_leaf_write_verify > /root/tmp/syslog/syslog* > Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906268] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906448] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.509891] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.510414] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > xfs_check > xfs_repair -> no fault > Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298098] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298259] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.969849] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01a66f0 > Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.970217] > [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [ 08:06:37 ] - root at gluster-ger-ber-10 > /var/log/glusterfs/geo-replication $ > > [ 08:04:51 ] - root at gluster-ger-ber-12 ~/tmp/syslog $grep > xfs_attr3_leaf_write_verify syslog* > Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772229] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa019a6f0 > Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772504] > [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.198659] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa019a6f0 > Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.199085] > [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > xfs_check > xfs_repair -> no fault > Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342473] XFS > (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa019a6f0 > Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342850] > [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493190] XFS (sdc1): > Internal error xfs_attr3_leaf_write_verify at line 216 of file > /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller > 0xffffffffa01936f0 > Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493550] > [<ffffffffa01b4b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] > [ 08:05:02 ] - root at gluster-ger-ber-12 ~/tmp/syslog $ > > gluster-ger-ber-10-int: > glustershd.log : > [2015-12-02 23:45:33.160852] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data > available) > [2015-12-02 23:45:33.170590] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:43.784388] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-3: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if > brick process is running. > [2015-12-02 23:45:43.784543] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:50.000203] W > [client-rpc-fops.c:1090:client3_3_getxattr_cbk] 0-ger-ber-01-client-3: > remote operation failed: Transport endpoint is not connected. Path: / > (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo > [2015-12-02 23:49:33.524740] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-1: readv on 10.0.1.107:49152 failed (No data > available) > [2015-12-02 23:49:33.524934] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-1: disconnected from ger-ber-01-client-1. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:49:43.882976] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-1: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if > brick process is running. > > sdn.log : > [2015-12-02 23:45:33.160963] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data > available) > [2015-12-02 23:45:33.168504] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:43.395787] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-3: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if > brick process is running. > > nfs.log : > [2015-12-02 23:45:33.160856] W [socket.c:620:__socket_rwv] > 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data > available) > [2015-12-02 23:45:33.180366] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > [2015-12-02 23:45:43.780186] E > [client-handshake.c:1496:client_query_portmap_cbk] > 0-ger-ber-01-client-3: failed to get the port number for remote > subvolume. Please run 'gluster volume status' on server to see if > brick process is running. > [2015-12-02 23:45:43.780340] I [client.c:2203:client_rpc_notify] > 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client > process will keep trying to connect to glusterd until brick's port is > available > > geo-replication log : > [2015-12-02 23:44:34.624957] I [master(/gluster-export):514:crawlwrap] > _GMaster: 0 crawls, 0 turns > [2015-12-02 23:44:54.798414] E > [syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL: > Traceback (most recent call last): > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", > line 164, in main main_i() > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", > line 643, in main_i local.service_loop(*[r for r in [remote] if r]) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 1325, in service_loop g3.crawlwrap(oneshot=True) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", > line 527, in crawlwrap brick_stime = self.xtime('.', self.slave) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", > line 362, in xtime return self.xtime_low(rsc, path, **opts) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", > line 132, in xtime_low xt = rsc.server.stime(path, self.uuid) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 1259, in <lambda> uuid + '.' + gconf.slave_id) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 322, in ff return f(*a) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", > line 510, in stime 8) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", > line 55, in lgetxattr return cls._query_xattr(path, siz, > 'lgetxattr', attr) > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", > line 47, in _query_xattr cls.raise_oserr() > File > "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", > line 37, in raise_oserr raise OSError(errn, os.strerror(errn)) > OSError: [Errno 5] Input/output error > [2015-12-02 23:44:54.845763] I > [syncdutils(/gluster-export):214:finalize] <top>: exiting. > [2015-12-02 23:44:54.847527] I [repce(agent):92:service_loop] > RepceServer: terminating on reaching EOF. > [2015-12-02 23:44:54.847784] I [syncdutils(agent):214:finalize] <top>: > exiting. > [2015-12-02 23:44:54.849092] I [monitor(monitor):141:set_state] > Monitor: new state: faulty > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-- Mit freundlichen Gr??en Julius Thomas Gesch?ftsf?hrer 3Q Medien GmbH Wetzlarer Str. 86 D-14482 Potsdam Fon +49 (0)331 / 27 97 866 - 0 Fax +49 (0)331 / 27 97 866 - 1 www.3qsdn.com Vertrieb: sales at 3qsdn.com Hauptsitz der Gesellschaft: Siemensstr. 3, 84478 Waldkraiburg Vertretungsberechtigte Gesch?ftsf?hrer: Holm Kr?mer, Julius Thomas Registergericht: Handelsregister Traunstein, HRB 19129 Diese E-Mail und eventuelle Anlagen k?nnen vertrauliche und/oder rechtlich gesch?tzte Informationen enthalten. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet. Sie k?nnen der Nutzung Ihrer Daten widersprechen. Wenn Sie zuk?nftig keine Informationen mehr zu interessanten Leistungen von 3Q Medien GmbH erhalten m?chten, teilen Sie dies bitte an oben genannte Adresse mit.