Xavier Hernandez
2017-Jan-10 12:53 UTC
[Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume
Hi Ram, the error is caused by an extended attribute that does not match on all 3 bricks of the disperse set. Most probable value is trusted.ec.version, but could be others. At first sight, I don't see any change from 3.7.8 that could have caused this. I'll check again. What kind of operations are you doing ? this can help me narrow the search. Xavi On 10/01/17 13:43, Ankireddypalle Reddy wrote:> Xavi, > Thanks. If you could please explain what to look for in the extended attributes then I will check and let you know if I find anything suspicious. Also we noticed that some of these operations would succeed if retried. Do you know of any communicated related errors that are being reported/triaged. > > Thanks and Regards, > Ram > > -----Original Message----- > From: Xavier Hernandez [mailto:xhernandez at datalab.es] > Sent: Tuesday, January 10, 2017 7:23 AM > To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org); gluster-users at gluster.org > Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume > > Hi Ram, > > On 10/01/17 13:14, Ankireddypalle Reddy wrote: >> Attachment (1): >> >> 1 >> >> >> >> ecxattrs.txt >> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co >> mmvault.com/webconsole/api/drive/publicshare/346714/file/1272e68278744 >> f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault. >> com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744f >> 15bf1a54f2b31b559d/action/download> >> [Download] >> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34 >> 6714/file/1272e68278744f15bf1a54f2b31b559d/action/download>(5.92 >> KB) >> >> Xavi, >> Please find attached the extended attributes for a >> directory from all the bricks. Free space check failed for this with >> error number EIO. > > What do you mean ? what operation have you made to check the free space on that directory ? > > If it's a recursive check, I need the extended attributes from the exact file that triggers the EIO. The attached attributes seem consistent and that directory shouldn't cause any problem. Does an 'ls' on that directory fail or does it show the contents ? > > Xavi > >> >> Thanks and Regards, >> Ram >> >> -----Original Message----- >> From: Xavier Hernandez [mailto:xhernandez at datalab.es] >> Sent: Tuesday, January 10, 2017 6:45 AM >> To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org); >> gluster-users at gluster.org >> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume >> >> Hi Ram, >> >> can you execute the following command on all bricks on a file that is >> giving EIO ? >> >> getfattr -m. -e hex -d <path to file in brick> >> >> Xavi >> >> On 10/01/17 12:41, Ankireddypalle Reddy wrote: >>> Xavi, >>> We have been running 3.7.8 on these servers. We upgraded >> to 3.7.18 yesterday. We upgraded all the servers at a time. The >> volume was brought down during upgrade. >>> >>> Thanks and Regards, >>> Ram >>> >>> -----Original Message----- >>> From: Xavier Hernandez [mailto:xhernandez at datalab.es] >>> Sent: Tuesday, January 10, 2017 6:35 AM >>> To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org); >>> gluster-users at gluster.org >>> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume >>> >>> Hi Ram, >>> >>> how did you upgrade gluster ? from which version ? >>> >>> Did you upgrade one server at a time and waited until self-heal >> finished before upgrading the next server ? >>> >>> Xavi >>> >>> On 10/01/17 11:39, Ankireddypalle Reddy wrote: >>>> Hi, >>>> >>>> We upgraded to GlusterFS 3.7.18 yesterday. We see lot of >>>> failures in our applications. Most of the errors are EIO. The >>>> following log lines are commonly seen in the logs: >>>> >>>> >>>> >>>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] >>>> 0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'" >>>> repeated 2 times between [2017-01-10 02:46:25.069809] and >>>> [2017-01-10 02:46:25.069835] >>>> >>>> [2017-01-10 02:46:25.069852] W [MSGID: 122056] >>>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5: >>>> Mismatching xdata in answers of 'LOOKUP' >>>> >>>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] >>>> 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'" >>>> repeated 2 times between [2017-01-10 02:46:25.069852] and >>>> [2017-01-10 02:46:25.069873] >>>> >>>> [2017-01-10 02:46:25.069910] W [MSGID: 122056] >>>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6: >>>> Mismatching xdata in answers of 'LOOKUP' >>>> >>>> ... >>>> >>>> [2017-01-10 02:46:26.520774] I [MSGID: 109036] >>>> [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal] >>>> 0-StoragePool-dht: Setting layout of >>>> /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585 with >>>> [Subvol_name: StoragePool-disperse-0, Err: -1 , Start: 3221225466 , >>>> Stop: 3758096376 , Hash: 1 ], [Subvol_name: StoragePool-disperse-1, Err: >>>> -1 , Start: 3758096377 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: >>>> StoragePool-disperse-2, Err: -1 , Start: 0 , Stop: 536870910 , Hash: >>>> 1 ], [Subvol_name: StoragePool-disperse-3, Err: -1 , Start: >>>> 536870911 , >>>> Stop: 1073741821 , Hash: 1 ], [Subvol_name: StoragePool-disperse-4, Err: >>>> -1 , Start: 1073741822 , Stop: 1610612732 , Hash: 1 ], [Subvol_name: >>>> StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop: >>>> 2147483643 , >>>> Hash: 1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 , Start: >>>> 2147483644 , Stop: 2684354554 , Hash: 1 ], [Subvol_name: >>>> StoragePool-disperse-7, Err: -1 , Start: 2684354555 , Stop: >>>> 3221225465 , >>>> Hash: 1 ], >>>> >>>> [2017-01-10 02:46:26.522841] N [MSGID: 122031] >>>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3: >>>> Mismatching dictionary in answers of 'GF_FOP_XATTROP' >>>> >>>> The message "N [MSGID: 122031] >>>> [ec-generic.c:1130:ec_combine_xattrop] >>>> 0-StoragePool-disperse-3: Mismatching dictionary in answers of >>>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 >>>> 02:46:26.522841] and [2017-01-10 02:46:26.522894] >>>> >>>> [2017-01-10 02:46:26.522898] W [MSGID: 122040] >>>> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-3: >>>> Failed to get size and version [Input/output error] >>>> >>>> [2017-01-10 02:46:26.523115] N [MSGID: 122031] >>>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-6: >>>> Mismatching dictionary in answers of 'GF_FOP_XATTROP' >>>> >>>> The message "N [MSGID: 122031] >>>> [ec-generic.c:1130:ec_combine_xattrop] >>>> 0-StoragePool-disperse-6: Mismatching dictionary in answers of >>>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 >>>> 02:46:26.523115] and [2017-01-10 02:46:26.523143] >>>> >>>> [2017-01-10 02:46:26.523147] W [MSGID: 122040] >>>> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-6: >>>> Failed to get size and version [Input/output error] >>>> >>>> [2017-01-10 02:46:26.523302] N [MSGID: 122031] >>>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-2: >>>> Mismatching dictionary in answers of 'GF_FOP_XATTROP' >>>> >>>> The message "N [MSGID: 122031] >>>> [ec-generic.c:1130:ec_combine_xattrop] >>>> 0-StoragePool-disperse-2: Mismatching dictionary in answers of >>>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 >>>> 02:46:26.523302] and [2017-01-10 02:46:26.523324] >>>> >>>> [2017-01-10 02:46:26.523328] W [MSGID: 122040] >>>> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-2: >>>> Failed to get size and version [Input/output error] >>>> >>>> >>>> >>>> [root at glusterfs3 Log_Files]# gluster --version >>>> >>>> glusterfs 3.7.18 built on Dec 8 2016 06:34:26 >>>> >>>> >>>> >>>> [root at glusterfs3 Log_Files]# gluster volume info >>>> >>>> >>>> >>>> Volume Name: StoragePool >>>> >>>> Type: Distributed-Disperse >>>> >>>> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f >>>> >>>> Status: Started >>>> >>>> Number of Bricks: 8 x (2 + 1) = 24 >>>> >>>> Transport-type: tcp >>>> >>>> Bricks: >>>> >>>> Brick1: glusterfs1sds:/ws/disk1/ws_brick >>>> >>>> Brick2: glusterfs2sds:/ws/disk1/ws_brick >>>> >>>> Brick3: glusterfs3sds:/ws/disk1/ws_brick >>>> >>>> Brick4: glusterfs1sds:/ws/disk2/ws_brick >>>> >>>> Brick5: glusterfs2sds:/ws/disk2/ws_brick >>>> >>>> Brick6: glusterfs3sds:/ws/disk2/ws_brick >>>> >>>> Brick7: glusterfs1sds:/ws/disk3/ws_brick >>>> >>>> Brick8: glusterfs2sds:/ws/disk3/ws_brick >>>> >>>> Brick9: glusterfs3sds:/ws/disk3/ws_brick >>>> >>>> Brick10: glusterfs1sds:/ws/disk4/ws_brick >>>> >>>> Brick11: glusterfs2sds:/ws/disk4/ws_brick >>>> >>>> Brick12: glusterfs3sds:/ws/disk4/ws_brick >>>> >>>> Brick13: glusterfs1sds:/ws/disk5/ws_brick >>>> >>>> Brick14: glusterfs2sds:/ws/disk5/ws_brick >>>> >>>> Brick15: glusterfs3sds:/ws/disk5/ws_brick >>>> >>>> Brick16: glusterfs1sds:/ws/disk6/ws_brick >>>> >>>> Brick17: glusterfs2sds:/ws/disk6/ws_brick >>>> >>>> Brick18: glusterfs3sds:/ws/disk6/ws_brick >>>> >>>> Brick19: glusterfs1sds:/ws/disk7/ws_brick >>>> >>>> Brick20: glusterfs2sds:/ws/disk7/ws_brick >>>> >>>> Brick21: glusterfs3sds:/ws/disk7/ws_brick >>>> >>>> Brick22: glusterfs1sds:/ws/disk8/ws_brick >>>> >>>> Brick23: glusterfs2sds:/ws/disk8/ws_brick >>>> >>>> Brick24: glusterfs3sds:/ws/disk8/ws_brick >>>> >>>> Options Reconfigured: >>>> >>>> performance.readdir-ahead: on >>>> >>>> diagnostics.client-log-level: INFO >>>> >>>> >>>> >>>> Thanks and Regards, >>>> >>>> Ram >>>> >>>> ***************************Legal >>>> Disclaimer*************************** >>>> "This communication may contain confidential and privileged material >>>> for the sole use of the intended recipient. Any unauthorized review, >>>> use or distribution by others is strictly prohibited. If you have >>>> received the message by mistake, please advise the sender by reply >>>> email and delete the message. Thank you." >>>> ******************************************************************** >>>> * >>>> * >>>> >>>> >>>> _______________________________________________ >>>> Gluster-devel mailing list >>>> Gluster-devel at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-devel >>>> >>> >>> ***************************Legal >>> Disclaimer*************************** >>> "This communication may contain confidential and privileged material >>> for the sole use of the intended recipient. Any unauthorized review, >>> use or distribution by others is strictly prohibited. If you have >>> received the message by mistake, please advise the sender by reply >> email and delete the message. Thank you." >>> ********************************************************************* >>> * >>> >> >> ***************************Legal Disclaimer*************************** >> "This communication may contain confidential and privileged material >> for the sole use of the intended recipient. Any unauthorized review, >> use or distribution by others is strictly prohibited. If you have >> received the message by mistake, please advise the sender by reply >> email and delete the message. Thank you." >> ********************************************************************** > > ***************************Legal Disclaimer*************************** > "This communication may contain confidential and privileged material for the > sole use of the intended recipient. Any unauthorized review, use or distribution > by others is strictly prohibited. If you have received the message by mistake, > please advise the sender by reply email and delete the message. Thank you." > ********************************************************************** >
Ankireddypalle Reddy
2017-Jan-10 13:42 UTC
[Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume
Attachments (2):
1
ec.txt<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/download>
[Download]<https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/download>
(11.50 KB)
2
ws-glus.log<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/cff3e0506e754b9a939db02da1cbbd58/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/cff3e0506e754b9a939db02da1cbbd58/action/download>
[Download]<https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/cff3e0506e754b9a939db02da1cbbd58/action/download>
(3.48 MB)
Xavi,
We are encountering errors for different kinds of FOPS.
The open failed for the following file:
cvd_2017_01_10_02_28_26.log:98182 1f9fe 01/10 00:57:10 8414465
[MEDIAFS ] 20117519-52075477 SingleInstancer_FS::StartDataFile2: Failed to
create the data file
[/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720/SFILE_CONTAINER_062],
error=0xECCC0005:{CQiFile::Open(92)} +
{CQiUTFOSAPI::open(96)/ErrNo.5.(Input/output error)-Open failed,
File=/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720/SFILE_CONTAINER_062,
OperationFlag=0xC1, PermissionMode=0x1FF}
I've attached the extended attributes for the directories
/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/ and
/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720
from all the bricks.
The attributes look fine to me. I've also attached some log cuts to
illustrate the problem.
Thanks and Regards,
Ram
-----Original Message-----
From: Xavier Hernandez [mailto:xhernandez at datalab.es]
Sent: Tuesday, January 10, 2017 7:53 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org);
gluster-users at gluster.org
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
Hi Ram,
the error is caused by an extended attribute that does not match on all
3 bricks of the disperse set. Most probable value is trusted.ec.version, but
could be others.
At first sight, I don't see any change from 3.7.8 that could have caused
this. I'll check again.
What kind of operations are you doing ? this can help me narrow the search.
Xavi
On 10/01/17 13:43, Ankireddypalle Reddy wrote:> Xavi,
> Thanks. If you could please explain what to look for in the
extended attributes then I will check and let you know if I find anything
suspicious. Also we noticed that some of these operations would succeed if
retried. Do you know of any communicated related errors that are being
reported/triaged.
>
> Thanks and Regards,
> Ram
>
> -----Original Message-----
> From: Xavier Hernandez [mailto:xhernandez at datalab.es]
> Sent: Tuesday, January 10, 2017 7:23 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org);
> gluster-users at gluster.org
> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>
> Hi Ram,
>
> On 10/01/17 13:14, Ankireddypalle Reddy wrote:
>> Attachment (1):
>>
>> 1
>>
>>
>>
>> ecxattrs.txt
>>
<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.c
>> o
>> mmvault.com/webconsole/api/drive/publicshare/346714/file/1272e6827874
>> 4
>>
f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault.
>> com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744
>> f
>> 15bf1a54f2b31b559d/action/download>
>> [Download]
>>
<https://imap.commvault.com/webconsole/api/contentstore/publicshare/3
>> 4
>> 6714/file/1272e68278744f15bf1a54f2b31b559d/action/download>(5.92
>> KB)
>>
>> Xavi,
>> Please find attached the extended attributes for a
>> directory from all the bricks. Free space check failed for this with
>> error number EIO.
>
> What do you mean ? what operation have you made to check the free space on
that directory ?
>
> If it's a recursive check, I need the extended attributes from the
exact file that triggers the EIO. The attached attributes seem consistent and
that directory shouldn't cause any problem. Does an 'ls' on that
directory fail or does it show the contents ?
>
> Xavi
>
>>
>> Thanks and Regards,
>> Ram
>>
>> -----Original Message-----
>> From: Xavier Hernandez [mailto:xhernandez at datalab.es]
>> Sent: Tuesday, January 10, 2017 6:45 AM
>> To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org);
>> gluster-users at gluster.org
>> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>>
>> Hi Ram,
>>
>> can you execute the following command on all bricks on a file that is
>> giving EIO ?
>>
>> getfattr -m. -e hex -d <path to file in brick>
>>
>> Xavi
>>
>> On 10/01/17 12:41, Ankireddypalle Reddy wrote:
>>> Xavi,
>>> We have been running 3.7.8 on these servers. We
upgraded
>> to 3.7.18 yesterday. We upgraded all the servers at a time. The
>> volume was brought down during upgrade.
>>>
>>> Thanks and Regards,
>>> Ram
>>>
>>> -----Original Message-----
>>> From: Xavier Hernandez [mailto:xhernandez at datalab.es]
>>> Sent: Tuesday, January 10, 2017 6:35 AM
>>> To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at
gluster.org);
>>> gluster-users at gluster.org
>>> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>>>
>>> Hi Ram,
>>>
>>> how did you upgrade gluster ? from which version ?
>>>
>>> Did you upgrade one server at a time and waited until self-heal
>> finished before upgrading the next server ?
>>>
>>> Xavi
>>>
>>> On 10/01/17 11:39, Ankireddypalle Reddy wrote:
>>>> Hi,
>>>>
>>>> We upgraded to GlusterFS 3.7.18 yesterday. We see lot of
>>>> failures in our applications. Most of the errors are EIO. The
>>>> following log lines are commonly seen in the logs:
>>>>
>>>>
>>>>
>>>> The message "W [MSGID: 122056]
[ec-combine.c:873:ec_combine_check]
>>>> 0-StoragePool-disperse-4: Mismatching xdata in answers of
'LOOKUP'"
>>>> repeated 2 times between [2017-01-10 02:46:25.069809] and
>>>> [2017-01-10 02:46:25.069835]
>>>>
>>>> [2017-01-10 02:46:25.069852] W [MSGID: 122056]
>>>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5:
>>>> Mismatching xdata in answers of 'LOOKUP'
>>>>
>>>> The message "W [MSGID: 122056]
[ec-combine.c:873:ec_combine_check]
>>>> 0-StoragePool-disperse-5: Mismatching xdata in answers of
'LOOKUP'"
>>>> repeated 2 times between [2017-01-10 02:46:25.069852] and
>>>> [2017-01-10 02:46:25.069873]
>>>>
>>>> [2017-01-10 02:46:25.069910] W [MSGID: 122056]
>>>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6:
>>>> Mismatching xdata in answers of 'LOOKUP'
>>>>
>>>> ...
>>>>
>>>> [2017-01-10 02:46:26.520774] I [MSGID: 109036]
>>>> [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal]
>>>> 0-StoragePool-dht: Setting layout of
>>>> /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585
with
>>>> [Subvol_name: StoragePool-disperse-0, Err: -1 , Start:
3221225466 ,
>>>> Stop: 3758096376 , Hash: 1 ], [Subvol_name:
StoragePool-disperse-1, Err:
>>>> -1 , Start: 3758096377 , Stop: 4294967295 , Hash: 1 ],
[Subvol_name:
>>>> StoragePool-disperse-2, Err: -1 , Start: 0 , Stop: 536870910 ,
Hash:
>>>> 1 ], [Subvol_name: StoragePool-disperse-3, Err: -1 , Start:
>>>> 536870911 ,
>>>> Stop: 1073741821 , Hash: 1 ], [Subvol_name:
StoragePool-disperse-4, Err:
>>>> -1 , Start: 1073741822 , Stop: 1610612732 , Hash: 1 ],
[Subvol_name:
>>>> StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop:
>>>> 2147483643 ,
>>>> Hash: 1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 ,
Start:
>>>> 2147483644 , Stop: 2684354554 , Hash: 1 ], [Subvol_name:
>>>> StoragePool-disperse-7, Err: -1 , Start: 2684354555 , Stop:
>>>> 3221225465 ,
>>>> Hash: 1 ],
>>>>
>>>> [2017-01-10 02:46:26.522841] N [MSGID: 122031]
>>>> [ec-generic.c:1130:ec_combine_xattrop]
0-StoragePool-disperse-3:
>>>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>>>>
>>>> The message "N [MSGID: 122031]
>>>> [ec-generic.c:1130:ec_combine_xattrop]
>>>> 0-StoragePool-disperse-3: Mismatching dictionary in answers of
>>>> 'GF_FOP_XATTROP'" repeated 2 times between
[2017-01-10
>>>> 02:46:26.522841] and [2017-01-10 02:46:26.522894]
>>>>
>>>> [2017-01-10 02:46:26.522898] W [MSGID: 122040]
>>>> [ec-common.c:919:ec_prepare_update_cbk]
0-StoragePool-disperse-3:
>>>> Failed to get size and version [Input/output error]
>>>>
>>>> [2017-01-10 02:46:26.523115] N [MSGID: 122031]
>>>> [ec-generic.c:1130:ec_combine_xattrop]
0-StoragePool-disperse-6:
>>>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>>>>
>>>> The message "N [MSGID: 122031]
>>>> [ec-generic.c:1130:ec_combine_xattrop]
>>>> 0-StoragePool-disperse-6: Mismatching dictionary in answers of
>>>> 'GF_FOP_XATTROP'" repeated 2 times between
[2017-01-10
>>>> 02:46:26.523115] and [2017-01-10 02:46:26.523143]
>>>>
>>>> [2017-01-10 02:46:26.523147] W [MSGID: 122040]
>>>> [ec-common.c:919:ec_prepare_update_cbk]
0-StoragePool-disperse-6:
>>>> Failed to get size and version [Input/output error]
>>>>
>>>> [2017-01-10 02:46:26.523302] N [MSGID: 122031]
>>>> [ec-generic.c:1130:ec_combine_xattrop]
0-StoragePool-disperse-2:
>>>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>>>>
>>>> The message "N [MSGID: 122031]
>>>> [ec-generic.c:1130:ec_combine_xattrop]
>>>> 0-StoragePool-disperse-2: Mismatching dictionary in answers of
>>>> 'GF_FOP_XATTROP'" repeated 2 times between
[2017-01-10
>>>> 02:46:26.523302] and [2017-01-10 02:46:26.523324]
>>>>
>>>> [2017-01-10 02:46:26.523328] W [MSGID: 122040]
>>>> [ec-common.c:919:ec_prepare_update_cbk]
0-StoragePool-disperse-2:
>>>> Failed to get size and version [Input/output error]
>>>>
>>>>
>>>>
>>>> [root at glusterfs3 Log_Files]# gluster --version
>>>>
>>>> glusterfs 3.7.18 built on Dec 8 2016 06:34:26
>>>>
>>>>
>>>>
>>>> [root at glusterfs3 Log_Files]# gluster volume info
>>>>
>>>>
>>>>
>>>> Volume Name: StoragePool
>>>>
>>>> Type: Distributed-Disperse
>>>>
>>>> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
>>>>
>>>> Status: Started
>>>>
>>>> Number of Bricks: 8 x (2 + 1) = 24
>>>>
>>>> Transport-type: tcp
>>>>
>>>> Bricks:
>>>>
>>>> Brick1: glusterfs1sds:/ws/disk1/ws_brick
>>>>
>>>> Brick2: glusterfs2sds:/ws/disk1/ws_brick
>>>>
>>>> Brick3: glusterfs3sds:/ws/disk1/ws_brick
>>>>
>>>> Brick4: glusterfs1sds:/ws/disk2/ws_brick
>>>>
>>>> Brick5: glusterfs2sds:/ws/disk2/ws_brick
>>>>
>>>> Brick6: glusterfs3sds:/ws/disk2/ws_brick
>>>>
>>>> Brick7: glusterfs1sds:/ws/disk3/ws_brick
>>>>
>>>> Brick8: glusterfs2sds:/ws/disk3/ws_brick
>>>>
>>>> Brick9: glusterfs3sds:/ws/disk3/ws_brick
>>>>
>>>> Brick10: glusterfs1sds:/ws/disk4/ws_brick
>>>>
>>>> Brick11: glusterfs2sds:/ws/disk4/ws_brick
>>>>
>>>> Brick12: glusterfs3sds:/ws/disk4/ws_brick
>>>>
>>>> Brick13: glusterfs1sds:/ws/disk5/ws_brick
>>>>
>>>> Brick14: glusterfs2sds:/ws/disk5/ws_brick
>>>>
>>>> Brick15: glusterfs3sds:/ws/disk5/ws_brick
>>>>
>>>> Brick16: glusterfs1sds:/ws/disk6/ws_brick
>>>>
>>>> Brick17: glusterfs2sds:/ws/disk6/ws_brick
>>>>
>>>> Brick18: glusterfs3sds:/ws/disk6/ws_brick
>>>>
>>>> Brick19: glusterfs1sds:/ws/disk7/ws_brick
>>>>
>>>> Brick20: glusterfs2sds:/ws/disk7/ws_brick
>>>>
>>>> Brick21: glusterfs3sds:/ws/disk7/ws_brick
>>>>
>>>> Brick22: glusterfs1sds:/ws/disk8/ws_brick
>>>>
>>>> Brick23: glusterfs2sds:/ws/disk8/ws_brick
>>>>
>>>> Brick24: glusterfs3sds:/ws/disk8/ws_brick
>>>>
>>>> Options Reconfigured:
>>>>
>>>> performance.readdir-ahead: on
>>>>
>>>> diagnostics.client-log-level: INFO
>>>>
>>>>
>>>>
>>>> Thanks and Regards,
>>>>
>>>> Ram
>>>>
>>>> ***************************Legal
>>>> Disclaimer***************************
>>>> "This communication may contain confidential and
privileged
>>>> material for the sole use of the intended recipient. Any
>>>> unauthorized review, use or distribution by others is strictly
>>>> prohibited. If you have received the message by mistake, please
>>>> advise the sender by reply email and delete the message. Thank
you."
>>>>
*******************************************************************
>>>> *
>>>> *
>>>> *
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>
>>> ***************************Legal
>>> Disclaimer***************************
>>> "This communication may contain confidential and privileged
material
>>> for the sole use of the intended recipient. Any unauthorized
review,
>>> use or distribution by others is strictly prohibited. If you have
>>> received the message by mistake, please advise the sender by reply
>> email and delete the message. Thank you."
>>>
********************************************************************
>>> *
>>> *
>>>
>>
>> ***************************Legal
>> Disclaimer***************************
>> "This communication may contain confidential and privileged
material
>> for the sole use of the intended recipient. Any unauthorized review,
>> use or distribution by others is strictly prohibited. If you have
>> received the message by mistake, please advise the sender by reply
>> email and delete the message. Thank you."
>> *********************************************************************
>> *
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply email
and delete the message. Thank you."
> **********************************************************************
>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for
the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20170110/8847a712/attachment.html>