Gluster users - Dec 2016 - File operation failure on simple distributed volume

Hi,

When my application moves a file from it's local disk to FUSE-mounted
GlusterFS volume, the client outputs many warnings and errors not
always but occasionally. The volume is a simple distributed volume.

A sample of logs pasted: http://pastebin.com/axkTCRJX

It seems to come from something like a network disconnection
("Transport endpoint is not connected") at a glance, but other
networking applications on the same machine don't observe such a
thing. So I guess there may be a problem somewhere in GlusterFS stack.

It ended in failing to rename a file, logging PHP Warning like below:

    PHP Warning:  rename(/glusterfs01/db1/stack/f0/13a9a2f0): failed
to open stream: Input/output error in [snipped].php on line 278
    PHP Warning:
rename(/var/stack/13a9a2f0,/glusterfs01/db1/stack/f0/13a9a2f0):
Input/output error in [snipped].php on line 278

Conditions:

- GlusterFS 3.8.5 installed via yum CentOS-Gluster-3.8.repo
- Volume info and status pasted: http://pastebin.com/JPt2KeD8
- Client machines' OS: Scientific Linux 6 or CentOS 6.
- Server machines' OS: CentOS 6.
- Kernel version is 2.6.32-642.6.2.el6.x86_64 on all machines.
- The number of connected FUSE clients is 260.
- No firewall between connected machines.
- Neither remounting volumes nor rebooting client machines take effect.
- It is caused by not only rename() but also copy() and filesize() operation.
- No outputs in brick logs when it happens.

Any ideas? I'd appreciate any help.

Regards.

Hi Yonex,

Is this consistently reproducible ? if so, Can you enable debug log [1]
and check for any message similar to [2]. Basically you can even search
for "EOF on socket".

You can set your log level back to default (INFO) after capturing for
some time.


[1] : gluster volume set <volname> diagnostics.brick-log-level DEBUG and
gluster volume set <volname> diagnostics.client-log-level DEBUG

[2] : http://pastebin.com/xn8QHXWa


Regards

Rafi KC

On 12/12/2016 09:35 PM, yonex wrote:
> Hi,
>
> When my application moves a file from it's local disk to FUSE-mounted
> GlusterFS volume, the client outputs many warnings and errors not
> always but occasionally. The volume is a simple distributed volume.
>
> A sample of logs pasted: http://pastebin.com/axkTCRJX
>
> It seems to come from something like a network disconnection
> ("Transport endpoint is not connected") at a glance, but other
> networking applications on the same machine don't observe such a
> thing. So I guess there may be a problem somewhere in GlusterFS stack.
>
> It ended in failing to rename a file, logging PHP Warning like below:
>
>     PHP Warning:  rename(/glusterfs01/db1/stack/f0/13a9a2f0): failed
> to open stream: Input/output error in [snipped].php on line 278
>     PHP Warning:
> rename(/var/stack/13a9a2f0,/glusterfs01/db1/stack/f0/13a9a2f0):
> Input/output error in [snipped].php on line 278
>
> Conditions:
>
> - GlusterFS 3.8.5 installed via yum CentOS-Gluster-3.8.repo
> - Volume info and status pasted: http://pastebin.com/JPt2KeD8
> - Client machines' OS: Scientific Linux 6 or CentOS 6.
> - Server machines' OS: CentOS 6.
> - Kernel version is 2.6.32-642.6.2.el6.x86_64 on all machines.
> - The number of connected FUSE clients is 260.
> - No firewall between connected machines.
> - Neither remounting volumes nor rebooting client machines take effect.
> - It is caused by not only rename() but also copy() and filesize()
operation.
> - No outputs in brick logs when it happens.
>
> Any ideas? I'd appreciate any help.
>
> Regards.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users