Sam Wouters
2013-Jul-04 09:20 UTC
[Gluster-users] Geo-replication causing input/output errors on nfs clients
Hi, we have a setup with a 2 node gluster replication cluster where the volume is exported through the gluster build-in nfs service. This volume is also replicated to a secondary site with geo-replication. The problem is, that when clients are writing files larger than 10M, once every time the write gets aborted with a input/output error. At that moment, following lines appear in the gluster/bricks log: [2013-07-01 12:41:39.800465] E [posix.c:2128:posix_writev] 0-<vol>-posix: write failed: offset 1069056, Bad file descriptor [2013-07-01 12:41:39.800510] I [server3_1-fops.c:1414:server_writev_cbk] 0-<vol>-server: 114834: WRITEV -2 (8e01b426-11dd-4147-9e5c-9256ca118be7) ==> -1 (Bad file descriptor) When geo-replication is stopped, these errors do not occur, so I suspect this is some kind of locking issue between the integrated nfs server en the crawler for the geo-rep. Anybody any ideas how to circumvent this? Version information of the gluster servers: CentOS release 6.4 (Final) glusterfs-3.3.1-1.el6.x86_64 glusterfs-server-3.3.1-1.el6.x86_64 glusterfs-fuse-3.3.1-1.el6.x86_64 glusterfs-geo-replication-3.3.1-1.el6.x86_64 regards, Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130704/8e1e3e3b/attachment.html>
Vijay Bellur
2013-Jul-04 16:11 UTC
[Gluster-users] Geo-replication causing input/output errors on nfs clients
On 07/04/2013 02:50 PM, Sam Wouters wrote:> Hi, > > we have a setup with a 2 node gluster replication cluster where the > volume is exported through the gluster build-in nfs service. > This volume is also replicated to a secondary site with geo-replication. > > The problem is, that when clients are writing files larger than 10M, > once every time the write gets aborted with a input/output error. At > that moment, following lines appear in the gluster/bricks log: > > [2013-07-01 12:41:39.800465] E [posix.c:2128:posix_writev] > 0-<vol>-posix: write failed: offset 1069056, Bad file descriptor > [2013-07-01 12:41:39.800510] I [server3_1-fops.c:1414:server_writev_cbk] > 0-<vol>-server: 114834: WRITEV -2 (8e01b426-11dd-4147-9e5c-9256ca118be7) > ==> -1 (Bad file descriptor) > > When geo-replication is stopped, these errors do not occur, so I suspect > this is some kind of locking issue between the integrated nfs server en > the crawler for the geo-rep. > > Anybody any ideas how to circumvent this? >This patch is needed to circumvent the issue: http://review.gluster.org/#/c/5283/ -Vijay