Gluster users - Dec 2013 - glusterfs-3.4.2.qa4: problem accessing files when adding a replica brick

Hi All,

I'm testing the 3.4.2.qa4 to see if the "Structure Needs cleaning"
fix
is working.
During my deployement I got a new problem which wasn't there in 3.4.1

I have a KVM server running multiple vhosts with there disk stored as a 
file on a fuse-glusterfs mountpoint.
These machines are running without any problems when 1 brick is configured.

I add an new brick as replica 2 with following command:
# gluster volume add-brick testvolume replica 2 SRV-2:/gluster/brick2

 From that moment my virtual machine (BSC-1) looses there disk (ata 
device error).
When I stop and restart my virtual machines the problem is solved.

I've remount the mountpoint with DEBUG resulting in big logfile:
http://tempsend.com/53ABEE479C

This are some main errors:
[2013-12-19 11:49:21.440618] W [fuse-resolve.c:546:fuse_resolve_fd] 
0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF 
(old-subvolume:testvolume-0 new-subvolume:testvolume-1)
[2013-12-19 11:49:21.440731] W [fuse-bridge.c:2167:fuse_writev_cbk] 
0-glusterfs-fuse: 47363: WRITE => -1 (Invalid argument)
[2013-12-19 11:49:21.444920] W [fuse-resolve.c:546:fuse_resolve_fd] 
0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF 
(old-subvolume:testvolume-0 new-subvolume:testvolume-1)
[2013-12-19 11:49:21.445042] W [fuse-bridge.c:2167:fuse_writev_cbk] 
0-glusterfs-fuse: 47365: WRITE => -1 (Bad file descriptor)
[2013-12-19 11:49:21.449096] W [fuse-resolve.c:546:fuse_resolve_fd] 
0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF 
(old-subvolume:testvolume-0 new-subvolume:testvolume-1)



When reading and writing small files (every 0.5 sec) during that 
add-brick command I don't see any problems.
When have a file open with vim during the add-brick , it gives an error 
when closing the file:
E72: Close error on swap file

Is this normal behaviour or a bug (as this was not a problem in a 
previous version I suspect a bug)
Can this be fixed in 3.4.2 or is there some tweaking that can prevent 
this from happening ?

Thanks for your time!
Greetings,
Johan Huysmans

Hi All,

testing the beta release of 3.5.0 I encountered the same issue while 
adding a brick.
Removing a brick did trigger this problem.

gr.
Johan Huysmans

On 19-12-13 13:18, Johan Huysmans wrote:
> Hi All,
>
> I'm testing the 3.4.2.qa4 to see if the "Structure Needs
cleaning" fix
> is working.
> During my deployement I got a new problem which wasn't there in 3.4.1
>
> I have a KVM server running multiple vhosts with there disk stored as 
> a file on a fuse-glusterfs mountpoint.
> These machines are running without any problems when 1 brick is 
> configured.
>
> I add an new brick as replica 2 with following command:
> # gluster volume add-brick testvolume replica 2 SRV-2:/gluster/brick2
>
> From that moment my virtual machine (BSC-1) looses there disk (ata 
> device error).
> When I stop and restart my virtual machines the problem is solved.
>
> I've remount the mountpoint with DEBUG resulting in big logfile:
> http://tempsend.com/53ABEE479C
>
> This are some main errors:
> [2013-12-19 11:49:21.440618] W [fuse-resolve.c:546:fuse_resolve_fd] 
> 0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
> b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF 
> (old-subvolume:testvolume-0 new-subvolume:testvolume-1)
> [2013-12-19 11:49:21.440731] W [fuse-bridge.c:2167:fuse_writev_cbk] 
> 0-glusterfs-fuse: 47363: WRITE => -1 (Invalid argument)
> [2013-12-19 11:49:21.444920] W [fuse-resolve.c:546:fuse_resolve_fd] 
> 0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
> b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF 
> (old-subvolume:testvolume-0 new-subvolume:testvolume-1)
> [2013-12-19 11:49:21.445042] W [fuse-bridge.c:2167:fuse_writev_cbk] 
> 0-glusterfs-fuse: 47365: WRITE => -1 (Bad file descriptor)
> [2013-12-19 11:49:21.449096] W [fuse-resolve.c:546:fuse_resolve_fd] 
> 0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
> b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF 
> (old-subvolume:testvolume-0 new-subvolume:testvolume-1)
>
>
>
> When reading and writing small files (every 0.5 sec) during that 
> add-brick command I don't see any problems.
> When have a file open with vim during the add-brick , it gives an 
> error when closing the file:
> E72: Close error on swap file
>
> Is this normal behaviour or a bug (as this was not a problem in a 
> previous version I suspect a bug)
> Can this be fixed in 3.4.2 or is there some tweaking that can prevent 
> this from happening ?
>
> Thanks for your time!
> Greetings,
> Johan Huysmans
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

On 12/19/2013 05:48 PM, Johan Huysmans wrote:
> Hi All,
>
> I'm testing the 3.4.2.qa4 to see if the "Structure Needs
cleaning" fix
> is working.
> During my deployement I got a new problem which wasn't there in 3.4.1
>
> I have a KVM server running multiple vhosts with there disk stored as a
> file on a fuse-glusterfs mountpoint.
> These machines are running without any problems when 1 brick is configured.
>
> I add an new brick as replica 2 with following command:
> # gluster volume add-brick testvolume replica 2 SRV-2:/gluster/brick2
>
>  From that moment my virtual machine (BSC-1) looses there disk (ata
> device error).
> When I stop and restart my virtual machines the problem is solved.
>
> I've remount the mountpoint with DEBUG resulting in big logfile:
> http://tempsend.com/53ABEE479C
>
> This are some main errors:
> [2013-12-19 11:49:21.440618] W [fuse-resolve.c:546:fuse_resolve_fd]
> 0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
> b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF
> (old-subvolume:testvolume-0 new-subvolume:testvolume-1)
> [2013-12-19 11:49:21.440731] W [fuse-bridge.c:2167:fuse_writev_cbk]
> 0-glusterfs-fuse: 47363: WRITE => -1 (Invalid argument)
> [2013-12-19 11:49:21.444920] W [fuse-resolve.c:546:fuse_resolve_fd]
> 0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
> b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF
> (old-subvolume:testvolume-0 new-subvolume:testvolume-1)
> [2013-12-19 11:49:21.445042] W [fuse-bridge.c:2167:fuse_writev_cbk]
> 0-glusterfs-fuse: 47365: WRITE => -1 (Bad file descriptor)
> [2013-12-19 11:49:21.449096] W [fuse-resolve.c:546:fuse_resolve_fd]
> 0-fuse-resolve: migration of basefd (ptr:0x15ed8ec inode-gfid:e31eb690-0
> b47-4204-8e6c-7a10116458c8) did not complete, failing fop with EBADF
> (old-subvolume:testvolume-0 new-subvolume:testvolume-1)
>
>
>
Would it be possible to upload the log file again? I seem to be getting 
a 404 now when I access the tempsend link.

fd migration errors are not common as part of an add-brick command.

-Vijay