VAN CAUSBROECK Wannes
2014-Sep-08 14:27 UTC
[Gluster-users] distributed files disappearing
Hello all, I'm running gluster 3.4.2-1 on rhel 6.5 (don't worry, we'll be moving to RHS soon ;) I just added a brick to an existing volume and suddenly my files can no longer be accessed by the client. This is what I did: [root at lpr-nas01 in]# gluster vol info prd_yearlyextract Volume Name: prd_yearlyextract Type: Distribute Volume ID: d3849bfa-7d1e-43bb-9dcf-af6ff0746c19 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: lpr-nas01:/brick-xiv2/prd_yearlyextract [root at lpr-nas01 in]# gluster vol brick-add prd_yearlyextract lpr-nas01:/brick-xiv1/prd_yearlyextract [root at lpr-nas01 in]# gluster vol rebalance prd_yearlyextract Now in the two bricks I have the following folders: [root at lpr-nas01 in]# ll /brick-xiv1/prd_yearlyextract/sigedis/in/ total 0 ---------T 2 444 ftpsiged 129337731 8. Sep 15:40 YCR_20130121_44.xml.DONE ---------T 2 444 ftpsiged 128549601 8. Sep 15:40 YCR_20130121_48.xml ---------T 2 444 ftpsiged 128166675 8. Sep 15:40 YCR_20130121_50.xml ---------T 2 444 ftpsiged 128227308 8. Sep 15:40 YCR_20130121_52.xml <TRUNCATED> ---------T 2 998 ftpsiged 10544189 8. Sep 15:40 YCR_20140128_98.xml.GZ [root at lpr-nas01 in]# ll /brick-xiv2/prd_yearlyextract/sigedis/in/ total 5648632 -rw-rw-r-- 2 444 ftpsiged 129337731 8. Mai 2013 YCR_20130121_44.xml.DONE -rw-rw-r-- 2 444 ftpsiged 125619849 8. Mai 2013 YCR_20130121_45.xml -rw-rw-r-- 2 444 ftpsiged 129315886 8. Mai 2013 YCR_20130121_46.xml -rw-rw-r-- 2 444 ftpsiged 125900420 8. Mai 2013 YCR_20130121_47.xml -rw-rw-r-- 2 444 ftpsiged 128549601 8. Mai 2013 YCR_20130121_48.xml -rw-rw-r-- 2 444 ftpsiged 126278398 8. Mai 2013 YCR_20130121_49.xml -rw-rw-r-- 2 444 ftpsiged 128166675 8. Mai 2013 YCR_20130121_50.xml <TRUNCATED> -rw-rw-r-- 2 998 ftpsiged 10425421 12. Feb 2014 YCR_20140128_99.xml.GZ On the client I can only see the files that were in the original brick (brick2). The files that were put there afterwards are gone. In the rebalance log I lines like this: +------------------------------------------------------------------------------+ [2014-09-08 12:14:41.426033] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-prd_yearlyextract-client-0: changing port to 49282 (from 0) [2014-09-08 12:14:41.426069] W [socket.c:514:__socket_rwv] 0-prd_yearlyextract-client-0: readv failed (No data available) [2014-09-08 12:14:41.428805] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-prd_yearlyextract-client-1: changing port to 49321 (from 0) [2014-09-08 12:14:41.428854] W [socket.c:514:__socket_rwv] 0-prd_yearlyextract-client-1: readv failed (No data available) [2014-09-08 12:14:41.431634] I [client-handshake.c:1659:select_server_supported_programs] 0-prd_yearlyextract-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-09-08 12:14:41.431698] I [client-handshake.c:1659:select_server_supported_programs] 0-prd_yearlyextract-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-09-08 12:14:41.431895] I [client-handshake.c:1456:client_setvolume_cbk] 0-prd_yearlyextract-client-0: Connected to 192.168.151.21:49282, attached to remote volume '/brick-xiv2/prd_yearlyextract'. [2014-09-08 12:14:41.431908] I [client-handshake.c:1468:client_setvolume_cbk] 0-prd_yearlyextract-client-0: Server and Client lk-version numbers are not same, reopening the fds [2014-09-08 12:14:41.431991] I [client-handshake.c:1456:client_setvolume_cbk] 0-prd_yearlyextract-client-1: Connected to 192.168.151.21:49321, attached to remote volume '/brick-xiv1/prd_yearlyextract'. [2014-09-08 12:14:41.432001] I [client-handshake.c:1468:client_setvolume_cbk] 0-prd_yearlyextract-client-1: Server and Client lk-version numbers are not same, reopening the fds [2014-09-08 12:14:41.432104] I [client-handshake.c:450:client_set_lk_version_cbk] 0-prd_yearlyextract-client-0: Server lk version = 1 [2014-09-08 12:14:41.432130] I [client-handshake.c:450:client_set_lk_version_cbk] 0-prd_yearlyextract-client-1: Server lk version = 1 [2014-09-08 12:14:41.437758] I [dht-common.c:2618:dht_setxattr] 0-prd_yearlyextract-dht: fixing the layout of / [2014-09-08 12:14:41.438260] I [dht-rebalance.c:1085:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: migrate data called on / [2014-09-08 12:14:41.445939] I [dht-rebalance.c:1296:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: Migration operation on dir / took 0.01 secs [2014-09-08 12:14:41.448304] I [dht-common.c:2618:dht_setxattr] 0-prd_yearlyextract-dht: fixing the layout of /sigedis [2014-09-08 12:14:41.448800] I [dht-rebalance.c:1085:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: migrate data called on /sigedis [2014-09-08 12:14:41.475828] I [dht-rebalance.c:1296:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: Migration operation on dir /sigedis took 0.03 secs [2014-09-08 12:14:41.478187] I [dht-common.c:2618:dht_setxattr] 0-prd_yearlyextract-dht: fixing the layout of /sigedis/in [2014-09-08 12:14:41.492970] I [dht-rebalance.c:1085:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: migrate data called on /sigedis/in [2014-09-08 12:14:41.924592] I [dht-rebalance.c:666:dht_migrate_file] 0-prd_yearlyextract-dht: /sigedis/in/YCR_20130121_44.xml.DONE: attempting to move from prd_yearlyextract-client-0 to prd_yearlyextract-client-1 [2014-09-08 12:14:41.925840] W [dht-rebalance.c:382:__dht_check_free_space] 0-prd_yearlyextract-dht: data movement attempted from node (prd_yearlyextract-client-0) with higher disk space to a node (prd_yearlyextract-client-1) with lesser disk space (/sigedis/in/YCR_20130121_44.xml.DONE) [2014-09-08 12:14:41.928050] I [dht-rebalance.c:666:dht_migrate_file] 0-prd_yearlyextract-dht: /sigedis/in/YCR_20140128_112.xml.GZ: attempting to move from prd_yearlyextract-client-0 to prd_yearlyextract-client-1 [2014-09-08 12:14:41.929562] W [dht-rebalance.c:382:__dht_check_free_space] 0-prd_yearlyextract-dht: data movement attempted from node (prd_yearlyextract-client-0) with higher disk space to a node (prd_yearlyextract-client-1) with lesser disk space (/sigedis/in/YCR_20140128_112.xml.GZ) The lines at the bottom concern the files that are invisible. What did I do wrong? Wannes -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140908/25f321cd/attachment.html>