james.bellinger at icecube.wisc.edu
2013-Sep-16 21:56 UTC
[Gluster-users] remove-brick question
I inherited a system with a wide mix of array sizes (no replication) in 3.2.2, and wanted to drain data from a failing array. I upgraded to 3.3.2, and began a gluster volume remove-brick scratch "gfs-node01:/sda" start After some time I got this: gluster volume remove-brick scratch "gfs-node01:/sda" status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 0 0Bytes 0 0 not started gfs-node06 0 0Bytes 0 0 not started gfs-node03 0 0Bytes 0 0 not started gfs-node05 0 0Bytes 0 0 not started gfs-node01 2257394624 2.8TB 5161640 208878 completed Two things jump instantly to mind: 1) The number of failures is rather large 2) A _different_ disk seems to have been _partially_ drained. /dev/sda 2.8T 2.7T 12G 100% /sda /dev/sdb 2.8T 769G 2.0T 28% /sdb /dev/sdc 2.8T 2.1T 698G 75% /sdc /dev/sdd 2.8T 2.2T 589G 79% /sdd When I mount the system it is read-only (another problem I want to fix ASAP) so I'm pretty sure the failures aren't due to users changing the system underneath me. Thanks for any pointers. James Bellinger
On 09/17/2013 03:26 AM, james.bellinger at icecube.wisc.edu wrote:> I inherited a system with a wide mix of array sizes (no replication) in > 3.2.2, and wanted to drain data from a failing array. > > I upgraded to 3.3.2, and began a > gluster volume remove-brick scratch "gfs-node01:/sda" start > > After some time I got this: > gluster volume remove-brick scratch "gfs-node01:/sda" status > Node Rebalanced-files size scanned failures > status > --------- ----------- ----------- ----------- ----------- > ------------ > localhost 0 0Bytes 0 0 > not started > gfs-node06 0 0Bytes 0 0 > not started > gfs-node03 0 0Bytes 0 0 > not started > gfs-node05 0 0Bytes 0 0 > not started > gfs-node01 2257394624 2.8TB 5161640 208878 > completed > > Two things jump instantly to mind: > 1) The number of failures is rather largeCan you see the rebalance logs (/var/log/scratch-rebalance.log) to figure out what the error messages are?> 2) A _different_ disk seems to have been _partially_ drained. > /dev/sda 2.8T 2.7T 12G 100% /sda > /dev/sdb 2.8T 769G 2.0T 28% /sdb > /dev/sdc 2.8T 2.1T 698G 75% /sdc > /dev/sdd 2.8T 2.2T 589G 79% /sdd > >I know this sounds silly, but just to be sure, is /dev/sda actually mounted on "gfs-node01:sda"? If yes,the files that _were_ successfully rebalanced should have been moved from gfs-node01:sda to one of the other bricks. Is that the case?> When I mount the system it is read-only (another problem I want to fixAgain, the mount logs could shed some information .. (btw a successful rebalance start/status sequence should be followed by the rebalance 'commit' command to ensure the volume information gets updated)> ASAP) so I'm pretty sure the failures aren't due to users changing the > system underneath me. > > Thanks for any pointers. > > James Bellinger > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users
Hi James, Could you please check rebalance logs from the node where failures are happening. path: /var/log/glusterfs/<volname>-rebalance.log Also please check the dht layout xattrs from all the bricks to make sure that layout is not zeroed out for the wrong brick. https://bugzilla.redhat.com/show_bug.cgi?id=963896 Thanks, Shylesh james.bellinger at icecube.wisc.edu wrote:> I inherited a system with a wide mix of array sizes (no replication) in > 3.2.2, and wanted to drain data from a failing array. > > I upgraded to 3.3.2, and began a > gluster volume remove-brick scratch "gfs-node01:/sda" start > > After some time I got this: > gluster volume remove-brick scratch "gfs-node01:/sda" status > Node Rebalanced-files size scanned failures > status > --------- ----------- ----------- ----------- ----------- > ------------ > localhost 0 0Bytes 0 0 > not started > gfs-node06 0 0Bytes 0 0 > not started > gfs-node03 0 0Bytes 0 0 > not started > gfs-node05 0 0Bytes 0 0 > not started > gfs-node01 2257394624 2.8TB 5161640 208878 > completed > > Two things jump instantly to mind: > 1) The number of failures is rather large > 2) A _different_ disk seems to have been _partially_ drained. > /dev/sda 2.8T 2.7T 12G 100% /sda > /dev/sdb 2.8T 769G 2.0T 28% /sdb > /dev/sdc 2.8T 2.1T 698G 75% /sdc > /dev/sdd 2.8T 2.2T 589G 79% /sdd > > > > When I mount the system it is read-only (another problem I want to fix > ASAP) so I'm pretty sure the failures aren't due to users changing the > system underneath me. > > Thanks for any pointers. > > James Bellinger > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users