Gluster users - Apr 2013 - Re-balancing after expansion

Hi, I have a system using 3.2.6-1, running ext4. I recently expanded the
system from 4 to 5 servers. I have NOT yet done re-balancing of the
system - so at the moment most of the writing of new files goes to the
new server.



My impression is that it would be possible to re balance while the
system is in use, is this correct? Or are there any gotchas involved in
this? The servers consist of about 11 TB each, approximately how long
time would a re-balance take (we are using RDMA here, and assuming four
are full and one is empty)?




Thanks,




/jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130426/d64ce5e9/attachment.html>

Hi Jon,

Jon Tegner wrote on 20130426:
> Hi, I have a system using 3.2.6-1, running ext4. I recently expanded
> the system from 4 to 5 servers. I have NOT yet done re-balancing of
> the system - so at the moment most of the writing of new files goes to
> the new server.
> 
> My impression is that it would be possible to re balance while the
> system is in use, is this correct?
Yes.
> Or are there any gotchas involved in this?
High IOPS load on all storage nodes.
> The servers consist of about 11 TB each, approximately how long time
> would a re-balance take (we are using RDMA here, and assuming four are
> full and one is empty)?
I think it greatly depends on the amount of directories and files in
use. For instance; I did a rebalance of a 15 TiB volume with about 1 M
directories and 20 M files on 3.2.5 on 2 storage nodes each consisting
of 8*2TiB drives, and that took 7 days for the fix layout rebalance step
and 40 days for the data rebalance. The data rebalance step even failed,
but data was spread out quite well.

regards,
   Hans Lambermont

ps.  I'm now on 3.3.1 and looking at upgrading to release-3.3 to get the
recent patch for the file descriptor leak
https://bugzilla.redhat.com/show_bug.cgi?id=928631 .  I'm not sure if
3.2.6 also has this fd leak bug.
A test setup however showed that release-3.3 is not usable, see
https://bugzilla.redhat.com/show_bug.cgi?id=956245 . I will test again
to see if things have improved.
-- 
Hans Lambermont | Senior Architect
(t) +31407370104 (w) www.shapeways.com