We have a glusterfs with 8 bricks on 4 servers. The filesystem is at ~73% full, but today I got a 'device full error' from a failed batch job and sure enough, one brick had filled. I was able to probe the brick, find some duplicate files and delete enough to allow the system to continue, but obviously this will be a continuing problem. I suspect that this is due to some very large files placed early in the fs lifetime. I understand that gluster does not allow gluster-rebalancing in the same config (ie, without adding a brick), but would moving some offending files to another fs and then copying them back tend to rebalance the FS by distributing the copied files more evenly? Or is the only solution to add (or clear) a brick and then rebalance across it when added back in? hjm -- -- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130611/fbdd556c/attachment.html>
On 06/11/2013 05:55 PM, Harry Mangalam wrote:> I understand that gluster does not allow gluster-rebalancing in the same > config (ie, without adding a brick), but would moving some offending > files to another fs and then copying them back tend to rebalance the FS > by distributing the copied files more evenly?The problem is that no files will move unless the layout (the hash-range information used to place files) changes, and the layout won't change unless the set of bricks does, so a rebalance without adding/removing bricks would end up being a no-op. However, there are two other ways this kind of imbalance can be addressed. Firstly, we can observe that GlusterFS will still find files even if they're in the "wrong" place according to the layout. It is possible to move files manually from one brick to another, so long as appropriate care is taken e.g. to ensure that all extended attributes are preserved. The problem with this approach is that such careful placement might be undone the next time you do a rebalance for other reasons. The second approach avoids that problem, but is a bit trickier in other ways. There is a feature that allows a user to define their own layout for a directory, so that it will be left alone by rebalance. There's even a script in your friendly neighborhood source tree (extras/rebalance.py) that will use this to set a layout that's weighted according to the size of bricks (or free space) instead of just treating all as equal. You can use that as an example of how to construct and apply a valid user-defined layout, plus other techniques that are likely to be useful. Lastly, we have a feature that setting the fake "distribute.migrate-data" xattr on a file will cause use to re-evaluate where it should go and migrate it to the correct brick if necessary. Since you seem to be dealing with a relatively small number of files that need to be moved, it shouldn't be too hard to combine this little bag of tricks into a solution that meets your needs. Just let me know if you'd like me to assist.