Dear All, We are running a lustre-1.6.6 on Linux vanilla kernel 2.6.22.19. Recently we suffer a serious un-balanced OSTs problem. If I run the command: # /opt/lustre-1.8/bin/lfs df UUID 1K-blocks Used Available Use% Mounted on cwarp-MDT0000_UUID 119627860 659184 112132220 0% /mnt/src[MDT:0] cwarp-OST0000_UUID 1441859112 1343925184 24691744 93% /mnt/src[OST:0] cwarp-OST0001_UUID 1441859128 799739136 568869616 55% /mnt/src[OST:1] cwarp-OST0002_UUID 1441859128 643666316 724950624 44% /mnt/src[OST:2] cwarp-OST0003_UUID 1441859112 745288308 623015556 51% /mnt/src[OST:3] cwarp-OST0004_UUID 1441859128 654020352 714567920 45% /mnt/src[OST:4] cwarp-OST0005_UUID 1441859128 658416232 709949996 45% /mnt/src[OST:5] filesystem summary: 8651154736 4845055528 3366045456 56% /mnt/src It is clear that the cwarp-OST0000_UUID is almost full, but the other OSTs are still quite empty. No matter what, we are trying to do something in order to re-balance the OSTs. Our procedure is (which is indicated in the Lustre manual): 1. In MDS, disable the cwarp-OST0000 so that newly created file will go to other OSTs: echo 0 > /proc/fs/lustre/osc/cwarp-OST0000-osc/active 2. In one of the client node, we copy and rename files, in hope that some files can be pull out of the cwarp-OST0000 and then go to other OSTs: cp /path/to/some/files /path/to/some/files.tmp mv /path/to/some/files.tmp /path/to/some/files However, this way seems does not help too much. I guess this is because we do not pull out the files which locate in cwarp-OST0000 exactly. Actually we are not sure whether this is the correct way to solve this problem. Any suggestion to re-balance or to prevent un-balanced OSTs are very appriciated. Best Regards, T.H.Hsieh
On Fri, 2009-12-04 at 15:55 +0800, thhsieh wrote:> 2. In one of the client node, we copy and rename files, in hope that > some files can be pull out of the cwarp-OST0000 and then go to other > OSTs: > > cp /path/to/some/files /path/to/some/files.tmp > mv /path/to/some/files.tmp /path/to/some/files > > However, this way seems does not help too much. I guess this is because > we do not pull out the files which locate in cwarp-OST0000 exactly.Yes. You need to get the list of files you run through step 2 above from an lfs find that targets files on the specific OSTs you want to reduce the usage of. Personally, I would target singly-striped files before I started messing around with moving striped files around. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091204/9758679e/attachment.bin
On 2009-12-04, at 00:55, thhsieh wrote:> We are running a lustre-1.6.6 on Linux vanilla kernel 2.6.22.19. > Recently we suffer a serious un-balanced OSTs problem. If I run > the command: > > # /opt/lustre-1.8/bin/lfs df > UUID 1K-blocks Used Available Use% Mounted on > cwarp-MDT0000_UUID 119627860 659184 112132220 0% /mnt/ > src[MDT:0] > cwarp-OST0000_UUID 1441859112 1343925184 24691744 93% /mnt/ > src[OST:0] > cwarp-OST0001_UUID 1441859128 799739136 568869616 55% /mnt/ > src[OST:1] > cwarp-OST0002_UUID 1441859128 643666316 724950624 44% /mnt/ > src[OST:2] > cwarp-OST0003_UUID 1441859112 745288308 623015556 51% /mnt/ > src[OST:3] > cwarp-OST0004_UUID 1441859128 654020352 714567920 45% /mnt/ > src[OST:4] > cwarp-OST0005_UUID 1441859128 658416232 709949996 45% /mnt/ > src[OST:5] > > filesystem summary: 8651154736 4845055528 3366045456 56% /mnt/src > > It is clear that the cwarp-OST0000_UUID is almost full, but the > other OSTs are still quite empty.Normally this happens when a single large file like a tarball. You can use "lfs find" to find the few largest files on the full OST, and either stripe them over all the OSTs or just distribute them individually over the OSTs.> No matter what, we are trying > to do something in order to re-balance the OSTs. Our procedure is > (which is indicated in the Lustre manual): > > 1. In MDS, disable the cwarp-OST0000 so that newly created file > will go to other OSTs: > > echo 0 > /proc/fs/lustre/osc/cwarp-OST0000-osc/activeDepending on how much "turnover" you have of files (deletion of old files, creating new files) this will slowly empty OST0000 and fill the other OSTs.> 2. In one of the client node, we copy and rename files, in hope that > some files can be pull out of the cwarp-OST0000 and then go to other > OSTs: > > cp /path/to/some/files /path/to/some/files.tmp > mv /path/to/some/files.tmp /path/to/some/files > > However, this way seems does not help too much. I guess this is > because > we do not pull out the files which locate in cwarp-OST0000 exactly.You can find files that have stripes that OST with: lfs find -obd cwarp-OST0000_UUID /path/to/lustre You can further limit this to large files with lfs find -obd cwarp-OST0000_UUID -size +1G /path/to/lustre Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.