Alex
2008-Nov-04 13:51 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
[root at rs1 ~]# lfs df -h UUID bytes Used Available Use% Mounted on testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0% /mnt/lustre[MDT:0] testfs-OST0000_UUID 18.3G 17.4G 2.0M 94% /mnt/lustre[OST:0] testfs-OST0001_UUID 18.3G 15.5G 2.0G 84% /mnt/lustre[OST:1] testfs-OST0002_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:2] testfs-OST0003_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:3] filesystem summary: 110.0G 63.8G 40.7G 57% /mnt/lustre [root at rs1 ~]# As you can see OST:0 is full! Question1: Is any command to use in order to move arround 60% of OST:0 data from OST:0 to other OST or to all other OSTes (arround 20% to each other)? Question2: How can i see larger objects (bigger then 100MB) stored on OST:0 and correspondig files on /mnt/lustre? I want to erase them! lfs getstripe did not produce useful info! Regards, Alx
Brian J. Murrell
2008-Nov-04 14:37 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Tue, 2008-11-04 at 15:51 +0200, Alex wrote:> [root at rs1 ~]# lfs df -h > UUID bytes Used Available Use% Mounted on > testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0% /mnt/lustre[MDT:0] > testfs-OST0000_UUID 18.3G 17.4G 2.0M 94% /mnt/lustre[OST:0] > testfs-OST0001_UUID 18.3G 15.5G 2.0G 84% /mnt/lustre[OST:1] > testfs-OST0002_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:2] > testfs-OST0003_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:3] > filesystem summary: 110.0G 63.8G 40.7G 57% /mnt/lustre > [root at rs1 ~]# > > As you can see OST:0 is full!Indeed. Is this a 1.6 or a 1.4 installation? As long as you have not forced an allocation policy with striping for example, with 1.6 your OSTs should fill in a more-or-less even manner and you should not see this type of problem.> Question1: Is any command to use in order to move arround 60% of OST:0 data > from OST:0 to other OST or to all other OSTes (arround 20% to each other)?Deactivate allocation to the full OST on the MDS and do some "mv" (or cp/rm if you are more comfortable with that) operations which will result in moving the objects for the file to other OSTs.> Question2: How can i see larger objects (bigger then 100MB) stored on OST:0man lfs The find option to lfs has all sorts of options for finding files on a Lustre filesystem including some Lustre specific options. You probably want: # lfs find /mnt/lustre -size +100M --obd testfs-OST0000_UUID b.
Alex
2008-Nov-04 15:47 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Tuesday 04 November 2008 16:37, Brian J. Murrell wrote:> On Tue, 2008-11-04 at 15:51 +0200, Alex wrote: > > [root at rs1 ~]# lfs df -h > > UUID bytes Used Available Use% Mounted on > > testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0% > > /mnt/lustre[MDT:0] testfs-OST0000_UUID 18.3G 17.4G 2.0M > > 94% /mnt/lustre[OST:0] testfs-OST0001_UUID 18.3G 15.5G 2.0G > > 84% /mnt/lustre[OST:1] testfs-OST0002_UUID 36.7G 15.5G > > 19.4G 42% /mnt/lustre[OST:2] testfs-OST0003_UUID 36.7G 15.5G > > 19.4G 42% /mnt/lustre[OST:3] filesystem summary: 110.0G 63.8G > > 40.7G 57% /mnt/lustre [root at rs1 ~]# > > > > As you can see OST:0 is full! > > Indeed. Is this a 1.6 or a 1.4 installation?Hi Brian, Many thanks for your reply. See my answers inline. [root at rs1 ~]# rpm -q lustre lustre-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp [root at rs1 ~]#> As long as you have not > forced an allocation policy with striping for example, with 1.6 your > OSTs should fill in a more-or-less even manner and you should not see > this type of problem.I have activated striping on /mnt/lustre after some OSTs has been filled arbitray, in order to be able to use entire /mnt/lustre size (110GB), so thats the cause of my present problem.> > Question1: Is any command to use in order to move arround 60% of OST:0 > > data from OST:0 to other OST or to all other OSTes (arround 20% to each > > other)?Now i have: [root at rhclm ~]# cat /proc/fs/lustre/lov/testfs-mdtlov/qos_prio_free 90% [root at rhclm ~]#> Deactivate allocation to the full OST on the MDSHow can i do this? lctl conf_param testfs-MDT0000.lov.qos_prio_free=100 will do the trick?> and do some "mv" (or > cp/rm if you are more comfortable with that) operations which will > result in moving the objects for the file to other OSTs. >All right. Let me explain better... I found that /mnt/lustre/test3.img is 12Gb in size and is striped between all OSTs (arround 3GB/OST). Indeed, if i remove /mnt/lustre/test3.img i''ll get some free space on OST:0, as well as on other OSTs. Now, i want to make up some free space ONLY on OST:0, MOVING some objects from it to another OST. [root at rs1 ~]# lfs getstripe /mnt/lustre/test3.img OBDS: 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE 2: testfs-OST0002_UUID ACTIVE 3: testfs-OST0003_UUID ACTIVE /mnt/lustre/test3.img obdidx objid objid group 0 7 0x7 0 2 6 0x6 0 3 70 0x46 0 1 69 0x45 0 [root at rs1 ~]# So, how can i move objid=7 from OST:0 to OST:2, keeping intact /mnt/lustre/test3.img file? Regards, Alx
Wojciech Turek
2008-Nov-04 16:23 UTC
Re: redistribute used space to other OSTs (free space)
Alex wrote:
On Tuesday 04 November 2008 16:37, Brian J. Murrell wrote:
On Tue, 2008-11-04 at 15:51 +0200, Alex wrote:
[root@rs1 ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0%
/mnt/lustre[MDT:0] testfs-OST0000_UUID 18.3G 17.4G 2.0M
94% /mnt/lustre[OST:0] testfs-OST0001_UUID 18.3G 15.5G 2.0G
84% /mnt/lustre[OST:1] testfs-OST0002_UUID 36.7G 15.5G
19.4G 42% /mnt/lustre[OST:2] testfs-OST0003_UUID 36.7G 15.5G
19.4G 42% /mnt/lustre[OST:3] filesystem summary: 110.0G 63.8G
40.7G 57% /mnt/lustre [root@rs1 ~]#
As you can see OST:0 is full!
Indeed. Is this a 1.6 or a 1.4 installation?
Hi Brian,
Many thanks for your reply. See my answers inline.
[root@rs1 ~]# rpm -q lustre
lustre-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp
[root@rs1 ~]#
As long as you have not
forced an allocation policy with striping for example, with 1.6 your
OSTs should fill in a more-or-less even manner and you should not see
this type of problem.
I have activated striping on /mnt/lustre after some OSTs has been filled
arbitray, in order to be able to use entire /mnt/lustre size (110GB), so
thats the cause of my present problem.
Question1: Is any command to use in order to move arround 60% of OST:0
data from OST:0 to other OST or to all other OSTes (arround 20% to each
other)?
Now i have:
[root@rhclm ~]# cat /proc/fs/lustre/lov/testfs-mdtlov/qos_prio_free
90%
[root@rhclm ~]#
Deactivate allocation to the full OST on the MDS
How can i do this? lctl conf_param testfs-MDT0000.lov.qos_prio_free=100 will
do the trick?
You need to disable an OST on the MDS server but not on the clients.
This will prevent the clients from creating new object on the full OST
but you still will be able to access it for reading.
lctl --device N deactivate
where N is the device number which you can obtain by running lctl dl
command
and do some "mv" (or
cp/rm if you are more comfortable with that) operations which will
result in moving the objects for the file to other OSTs.
All right. Let me explain better... I found that /mnt/lustre/test3.img is 12Gb
in size and is striped between all OSTs (arround 3GB/OST). Indeed, if i
remove /mnt/lustre/test3.img i''ll get some free space on OST:0, as well
as on
other OSTs. Now, i want to make up some free space ONLY on OST:0, MOVING some
objects from it to another OST.
[root@rs1 ~]# lfs getstripe /mnt/lustre/test3.img
OBDS:
0: testfs-OST0000_UUID ACTIVE
1: testfs-OST0001_UUID ACTIVE
2: testfs-OST0002_UUID ACTIVE
3: testfs-OST0003_UUID ACTIVE
/mnt/lustre/test3.img
obdidx objid objid group
0 7 0x7 0
2 6 0x6 0
3 70 0x46 0
1 69 0x45 0
[root@rs1 ~]#
So, how can i move objid=7 from OST:0 to OST:2, keeping
intact /mnt/lustre/test3.img file?
I don''t know how to move a particular object but you could move a whole
file to another OST and that would release some space from the full OST.
mkdir /mnt/lustre/tmp
lfs setstripe -i 2 -c 1 /mnt/lustre/tmp
and then copy or move file test3.img into tmp dir
Regards,
Alx
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
Brian J. Murrell
2008-Nov-04 16:49 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Tue, 2008-11-04 at 17:47 +0200, Alex wrote:> Now i have: > [root at rhclm ~]# cat /proc/fs/lustre/lov/testfs-mdtlov/qos_prio_free > 90%That is the default value, which should result in a more-or-less equal usage (in terms of percentages) of all OSTs.> How can i do this? lctl conf_param testfs-MDT0000.lov.qos_prio_free=100 will > do the trick?No. Look at the deactivate option of lctl in the manpage.> All right. Let me explain better... I found that /mnt/lustre/test3.img is 12Gb > in size and is striped between all OSTs (arround 3GB/OST). Indeed, if i > remove /mnt/lustre/test3.img i''ll get some free space on OST:0, as well as on > other OSTs.Right.> Now, i want to make up some free space ONLY on OST:0, MOVING some > objects from it to another OST.No, you don''t manipulate objects directly, you let the MDS take care of allocation.> [root at rs1 ~]# lfs getstripe /mnt/lustre/test3.img > OBDS: > 0: testfs-OST0000_UUID ACTIVE > 1: testfs-OST0001_UUID ACTIVE > 2: testfs-OST0002_UUID ACTIVE > 3: testfs-OST0003_UUID ACTIVE > /mnt/lustre/test3.img > obdidx objid objid group > 0 7 0x7 0 > 2 6 0x6 0 > 3 70 0x46 0 > 1 69 0x45 0 > > [root at rs1 ~]# > > So, how can i move objid=7 from OST:0 to OST:2, keeping > intact /mnt/lustre/test3.img file?You can''t work with that much precision. You first deactivate the OST on the MDS, then when you cp the file, the destination file will be restriped, but will not get striped to the deactivated OST. You must note that the copied file does not inherit the striping policy of the original automatically. If the files you want to move are just using the filesystem default striping then this is not a problem. However if the striping for a directory is different than the filesystem, you will want to create your target directory first and set the stripe policy and then cp/rm the files. If you have striping policies on individual files then you have to address them on a case by case basis. I said earlier that you could use mv. I''m not positive that will always have the desired results so likely you should stick with cp/rm. b.
Brian J. Murrell
2008-Nov-04 16:52 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Tue, 2008-11-04 at 16:23 +0000, Wojciech Turek wrote:> I don''t know how to move a particular object but you could move a > whole file to another OST and that would release some space from the > full OST. > mkdir /mnt/lustre/tmp > lfs setstripe -i 2 -c 1 /mnt/lustre/tmp > and then copy or move file test3.img into tmp dirHe could if he wanted to be sure to control exactly which OST to move to, but typically the above is more work than is needed. It should be sufficient to disable the OST on the MDS and let the round-robin allocator take care of moving the objects to less full OSTs during the cp/rm. b.
Alex
2008-Nov-05 15:18 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Tuesday 04 November 2008 18:52, Brian J. Murrell wrote:> On Tue, 2008-11-04 at 16:23 +0000, Wojciech Turek wrote: > > I don''t know how to move a particular object but you could move a > > whole file to another OST and that would release some space from the > > full OST. > > mkdir /mnt/lustre/tmp > > lfs setstripe -i 2 -c 1 /mnt/lustre/tmpsupposing that i have enough space on all OSTs, but i want to create file1 on 2 OSTs, the next command will be enough: lfs setstripe -c 2 /mnt/lustre/file1 but... i can predict which OSTs will be used! Is possible to specify somehow OSTs index, for more then one OST? someting like: lfs setstripe -i 2 -i 4 -c 2 /mnt/lustre/file1 in order to create file1 on OST:2 and OST4?> > and then copy or move file test3.img into tmp dirusing cp/rm command after i deactivated my full OST:0 caused that the new file to be restriped (that''s good and what i want). but this technique (cp followed by rm) it has a major disavantage: you need more free space on /mnt/lustre. what is happen if we are trying to move a test3.img and we have /mnt/lustre 99% full -> we need free space equivalent with minimum another test3.img size -> cp && rm command can not be used -> we go to MOVE command but here comes my problem again. See below: ON CLIENT: [root at rs1 ~]# lfs df -h UUID bytes Used Available Use% Mounted on testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0% /mnt/lustre[MDT:0] testfs-OST0000_UUID 18.3G 17.4G 2.0M 94% /mnt/lustre[OST:0] testfs-OST0001_UUID 18.3G 15.5G 2.0G 84% /mnt/lustre[OST:1] testfs-OST0002_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:2] testfs-OST0003_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:3] filesystem summary: 110.0G 63.8G 40.7G 57% /mnt/lustre [root at rs1 ~]# ON MGS: [root at rhclm ~]# lctl --device testfs-OST0000-osc deactivate [root at rhclm ~]# lctl --device testfs-OST0001-osc deactivate [root at rhclm ~]# lctl dl|grep -i in 5 IN osc testfs-OST0000-osc testfs-mdtlov_UUID 5 6 IN osc testfs-OST0001-osc testfs-mdtlov_UUID 5 [root at rhclm ~]# ON CLIENT AGAIN: [root at rs1 ~]# mkdir /mnt/lustre/tmp [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/ OBDS: 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE 2: testfs-OST0002_UUID ACTIVE 3: testfs-OST0003_UUID ACTIVE /mnt/lustre/tmp/ stripe_count: -1 stripe_size: 0 stripe_offset: -1 [root at rs1 ~]# [root at rs1 ~]# mv /mnt/lustre/test3.img /mnt/lustre/tmp/ [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/test3.img OBDS: 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE 2: testfs-OST0002_UUID ACTIVE 3: testfs-OST0003_UUID ACTIVE /mnt/lustre/tmp/test3.img obdidx objid objid group 0 7 0x7 0 2 6 0x6 0 3 70 0x46 0 1 69 0x45 0 [root at rs1 ~]# So, using move command, test3.img is not restriped when moved in /mnt/lustre/tmp directory, even OST:0 and OST:1 are deactivated! In this case why round-robin allocator is not working? Is any way to use mv command and have test3.img file restriped on the fly when moved to new tmp directory? If yes, how can i do that? I would like to avoid cp usage (which i tested and is working)? Regards, Alx
Wojciech Turek
2008-Nov-05 15:27 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
Alex wrote:> On Tuesday 04 November 2008 18:52, Brian J. Murrell wrote: > >> On Tue, 2008-11-04 at 16:23 +0000, Wojciech Turek wrote: >> >>> I don''t know how to move a particular object but you could move a >>> whole file to another OST and that would release some space from the >>> full OST. >>> mkdir /mnt/lustre/tmp >>> lfs setstripe -i 2 -c 1 /mnt/lustre/tmp >>> > > supposing that i have enough space on all OSTs, but i want to create file1 on > 2 OSTs, the next command will be enough: lfs setstripe -c 2 /mnt/lustre/file1 > but... i can predict which OSTs will be used! > > Is possible to specify somehow OSTs index, for more then one OST? someting > like: lfs setstripe -i 2 -i 4 -c 2 /mnt/lustre/file1 in order to create file1 > on OST:2 and OST4? >AFAIK no it isn''t possible. Currently when using lfs setstripe command with --index switch you can only specify index of the first OST of the striping pattern.> >>> and then copy or move file test3.img into tmp dir >>> > > using cp/rm command after i deactivated my full OST:0 caused that the new file > to be restriped (that''s good and what i want). > > but this technique (cp followed by rm) it has a major disavantage: you need > more free space on /mnt/lustre. what is happen if we are trying to move a > test3.img and we have /mnt/lustre 99% full -> we need free space equivalent > with minimum another test3.img size -> cp && rm command can not be used -> we > go to MOVE command but here comes my problem again. See below: > > ON CLIENT: > [root at rs1 ~]# lfs df -h > UUID bytes Used Available Use% Mounted on > testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0% /mnt/lustre[MDT:0] > testfs-OST0000_UUID 18.3G 17.4G 2.0M 94% /mnt/lustre[OST:0] > testfs-OST0001_UUID 18.3G 15.5G 2.0G 84% /mnt/lustre[OST:1] > testfs-OST0002_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:2] > testfs-OST0003_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:3] > filesystem summary: 110.0G 63.8G 40.7G 57% /mnt/lustre > [root at rs1 ~]# > > ON MGS: > [root at rhclm ~]# lctl --device testfs-OST0000-osc deactivate > [root at rhclm ~]# lctl --device testfs-OST0001-osc deactivate > > [root at rhclm ~]# lctl dl|grep -i in > 5 IN osc testfs-OST0000-osc testfs-mdtlov_UUID 5 > 6 IN osc testfs-OST0001-osc testfs-mdtlov_UUID 5 > [root at rhclm ~]# > > ON CLIENT AGAIN: > [root at rs1 ~]# mkdir /mnt/lustre/tmp > [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/ > OBDS: > 0: testfs-OST0000_UUID ACTIVE > 1: testfs-OST0001_UUID ACTIVE > 2: testfs-OST0002_UUID ACTIVE > 3: testfs-OST0003_UUID ACTIVE > /mnt/lustre/tmp/ > stripe_count: -1 stripe_size: 0 stripe_offset: -1 > [root at rs1 ~]# > [root at rs1 ~]# mv /mnt/lustre/test3.img /mnt/lustre/tmp/ > [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/test3.img > OBDS: > 0: testfs-OST0000_UUID ACTIVE > 1: testfs-OST0001_UUID ACTIVE > 2: testfs-OST0002_UUID ACTIVE > 3: testfs-OST0003_UUID ACTIVE > /mnt/lustre/tmp/test3.img > obdidx objid objid group > 0 7 0x7 0 > 2 6 0x6 0 > 3 70 0x46 0 > 1 69 0x45 0 > [root at rs1 ~]# > > So, using move command, test3.img is not restriped when moved > in /mnt/lustre/tmp directory, even OST:0 and OST:1 are deactivated! In this > case why round-robin allocator is not working? Is any way to use mv command > and have test3.img file restriped on the fly when moved to new tmp directory? > If yes, how can i do that? I would like to avoid cp usage (which i tested and > is working)? > > Regards, > Alx > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- Wojciech Turek Assistant System Manager High Performance Computing Service University of Cambridge Email: wjt27 at cam.ac.uk Tel: (+)44 1223 763517
Brian J. Murrell
2008-Nov-05 15:49 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Wed, 2008-11-05 at 17:18 +0200, Alex wrote:> > supposing that i have enough space on all OSTs, but i want to create file1 on > 2 OSTs, the next command will be enough: lfs setstripe -c 2 /mnt/lustre/file1 > but... i can predict which OSTs will be used!No, you can only predict the first OST.> Is possible to specify somehow OSTs index, for more then one OST?No.> using cp/rm command after i deactivated my full OST:0 caused that the new file > to be restriped (that''s good and what i want).Good.> but this technique (cp followed by rm) it has a major disavantage: you need > more free space on /mnt/lustre.Of course.> what is happen if we are trying to move a > test3.img and we have /mnt/lustre 99% fullWell, if all of your OSTs are full then you don''t have a "rebalancing" problem but a free space problem. That is solved by adding more OSTs. Once you add space you can proceed with your rebalance if you wish, or simply let lustre fill up your new OST(s) in due course. You of course won''t benefit from the bandwidth of having as many OSTs as you do since the existing ones are full.> with minimum another test3.img size -> cp && rm command can not be usedIf it cannot be used then all of your OSTs are full and you need to add more space.> ON CLIENT: > [root at rs1 ~]# lfs df -h > UUID bytes Used Available Use% Mounted on > testfs-MDT0000_UUID 130.4G 460.1M 122.5G 0% /mnt/lustre[MDT:0] > testfs-OST0000_UUID 18.3G 17.4G 2.0M 94% /mnt/lustre[OST:0] > testfs-OST0001_UUID 18.3G 15.5G 2.0G 84% /mnt/lustre[OST:1] > testfs-OST0002_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:2] > testfs-OST0003_UUID 36.7G 15.5G 19.4G 42% /mnt/lustre[OST:3] > filesystem summary: 110.0G 63.8G 40.7G 57% /mnt/lustre > [root at rs1 ~]# > > ON MGS: > [root at rhclm ~]# lctl --device testfs-OST0000-osc deactivate > [root at rhclm ~]# lctl --device testfs-OST0001-osc deactivate > > [root at rhclm ~]# lctl dl|grep -i in > 5 IN osc testfs-OST0000-osc testfs-mdtlov_UUID 5 > 6 IN osc testfs-OST0001-osc testfs-mdtlov_UUID 5 > [root at rhclm ~]# > > ON CLIENT AGAIN: > [root at rs1 ~]# mkdir /mnt/lustre/tmp > [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/ > OBDS: > 0: testfs-OST0000_UUID ACTIVE > 1: testfs-OST0001_UUID ACTIVE > 2: testfs-OST0002_UUID ACTIVE > 3: testfs-OST0003_UUID ACTIVE > /mnt/lustre/tmp/ > stripe_count: -1 stripe_size: 0 stripe_offset: -1 > [root at rs1 ~]# > [root at rs1 ~]# mv /mnt/lustre/test3.img /mnt/lustre/tmp/ > [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/test3.img > OBDS: > 0: testfs-OST0000_UUID ACTIVE > 1: testfs-OST0001_UUID ACTIVE > 2: testfs-OST0002_UUID ACTIVE > 3: testfs-OST0003_UUID ACTIVE > /mnt/lustre/tmp/test3.img > obdidx objid objid group > 0 7 0x7 0 > 2 6 0x6 0 > 3 70 0x46 0 > 1 69 0x45 0 > [root at rs1 ~]# > > So, using move command, test3.img is not restriped when moved > in /mnt/lustre/tmp directory, even OST:0 and OST:1 are deactivated! In this > case why round-robin allocator is not working? Is any way to use mv command > and have test3.img file restriped on the fly when moved to new tmp directory? > If yes, how can i do that? I would like to avoid cp usage (which i tested and > is working)?As I said before, I suspect that mv won''t work. You must use cp/rm. I don''t understand why in your above scenario you can''t use cp/rm. There is plenty of space for it to work. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20081105/430a6773/attachment.bin
Andreas Dilger
2008-Nov-05 19:19 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Nov 04, 2008 11:49 -0500, Brian J. Murrell wrote:> > [root at rs1 ~]# lfs getstripe /mnt/lustre/test3.img > > OBDS: > > 0: testfs-OST0000_UUID ACTIVE > > 1: testfs-OST0001_UUID ACTIVE > > 2: testfs-OST0002_UUID ACTIVE > > 3: testfs-OST0003_UUID ACTIVE > > /mnt/lustre/test3.img > > obdidx objid objid group > > 0 7 0x7 0 > > 2 6 0x6 0 > > 3 70 0x46 0 > > 1 69 0x45 0 > > > > [root at rs1 ~]# > > > > So, how can i move objid=7 from OST:0 to OST:2, keeping > > intact /mnt/lustre/test3.img file? > > You can''t work with that much precision. You first deactivate the OST > on the MDS, then when you cp the file, the destination file will be > restriped, but will not get striped to the deactivated OST.As an FYI, being able to migrate data between OSTs arbitrarily is a project we are working on, but it won''t be ready for some time.> You must note that the copied file does not inherit the striping policy of > the original automatically. If the files you want to move are just using > the filesystem default striping then this is not a problem. However if > the striping for a directory is different than the filesystem, you will > want to create your target directory first and set the stripe policy and > then cp/rm the files.If the target file is created in the same directory as the source, then it will get the same directory default striping, unless a specific striping was specified. It is easier to cp/rm the files one at a time in the same directory tree than to move the whole tree.> I said earlier that you could use mv. I''m not positive that will always > have the desired results so likely you should stick with cp/rm.Using "mv" will only rename the inode on the MDS, it doesn''t change any of the data. That is true for local filesystems as well as Lustre. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Brian J. Murrell
2008-Nov-05 19:24 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Wed, 2008-11-05 at 12:19 -0700, Andreas Dilger wrote:> On Nov 04, 2008 11:49 -0500, Brian J. Murrell wrote: > > > I said earlier that you could use mv. I''m not positive that will always > > have the desired results so likely you should stick with cp/rm. > > Using "mv" will only rename the inode on the MDS, it doesn''t change any > of the data. That is true for local filesystems as well as Lustre.Indeed. This is exactly the problem I was thinking of when I corrected myself and is very important to note with regard to the OP''s followup about using mv vs. cp/rm. Thanx for clarifying, Andreas. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20081105/a95787ca/attachment.bin
Andreas Dilger
2008-Nov-05 19:31 UTC
[Lustre-discuss] redistribute used space to other OSTs (free space)
On Nov 05, 2008 17:18 +0200, Alex wrote:> supposing that i have enough space on all OSTs, but i want to create file1 on > 2 OSTs, the next command will be enough: lfs setstripe -c 2 /mnt/lustre/file1 > but... i can predict which OSTs will be used!I question why you need to know which OST will be used? The files should not be allocated on the full OST, which is a problem for the MDS to handle, but otherwise it shouldn''t really matter where the files are stored.> Is possible to specify somehow OSTs index, for more then one OST? someting > like: lfs setstripe -i 2 -i 4 -c 2 /mnt/lustre/file1 in order to create file1 > on OST:2 and OST4?In the 1.8.0 release it will be possible to specify more arbitrary groupings of OSTs using the "OST Pools" feature. Until then you can only specify the starting OST index.> but this technique (cp followed by rm) it has a major disavantage: you need > more free space on /mnt/lustre. what is happen if we are trying to move a > test3.img and we have /mnt/lustre 99% full -> we need free space equivalent > with minimum another test3.img size -> cp && rm command can not be used -> we > go to MOVE command but here comes my problem again.I agree with Brian - if the target OSTs do not have enough free space to hold the file data, how can you move the file there?> [root at rs1 ~]# lfs getstripe /mnt/lustre/tmp/ > /mnt/lustre/tmp/ > stripe_count: -1 stripe_size: 0 stripe_offset: -1Note that you are requesting that all files be striped over all OSTs, which will always result in the full OST being used. If you specify the files be striped over only 2 or 3 OSTs then the full OST can be skipped.> I found that /mnt/lustre/test3.img is 12Gb in size and is striped between > all OSTs (arround 3GB/OST)I would also suggest that instead of having such small OSTs (18GB and 36GB) it is much better to have fewer large OSTs. Even a single disk today is hundreds of GB or more, so it seems you are only using a small partition for each OST, and you will run into this sort of "free space fragmentation" problem. This is especially true if the size of a single file is a very large fraction of the total OST size. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Alex
2008-Nov-06 09:40 UTC
[Lustre-discuss] redistribute used space to other OSTs ( free space )
On Wednesday 05 November 2008 21:19, Andreas Dilger wrote:> On Nov 04, 2008 11:49 -0500, Brian J. Murrell wrote: > > > [root at rs1 ~]# lfs getstripe /mnt/lustre/test3.img > > > OBDS: > > > 0: testfs-OST0000_UUID ACTIVE > > > 1: testfs-OST0001_UUID ACTIVE > > > 2: testfs-OST0002_UUID ACTIVE > > > 3: testfs-OST0003_UUID ACTIVE > > > /mnt/lustre/test3.img > > > obdidx objid objid group > > > 0 7 0x7 0 > > > 2 6 0x6 0 > > > 3 70 0x46 0 > > > 1 69 0x45 0 > > > > > > [root at rs1 ~]# > > > > > > So, how can i move objid=7 from OST:0 to OST:2, keeping > > > intact /mnt/lustre/test3.img file? > > > > You can''t work with that much precision. You first deactivate the OST > > on the MDS, then when you cp the file, the destination file will be > > restriped, but will not get striped to the deactivated OST. > > As an FYI, being able to migrate data between OSTs arbitrarily is a > project we are working on, but it won''t be ready for some time. > > > You must note that the copied file does not inherit the striping policy > > of the original automatically. If the files you want to move are just > > using the filesystem default striping then this is not a problem. > > However if the striping for a directory is different than the filesystem, > > you will want to create your target directory first and set the stripe > > policy and then cp/rm the files. > > If the target file is created in the same directory as the source, then > it will get the same directory default striping, unless a specific > striping was specified. It is easier to cp/rm the files one at a time > in the same directory tree than to move the whole tree. > > > I said earlier that you could use mv. I''m not positive that will always > > have the desired results so likely you should stick with cp/rm. > > Using "mv" will only rename the inode on the MDS, it doesn''t change any > of the data. That is true for local filesystems as well as Lustre.That''s brilliant explanation, and conclude some things for clear: - if you want to influence objects redistribution on OSTs when "moving" a file, is a must to use CP&&RM commands. MV command will never influence that and is not helpful at all, because it will affect ONLY MDS (as you specified above, will only rename the inode on the MDS) - cp always require to have more free space on your filesystem then mv - if you don''t have enough free space to complete these operations and you don''t want to erase some files -> add another OST. Thanks again for answers to all has been part of this thread. Regards, Alx