Hello I?m newbie on lustre ? Currently, i make some test on virtual machines. I?ve: - Server mds1 for mdt mgs - Server ost1 for data, with 2 drives 1Go (for test, it''s some virtual machines) - Server client to mount mds On server mds1 /dev/sda3 93G 461M 87G 1% /mds On server ost1 /dev/sdc 1008M 64M 967M 2% /home/ost2 /dev/sdb 1008M 957M 68K 100% /home/ost1 ( Or On server ost1 /dev/sdb 1008M 957M 68K 100% /home/ost1 On server ost2 /dev/sdb 1008M 64M 967M 2% /home/ost2 } On client 192.168.45.20 at tcp0:/large-fs 2,0G 1,3G 629M 68% /home At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want.... For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) The problem, is, when sdb is full, he stop the copy, and don''t continue on ost2 Maybe I must create a lvm, but I don''t found howto to (maybe it''s possible to create lvm without format when I add others ost later..) Questions: - it''s the good way? - How I can write a 1.8Go (one file) on the 2 os, one part on ost1, one and final part on ost2 Regards
Phillipe, One way to circumvent this would be to create one file on each OST, loop mount them on the client and use pvcreate on each to size them at ~950 MB. Once you''ve done that create a VG (striped, stripe size should be exactly 1M (lustre''s preferred I/O size), and you should be good to go. It''s been awhile since I''ve played around with doing this, but it''s the basic idea. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Eudes PHILIPPE Sent: Wednesday, December 15, 2010 12:06 PM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] howto make a lvm, or virtual lvm? Hello I?m newbie on lustre ? Currently, i make some test on virtual machines. I?ve: - Server mds1 for mdt mgs - Server ost1 for data, with 2 drives 1Go (for test, it''s some virtual machines) - Server client to mount mds On server mds1 /dev/sda3 93G 461M 87G 1% /mds On server ost1 /dev/sdc 1008M 64M 967M 2% /home/ost2 /dev/sdb 1008M 957M 68K 100% /home/ost1 ( Or On server ost1 /dev/sdb 1008M 957M 68K 100% /home/ost1 On server ost2 /dev/sdb 1008M 64M 967M 2% /home/ost2 } On client 192.168.45.20 at tcp0:/large-fs 2,0G 1,3G 629M 68% /home At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want.... For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) The problem, is, when sdb is full, he stop the copy, and don''t continue on ost2 Maybe I must create a lvm, but I don''t found howto to (maybe it''s possible to create lvm without format when I add others ost later..) Questions: - it''s the good way? - How I can write a 1.8Go (one file) on the 2 os, one part on ost1, one and final part on ost2 Regards _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Mhhh... it''s strange, but it''s done! Except... if each ost are on different servers... the creation of lvm you mean, is on one ost server. And on the client server which mount the mds partition, there is one or several ost on several servers. I want to merge ALL ost on a "similar" lvm, or, a raid 5(?) over network Are you sure it is the better way in a production environment ? -----Message d''origine----- De : Ben Evans [mailto:Ben.Evans at terascala.com] Envoy? : mercredi 15 d?cembre 2010 19:55 ? : Eudes PHILIPPE; lustre-discuss at lists.lustre.org Objet : RE: [Lustre-discuss] howto make a lvm, or virtual lvm? Phillipe, One way to circumvent this would be to create one file on each OST, loop mount them on the client and use pvcreate on each to size them at ~950 MB. Once you''ve done that create a VG (striped, stripe size should be exactly 1M (lustre''s preferred I/O size), and you should be good to go. It''s been awhile since I''ve played around with doing this, but it''s the basic idea. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Eudes PHILIPPE Sent: Wednesday, December 15, 2010 12:06 PM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] howto make a lvm, or virtual lvm? Hello I?m newbie on lustre ? Currently, i make some test on virtual machines. I?ve: - Server mds1 for mdt mgs - Server ost1 for data, with 2 drives 1Go (for test, it''s some virtual machines) - Server client to mount mds On server mds1 /dev/sda3 93G 461M 87G 1% /mds On server ost1 /dev/sdc 1008M 64M 967M 2% /home/ost2 /dev/sdb 1008M 957M 68K 100% /home/ost1 ( Or On server ost1 /dev/sdb 1008M 957M 68K 100% /home/ost1 On server ost2 /dev/sdb 1008M 64M 967M 2% /home/ost2 } On client 192.168.45.20 at tcp0:/large-fs 2,0G 1,3G 629M 68% /home At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want.... For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) The problem, is, when sdb is full, he stop the copy, and don''t continue on ost2 Maybe I must create a lvm, but I don''t found howto to (maybe it''s possible to create lvm without format when I add others ost later..) Questions: - it''s the good way? - How I can write a 1.8Go (one file) on the 2 os, one part on ost1, one and final part on ost2 Regards _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
The problem is if i have: Server MDS1 for mdt and mgt Server OSS1 for ost1 Server OSS2 for ost2 Server OSS3 for ost3 . . . -----Message d''origine----- De : Ben Evans [mailto:Ben.Evans at terascala.com] Envoy? : mercredi 15 d?cembre 2010 20:33 ? : Eudes PHILIPPE Objet : RE: [Lustre-discuss] howto make a lvm, or virtual lvm? First to clear up some terminology: You have Server MDS1 for mdt and mgt Server OSS1 for ost1 and ost2 Client1 which mounts the filesystem from all the MDS and OSS servers. On Client1, you create the two files, each mapped using the "lfs setstripe" command to a specific OST. When you run "lfs getstripe $filename" you should see a list of all the OSTs that it is mapped to. Loop mount the files on client1.>From LVM, use pvcreate to create virtual disks of a fixed size.Use VG create on client1 to create your single, large volume with a stripe size of 1MB. At this point, you should have a RAID-0 group over those two files on the back-end, that you could now create a filesystem on and export through NFS (for example), or whatever you want to do. LVM may or may not have a RAID-5 setting (I don''t recall), but other utilities in Linux have it, and you should be able to leverage them to create something extensible. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Eudes PHILIPPE Sent: Wednesday, December 15, 2010 2:14 PM To: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] howto make a lvm, or virtual lvm? Mhhh... it''s strange, but it''s done! Except... if each ost are on different servers... the creation of lvm you mean, is on one ost server. And on the client server which mount the mds partition, there is one or several ost on several servers. I want to merge ALL ost on a "similar" lvm, or, a raid 5(?) over network Are you sure it is the better way in a production environment ? -----Message d''origine----- De : Ben Evans [mailto:Ben.Evans at terascala.com] Envoy? : mercredi 15 d?cembre 2010 19:55 ? : Eudes PHILIPPE; lustre-discuss at lists.lustre.org Objet : RE: [Lustre-discuss] howto make a lvm, or virtual lvm? Phillipe, One way to circumvent this would be to create one file on each OST, loop mount them on the client and use pvcreate on each to size them at ~950 MB. Once you''ve done that create a VG (striped, stripe size should be exactly 1M (lustre''s preferred I/O size), and you should be good to go. It''s been awhile since I''ve played around with doing this, but it''s the basic idea. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Eudes PHILIPPE Sent: Wednesday, December 15, 2010 12:06 PM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] howto make a lvm, or virtual lvm? Hello I?m newbie on lustre ? Currently, i make some test on virtual machines. I?ve: - Server mds1 for mdt mgs - Server ost1 for data, with 2 drives 1Go (for test, it''s some virtual machines) - Server client to mount mds On server mds1 /dev/sda3 93G 461M 87G 1% /mds On server ost1 /dev/sdc 1008M 64M 967M 2% /home/ost2 /dev/sdb 1008M 957M 68K 100% /home/ost1 ( Or On server ost1 /dev/sdb 1008M 957M 68K 100% /home/ost1 On server ost2 /dev/sdb 1008M 64M 967M 2% /home/ost2 } On client 192.168.45.20 at tcp0:/large-fs 2,0G 1,3G 629M 68% /home At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want.... For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) The problem, is, when sdb is full, he stop the copy, and don''t continue on ost2 Maybe I must create a lvm, but I don''t found howto to (maybe it''s possible to create lvm without format when I add others ost later..) Questions: - it''s the good way? - How I can write a 1.8Go (one file) on the 2 os, one part on ost1, one and final part on ost2 Regards _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 2010-12-15, at 10:06, Eudes PHILIPPE wrote:> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want....Lustre itself can not do RAID over the network, if that is what you are looking for...> For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) > The problem, is, when sdb is full, he stop the copy, and don''t continue on ost2If you create your file to be striped over both OSTs, then it should work. Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
Ok, so, i''ll try this : - One mds - 2 physical oss with one drive (1 GB) (one ost on one oss) On client, mount mds on /home.. lfs setstripe -c2 /home I upload (in sftp) one file, 300 MB - On Oss 1, he use 150 Mb of 1000 - On Oss 2, he use 150 Mb of 1000 All right! I continue... copy my first file 4 times (so there is 5 * 300 MB = 1500 MB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 ************************* Now, I add a new oss server, with one ost (1GB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 - On Oss 3, he use 0 Mb of 1000 lfs setstripe -c3 /home on client I upload a big file, 1.3 Go He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop (Couldn''t write to remote file "/home/big_log.log": Failure) ****************************** So now, - On Oss 1, he use 1000 Mb of 1000 - On Oss 2, he use 1000 Mb of 1000 - On Oss 3, he use 250 Mb of 1000 I upload again, just for see, my first file (300 MB), he copy the file only on Oss3 (oss 1 and 2 are full of course), it''s ok :) Is there a solution for this problem? Regards -----Message d''origine----- De?: Andreas Dilger [mailto:andreas.dilger at oracle.com] Envoy??: mercredi 15 d?cembre 2010 22:39 ??: Eudes PHILIPPE Cc?: lustre-discuss at lists.lustre.org Objet?: Re: [Lustre-discuss] howto make a lvm, or virtual lvm? On 2010-12-15, at 10:06, Eudes PHILIPPE wrote:> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1physical raid 5 on each ostX and a big lvm I can extend as I want.... Lustre itself can not do RAID over the network, if that is what you are looking for...> For my first test, I upload on client a file (1.8 Go) (each ost have 1 > Go) The problem, is, when sdb is full, he stop the copy, and don''t > continue on ost2If you create your file to be striped over both OSTs, then it should work. Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
On 2010-12-16, at 7:49, Eudes PHILIPPE <eudes at cisneo.fr> wrote:> Now, I add a new oss server, with one ost (1GB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > - On Oss 3, he use 0 Mb of 1000 > > lfs setstripe -c3 /home on client > > I upload a big file, 1.3 Go > He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop > (Couldn''t write to remote file "/home/big_log.log": Failure) > ****************************** > > So now, > - On Oss 1, he use 1000 Mb of 1000 > - On Oss 2, he use 1000 Mb of 1000 > - On Oss 3, he use 250 Mb of 1000 > I upload again, just for see, my first file (300 MB), he copy the file only > on Oss3 (oss 1 and 2 are full of course), it''s ok :) > > Is there a solution for this problem?This is much less of a problem in real usage of Lustre, when the size of the file is not a large fraction of thectotal OST size. Lustre will balance object allocation dynamically based on how full an OST is, but cannot currently do this in the middle of a single file. With typical OST sizes in the 8-16TB range today, this is rarely a problem. Cheers, Andreas
? 2010-12-16???10:49? Eudes PHILIPPE ???> Ok, so, i''ll try this : > - One mds > - 2 physical oss with one drive (1 GB) (one ost on one oss) > > On client, mount mds on /home.. > lfs setstripe -c2 /home > > I upload (in sftp) one file, 300 MB > - On Oss 1, he use 150 Mb of 1000 > - On Oss 2, he use 150 Mb of 1000 > > All right! > > I continue... copy my first file 4 times (so there is 5 * 300 MB = 1500 MB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > > ************************* > Now, I add a new oss server, with one ost (1GB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > - On Oss 3, he use 0 Mb of 1000 > > lfs setstripe -c3 /home on client > > I upload a big file, 1.3 Go > He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop > (Couldn''t write to remote file "/home/big_log.log": Failure) > ******************************All files in a directory inherits its parent dir''s stripe attributes. As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. As OST 1 and 2 are full, surely you''ll get write failure with ENOSPC.> > So now, > - On Oss 1, he use 1000 Mb of 1000 > - On Oss 2, he use 1000 Mb of 1000 > - On Oss 3, he use 250 Mb of 1000 > I upload again, just for see, my first file (300 MB), he copy the file only > on Oss3 (oss 1 and 2 are full of course), it''s ok :) > > Is there a solution for this problem?If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2).> > Regards > > > > -----Message d''origine----- > De : Andreas Dilger [mailto:andreas.dilger at oracle.com] > Envoy? : mercredi 15 d?cembre 2010 22:39 > ? : Eudes PHILIPPE > Cc : lustre-discuss at lists.lustre.org > Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? > > On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 > physical raid 5 on each ostX and a big lvm I can extend as I want.... > > Lustre itself can not do RAID over the network, if that is what you are > looking for... > >> For my first test, I upload on client a file (1.8 Go) (each ost have 1 >> Go) The problem, is, when sdb is full, he stop the copy, and don''t >> continue on ost2 > > If you create your file to be striped over both OSTs, then it should work. > > Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
I was wrong about what I say If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok (sftp said nothing), but the file is different (the md5sum si different!) It''s very dangerous!! Is there a solution, if I see all ossX are almost full, when I add (some) new oss, distribute data on these new oss to always have the same poucentage use on all oss? -----Message d''origine----- De : Wang Yibin [mailto:wang.yibin at oracle.com] Envoy? : jeudi 16 d?cembre 2010 16:09 ? : Eudes PHILIPPE Cc : lustre-discuss Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? ? 2010-12-16???10:49? Eudes PHILIPPE ???> Ok, so, i''ll try this : > - One mds > - 2 physical oss with one drive (1 GB) (one ost on one oss) > > On client, mount mds on /home.. > lfs setstripe -c2 /home > > I upload (in sftp) one file, 300 MB > - On Oss 1, he use 150 Mb of 1000 > - On Oss 2, he use 150 Mb of 1000 > > All right! > > I continue... copy my first file 4 times (so there is 5 * 300 MB = > 1500 MB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > > ************************* > Now, I add a new oss server, with one ost (1GB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > - On Oss 3, he use 0 Mb of 1000 > > lfs setstripe -c3 /home on client > > I upload a big file, 1.3 Go > He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop > (Couldn''t write to remote file "/home/big_log.log": Failure) > ******************************All files in a directory inherits its parent dir''s stripe attributes. As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. As OST 1 and 2 are full, surely you''ll get write failure with ENOSPC.> > So now, > - On Oss 1, he use 1000 Mb of 1000 > - On Oss 2, he use 1000 Mb of 1000 > - On Oss 3, he use 250 Mb of 1000 > I upload again, just for see, my first file (300 MB), he copy the file > only on Oss3 (oss 1 and 2 are full of course), it''s ok :) > > Is there a solution for this problem?If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2).> > Regards > > > > -----Message d''origine----- > De : Andreas Dilger [mailto:andreas.dilger at oracle.com] > Envoy? : mercredi 15 d?cembre 2010 22:39 ? : Eudes PHILIPPE Cc : > lustre-discuss at lists.lustre.org Objet : Re: [Lustre-discuss] howto > make a lvm, or virtual lvm? > > On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 > physical raid 5 on each ostX and a big lvm I can extend as I want.... > > Lustre itself can not do RAID over the network, if that is what you > are looking for... > >> For my first test, I upload on client a file (1.8 Go) (each ost have >> 1 >> Go) The problem, is, when sdb is full, he stop the copy, and don''t >> continue on ost2 > > If you create your file to be striped over both OSTs, then it should work. > > Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Lustre has its own load-balancing algorithms (either round-robin or weighted) depending on the space usage of the OSTs. ? 2010-12-16???11:25? Eudes PHILIPPE ???> I was wrong about what I say > If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok (sftp said nothing), but the file is different (the md5sum si different!) > It''s very dangerous!! > > Is there a solution, if I see all ossX are almost full, when I add (some) new oss, distribute data on these new oss to always have the same poucentage use on all oss? > > > > -----Message d''origine----- > De : Wang Yibin [mailto:wang.yibin at oracle.com] > Envoy? : jeudi 16 d?cembre 2010 16:09 > ? : Eudes PHILIPPE > Cc : lustre-discuss > Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? > > > ? 2010-12-16???10:49? Eudes PHILIPPE ??? > >> Ok, so, i''ll try this : >> - One mds >> - 2 physical oss with one drive (1 GB) (one ost on one oss) >> >> On client, mount mds on /home.. >> lfs setstripe -c2 /home >> >> I upload (in sftp) one file, 300 MB >> - On Oss 1, he use 150 Mb of 1000 >> - On Oss 2, he use 150 Mb of 1000 >> >> All right! >> >> I continue... copy my first file 4 times (so there is 5 * 300 MB = >> 1500 MB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> >> ************************* >> Now, I add a new oss server, with one ost (1GB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> - On Oss 3, he use 0 Mb of 1000 >> >> lfs setstripe -c3 /home on client >> >> I upload a big file, 1.3 Go >> He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop >> (Couldn''t write to remote file "/home/big_log.log": Failure) >> ****************************** > > All files in a directory inherits its parent dir''s stripe attributes. > As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. > As OST 1 and 2 are full, surely you''ll get write failure with ENOSPC. > >> >> So now, >> - On Oss 1, he use 1000 Mb of 1000 >> - On Oss 2, he use 1000 Mb of 1000 >> - On Oss 3, he use 250 Mb of 1000 >> I upload again, just for see, my first file (300 MB), he copy the file >> only on Oss3 (oss 1 and 2 are full of course), it''s ok :) >> >> Is there a solution for this problem? > > If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. > In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). > >> >> Regards >> >> >> >> -----Message d''origine----- >> De : Andreas Dilger [mailto:andreas.dilger at oracle.com] >> Envoy? : mercredi 15 d?cembre 2010 22:39 ? : Eudes PHILIPPE Cc : >> lustre-discuss at lists.lustre.org Objet : Re: [Lustre-discuss] howto >> make a lvm, or virtual lvm? >> >> On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >>> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 >> physical raid 5 on each ostX and a big lvm I can extend as I want.... >> >> Lustre itself can not do RAID over the network, if that is what you >> are looking for... >> >>> For my first test, I upload on client a file (1.8 Go) (each ost have >>> 1 >>> Go) The problem, is, when sdb is full, he stop the copy, and don''t >>> continue on ost2 >> >> If you create your file to be striped over both OSTs, then it should work. >> >> Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 2010-12-16, at 08:25, Eudes PHILIPPE wrote:> If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok (sftp said nothing), but the file is different (the md5sum is different!) > It''s very dangerous!!What version of Lustre are you using? This should definitely not happen (md5sum is different, without any errors). Is the size of the new file in Lustre matching the size of the original file? Does it work OK if you are using "cp" instead of sftp?> Is there a solution, if I see all ossX are almost full, when I add (some) new oss, distribute data on these new oss to always have the same poucentage use on all oss?This already works today, so long as the file sizes are not so large that they fill most of an OST.> -----Message d''origine----- > De : Wang Yibin [mailto:wang.yibin at oracle.com] > Envoy? : jeudi 16 d?cembre 2010 16:09 > ? : Eudes PHILIPPE > Cc : lustre-discuss > Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? > > > ? 2010-12-16???10:49? Eudes PHILIPPE ??? > >> Ok, so, i''ll try this : >> - One mds >> - 2 physical oss with one drive (1 GB) (one ost on one oss) >> >> On client, mount mds on /home.. >> lfs setstripe -c2 /home >> >> I upload (in sftp) one file, 300 MB >> - On Oss 1, he use 150 Mb of 1000 >> - On Oss 2, he use 150 Mb of 1000 >> >> All right! >> >> I continue... copy my first file 4 times (so there is 5 * 300 MB = >> 1500 MB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> >> ************************* >> Now, I add a new oss server, with one ost (1GB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> - On Oss 3, he use 0 Mb of 1000 >> >> lfs setstripe -c3 /home on client >> >> I upload a big file, 1.3 Go >> He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop >> (Couldn''t write to remote file "/home/big_log.log": Failure) >> ****************************** > > All files in a directory inherits its parent dir''s stripe attributes. > As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. > As OST 1 and 2 are full, surely you''ll get write failure with ENOSPC. > >> >> So now, >> - On Oss 1, he use 1000 Mb of 1000 >> - On Oss 2, he use 1000 Mb of 1000 >> - On Oss 3, he use 250 Mb of 1000 >> I upload again, just for see, my first file (300 MB), he copy the file >> only on Oss3 (oss 1 and 2 are full of course), it''s ok :) >> >> Is there a solution for this problem? > > If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. > In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). > >> >> Regards >> >> >> >> -----Message d''origine----- >> De : Andreas Dilger [mailto:andreas.dilger at oracle.com] >> Envoy? : mercredi 15 d?cembre 2010 22:39 ? : Eudes PHILIPPE Cc : >> lustre-discuss at lists.lustre.org Objet : Re: [Lustre-discuss] howto >> make a lvm, or virtual lvm? >> >> On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >>> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 >> physical raid 5 on each ostX and a big lvm I can extend as I want.... >> >> Lustre itself can not do RAID over the network, if that is what you >> are looking for... >> >>> For my first test, I upload on client a file (1.8 Go) (each ost have >>> 1 >>> Go) The problem, is, when sdb is full, he stop the copy, and don''t >>> continue on ost2 >> >> If you create your file to be striped over both OSTs, then it should work. >> >> Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
Okkkkk..... I use package on http://www.pdsi-scidac.org/repository/debian/ and it''s not 1.8 but 1.6 (see on luster-source package) I''ll try with good sources (1.8) and reply after Thanks a lot -----Message d''origine----- De : Andreas Dilger [mailto:andreas.dilger at oracle.com] Envoy? : jeudi 16 d?cembre 2010 16:57 ? : Eudes PHILIPPE Cc : ''Wang Yibin''; ''lustre-discuss'' Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? On 2010-12-16, at 08:25, Eudes PHILIPPE wrote:> If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload > is ok (sftp said nothing), but the file is different (the md5sum is different!) It''s very dangerous!!What version of Lustre are you using? This should definitely not happen (md5sum is different, without any errors). Is the size of the new file in Lustre matching the size of the original file? Does it work OK if you are using "cp" instead of sftp?> Is there a solution, if I see all ossX are almost full, when I add (some) new oss, distribute data on these new oss to always have the same poucentage use on all oss?This already works today, so long as the file sizes are not so large that they fill most of an OST.> -----Message d''origine----- > De : Wang Yibin [mailto:wang.yibin at oracle.com] Envoy? : jeudi 16 > d?cembre 2010 16:09 ? : Eudes PHILIPPE Cc : lustre-discuss Objet : Re: > [Lustre-discuss] howto make a lvm, or virtual lvm? > > > ? 2010-12-16???10:49? Eudes PHILIPPE ??? > >> Ok, so, i''ll try this : >> - One mds >> - 2 physical oss with one drive (1 GB) (one ost on one oss) >> >> On client, mount mds on /home.. >> lfs setstripe -c2 /home >> >> I upload (in sftp) one file, 300 MB >> - On Oss 1, he use 150 Mb of 1000 >> - On Oss 2, he use 150 Mb of 1000 >> >> All right! >> >> I continue... copy my first file 4 times (so there is 5 * 300 MB = >> 1500 MB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> >> ************************* >> Now, I add a new oss server, with one ost (1GB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> - On Oss 3, he use 0 Mb of 1000 >> >> lfs setstripe -c3 /home on client >> >> I upload a big file, 1.3 Go >> He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop >> (Couldn''t write to remote file "/home/big_log.log": Failure) >> ****************************** > > All files in a directory inherits its parent dir''s stripe attributes. > As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. > As OST 1 and 2 are full, surely you''ll get write failure with ENOSPC. > >> >> So now, >> - On Oss 1, he use 1000 Mb of 1000 >> - On Oss 2, he use 1000 Mb of 1000 >> - On Oss 3, he use 250 Mb of 1000 >> I upload again, just for see, my first file (300 MB), he copy the file >> only on Oss3 (oss 1 and 2 are full of course), it''s ok :) >> >> Is there a solution for this problem? > > If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. > In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). > >> >> Regards >> >> >> >> -----Message d''origine----- >> De : Andreas Dilger [mailto:andreas.dilger at oracle.com] >> Envoy? : mercredi 15 d?cembre 2010 22:39 ? : Eudes PHILIPPE Cc : >> lustre-discuss at lists.lustre.org Objet : Re: [Lustre-discuss] howto >> make a lvm, or virtual lvm? >> >> On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >>> At the end, I want (if it''s possible), a raid 5 over Ethernet, or, 1 >> physical raid 5 on each ostX and a big lvm I can extend as I want.... >> >> Lustre itself can not do RAID over the network, if that is what you >> are looking for... >> >>> For my first test, I upload on client a file (1.8 Go) (each ost have >>> 1 >>> Go) The problem, is, when sdb is full, he stop the copy, and don''t >>> continue on ost2 >> >> If you create your file to be striped over both OSTs, then it should work. >> >> Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.