Lex wrote:> Hi guys > > I have some questions about file joining mechanism in lustre, so i > post here and hope you can help me to clarify it :) > > As we know, lustre use the split writing and hierarchy striping to > distribute file object ( subfile) to many osts in the cluster. When a > client request one file ( over mds metadata ), these objects are > retrieve from each ost to client. > > So, the question is: when and where these objects are joined together > to become a real file ?File joining is actually joining the file stripe information. which is stored in MDS, so the joining actually happens in MDS, instead of OSS. Btw: Because of some reasons, joining file feature will be removed temporarily in lustre2.0. Thanks WangDi> > Just guess, are there joined together in OSSs? If it''s true, how these > OSSs connect together to join its objects together? ( For example, > assuming we have 2 OSSs, each OSS have 1 OST, file are striped across > to 2 these OSTs. When a client request a file, i get to object in 2 > different OST from 2 different OSSs, so how these objects can combined > together? ) > > Any help would be appreciated :) > > Many thanks > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Hi guys I have some questions about file joining mechanism in lustre, so i post here and hope you can help me to clarify it :) As we know, lustre use the split writing and hierarchy striping to distribute file object ( subfile) to many osts in the cluster. When a client request one file ( over mds metadata ), these objects are retrieve from each ost to client. So, the question is: when and where these objects are joined together to become a real file ? Just guess, are there joined together in OSSs? If it''s true, how these OSSs connect together to join its objects together? ( For example, assuming we have 2 OSSs, each OSS have 1 OST, file are striped across to 2 these OSTs. When a client request a file, i get to object in 2 different OST from 2 different OSSs, so how these objects can combined together? ) Any help would be appreciated :) Many thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100113/f9cbb051/attachment.html
On Wed, Jan 13, 2010 at 10:23:06PM +0700, Lex wrote:> I have some questions about file joining mechanism in lustre, so i post here > and hope you can help me to clarify it :) > > As we know, lustre use the split writing and hierarchy striping to > distribute file object ( subfile) to many osts in the cluster. When a > client request one file ( over mds metadata ), these objects are retrieve > from each ost to client. > > So, the question is: when and where these objects are joined together to > become a real file ? > > Just guess, are there joined together in OSSs? If it''s true, how these OSSs > connect together to join its objects together? ( For example, assuming we > have 2 OSSs, each OSS have 1 OST, file are striped across to 2 these OSTs. > When a client request a file, i get to object in 2 different OST from 2 > different OSSs, so how these objects can combined together? )On the client. The MDS knows the layout of each file: the number and location of the file''s stripes. The client obtains the layout from the MDS. The client does I/O against the stripes of a file by talking directly to the relevant OSTs. Nico --
On Wed, Jan 13, 2010 at 5:40 PM, Tom.Wang <Tom.Wang at sun.com> wrote:> Lex wrote: > >> Hi guys >> >> I have some questions about file joining mechanism in lustre, so i post >> here and hope you can help me to clarify it :) >> >> As we know, lustre use the split writing and hierarchy striping to >> distribute file object ( subfile) to many osts in the cluster. When a >> client request one file ( over mds metadata ), these objects are retrieve >> from each ost to client. >> >> So, the question is: when and where these objects are joined together to >> become a real file ? >> > File joining is actually joining the file stripe information. which is > stored in MDS, so the joining actually happens in MDS, instead of OSS. > >Thank you for you fast reply, but i mean the physical joining file process, not just information or metadata. As i know, file ( data ) doesn''t go over MDS, right ? Sorry if i''m misunderstanding you> Btw: Because of some reasons, joining file feature will be removed > temporarily in lustre2.0. >Could you please tell me exactly what the reasons are ?> > Thanks > WangDi > >> >> Just guess, are there joined together in OSSs? If it''s true, how these >> OSSs connect together to join its objects together? ( For example, assuming >> we have 2 OSSs, each OSS have 1 OST, file are striped across to 2 these >> OSTs. When a client request a file, i get to object in 2 different OST from >> 2 different OSSs, so how these objects can combined together? ) >> >> Any help would be appreciated :) >> >> Many thanks >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100114/ef485373/attachment.html
You mean lustre client? Our lustre client work as a web server for downloading file, it''s using ssd drive and the free space is quite small, just few Gbs, but it''s serving many 10GB size file quite well. If the joining file process is on the client, how the big file can fit the poor free space well like this ? Could you please explain it for me ? Best regards> > On the client. > > The MDS knows the layout of each file: the number and location of the > file''s stripes. > > The client obtains the layout from the MDS. > > The client does I/O against the stripes of a file by talking directly to > the relevant OSTs. > > Nico > -- >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100114/482b3396/attachment.html
Lex wrote:> > > On Wed, Jan 13, 2010 at 5:40 PM, Tom.Wang <Tom.Wang at sun.com > <mailto:Tom.Wang at sun.com>> wrote: > > Lex wrote: > > Hi guys > > I have some questions about file joining mechanism in lustre, > so i post here and hope you can help me to clarify it :) > > As we know, lustre use the split writing and hierarchy > striping to distribute file object ( subfile) to many osts in > the cluster. When a client request one file ( over mds > metadata ), these objects are retrieve from each ost to client. > > So, the question is: when and where these objects are joined > together to become a real file ? > > File joining is actually joining the file stripe information. > which is stored in MDS, so the joining actually happens in MDS, > instead of OSS. > > > Thank you for you fast reply, but i mean the physical joining file > process, not just information or metadata. As i know, file ( data ) > doesn''t go over MDS, right ? Sorry if i''m misunderstanding youThe stripe information is the layout information of file data, which indicates where those data is really stored on OST. And the joining process is actually joining these layout information, so the two file_data looks like joined together. For example you have file_A size 8k stripe_A file_B size 16k stripe_B then the stripe information of joinAB will be sth like file extent [0, 8k] stripe_A ---------------> original file_A data on OST [8k, 24k] stripe_B --------------> original file_B data on OST> > > Btw: Because of some reasons, joining file feature will be removed > temporarily in lustre2.0. > > > Could you please tell me exactly what the reasons are ?On 2.0, MDS is completely restructured (you can see that on lustre roadmap) because of cluster MDS, so join file also needs to be re-implemented. But it is a low priority project, and we do not have enough resource to do that now. We might do that in some later version. Thanks Wangdi> > > > Thanks > WangDi > > > Just guess, are there joined together in OSSs? If it''s true, > how these OSSs connect together to join its objects together? > ( For example, assuming we have 2 OSSs, each OSS have 1 OST, > file are striped across to 2 these OSTs. When a client request > a file, i get to object in 2 different OST from 2 different > OSSs, so how these objects can combined together? ) > > Any help would be appreciated :) > > Many thanks > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > <mailto:Lustre-discuss at lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > >
Hi Wangdi I understand the process that you said and give me a clearly example. So, In the lustre system internal, there is no "physical" file joining process ? My lustre client works as web server, serving download. In this case, the "physical" data joining process are on our web client , isn''t it ? One more thing, just need a confirmation, in the joining file process, OSSs doesn''t have any role, right ? They just handle I/O action? Sorry if i''m rising such a stupid question :) Many thanks On Thu, Jan 14, 2010 at 10:55 AM, Tom.Wang <Tom.Wang at sun.com> wrote:> Lex wrote: > > >> >> On Wed, Jan 13, 2010 at 5:40 PM, Tom.Wang <Tom.Wang at sun.com <mailto: >> Tom.Wang at sun.com>> wrote: >> >> Lex wrote: >> >> Hi guys >> >> I have some questions about file joining mechanism in lustre, >> so i post here and hope you can help me to clarify it :) >> >> As we know, lustre use the split writing and hierarchy >> striping to distribute file object ( subfile) to many osts in >> the cluster. When a client request one file ( over mds >> metadata ), these objects are retrieve from each ost to client. >> >> So, the question is: when and where these objects are joined >> together to become a real file ? >> >> File joining is actually joining the file stripe information. >> which is stored in MDS, so the joining actually happens in MDS, >> instead of OSS. >> >> Thank you for you fast reply, but i mean the physical joining file >> process, not just information or metadata. As i know, file ( data ) doesn''t >> go over MDS, right ? Sorry if i''m misunderstanding you >> > The stripe information is the layout information of file data, which > indicates where those data is really stored on OST. And the joining process > is actually joining these layout information, so the two file_data looks > like joined together. For example you have > > file_A size 8k stripe_A file_B size 16k stripe_B > > then the stripe information of joinAB will be sth like file extent > > [0, 8k] stripe_A ---------------> original file_A data on OST > [8k, 24k] stripe_B --------------> original file_B data on OST > > > >> Btw: Because of some reasons, joining file feature will be removed >> temporarily in lustre2.0. >> >> >> Could you please tell me exactly what the reasons are ? >> > On 2.0, MDS is completely restructured (you can see that on lustre roadmap) > because of cluster MDS, so join file also needs to be re-implemented. But it > is a low priority project, and we do not have enough resource to do that > now. We might do that in some later version. > > Thanks > Wangdi > >> >> >> Thanks >> WangDi >> >> >> Just guess, are there joined together in OSSs? If it''s true, >> how these OSSs connect together to join its objects together? >> ( For example, assuming we have 2 OSSs, each OSS have 1 OST, >> file are striped across to 2 these OSTs. When a client request >> a file, i get to object in 2 different OST from 2 different >> OSSs, so how these objects can combined together? ) >> >> Any help would be appreciated :) >> >> Many thanks >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> <mailto:Lustre-discuss at lists.lustre.org> >> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100114/a58623b4/attachment-0001.html
On Thu, Jan 14, 2010 at 09:49:13AM +0700, Lex wrote:> You mean lustre client? Our lustre client work as a web server for > downloading file, it''s using ssd drive and the free space is quite small, > just few Gbs, but it''s serving many 10GB size file quite well. If the > joining file process is on the client, how the big file can fit the poor > free space well like this ? Could you please explain it for me ?First, I thought you meant by "file join" something very different than what we (Lustre team) mean by "file join". We mean this: an optimization of "cat file1 file2 ... > joined-file". I think you meant: how the illusion of a single file is created out of multiple stripes. Do correct me if my interpretation of your question was incorrect. If I understood you correctly, then there is no "physically joined" version of a file, except in the case of files with a single stripe. The illusion of "joined" stripes is presented by the client. The client need not have enough memory to store a whole file. It only needs enough for the application''s needs. E.g., if you have a 1TB file and 4GB of virtual memory, then you can''t read the whole file into memory, but you can read a couple of GBs at a time, do some work, then read the next couple of GBs. Nico --
Thanks all you guys for helping me. Sorry for my late reply. You gave me some very useful info. I will consider it carefully and maybe will ask your help once again :) Many thanks On Sat, Jan 16, 2010 at 7:02 AM, Nicolas Williams <Nicolas.Williams at sun.com>wrote:> On Thu, Jan 14, 2010 at 09:49:13AM +0700, Lex wrote: > > You mean lustre client? Our lustre client work as a web server for > > downloading file, it''s using ssd drive and the free space is quite small, > > just few Gbs, but it''s serving many 10GB size file quite well. If the > > joining file process is on the client, how the big file can fit the poor > > free space well like this ? Could you please explain it for me ? > > First, I thought you meant by "file join" something very different than > what we (Lustre team) mean by "file join". We mean this: an > optimization of "cat file1 file2 ... > joined-file". I think you meant: > how the illusion of a single file is created out of multiple stripes. > > Do correct me if my interpretation of your question was incorrect. > > If I understood you correctly, then there is no "physically joined" > version of a file, except in the case of files with a single stripe. > > The illusion of "joined" stripes is presented by the client. > > The client need not have enough memory to store a whole file. It only > needs enough for the application''s needs. E.g., if you have a 1TB file > and 4GB of virtual memory, then you can''t read the whole file into > memory, but you can read a couple of GBs at a time, do some work, then > read the next couple of GBs. > > Nico > -- >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100125/3e99696e/attachment.html