Roger Spellman
2011-Jul-29 20:49 UTC
[Lustre-discuss] Random OST Numbers chosen in a stripe
Suppose that I stripe a directory with the following command:
lfs setstripe -c 4 .
On some of my systems, when I create file in the directory, the list of OSTs for
a particular file is sequential, e.g.
obdidx objid objid group
12 2 0x2 0
13 2 0x2 0
14 2 0x2 0
15 2 0x2 0
On another one of my systems, when I create files in a similarly striped
directory, I get seemingly random assignment, e.g.
For one file:
?? obdidx?????????? objid????????? objid??????????? group
??? 14??????????? 6884???????? 0x1ae4??????????????? 0
??? 46??????????? 6880???????? 0x1ae0??????? ????????0
???? 8??????????? 6883???????? 0x1ae3??????????????? 0
?? 29??????????? 6880???????? 0x1ae0??????????????? 0
For a different file:
?? obdidx?????????? objid????????? objid??????????? group
???? 13??????? ????6884???????? 0x1ae4??????????????? 0
???? 28??????????? 6880???????? 0x1ae0??????????????? 0
?? 44??????????? 6880???????? 0x1ae0??????????????? 0
???? 27??????????? 6880???????? 0x1ae0??????????????? 0
Why is this?
How can I control it to always be sequential?
Thanks.
Roger Spellman
Staff Engineer
Terascala, Inc.
508-588-1501
www.terascala.com <http://www.terascala.com/>
Atul Vidwansa
2011-Aug-01 08:08 UTC
[Lustre-discuss] Random OST Numbers chosen in a stripe
Try specifying start OST to setstripe command. Look at lfs manpage.
Cheers,
_Atul
-----Original Message-----
From: Roger Spellman [mailto:Roger.Spellman at terascala.com]
Sent: Saturday, 30 July 2011 2:19 AM
To: lustre-discuss at lists.lustre.org; wc-discuss at whamcloud.com
Subject: [wc-discuss] Random OST Numbers chosen in a stripe
Suppose that I stripe a directory with the following command:
lfs setstripe -c 4 .
On some of my systems, when I create file in the directory, the list of OSTs for
a particular file is sequential, e.g.
obdidx objid objid group
12 2 0x2 0
13 2 0x2 0
14 2 0x2 0
15 2 0x2 0
On another one of my systems, when I create files in a similarly striped
directory, I get seemingly random assignment, e.g.
For one file:
?? obdidx?????????? objid????????? objid??????????? group
??? 14??????????? 6884???????? 0x1ae4??????????????? 0
??? 46??????????? 6880???????? 0x1ae0??????? ????????0
???? 8??????????? 6883???????? 0x1ae3??????????????? 0
?? 29??????????? 6880???????? 0x1ae0??????????????? 0
For a different file:
?? obdidx?????????? objid????????? objid??????????? group
???? 13??????? ????6884???????? 0x1ae4??????????????? 0
???? 28??????????? 6880???????? 0x1ae0??????????????? 0
?? 44??????????? 6880???????? 0x1ae0??????????????? 0
???? 27??????????? 6880???????? 0x1ae0??????????????? 0
Why is this?
How can I control it to always be sequential?
Thanks.
Roger Spellman
Staff Engineer
Terascala, Inc.
508-588-1501
www.terascala.com <http://www.terascala.com/>
Johann Lombardi
2011-Aug-01 08:38 UTC
[Lustre-discuss] Random OST Numbers chosen in a stripe
On Fri, Jul 29, 2011 at 04:49:28PM -0400, Roger Spellman wrote:> For a different file: > > ?? obdidx?????????? objid????????? objid??????????? group > ???? 13??????? ????6884???????? 0x1ae4??????????????? 0 > ???? 28??????????? 6880???????? 0x1ae0??????????????? 0 > ?? 44??????????? 6880???????? 0x1ae0??????????????? 0 > ???? 27??????????? 6880???????? 0x1ae0??????????????? 0 > > Why is this? > > How can I control it to always be sequential?It depends on the OST usage imbalance and you can tune the stripe allocation policy with qos_threshold_rr. For more information, please refer to the lustre manual: http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html#50438271_pgfId-1296529 Cheers, Johann -- Johann Lombardi Whamcloud, Inc. www.whamcloud.com
Kevin Van Maren
2011-Aug-01 16:09 UTC
[Lustre-discuss] Random OST Numbers chosen in a stripe
Johann Lombardi wrote:> On Fri, Jul 29, 2011 at 04:49:28PM -0400, Roger Spellman wrote: > >> For a different file: >> >> obdidx objid objid group >> 13 6884 0x1ae4 0 >> 28 6880 0x1ae0 0 >> 44 6880 0x1ae0 0 >> 27 6880 0x1ae0 0 >> >> Why is this? >> >> How can I control it to always be sequential? >> > > It depends on the OST usage imbalance and you can tune the stripe allocation policy with qos_threshold_rr. For more information, please refer to the lustre manual: > http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html#50438271_pgfId-1296529 > > Cheers, > Johann >Also note that newer versions of Lustre sort the OST list even in RR mode, so that it will not allocate successive objects from the same OSS node. Kevin
Cory Spitz
2011-Aug-04 14:26 UTC
[Lustre-discuss] [wc-discuss] Re: Random OST Numbers chosen in a stripe
Hi, Roger. Forcing RR all the time by setting qos_threshold_rr to 100 still won''t necessarily allow sequential OST stripe placement. There is a set order when specifying a starting OST. The following bugs track related issues with stripe allocation. bz 20844: Can round-robin allocation be based on ost number? - this one essentially asks if stripes can land on sequential OSTs as you asked bz 21528: round-robin allocation not working as expected when a start ost index is specified. - this one tracks a bug where two stripes used the same OSS when it should not bz 18547: optimize QOS RR allocator - this one tracks enhancements to the stripe allocators Thanks, -Cory On 08/01/2011 03:38 AM, Johann Lombardi wrote:> On Fri, Jul 29, 2011 at 04:49:28PM -0400, Roger Spellman wrote: >> For a different file: >> >> obdidx objid objid group >> 13 6884 0x1ae4 0 >> 28 6880 0x1ae0 0 >> 44 6880 0x1ae0 0 >> 27 6880 0x1ae0 0 >> >> Why is this? >> >> How can I control it to always be sequential? > > It depends on the OST usage imbalance and you can tune the stripe allocation policy with qos_threshold_rr. For more information, please refer to the lustre manual: > http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html#50438271_pgfId-1296529 > > Cheers, > Johann