Hi, I''m having some performance issues on my Lustre filesystem and it
looks to me like it''s related to I/Os getting fragmented before being
written to disk, but I can''t figure out why. This system is RHEL5,
running Lustre 1.8.4.
All of my OSTs look pretty much the same-
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
1: 88811 38 38 | 46375 17 17
2: 1497 0 38 | 7733 2 20
4: 1161 0 39 | 1840 0 21
8: 1168 0 39 | 7148 2 24
16: 922 0 40 | 3297 1 25
32: 979 0 40 | 7602 2 28
64: 1576 0 41 | 9046 3 31
128: 7063 3 44 | 16284 6 37
256: 129282 55 100 | 162090 62 100
read | write
disk fragmented I/Os ios % cum % | ios % cum %
0: 51181 22 22 | 0 0 0
1: 45280 19 42 | 82206 31 31
2: 16615 7 49 | 29108 11 42
3: 3425 1 50 | 17392 6 49
4: 110445 48 98 | 129481 49 98
5: 1661 0 99 | 2702 1 99
read | write
disk I/O size ios % cum % | ios % cum %
4K: 45889 8 8 | 56240 7 7
8K: 3658 0 8 | 6416 0 8
16K: 7956 1 10 | 4703 0 9
32K: 4527 0 11 | 11951 1 10
64K: 114369 20 31 | 134128 18 29
128K: 5095 0 32 | 17229 2 31
256K: 7164 1 33 | 30826 4 35
512K: 369512 66 100 | 465719 64 100
Oddly, there''s no 1024K row in the I/O size table...
...and these seem small to me as well, but I can''t seem to change them.
Writing new values to either doesn''t change anything.
# cat /sys/block/sdb/queue/max_hw_sectors_kb
320
# cat /sys/block/sdb/queue/max_sectors_kb
320
Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID
controllers, with MD1000 and MD1200 arrays, respectively.
Any clues on where I should look next?
Thanks,
Kevin
Kevin Hildebrand
University of Maryland, College Park
Office of Information Technology
You didn''t say, but I think they are LSI-based: are you using the mptsas driver with the PERC cards? Which driver version? First, max_sectors_kb should normally be set to a power of 2 number, like 256, over an odd size like 320. This number should also match the native raid size of the device, to avoid read-modify-write cycles. (See Bug 22886 on why not to make it > 1024 in general). See Bug 17086 for patches to increase the max_sectors_kb limitation for the mptsas driver to 1MB, or the true hardware maximum, rather than a driver limit; however, the hardware may still be limited to sizes < 1MB. Also, to clarify the sizes: the smallest bucket >= transfer_size is the one incremented, so a 320KB IO increments the 512KB bucket. Since your HW says it can only do a 320KB IO, there will never be a 1MB IO. You may want to instrument your HBA driver to see what is going on (ie, why the max_hw_sectors_kb is < 1024). Kevin Kevin Hildebrand wrote:> Hi, I''m having some performance issues on my Lustre filesystem and it > looks to me like it''s related to I/Os getting fragmented before being > written to disk, but I can''t figure out why. This system is RHEL5, > running Lustre 1.8.4. > > All of my OSTs look pretty much the same- > > read | write > pages per bulk r/w rpcs % cum % | rpcs % cum % > 1: 88811 38 38 | 46375 17 17 > 2: 1497 0 38 | 7733 2 20 > 4: 1161 0 39 | 1840 0 21 > 8: 1168 0 39 | 7148 2 24 > 16: 922 0 40 | 3297 1 25 > 32: 979 0 40 | 7602 2 28 > 64: 1576 0 41 | 9046 3 31 > 128: 7063 3 44 | 16284 6 37 > 256: 129282 55 100 | 162090 62 100 > > > read | write > disk fragmented I/Os ios % cum % | ios % cum % > 0: 51181 22 22 | 0 0 0 > 1: 45280 19 42 | 82206 31 31 > 2: 16615 7 49 | 29108 11 42 > 3: 3425 1 50 | 17392 6 49 > 4: 110445 48 98 | 129481 49 98 > 5: 1661 0 99 | 2702 1 99 > > read | write > disk I/O size ios % cum % | ios % cum % > 4K: 45889 8 8 | 56240 7 7 > 8K: 3658 0 8 | 6416 0 8 > 16K: 7956 1 10 | 4703 0 9 > 32K: 4527 0 11 | 11951 1 10 > 64K: 114369 20 31 | 134128 18 29 > 128K: 5095 0 32 | 17229 2 31 > 256K: 7164 1 33 | 30826 4 35 > 512K: 369512 66 100 | 465719 64 100 > > Oddly, there''s no 1024K row in the I/O size table... > > > ...and these seem small to me as well, but I can''t seem to change them. > Writing new values to either doesn''t change anything. > > # cat /sys/block/sdb/queue/max_hw_sectors_kb > 320 > # cat /sys/block/sdb/queue/max_sectors_kb > 320 > > Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID > controllers, with MD1000 and MD1200 arrays, respectively. > > > Any clues on where I should look next? > > Thanks, > > Kevin > > Kevin Hildebrand > University of Maryland, College Park > Office of Information Technology > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
The PERC 6 and H800 use megaraid_sas, I''m currently running 00.00.04.17-RH1. The max_sectors numbers (320) are what is being set by default- I am able to set it to something smaller than 320, but not larger. Kevin On Wed, 11 May 2011, Kevin Van Maren wrote:> You didn''t say, but I think they are LSI-based: are you using the mptsas > driver with the PERC cards? Which driver version? > > First, max_sectors_kb should normally be set to a power of 2 number, > like 256, over an odd size like 320. This number should also match the > native raid size of the device, to avoid read-modify-write cycles. (See > Bug 22886 on why not to make it > 1024 in general). > > See Bug 17086 for patches to increase the max_sectors_kb limitation for > the mptsas driver to 1MB, or the true hardware maximum, rather than a > driver limit; however, the hardware may still be limited to sizes < 1MB. > > Also, to clarify the sizes: the smallest bucket >= transfer_size is the > one incremented, so a 320KB IO increments the 512KB bucket. Since your > HW says it can only do a 320KB IO, there will never be a 1MB IO. > > You may want to instrument your HBA driver to see what is going on (ie, > why the max_hw_sectors_kb is < 1024). > > Kevin > > > Kevin Hildebrand wrote: >> Hi, I''m having some performance issues on my Lustre filesystem and it >> looks to me like it''s related to I/Os getting fragmented before being >> written to disk, but I can''t figure out why. This system is RHEL5, >> running Lustre 1.8.4. >> >> All of my OSTs look pretty much the same- >> >> read | write >> pages per bulk r/w rpcs % cum % | rpcs % cum % >> 1: 88811 38 38 | 46375 17 17 >> 2: 1497 0 38 | 7733 2 20 >> 4: 1161 0 39 | 1840 0 21 >> 8: 1168 0 39 | 7148 2 24 >> 16: 922 0 40 | 3297 1 25 >> 32: 979 0 40 | 7602 2 28 >> 64: 1576 0 41 | 9046 3 31 >> 128: 7063 3 44 | 16284 6 37 >> 256: 129282 55 100 | 162090 62 100 >> >> >> read | write >> disk fragmented I/Os ios % cum % | ios % cum % >> 0: 51181 22 22 | 0 0 0 >> 1: 45280 19 42 | 82206 31 31 >> 2: 16615 7 49 | 29108 11 42 >> 3: 3425 1 50 | 17392 6 49 >> 4: 110445 48 98 | 129481 49 98 >> 5: 1661 0 99 | 2702 1 99 >> >> read | write >> disk I/O size ios % cum % | ios % cum % >> 4K: 45889 8 8 | 56240 7 7 >> 8K: 3658 0 8 | 6416 0 8 >> 16K: 7956 1 10 | 4703 0 9 >> 32K: 4527 0 11 | 11951 1 10 >> 64K: 114369 20 31 | 134128 18 29 >> 128K: 5095 0 32 | 17229 2 31 >> 256K: 7164 1 33 | 30826 4 35 >> 512K: 369512 66 100 | 465719 64 100 >> >> Oddly, there''s no 1024K row in the I/O size table... >> >> >> ...and these seem small to me as well, but I can''t seem to change them. >> Writing new values to either doesn''t change anything. >> >> # cat /sys/block/sdb/queue/max_hw_sectors_kb >> 320 >> # cat /sys/block/sdb/queue/max_sectors_kb >> 320 >> >> Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID >> controllers, with MD1000 and MD1200 arrays, respectively. >> >> >> Any clues on where I should look next? >> >> Thanks, >> >> Kevin >> >> Kevin Hildebrand >> University of Maryland, College Park >> Office of Information Technology >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > >
Kevin Hildebrand wrote:> > The PERC 6 and H800 use megaraid_sas, I''m currently running > 00.00.04.17-RH1. > > The max_sectors numbers (320) are what is being set by default- I am > able to set it to something smaller than 320, but not larger.Right. You can not set max_sectors_kb larger than max_hw_sectors_kb (Linux normally defaults most drivers to 512, but Lustre sets them to be the same): you may want to instrument your HBA driver to see what is going on (ie, why the max_hw_sectors_kb is < 1024). I don''t know if it is due to a driver limitation or a true hardware limit. Most drivers have a limit of 512KB by default; see Bug 22850 for the patches that fixed the QLogic and Emulex fibre channel drivers. Kevin> Kevin > > On Wed, 11 May 2011, Kevin Van Maren wrote: > >> You didn''t say, but I think they are LSI-based: are you using the mptsas >> driver with the PERC cards? Which driver version? >> >> First, max_sectors_kb should normally be set to a power of 2 number, >> like 256, over an odd size like 320. This number should also match the >> native raid size of the device, to avoid read-modify-write cycles. (See >> Bug 22886 on why not to make it > 1024 in general). >> >> See Bug 17086 for patches to increase the max_sectors_kb limitation for >> the mptsas driver to 1MB, or the true hardware maximum, rather than a >> driver limit; however, the hardware may still be limited to sizes < 1MB. >> >> Also, to clarify the sizes: the smallest bucket >= transfer_size is the >> one incremented, so a 320KB IO increments the 512KB bucket. Since your >> HW says it can only do a 320KB IO, there will never be a 1MB IO. >> >> You may want to instrument your HBA driver to see what is going on (ie, >> why the max_hw_sectors_kb is < 1024). >> >> Kevin >> >> >> Kevin Hildebrand wrote: >>> Hi, I''m having some performance issues on my Lustre filesystem and it >>> looks to me like it''s related to I/Os getting fragmented before being >>> written to disk, but I can''t figure out why. This system is RHEL5, >>> running Lustre 1.8.4. >>> >>> All of my OSTs look pretty much the same- >>> >>> read | write >>> pages per bulk r/w rpcs % cum % | rpcs % cum % >>> 1: 88811 38 38 | 46375 17 17 >>> 2: 1497 0 38 | 7733 2 20 >>> 4: 1161 0 39 | 1840 0 21 >>> 8: 1168 0 39 | 7148 2 24 >>> 16: 922 0 40 | 3297 1 25 >>> 32: 979 0 40 | 7602 2 28 >>> 64: 1576 0 41 | 9046 3 31 >>> 128: 7063 3 44 | 16284 6 37 >>> 256: 129282 55 100 | 162090 62 100 >>> >>> >>> read | write >>> disk fragmented I/Os ios % cum % | ios % cum % >>> 0: 51181 22 22 | 0 0 0 >>> 1: 45280 19 42 | 82206 31 31 >>> 2: 16615 7 49 | 29108 11 42 >>> 3: 3425 1 50 | 17392 6 49 >>> 4: 110445 48 98 | 129481 49 98 >>> 5: 1661 0 99 | 2702 1 99 >>> >>> read | write >>> disk I/O size ios % cum % | ios % cum % >>> 4K: 45889 8 8 | 56240 7 7 >>> 8K: 3658 0 8 | 6416 0 8 >>> 16K: 7956 1 10 | 4703 0 9 >>> 32K: 4527 0 11 | 11951 1 10 >>> 64K: 114369 20 31 | 134128 18 29 >>> 128K: 5095 0 32 | 17229 2 31 >>> 256K: 7164 1 33 | 30826 4 35 >>> 512K: 369512 66 100 | 465719 64 100 >>> >>> Oddly, there''s no 1024K row in the I/O size table... >>> >>> >>> ...and these seem small to me as well, but I can''t seem to change them. >>> Writing new values to either doesn''t change anything. >>> >>> # cat /sys/block/sdb/queue/max_hw_sectors_kb >>> 320 >>> # cat /sys/block/sdb/queue/max_sectors_kb >>> 320 >>> >>> Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID >>> controllers, with MD1000 and MD1200 arrays, respectively. >>> >>> >>> Any clues on where I should look next? >>> >>> Thanks, >>> >>> Kevin >>> >>> Kevin Hildebrand >>> University of Maryland, College Park >>> Office of Information Technology >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >> >>
One of the oddities that I''m seeing that has me grasping at write fragmentation and I/O sizes may not be directly related to these things at all. Periodically, iostat will show that one or more of my OST disks will be running at 99% utilization. Reads per second is somewhere in the 150-200 range, while read kB/second is quite small. In addition, average request size is also very small. llobdstat output on the OST in question usually has zero, or very small values for reads and writes, and values for stats/punches/creates/deletes in the ones and twos. While this is happening, lustre starts complaining about ''slow commitrw'', ''slow direct_io'', etc. At this time, accesses from clients are usually hanging. Why would the disk(s) be pegged while llobdstat shows zero activity? After a few minutes in this state, the %util drops back down to single digit percentages and normal I/O resumes on the clients. Thanks, Kevin On Thu, 12 May 2011, Kevin Van Maren wrote:> Kevin Hildebrand wrote: >> >> The PERC 6 and H800 use megaraid_sas, I''m currently running >> 00.00.04.17-RH1. >> >> The max_sectors numbers (320) are what is being set by default- I am >> able to set it to something smaller than 320, but not larger. > > Right. You can not set max_sectors_kb larger than max_hw_sectors_kb > (Linux normally defaults most drivers to 512, but Lustre sets them to be > the same): you may want to instrument your HBA driver to see what is > going on (ie, why the max_hw_sectors_kb is < 1024). I don''t know if it > is due to a driver limitation or a true hardware limit. > > Most drivers have a limit of 512KB by default; see Bug 22850 for the > patches that fixed the QLogic and Emulex fibre channel drivers. > > Kevin > >> Kevin >> >> On Wed, 11 May 2011, Kevin Van Maren wrote: >> >>> You didn''t say, but I think they are LSI-based: are you using the mptsas >>> driver with the PERC cards? Which driver version? >>> >>> First, max_sectors_kb should normally be set to a power of 2 number, >>> like 256, over an odd size like 320. This number should also match the >>> native raid size of the device, to avoid read-modify-write cycles. (See >>> Bug 22886 on why not to make it > 1024 in general). >>> >>> See Bug 17086 for patches to increase the max_sectors_kb limitation for >>> the mptsas driver to 1MB, or the true hardware maximum, rather than a >>> driver limit; however, the hardware may still be limited to sizes < 1MB. >>> >>> Also, to clarify the sizes: the smallest bucket >= transfer_size is the >>> one incremented, so a 320KB IO increments the 512KB bucket. Since your >>> HW says it can only do a 320KB IO, there will never be a 1MB IO. >>> >>> You may want to instrument your HBA driver to see what is going on (ie, >>> why the max_hw_sectors_kb is < 1024). >>> >>> Kevin >>> >>> >>> Kevin Hildebrand wrote: >>>> Hi, I''m having some performance issues on my Lustre filesystem and it >>>> looks to me like it''s related to I/Os getting fragmented before being >>>> written to disk, but I can''t figure out why. This system is RHEL5, >>>> running Lustre 1.8.4. >>>> >>>> All of my OSTs look pretty much the same- >>>> >>>> read | write >>>> pages per bulk r/w rpcs % cum % | rpcs % cum % >>>> 1: 88811 38 38 | 46375 17 17 >>>> 2: 1497 0 38 | 7733 2 20 >>>> 4: 1161 0 39 | 1840 0 21 >>>> 8: 1168 0 39 | 7148 2 24 >>>> 16: 922 0 40 | 3297 1 25 >>>> 32: 979 0 40 | 7602 2 28 >>>> 64: 1576 0 41 | 9046 3 31 >>>> 128: 7063 3 44 | 16284 6 37 >>>> 256: 129282 55 100 | 162090 62 100 >>>> >>>> >>>> read | write >>>> disk fragmented I/Os ios % cum % | ios % cum % >>>> 0: 51181 22 22 | 0 0 0 >>>> 1: 45280 19 42 | 82206 31 31 >>>> 2: 16615 7 49 | 29108 11 42 >>>> 3: 3425 1 50 | 17392 6 49 >>>> 4: 110445 48 98 | 129481 49 98 >>>> 5: 1661 0 99 | 2702 1 99 >>>> >>>> read | write >>>> disk I/O size ios % cum % | ios % cum % >>>> 4K: 45889 8 8 | 56240 7 7 >>>> 8K: 3658 0 8 | 6416 0 8 >>>> 16K: 7956 1 10 | 4703 0 9 >>>> 32K: 4527 0 11 | 11951 1 10 >>>> 64K: 114369 20 31 | 134128 18 29 >>>> 128K: 5095 0 32 | 17229 2 31 >>>> 256K: 7164 1 33 | 30826 4 35 >>>> 512K: 369512 66 100 | 465719 64 100 >>>> >>>> Oddly, there''s no 1024K row in the I/O size table... >>>> >>>> >>>> ...and these seem small to me as well, but I can''t seem to change them. >>>> Writing new values to either doesn''t change anything. >>>> >>>> # cat /sys/block/sdb/queue/max_hw_sectors_kb >>>> 320 >>>> # cat /sys/block/sdb/queue/max_sectors_kb >>>> 320 >>>> >>>> Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID >>>> controllers, with MD1000 and MD1200 arrays, respectively. >>>> >>>> >>>> Any clues on where I should look next? >>>> >>>> Thanks, >>>> >>>> Kevin >>>> >>>> Kevin Hildebrand >>>> University of Maryland, College Park >>>> Office of Information Technology >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >>> >>> > >
On May 12, 2011, at 7:52 AM, Kevin Hildebrand wrote:> > One of the oddities that I''m seeing that has me grasping at write > fragmentation and I/O sizes may not be directly related to these things at > all. Periodically, iostat will show that one or more of my OST disks will > be running at 99% utilization. Reads per second is somewhere in the > 150-200 range, while read kB/second is quite small.That sounds familiar. You''re probably experiencing these: https://bugzilla.lustre.org/show_bug.cgi?id=24183 http://jira.whamcloud.com/browse/LU-15 Jason> In addition, average > request size is also very small. llobdstat output on the OST in question > usually has zero, or very small values for reads and writes, and values > for stats/punches/creates/deletes in the ones and twos. > While this is happening, lustre starts complaining about ''slow commitrw'', > ''slow direct_io'', etc. At this time, accesses from clients are usually > hanging. > > Why would the disk(s) be pegged while llobdstat shows zero activity? > > After a few minutes in this state, the %util drops back down to single > digit percentages and normal I/O resumes on the clients. > > Thanks, > Kevin > > On Thu, 12 May 2011, Kevin Van Maren wrote: > >> Kevin Hildebrand wrote: >>> >>> The PERC 6 and H800 use megaraid_sas, I''m currently running >>> 00.00.04.17-RH1. >>> >>> The max_sectors numbers (320) are what is being set by default- I am >>> able to set it to something smaller than 320, but not larger. >> >> Right. You can not set max_sectors_kb larger than max_hw_sectors_kb >> (Linux normally defaults most drivers to 512, but Lustre sets them to be >> the same): you may want to instrument your HBA driver to see what is >> going on (ie, why the max_hw_sectors_kb is < 1024). I don''t know if it >> is due to a driver limitation or a true hardware limit. >> >> Most drivers have a limit of 512KB by default; see Bug 22850 for the >> patches that fixed the QLogic and Emulex fibre channel drivers. >> >> Kevin >> >>> Kevin >>> >>> On Wed, 11 May 2011, Kevin Van Maren wrote: >>> >>>> You didn''t say, but I think they are LSI-based: are you using the mptsas >>>> driver with the PERC cards? Which driver version? >>>> >>>> First, max_sectors_kb should normally be set to a power of 2 number, >>>> like 256, over an odd size like 320. This number should also match the >>>> native raid size of the device, to avoid read-modify-write cycles. (See >>>> Bug 22886 on why not to make it > 1024 in general). >>>> >>>> See Bug 17086 for patches to increase the max_sectors_kb limitation for >>>> the mptsas driver to 1MB, or the true hardware maximum, rather than a >>>> driver limit; however, the hardware may still be limited to sizes < 1MB. >>>> >>>> Also, to clarify the sizes: the smallest bucket >= transfer_size is the >>>> one incremented, so a 320KB IO increments the 512KB bucket. Since your >>>> HW says it can only do a 320KB IO, there will never be a 1MB IO. >>>> >>>> You may want to instrument your HBA driver to see what is going on (ie, >>>> why the max_hw_sectors_kb is < 1024). >>>> >>>> Kevin >>>> >>>> >>>> Kevin Hildebrand wrote: >>>>> Hi, I''m having some performance issues on my Lustre filesystem and it >>>>> looks to me like it''s related to I/Os getting fragmented before being >>>>> written to disk, but I can''t figure out why. This system is RHEL5, >>>>> running Lustre 1.8.4. >>>>> >>>>> All of my OSTs look pretty much the same- >>>>> >>>>> read | write >>>>> pages per bulk r/w rpcs % cum % | rpcs % cum % >>>>> 1: 88811 38 38 | 46375 17 17 >>>>> 2: 1497 0 38 | 7733 2 20 >>>>> 4: 1161 0 39 | 1840 0 21 >>>>> 8: 1168 0 39 | 7148 2 24 >>>>> 16: 922 0 40 | 3297 1 25 >>>>> 32: 979 0 40 | 7602 2 28 >>>>> 64: 1576 0 41 | 9046 3 31 >>>>> 128: 7063 3 44 | 16284 6 37 >>>>> 256: 129282 55 100 | 162090 62 100 >>>>> >>>>> >>>>> read | write >>>>> disk fragmented I/Os ios % cum % | ios % cum % >>>>> 0: 51181 22 22 | 0 0 0 >>>>> 1: 45280 19 42 | 82206 31 31 >>>>> 2: 16615 7 49 | 29108 11 42 >>>>> 3: 3425 1 50 | 17392 6 49 >>>>> 4: 110445 48 98 | 129481 49 98 >>>>> 5: 1661 0 99 | 2702 1 99 >>>>> >>>>> read | write >>>>> disk I/O size ios % cum % | ios % cum % >>>>> 4K: 45889 8 8 | 56240 7 7 >>>>> 8K: 3658 0 8 | 6416 0 8 >>>>> 16K: 7956 1 10 | 4703 0 9 >>>>> 32K: 4527 0 11 | 11951 1 10 >>>>> 64K: 114369 20 31 | 134128 18 29 >>>>> 128K: 5095 0 32 | 17229 2 31 >>>>> 256K: 7164 1 33 | 30826 4 35 >>>>> 512K: 369512 66 100 | 465719 64 100 >>>>> >>>>> Oddly, there''s no 1024K row in the I/O size table... >>>>> >>>>> >>>>> ...and these seem small to me as well, but I can''t seem to change them. >>>>> Writing new values to either doesn''t change anything. >>>>> >>>>> # cat /sys/block/sdb/queue/max_hw_sectors_kb >>>>> 320 >>>>> # cat /sys/block/sdb/queue/max_sectors_kb >>>>> 320 >>>>> >>>>> Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID >>>>> controllers, with MD1000 and MD1200 arrays, respectively. >>>>> >>>>> >>>>> Any clues on where I should look next? >>>>> >>>>> Thanks, >>>>> >>>>> Kevin >>>>> >>>>> Kevin Hildebrand >>>>> University of Maryland, College Park >>>>> Office of Information Technology >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>>> >>>> >>>> >> >> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- Jason Rappleye System Administrator NASA Advanced Supercomputing Division NASA Ames Research Center Moffett Field, CA 94035