Kshitij Mehta
2012-Feb-10 22:27 UTC
[Lustre-discuss] IOR writing to a shared file, performance does not scale
We have lustre 1.6.7 configured using 64 OSTs. I am testing the performance using IOR, which is a file system benchmark. When I run IOR using mpi such that processes write to a shared file, performance does not scale. I tested with 1,2 and 4 processes, and the performance remains constant at 230 MBps. When processes write to separate files, performance improves greatly, reaching 475 MBps. Note that all processes are spawned on a single node. Here is the output: Writing to a shared file:> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o > /fastfs/gabriel/ss_64/km_ior.out > Machine: Linux deimos102 > > Summary: > api = POSIX > test filename = /fastfs/gabriel/ss_64/km_ior.out > access = single-shared-file > ordering in a file = sequential offsets > ordering inter file= no tasks offsets > clients = 4 (4 per node) > repetitions = 1 > xfersize = 32 MiB > blocksize = 2 GiB > aggregate filesize = 8 GiB > > Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min > (OPs) Mean (OPs) Std Dev Mean (s) > --------- --------- --------- ---------- ------- --------- > --------- ---------- ------- -------- > write 233.61 233.61 233.61 0.00 7.30 > 7.30 7.30 0.00 35.06771 EXCEL > > Max Write: 233.61 MiB/sec (244.95 MB/sec)Writing to separate files:> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o > /fastfs/gabriel/ss_64/km_ior.out -F > Machine: Linux deimos102 > > Summary: > api = POSIX > test filename = /fastfs/gabriel/ss_64/km_ior.out > access = file-per-process > ordering in a file = sequential offsets > ordering inter file= no tasks offsets > clients = 4 (4 per node) > repetitions = 1 > xfersize = 32 MiB > blocksize = 2 GiB > aggregate filesize = 8 GiB > > Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min > (OPs) Mean (OPs) Std Dev Mean (s) > --------- --------- --------- ---------- ------- --------- > --------- ---------- ------- -------- > write 475.95 475.95 475.95 0.00 14.87 > 14.87 14.87 0.00 17.21191 EXCEL > > Max Write: 475.95 MiB/sec (499.07 MB/sec)I am trying to understand where the bottleneck is, when processes write to a shared file. Your help is appreciated. -- Kshitij Mehta PhD candidate Parallel Software Technologies Lab (pstl.cs.uh.edu) Dept. of Computer Science University of Houston Houston, Texas, USA
Hedges, Richard M.
2012-Feb-10 22:31 UTC
[Lustre-discuss] IOR writing to a shared file, performance does not scale
Did you set up the shared file for striping?> hype356{rhedges}395: lfs help setstripe > setstripe: Create a new file with a specific striping pattern or > set the default striping pattern on an existing directory or > delete the default striping pattern from an existing directory > usage: setstripe [--size|-s stripe_size] [--count|-c stripe_count] > [--index|-i|--offset|-o start_ost_index] > [--pool|-p <pool>] <directory|filename> > or > setstripe -d <directory> (to delete default striping) > stripe_size: Number of bytes on each OST (0 filesystem default) > Can be specified with k, m or g (in KB, MB and GB > respectively) > start_ost_index: OST index of first stripe (-1 default) > stripe_count: Number of OSTs to stripe over (0 default, -1 all) > pool: Name of OST pool to use (default none)On 2/10/12 2:27 PM, "Kshitij Mehta" <kmehta at cs.uh.edu> wrote:> We have lustre 1.6.7 configured using 64 OSTs. > I am testing the performance using IOR, which is a file system benchmark. > > When I run IOR using mpi such that processes write to a shared file, > performance does not scale. I tested with 1,2 and 4 processes, and the > performance remains constant at 230 MBps. > > When processes write to separate files, performance improves greatly, > reaching 475 MBps. > > Note that all processes are spawned on a single node. > > Here is the output: > Writing to a shared file: > >> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >> /fastfs/gabriel/ss_64/km_ior.out >> Machine: Linux deimos102 >> >> Summary: >> api = POSIX >> test filename = /fastfs/gabriel/ss_64/km_ior.out >> access = single-shared-file >> ordering in a file = sequential offsets >> ordering inter file= no tasks offsets >> clients = 4 (4 per node) >> repetitions = 1 >> xfersize = 32 MiB >> blocksize = 2 GiB >> aggregate filesize = 8 GiB >> >> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >> (OPs) Mean (OPs) Std Dev Mean (s) >> --------- --------- --------- ---------- ------- --------- >> --------- ---------- ------- -------- >> write 233.61 233.61 233.61 0.00 7.30 >> 7.30 7.30 0.00 35.06771 EXCEL >> >> Max Write: 233.61 MiB/sec (244.95 MB/sec) > > Writing to separate files: > >> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >> /fastfs/gabriel/ss_64/km_ior.out -F >> Machine: Linux deimos102 >> >> Summary: >> api = POSIX >> test filename = /fastfs/gabriel/ss_64/km_ior.out >> access = file-per-process >> ordering in a file = sequential offsets >> ordering inter file= no tasks offsets >> clients = 4 (4 per node) >> repetitions = 1 >> xfersize = 32 MiB >> blocksize = 2 GiB >> aggregate filesize = 8 GiB >> >> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >> (OPs) Mean (OPs) Std Dev Mean (s) >> --------- --------- --------- ---------- ------- --------- >> --------- ---------- ------- -------- >> write 475.95 475.95 475.95 0.00 14.87 >> 14.87 14.87 0.00 17.21191 EXCEL >> >> Max Write: 475.95 MiB/sec (499.07 MB/sec) > > I am trying to understand where the bottleneck is, when processes write > to a shared file. > Your help is appreciated.=================================================== Richard Hedges Customer Support and Test - File Systems Project Development Environment Group - Livermore Computing Lawrence Livermore National Laboratory 7000 East Avenue, MS L-557 Livermore, CA 94551 v: (925) 423-2699 f: (925) 423-6961 E: richard-hedges at llnl.gov
Kshitij Mehta
2012-Feb-10 22:34 UTC
[Lustre-discuss] IOR writing to a shared file, performance does not scale
An `lfs getstripe` on the output file shows that the file is being striped.> lfs getstripe km_ior_shared.out > OBDS: > 0: fastfs-OST0000_UUID ACTIVE > 1: fastfs-OST0001_UUID ACTIVE > 2: fastfs-OST0002_UUID ACTIVE > 3: fastfs-OST0003_UUID ACTIVE > 4: fastfs-OST0004_UUID ACTIVE > 5: fastfs-OST0005_UUID ACTIVE > 6: fastfs-OST0006_UUID ACTIVE > 7: fastfs-OST0007_UUID ACTIVE > 8: fastfs-OST0008_UUID ACTIVE > 9: fastfs-OST0009_UUID ACTIVE > 10: fastfs-OST000a_UUID ACTIVE > 11: fastfs-OST000b_UUID ACTIVE > 12: fastfs-OST000c_UUID ACTIVE > 13: fastfs-OST000d_UUID ACTIVE > 14: fastfs-OST000e_UUID ACTIVE > 15: fastfs-OST000f_UUID ACTIVE > 16: fastfs-OST0010_UUID ACTIVE > 17: fastfs-OST0011_UUID ACTIVE > 18: fastfs-OST0012_UUID ACTIVE > 19: fastfs-OST0013_UUID ACTIVE > 20: fastfs-OST0014_UUID ACTIVE > 21: fastfs-OST0015_UUID ACTIVE > 22: fastfs-OST0016_UUID ACTIVE > 23: fastfs-OST0017_UUID ACTIVE > 24: fastfs-OST0018_UUID ACTIVE > 25: fastfs-OST0019_UUID ACTIVE > 26: fastfs-OST001a_UUID ACTIVE > 27: fastfs-OST001b_UUID ACTIVE > 28: fastfs-OST001c_UUID ACTIVE > 29: fastfs-OST001d_UUID ACTIVE > 30: fastfs-OST001e_UUID ACTIVE > 31: fastfs-OST001f_UUID ACTIVE > 32: fastfs-OST0020_UUID ACTIVE > 33: fastfs-OST0021_UUID ACTIVE > 34: fastfs-OST0022_UUID ACTIVE > 35: fastfs-OST0023_UUID ACTIVE > 36: fastfs-OST0024_UUID ACTIVE > 37: fastfs-OST0025_UUID ACTIVE > 38: fastfs-OST0026_UUID ACTIVE > 39: fastfs-OST0027_UUID ACTIVE > 40: fastfs-OST0028_UUID ACTIVE > 41: fastfs-OST0029_UUID ACTIVE > 42: fastfs-OST002a_UUID ACTIVE > 43: fastfs-OST002b_UUID ACTIVE > 44: fastfs-OST002c_UUID ACTIVE > 45: fastfs-OST002d_UUID ACTIVE > 46: fastfs-OST002e_UUID ACTIVE > 47: fastfs-OST002f_UUID ACTIVE > 48: fastfs-OST0030_UUID ACTIVE > 49: fastfs-OST0031_UUID ACTIVE > 50: fastfs-OST0032_UUID ACTIVE > 51: fastfs-OST0033_UUID ACTIVE > 52: fastfs-OST0034_UUID ACTIVE > 53: fastfs-OST0035_UUID ACTIVE > 54: fastfs-OST0036_UUID ACTIVE > 55: fastfs-OST0037_UUID ACTIVE > 56: fastfs-OST0038_UUID ACTIVE > 57: fastfs-OST0039_UUID ACTIVE > 58: fastfs-OST003a_UUID ACTIVE > 59: fastfs-OST003b_UUID ACTIVE > 60: fastfs-OST003c_UUID ACTIVE > 61: fastfs-OST003d_UUID ACTIVE > 62: fastfs-OST003e_UUID ACTIVE > 63: fastfs-OST003f_UUID ACTIVE > km_ior_shared.out > obdidx objid objid group > 0 16799778 0x1005822 0 > 1 16940072 0x1027c28 0 > 2 17167283 0x105f3b3 0 > 3 15988448 0xf3f6e0 0 > 4 16859503 0x101416f 0 > 5 17364548 0x108f644 0 > 6 16849540 0x1011a84 0 > 7 12546146 0xbf7062 0 > 8 17143798 0x10597f6 0 > 9 17196912 0x1066770 0 > 10 16916803 0x1022143 0 > 11 7991491 0x79f0c3 0 > 12 16118156 0xf5f18c 0 > 13 16984031 0x10327df 0 > 14 17412454 0x109b166 0 > 15 16599931 0xfd4b7b 0 > 16 16323314 0xf912f2 0 > 17 16664028 0xfe45dc 0 > 18 17292865 0x107de41 0 > 19 17480546 0x10abb62 0 > 20 16527238 0xfc2f86 0 > 21 16436520 0xfacd28 0 > 22 17084101 0x104aec5 0 > 23 15701612 0xef966c 0 > 24 17139117 0x10585ad 0 > 25 16889152 0x101b540 0 > 26 17344287 0x108a71f 0 > 27 17290338 0x107d462 0 > 28 16794234 0x100427a 0 > 29 16153907 0xf67d33 0 > 30 16540825 0xfc6499 0 > 31 17413630 0x109b5fe 0 > 32 13592810 0xcf68ea 0 > 33 16801687 0x1005f97 0 > 34 17519073 0x10b51e1 0 > 35 16898364 0x101d93c 0 > 36 17455291 0x10a58bb 0 > 37 16185055 0xf6f6df 0 > 38 13208545 0xc98be1 0 > 39 16651852 0xfe164c 0 > 40 16845326 0x1010a0e 0 > 41 16305051 0xf8cb9b 0 > 42 9322279 0x8e3f27 0 > 43 17295009 0x107e6a1 0 > 44 16681757 0xfe8b1d 0 > 45 16674804 0xfe6ff4 0 > 46 17476089 0x10aa9f9 0 > 47 16979487 0x103161f 0 > 48 15884086 0xf25f36 0 > 49 16325833 0xf91cc9 0 > 50 15216532 0xe82f94 0 > 51 14917514 0xe39f8a 0 > 52 17016478 0x103a69e 0 > 53 17102674 0x104f752 0 > 54 16167305 0xf6b189 0 > 55 16794224 0x1004270 0 > 56 17047378 0x1041f52 0 > 57 16873050 0x101765a 0 > 58 16293381 0xf89e05 0 > 59 17106865 0x10507b1 0 > 60 16704274 0xfee312 0 > 61 16704006 0xfee206 0 > 62 16554665 0xfc9aa9 0 > 63 16949690 0x102a1ba 0On 02/10/2012 04:31 PM, Hedges, Richard M. wrote:> Did you set up the shared file for striping? > >> hype356{rhedges}395: lfs help setstripe >> setstripe: Create a new file with a specific striping pattern or >> set the default striping pattern on an existing directory or >> delete the default striping pattern from an existing directory >> usage: setstripe [--size|-s stripe_size] [--count|-c stripe_count] >> [--index|-i|--offset|-o start_ost_index] >> [--pool|-p<pool>]<directory|filename> >> or >> setstripe -d<directory> (to delete default striping) >> stripe_size: Number of bytes on each OST (0 filesystem default) >> Can be specified with k, m or g (in KB, MB and GB >> respectively) >> start_ost_index: OST index of first stripe (-1 default) >> stripe_count: Number of OSTs to stripe over (0 default, -1 all) >> pool: Name of OST pool to use (default none) > > > On 2/10/12 2:27 PM, "Kshitij Mehta"<kmehta at cs.uh.edu> wrote: > >> We have lustre 1.6.7 configured using 64 OSTs. >> I am testing the performance using IOR, which is a file system benchmark. >> >> When I run IOR using mpi such that processes write to a shared file, >> performance does not scale. I tested with 1,2 and 4 processes, and the >> performance remains constant at 230 MBps. >> >> When processes write to separate files, performance improves greatly, >> reaching 475 MBps. >> >> Note that all processes are spawned on a single node. >> >> Here is the output: >> Writing to a shared file: >> >>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >>> /fastfs/gabriel/ss_64/km_ior.out >>> Machine: Linux deimos102 >>> >>> Summary: >>> api = POSIX >>> test filename = /fastfs/gabriel/ss_64/km_ior.out >>> access = single-shared-file >>> ordering in a file = sequential offsets >>> ordering inter file= no tasks offsets >>> clients = 4 (4 per node) >>> repetitions = 1 >>> xfersize = 32 MiB >>> blocksize = 2 GiB >>> aggregate filesize = 8 GiB >>> >>> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >>> (OPs) Mean (OPs) Std Dev Mean (s) >>> --------- --------- --------- ---------- ------- --------- >>> --------- ---------- ------- -------- >>> write 233.61 233.61 233.61 0.00 7.30 >>> 7.30 7.30 0.00 35.06771 EXCEL >>> >>> Max Write: 233.61 MiB/sec (244.95 MB/sec) >> Writing to separate files: >> >>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >>> /fastfs/gabriel/ss_64/km_ior.out -F >>> Machine: Linux deimos102 >>> >>> Summary: >>> api = POSIX >>> test filename = /fastfs/gabriel/ss_64/km_ior.out >>> access = file-per-process >>> ordering in a file = sequential offsets >>> ordering inter file= no tasks offsets >>> clients = 4 (4 per node) >>> repetitions = 1 >>> xfersize = 32 MiB >>> blocksize = 2 GiB >>> aggregate filesize = 8 GiB >>> >>> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >>> (OPs) Mean (OPs) Std Dev Mean (s) >>> --------- --------- --------- ---------- ------- --------- >>> --------- ---------- ------- -------- >>> write 475.95 475.95 475.95 0.00 14.87 >>> 14.87 14.87 0.00 17.21191 EXCEL >>> >>> Max Write: 475.95 MiB/sec (499.07 MB/sec) >> I am trying to understand where the bottleneck is, when processes write >> to a shared file. >> Your help is appreciated. > > ===================================================> > Richard Hedges > Customer Support and Test - File Systems Project > Development Environment Group - Livermore Computing > Lawrence Livermore National Laboratory > 7000 East Avenue, MS L-557 > Livermore, CA 94551 > > v: (925) 423-2699 > f: (925) 423-6961 > E: richard-hedges at llnl.gov-- Kshitij Mehta PhD candidate Parallel Software Technologies Lab (pstl.cs.uh.edu) Dept. of Computer Science University of Houston Houston, Texas, USA
Carlos Thomaz
2012-Feb-10 23:26 UTC
[Lustre-discuss] IOR writing to a shared file, performance does not scale
Hi Kshitij Difficult to point put without knowing your exact configuration. Details like OSS configuration, storage and network details would help. The best to identify your bottleneck is to run other benchmarks to understand your raw performance and then compare it to what you get on top of lustre. You could use xdd to analyze your OSS raw performer and obdfilter to understand how your lustre backend is performing. However if your system is in production you won''t be able to do that since these benchmarks are destructive. What happens when you write, single client to a single OST, what figures are you getting? What about reads? Send us more information that we try to help you out what''s going on. Btw, how your network looks like? Rgds Carlos On Feb 10, 2012, at 4:28 PM, "Kshitij Mehta" <kmehta at cs.uh.edu> wrote:> We have lustre 1.6.7 configured using 64 OSTs. > I am testing the performance using IOR, which is a file system benchmark. > > When I run IOR using mpi such that processes write to a shared file, > performance does not scale. I tested with 1,2 and 4 processes, and the > performance remains constant at 230 MBps. > > When processes write to separate files, performance improves greatly, > reaching 475 MBps. > > Note that all processes are spawned on a single node. > > Here is the output: > Writing to a shared file: > >> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >> /fastfs/gabriel/ss_64/km_ior.out >> Machine: Linux deimos102 >> >> Summary: >> api = POSIX >> test filename = /fastfs/gabriel/ss_64/km_ior.out >> access = single-shared-file >> ordering in a file = sequential offsets >> ordering inter file= no tasks offsets >> clients = 4 (4 per node) >> repetitions = 1 >> xfersize = 32 MiB >> blocksize = 2 GiB >> aggregate filesize = 8 GiB >> >> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >> (OPs) Mean (OPs) Std Dev Mean (s) >> --------- --------- --------- ---------- ------- --------- >> --------- ---------- ------- -------- >> write 233.61 233.61 233.61 0.00 7.30 >> 7.30 7.30 0.00 35.06771 EXCEL >> >> Max Write: 233.61 MiB/sec (244.95 MB/sec) > > Writing to separate files: > >> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >> /fastfs/gabriel/ss_64/km_ior.out -F >> Machine: Linux deimos102 >> >> Summary: >> api = POSIX >> test filename = /fastfs/gabriel/ss_64/km_ior.out >> access = file-per-process >> ordering in a file = sequential offsets >> ordering inter file= no tasks offsets >> clients = 4 (4 per node) >> repetitions = 1 >> xfersize = 32 MiB >> blocksize = 2 GiB >> aggregate filesize = 8 GiB >> >> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >> (OPs) Mean (OPs) Std Dev Mean (s) >> --------- --------- --------- ---------- ------- --------- >> --------- ---------- ------- -------- >> write 475.95 475.95 475.95 0.00 14.87 >> 14.87 14.87 0.00 17.21191 EXCEL >> >> Max Write: 475.95 MiB/sec (499.07 MB/sec) > > I am trying to understand where the bottleneck is, when processes write > to a shared file. > Your help is appreciated. > > -- > Kshitij Mehta > PhD candidate > Parallel Software Technologies Lab (pstl.cs.uh.edu) > Dept. of Computer Science > University of Houston > Houston, Texas, USA > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Michael Kluge
2012-Feb-11 07:29 UTC
[Lustre-discuss] IOR writing to a shared file, performance does not scale
Hi Kshitij, I would recommend to run sgpdd-survey on the servers for one and for multiple disks and then obdfilter-survey. Then you know what your storage can deliver. Then you could do lnet tests as well to see wether the network works fine. If the disks and the network deliver the expected performance, IOR will most probably run with good performance as well. Please see: http://wiki.lustre.org/images/4/40/Wednesday_shpc-2009-benchmarking.pdf Regards, Michael On 10.02.2012 23:27, Kshitij Mehta wrote:> We have lustre 1.6.7 configured using 64 OSTs. > I am testing the performance using IOR, which is a file system benchmark. > > When I run IOR using mpi such that processes write to a shared file, > performance does not scale. I tested with 1,2 and 4 processes, and the > performance remains constant at 230 MBps. > > When processes write to separate files, performance improves greatly, > reaching 475 MBps. > > Note that all processes are spawned on a single node. > > Here is the output: > Writing to a shared file: > >> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >> /fastfs/gabriel/ss_64/km_ior.out >> Machine: Linux deimos102 >> >> Summary: >> api = POSIX >> test filename = /fastfs/gabriel/ss_64/km_ior.out >> access = single-shared-file >> ordering in a file = sequential offsets >> ordering inter file= no tasks offsets >> clients = 4 (4 per node) >> repetitions = 1 >> xfersize = 32 MiB >> blocksize = 2 GiB >> aggregate filesize = 8 GiB >> >> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >> (OPs) Mean (OPs) Std Dev Mean (s) >> --------- --------- --------- ---------- ------- --------- >> --------- ---------- ------- -------- >> write 233.61 233.61 233.61 0.00 7.30 >> 7.30 7.30 0.00 35.06771 EXCEL >> >> Max Write: 233.61 MiB/sec (244.95 MB/sec) > > Writing to separate files: > >> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >> /fastfs/gabriel/ss_64/km_ior.out -F >> Machine: Linux deimos102 >> >> Summary: >> api = POSIX >> test filename = /fastfs/gabriel/ss_64/km_ior.out >> access = file-per-process >> ordering in a file = sequential offsets >> ordering inter file= no tasks offsets >> clients = 4 (4 per node) >> repetitions = 1 >> xfersize = 32 MiB >> blocksize = 2 GiB >> aggregate filesize = 8 GiB >> >> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >> (OPs) Mean (OPs) Std Dev Mean (s) >> --------- --------- --------- ---------- ------- --------- >> --------- ---------- ------- -------- >> write 475.95 475.95 475.95 0.00 14.87 >> 14.87 14.87 0.00 17.21191 EXCEL >> >> Max Write: 475.95 MiB/sec (499.07 MB/sec) > > I am trying to understand where the bottleneck is, when processes write > to a shared file. > Your help is appreciated. >-- Dr.-Ing. Michael Kluge Technische Universit?t Dresden Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden Germany Contact: Willersbau, Room WIL A 208 Phone: (+49) 351 463-34217 Fax: (+49) 351 463-37773 e-mail: michael.kluge at tu-dresden.de WWW: http://www.tu-dresden.de/zih
Kshitij Mehta
2012-Feb-13 20:53 UTC
[Lustre-discuss] IOR writing to a shared file, performance does not scale
> Details like OSS configuration, storage and network details would help38 TByte Lustre filesystem, exported by 11 servers via Infiniband (64 OSTs)> What happens when you write, single client to a single OST, what figures are you getting?~ 280 MBps. I see similar results when I use multiple processes (from the same node) writing to a shared file.> how your network looks like?I/O network 4x Infiniband with up to 1 GiByte/s pro Link (192 Port Voltaire ISR 9288 IB-Switch) Do let me know if I can provide more details. Thanks, On 02/10/2012 05:26 PM, Carlos Thomaz wrote:> Hi Kshitij > > Difficult to point put without knowing your exact configuration. Details like OSS configuration, storage and network details would help. > > The best to identify your bottleneck is to run other benchmarks to understand your raw performance and then compare it to what you get on top of lustre. You could use xdd to analyze your OSS raw performer and obdfilter to understand how your lustre backend is performing. However if your system is in production you won''t be able to do that since these benchmarks are destructive. > > What happens when you write, single client to a single OST, what figures are you getting? What about reads? > > Send us more information that we try to help you out what''s going on. > > Btw, how your network looks like? > > Rgds > Carlos > > On Feb 10, 2012, at 4:28 PM, "Kshitij Mehta"<kmehta at cs.uh.edu> wrote: > >> We have lustre 1.6.7 configured using 64 OSTs. >> I am testing the performance using IOR, which is a file system benchmark. >> >> When I run IOR using mpi such that processes write to a shared file, >> performance does not scale. I tested with 1,2 and 4 processes, and the >> performance remains constant at 230 MBps. >> >> When processes write to separate files, performance improves greatly, >> reaching 475 MBps. >> >> Note that all processes are spawned on a single node. >> >> Here is the output: >> Writing to a shared file: >> >>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >>> /fastfs/gabriel/ss_64/km_ior.out >>> Machine: Linux deimos102 >>> >>> Summary: >>> api = POSIX >>> test filename = /fastfs/gabriel/ss_64/km_ior.out >>> access = single-shared-file >>> ordering in a file = sequential offsets >>> ordering inter file= no tasks offsets >>> clients = 4 (4 per node) >>> repetitions = 1 >>> xfersize = 32 MiB >>> blocksize = 2 GiB >>> aggregate filesize = 8 GiB >>> >>> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >>> (OPs) Mean (OPs) Std Dev Mean (s) >>> --------- --------- --------- ---------- ------- --------- >>> --------- ---------- ------- -------- >>> write 233.61 233.61 233.61 0.00 7.30 >>> 7.30 7.30 0.00 35.06771 EXCEL >>> >>> Max Write: 233.61 MiB/sec (244.95 MB/sec) >> Writing to separate files: >> >>> Command line used: ./IOR -a POSIX -b 2g -e -t 32m -w -o >>> /fastfs/gabriel/ss_64/km_ior.out -F >>> Machine: Linux deimos102 >>> >>> Summary: >>> api = POSIX >>> test filename = /fastfs/gabriel/ss_64/km_ior.out >>> access = file-per-process >>> ordering in a file = sequential offsets >>> ordering inter file= no tasks offsets >>> clients = 4 (4 per node) >>> repetitions = 1 >>> xfersize = 32 MiB >>> blocksize = 2 GiB >>> aggregate filesize = 8 GiB >>> >>> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min >>> (OPs) Mean (OPs) Std Dev Mean (s) >>> --------- --------- --------- ---------- ------- --------- >>> --------- ---------- ------- -------- >>> write 475.95 475.95 475.95 0.00 14.87 >>> 14.87 14.87 0.00 17.21191 EXCEL >>> >>> Max Write: 475.95 MiB/sec (499.07 MB/sec) >> I am trying to understand where the bottleneck is, when processes write >> to a shared file. >> Your help is appreciated. >> >> -- >> Kshitij Mehta >> PhD candidate >> Parallel Software Technologies Lab (pstl.cs.uh.edu) >> Dept. of Computer Science >> University of Houston >> Houston, Texas, USA >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss-- Kshitij Mehta PhD Candidate Parallel Software Technologies Lab (http://pstl.cs.uh.edu) Dept. of Computer Science University of Houston Texas, USA