Hello, I have a production script that do read operations to a lot of small files. I read that one can gain performance boost with small files by using a loop device on top of Lustre. So a created 500 GB file striped across all of my OSTs(which are 8). I formatted the file with ext2 fs, and mounted it on a client. Just for the sake of testing a simple bash script finds all files with a given file type and cat the first 10 lines in /dev/null. When I run the script on the Lustre cluster I get : time sh test.sh real 1m16.804s user 0m2.539s sys 0m5.363s If I immediately re-run the script the time is : real 0m12.158s user 0m2.218s sys 0m5.430s There are 5406 files that meet the filetype criteria. When I run the script on the mounted loop device I get : real 2m30.177s user 0m2.290s sys 0m4.880s And immediate re-run gives me : real 0m7.810s user 0m2.187s sys 0m5.360s I`m usig lustre-2.4.0-2.6.32_358.6.2.el6_lustre.g230b174.x86_64_gd3f91c4. Also set all of "small files" optimizations like, no striping for the dirs containing the small files, max_dirty_mb=256, max_rpcs_in_flight=32, staahead=8192 and lnet.debug=0. Is it normal to get two times slower access times with the mounted loop device ?? _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 2013/08/25 6:39 AM, "Nikolay Kvetsinski" <nkvecinski-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:>Hello, I have a production script that do read operations to a lot of >small files. I read that one can gain performance boost with small files >by using a loop device on top of Lustre. So a created 500 GB file striped >across all of my OSTs(which > are 8). I formatted the file with ext2 fs, and mounted it on a client. >Just for the sake of testing a simple bash script finds all files with a >given file type and cat the first 10 lines in /dev/null. > > >When I run the script on the Lustre cluster I get : > > >time sh test.sh > > >real 1m16.804s >user 0m2.539s >sys 0m5.363s > > > >If I immediately re-run the script the time is : > > >real 0m12.158s >user 0m2.218s >sys 0m5.430s > > > > >There are 5406 files that meet the filetype criteria. > > >When I run the script on the mounted loop device I get : > > >real 2m30.177s >user 0m2.290s >sys 0m4.880s > >And immediate re-run gives me : > >real 0m7.810s >user 0m2.187s >sys 0m5.360s > > >I`m usig lustre-2.4.0-2.6.32_358.6.2.el6_lustre.g230b174.x86_64_gd3f91c4. >Also >set all of "small files" optimizations like, no striping for the dirs >containing the small files, max_dirty_mb=256, max_rpcs_in_flight=32, >staahead=8192 and lnet.debug=0. >Is it normal to get two times slower access times with the mounted loop >device ??It depends on how the loop device is doing IO on the underlying objects. It may be that ext2 isn''t the best filesystem for this. You could try formatting it with: mke2fs -t ext4 -O ^journal {device} which will enable the flex_bg,extents and other ext4 features but disables the journal (which I assume you don''t need because you are formatting as ext2 originally). You should also mount with "-t ext4". The flex_bg and mballoc features of ext4 may help improve the IO going to the back-end storage and improve the performance when running over loop devices. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division
Thanks mate, I`ll try it right away. On Mon, Aug 26, 2013 at 9:11 PM, Dilger, Andreas <andreas.dilger-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>wrote:> On 2013/08/25 6:39 AM, "Nikolay Kvetsinski" <nkvecinski-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > >Hello, I have a production script that do read operations to a lot of > >small files. I read that one can gain performance boost with small files > >by using a loop device on top of Lustre. So a created 500 GB file striped > >across all of my OSTs(which > > are 8). I formatted the file with ext2 fs, and mounted it on a client. > >Just for the sake of testing a simple bash script finds all files with a > >given file type and cat the first 10 lines in /dev/null. > > > > > >When I run the script on the Lustre cluster I get : > > > > > >time sh test.sh > > > > > >real 1m16.804s > >user 0m2.539s > >sys 0m5.363s > > > > > > > >If I immediately re-run the script the time is : > > > > > >real 0m12.158s > >user 0m2.218s > >sys 0m5.430s > > > > > > > > > >There are 5406 files that meet the filetype criteria. > > > > > >When I run the script on the mounted loop device I get : > > > > > >real 2m30.177s > >user 0m2.290s > >sys 0m4.880s > > > >And immediate re-run gives me : > > > >real 0m7.810s > >user 0m2.187s > >sys 0m5.360s > > > > > >I`m usig lustre-2.4.0-2.6.32_358.6.2.el6_lustre.g230b174.x86_64_gd3f91c4. > >Also > >set all of "small files" optimizations like, no striping for the dirs > >containing the small files, max_dirty_mb=256, max_rpcs_in_flight=32, > >staahead=8192 and lnet.debug=0. > >Is it normal to get two times slower access times with the mounted loop > >device ?? > > It depends on how the loop device is doing IO on the underlying objects. > It may > be that ext2 isn''t the best filesystem for this. You could try formatting > it with: > > mke2fs -t ext4 -O ^journal {device} > > which will enable the flex_bg,extents and other ext4 features but disables > the > journal (which I assume you don''t need because you are formatting as ext2 > originally). You should also mount with "-t ext4". > > The flex_bg and mballoc features of ext4 may help improve the IO going to > the > back-end storage and improve the performance when running over loop > devices. > > Cheers, Andreas > -- > Andreas Dilger > > Lustre Software Architect > Intel High Performance Data Division > > >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss