Lustre devel - Jan 2007 - [Bug 10958] CSV graph generation for IOkit

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Created an attachment (id=9332)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9332&action=view)
scjody''s summary file

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



(In reply to comment #30)
> scjody, you were using an old version of sgpdd survey; the new one puts a
title
> of machine name/date as the first line in the summary file, so maybe we
don''t
> need the user specified title.
I want to say something useful about the survey, e.g. "1 OST, writeback
cache
disabled."  Coming to think of it, the titles for read and write graphs may
be
completely different, so I either need to be able to specify _two_ titles per
plot-sgpdd run, or tell plot-sgpdd to only consider writes or reads on this run
(at which point I could simply run it twice.)

Also note that the type of test(s) are specified by the user.  The sane choices
are "read", "write", "read write" and "write
read" but one could also specify
"write write read read write" or similar.  I won''t be
disappointed if you choose
not to handle that case, but please try and print a meaningful error message :)

Take a look at the Excel spreadsheets I posted to lustre-discuss a few weeks ago
if you want to see what I''m currently doing.  I''d like to
replace the excel
spreadsheet by a series of graphs (in PNG or whatever), the .summary files, plus
a brief text file describing the tests and settings.  So basically my
requirement is that plot-sgpdd be good enough to produce the graphs currently
made by Excel and save each one to its own file.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Created an attachment (id=9333)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9333&action=view)
excel graphs

Jitenrda, note the titles and the log scale on the X axis.
Can gnuplot make a 2-line title, e.g.:
sgpdd survey on host <hostname> dev <devlist> <date>
RSIZE=1M, 8 devices, chunk=256K

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Created an attachment (id=9362)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9362&action=view)
plot-sgpdd: improved as per scjody''s comments

did the following changes,
1. handle "read", "write", "write read" conditions
for individually plotting
graph for each of them.
2. It can handle multiple line graph title. The default is the first line of
log
and the additional is the string in command line (Please see syntax at the
begining). The graph title format that scjody mentioned can also be captured
from log, But its again appear hardcoded format in .scr file.
3. error handling and some minor changes.

This drop still doesn''t meet the exact requirement of scjody for
different rsz
values. there are two reasons behind that:
 - The script is designed to operate on the single output of sgpdd-survey.pl
and
AFAIK rsz is fixed for single run.(Please correct me)
 - It will create larger number of .dat and .scr files

scjody, let me know if you want me to develop a script based on plot-sgpdd to
solve the problem you are working on.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Hi Jitendra,

Thanks for the changes so far.

Yes, sgpdd-survey can produce single runs with many values of rsz.  Please take
a look at this script - you can specify rszlo and rszhi.  If they''re
the same,
only one rsz value will be used, but if not, it will increase exponentially as
with the other parameters.

I really need a tool that can produce separate graphs for each rsz, and separate
graphs for read and write (with different titles for read and write.)

Also, I need it to be able to produce PNG files or something similar for easy
viewing by others.

A few more minor things: please add a --help (see ''perldoc
Getopt::Long'' for one
way to do this) and the script produces fairly bad output when gnuplot is not
installed:

$ ~/cfs/plot-sgpdd sgpdd_survey_2006-12-06\@16\:45_ornl.summary 
Plotting Wed Dec 6 16:45:09 EST 2006 sgpdd-survey on /dev/sdb
Press RETURN to exitgnuplot does not exists, Please install it and try again.
$

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #9362 is|0                           |1
           obsolete|                            |


Created an attachment (id=9370)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9370&action=view)
plot-sgpdd: improved as per scjody''s comments #35

Jody, 
I did the modifications as per your suggestions in comment #35.
this script now generates multiple .scr and .dat files for each set of rsz
output.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Created an attachment (id=9381)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9381&action=view)
one summary file used by scjody

Hi Nathan,

This one''s fairly typical.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #9370 is|0                           |1
           obsolete|                            |


Created an attachment (id=9387)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9387&action=view)
plot-sgpdd.patch: improved as per nathan''s comment #42

did the improvements suggested by nathan in comment #42

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



(In reply to comment #44)
> comment #42">Created an attachment (id=9387)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 -->
(https://bugzilla.lustre.org/attachment.cgi?id=9387&action=view)
> plot-sgpdd.patch: improved as per nathan''s comment #42
> 
> did the improvements suggested by nathan in comment #42      
So close!
I think you must not understand what I mean by this:
>1. separate graphs for read and write tests with separate titles (just
prepend
>"read" or "write" to the second line of the title.)
Two different graphs, one "read" and one "write".  I
don''t care if you have a
combined .dat file or separate ones.  

Other than that, it''s great.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



(In reply to comment #45)
> Two different graphs, one "read" and one "write".  I
don''t care if you have a
> combined .dat file or separate ones.  
and different titles for each graph, to be specified by the user on the command
line.  Please :)

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #9387 is|0                           |1
           obsolete|                            |


Created an attachment (id=9403)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9403&action=view)
plot-sgpdd.patch: improved as per nathan and jody''s suggestions

did the improvements for separate read and wtite graphs with user specified
titles.
also assigned lagarithmic scale to x axis.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Hi Jitendra,

This is almost complete enough to use instead of excel :)  A few more things:

1. The lines for some series are wrong.  See, for example, the "Read, Rsize
1024" graph from the summary file I attached.  read16 is wrong, along with
a few
others.  If you want to see what I mean, take a look at the latest .xls
spreadsheet I posted to lustre-discuss.  This is the "Rsize=1M, 2
tiers" chart
on the right hand side of the "Read Charts" tab.

2. A white background seems to be more readable - can you drop the
''transparent''
flag to ''set terminal''?

3. For the y (Speed) axis: start it at 0.

4. Can we have a command line argument specifying the end of the y (Speed) axis?
 If it''s not present, do it automatically like now.

5. Can you add a line across the graph at each "tick" on the y axis?

6. I''d really like a similar tool for obdfilter-survey results. 
They''re really
similar in format.  Would that be possible?  It might even be worth extending
this code to cover both formats...

Thanks,
Jody

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Created an attachment (id=9408)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9408&action=view)
plot-sgpdd.patch: few more improvements

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



(In reply to comment #26)
> Created an attachment (id=9408)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 -->
(https://bugzilla.lustre.org/attachment.cgi?id=9408&action=view)
> plot-sgpdd.patch: few more improvements
>       
I''ve signed in this plus a few more fixes.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



Created an attachment (id=9432)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9432&action=view)
results from an obdfilter-survey run

I''m attaching results from an obdfilter-survey run.  Note that there
are _3_
types of test here: write, rewrite, and read.  They can appear in any order
except that any "read" must be preceded by at least one
"write".

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958



There are 2 more types of test, usually not used but they must be supported:
reread
rewrite_again

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pjones@clusterfs.com


 CSV file and graph generation for IOKit:
        There are scripte for csv file and graph generation for each 
tool in lustre-iiokit. It mainly includes following tools,
        -parse-ior.pl (plots graph for the output of ior-survey.pl)
        -plot-ost.pl  (plots graph for the output of ost-survey.pl)
        -plot-sgpdd   (plots graph for the output of sgpdd-survey.pl)
        
The information related to each script is inside the scripts itself.
all scripts usages gnuplot for graph plotting.
        
        For feedback on this tool please visit,
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=10958