Hi Bruce--
For future reference, when things go wrong there are usually many
messages on the console (in dmesg, or /var/log/messages) which can be
very helpful. If you include these in your bug reports, we may be able
to help you without requiring another iteration.
Bruce V. Schwartz wrote:>
> -- In a single node cluster (MDS, OST, and OSC on the same box)
I''ve been
> unable to run either Bonnie or iozone benchmarks to completion.
>
> Running Bonnie (which running many times slower than on a local ext3 file
> system) invariably generates errors during the multi-threaded
> random-seek/read phase of this form:
> Bonnie: drastic I/O error (read in doseek): Input/output error
> Bonnie: drastic I/O error (read in doseek): Input/output error
> Bonnie: drastic I/O error (pipe read): No such file or directory
> These errors result when the read() fails.
The case of running everything on one node -- specifically a client and
an OSS -- is not 100% robust yet. 1.0.1 already contains some
improvements, but anyone interested in this configuration is going to
need a fix for bug 2412.
We will make a new release available as soon as that bug is fixed.
Although it is not a common case for our big customers, it is of course
the first thing that everyone tries.
> Running iozone I ended up with a very unhappy file system. Whatever I do
> generates an "input/output error". e.g. in the /mnt/lustre
directory:
> # ls
> ls: output.wks: Input/output error
> ls: yy: Input/output error
> ls: x: Input/output error
> # touch yy
> touch: setting times of `yy'': Input/output error
> # cat x
> cat: x: Input/output error
Whenever anything goes wrong with a Lustre node, it usually manifests
itself as a timeout -- you can see these timeout messages on the
consoles. This is Lustre''s main mechanism for deciding that a node is
misbehaving; either a network RPC does not complete in a timely fashion,
or a client does not give up a lock, or something along those lines.
In a default Lustre configuration, if a client sees a timeout to an OST,
it will go into a fail-out mode. Until it is reconnected, it will
return errors when you try to access data stored on that OST.
It is not difficult to reconnect your client. First, find the device
associated with the failure. If you run "lctl device_list" you will
see
something like:
# lctl device_list
0 UP obdfilter OST_localhost OST_localhost_UUID 2
1 UP ost OSS OSS_UUID 2
2 UP mdt MDT MDT_UUID 2
3 UP mds mds1 mds1_UUID 2
4 UP osc OSC_localhost_OST_localhost_mds1
af8ce126-9401-42eb-8565-1c26228ad4e3 4
5 UP lov lov_mds1 af8ce126-9401-42eb-8565-1c26228ad4e3 4
6 UP osc OSC_localhost_OST_localhost_MNT_localhost 6fdef_lov1_ea7471fe7d 4
7 UP lov lov1 6fdef_lov1_ea7471fe7d 4
8 UP mdc MDC_localhost_mds1_MNT_localhost de062_MNT_localhost_1d7cc3d609 2
There are two object storage clients (OSCs):
#4 (OSC_localhost_OST_localhost_mds1) and
#6 (OSC_localhost_OST_localhost_MNT_localhost)
#4 is for the MDS, so you want #6. If you run "lctl --device 6
recover"
it should return fairly quickly, and your file system should work again.
We have wanted for a long time for this cumbersome process to be
automated. In the meantime, we have a script to do the job for you; I
will see about including that in the 1.0.2 packages.
> -- I haven''t been able to bring up a system without an LUV. Doing
so always
> results in this error when running with lconf -v:
In Lustre 1.x, a LOV is required, even for a single OSC configuration.
> -- It is quite difficult to bring up more complicated systems. For
example,
> if you want to run OSTs and OSCs on the same nodes it appears that you have
> to use the --minlevel and --maxlevel commands to lconf to bring things up
in
> a staged fashion. Usually I get errors on the last stage with --minlevel
65
> --maxlevel 75
It should not be necessary to stage configuration like this. Can you
send the list of lmc and lconf commands that you used to build the XML
and start your system? If there is a bug here, of course we want to fix
it. Also, there is probably a more useful message on the console about
exactly what went wrong.
Thanks for your patience! Your feedback is very much appreciated.
-Phil