Intro: CFS is developing an app to simulate thousands of clients, as a
bug- and performance bottlneck-finding tool.
In its current form, this runs locally on a server against a single OST.
This is an ongoing discussion about its development.
Andreas Dilger wrote:> On Nov 17, 2006 13:53 -0800, Nathaniel Rutman wrote:
>
>> My load generator is currently a lightweight app that starts low-level
>> create/write/destroy threads using the echo client. It tries hard to
>> clean up everything nicely when it encounters an error. Only requires
>> an OST.
>>
>
> Sounds like a good start at least. Would the addition of separate
> operation modes like "connect only", "enqueue locks",
etc be straight
> forward?
>
connect, create/destroy, c/write/read/d would be easy. locks - locks
divorced
from these operations?>
>> Other potential designs include using liblustre clients instead (these
>> would be much bigger, but might more closely simulate real clients.)
>> I could probably include both modes as options.
>>
>
> The only reason we proposed liblustre clients is that it would be possible
> to start them, kill -STOP, and have the userspace part of the task be
> swapped if needed. The MDS/OST/MGS wouldn''t evict them for
inactivity
> because they are liblustre clients.
>
> I think once you are running thousands of your clients on a single node
> it will inevitably have problems because of task switching latency,
> swapping, etc. That said, if we can reasonably simulate some hundreds or
> thousand(s) of clients per node this should give us the ability to simulate
> any reasonable sized cluster.
>
Indeed. On my admittedly poor test machine, I could only get about 500
clients with
everyone writing full-throttle to the same disk before I started getting
timeouts.
Throttling them down, I''m up to 1900. (But I don''t sync the
requests, so
they still might all
hit at once.)
Which leads me to a Q: apparently the echo ioctl''s expect immediate
results (within lock
or obd timeouts (I should investigate)) but shouldn''t real client VFS
ops just block?
Say everybody is writing to 1 disk, faster than the disk can go. I
would not expect errors,
but instead slowed responses. Pings in this case would continue to be
returned in a timely
manner, so I would expect that both the client and the server would both
agree that nobody
needs to be reconnected or evicted. (hmm - now that I''m thinking about
it, maybe I shouldn''t have
set the max_rpcs_in_flight to 1, which I did to reduce grant space).
>
>> <pbojanic> nathan: do you know where to check this in, or do you
need to
>> check with product management group first?
>> <nathan> uh, well, right now it''s convenient to put
under lustre/utils/
>> - anywhere else will require slightly more effort. Maybe lustre/tests
>> would be better though...
>> <pbojanic> nathan: please check with architecture where it should
be
>> signed in (arch@)
>>
>
> Usually the lustre/utils directory is for user-oriented tools, while
> lustre/tests is for testing infrastructure not intended to be shipped
> to customers.
>
I''m linking against libptlctl so I have direct access to obd creation
(script overhead would become crazy with 1000''s of clients).
So that''s why lustre/utils. But in any case, I think the idea is that
this will become a customer test tool as well...