Lustre discuss - May 2006 - Server and Client on the same machine

Would decreasing the max_cached_mb value in the client help
to prevent deadlock?  By default it''s set very high.
paul

Andreas Dilger wrote:
>On Dec 08, 2005  11:59 -0500, Brent A Nelson wrote:
>  
>
>>Note to moderator: please do not duplicate.
>>    
>>
>
>What does that mean, exactly?
>
>  
>
>>I noticed that the download page mentions that having a client on the
same
>>machine as a Lustre server is not 100% stable.  Could anyone elaborate?
I
>>would think a lot (most?) of the novel uses of Lustre would involve a 
>>Lustre server also having access to the Lustre filesystem...
>>    
>>
>
>Issue is as follows:
>- node is very short of memory
>- ''client'' is forced by VM to write dirty cache to free
memory
>- ''client'' does RPC to ''server'' (local)
>- ''server'' calls into VM to allocate memory to complete
request
>- VM is blocked, waiting for ''client'' to free memory
>
>We have worked to avoid allocations on the server during writes, but
>this is not entirely possible to fix 100%.  It is also not easily
>possible to set VM flags to avoid this recursion/dependency, since both
>bits of code are running in "filesystem" context so need to
allocate
>with the same GFP_NOFS flags.
>
>That said, we do a lot of our testing with client+server on same node
>without problem.  Warning is just to indicate that this is not yet
>bulletproof.  If this is a desirable operating mode for you I would say
>to give it a try in your environment+workload and it will likely work
>just fine.
>
>Cheers, Andreas
>--
>Andreas Dilger
>Principal Software Engineer
>Cluster File Systems, Inc.
>
>_______________________________________________
>Lustre-discuss mailing list
>Lustre-discuss@lists.clusterfs.com
>https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
>  
>

On Dec 08, 2005  11:59 -0500, Brent A Nelson wrote:
> Note to moderator: please do not duplicate.
What does that mean, exactly?
> I noticed that the download page mentions that having a client on the same 
> machine as a Lustre server is not 100% stable.  Could anyone elaborate? I 
> would think a lot (most?) of the novel uses of Lustre would involve a 
> Lustre server also having access to the Lustre filesystem...
Issue is as follows:
- node is very short of memory
- ''client'' is forced by VM to write dirty cache to free memory
- ''client'' does RPC to ''server'' (local)
- ''server'' calls into VM to allocate memory to complete
request
- VM is blocked, waiting for ''client'' to free memory

We have worked to avoid allocations on the server during writes, but
this is not entirely possible to fix 100%.  It is also not easily
possible to set VM flags to avoid this recursion/dependency, since both
bits of code are running in "filesystem" context so need to allocate
with the same GFP_NOFS flags.

That said, we do a lot of our testing with client+server on same node
without problem.  Warning is just to indicate that this is not yet
bulletproof.  If this is a desirable operating mode for you I would say
to give it a try in your environment+workload and it will likely work
just fine.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

On Dec 08, 2005  16:54 -0500, PAulN wrote:
> Would decreasing the max_cached_mb value in the client help
> to prevent deadlock?  By default it''s set very high.
There is a separate limit per OSC on the dirty data (32MB per OSC by
default), and the client will begin aggressively writing out dirty pages
as soon as they fill an RPC (1MB) instead of deferring the writeout until
there is VM pressure like most filesystems.

I agree that decreasing this limit would help somewhat, but the problem
is theoretically there in any case.  I honestly think that for normal
workloads and today''s RAM sizes that this is not a real issue (I have
not seen it myself in a long time).  The warning is there just as a
FYI/CYA so people are aware that this is not a very well tested configuration.

If someone wants to test in this environment I''d be happy to hear the
results and offer suggestions.  If it is to be a supported configuration
I think that is an issue for the customer and business folks to work out.

I do agree that running in this mode has a lot of appeal, but there are
drawbacks as well, such as what happens when a client node (holding part
of the filesystem data) fails?  There are generally many more clients
than storage servers, so the frequency of failure is correspondingly
higher.  They also do not generally have storage that is capable of
being shared between nodes, unless they are e.g. blades.

There are also optimizations that can be made to take advantage of
running in such an environment (e.g. clients prefer writing to new files
on their local storage).

The net result is that while the configuration may even work for
"normal" usage out-of-the-box today there is still some work to do
before considering it "complete".
> Andreas Dilger wrote:
> >On Dec 08, 2005  11:59 -0500, Brent A Nelson wrote:
> >>Note to moderator: please do not duplicate.
> >
> >What does that mean, exactly?
> >
> >>I noticed that the download page mentions that having a client on
the
> >>same machine as a Lustre server is not 100% stable.  Could anyone 
> >>elaborate? I would think a lot (most?) of the novel uses of Lustre
would
> >>involve a Lustre server also having access to the Lustre
filesystem...
> >
> >Issue is as follows:
> >- node is very short of memory
> >- ''client'' is forced by VM to write dirty cache to
free memory
> >- ''client'' does RPC to ''server''
(local)
> >- ''server'' calls into VM to allocate memory to
complete request
> >- VM is blocked, waiting for ''client'' to free memory
> >
> >We have worked to avoid allocations on the server during writes, but
> >this is not entirely possible to fix 100%.  It is also not easily
> >possible to set VM flags to avoid this recursion/dependency, since both
> >bits of code are running in "filesystem" context so need to
allocate
> >with the same GFP_NOFS flags.
> >
> >That said, we do a lot of our testing with client+server on same node
> >without problem.  Warning is just to indicate that this is not yet
> >bulletproof.  If this is a desirable operating mode for you I would say
> >to give it a try in your environment+workload and it will likely work
> >just fine.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

I suspected the warning was a bit overblown, but I just wanted to be sure. 
I''m hoping to develop a departmental server cluster, with Lustre being
at
the heart of it and enbd RAID1 network mirrors underneath.  I have 6 
machines to work with, and I''d like all of them to be pumping out
Lustre
to all of our Linux machines, as well as handling basic server tasks (e.g. 
Samba, EMail, web, etc.) in an LVS-style manner when possible or failover 
when not.  I didn''t really want to sacrifice a chunk of my little
cluster
as purely Luster servers (especially since all 6 machines have a good 
chunk of storage, most of which I''d like on Luster).

The moderator note was due to the fact that I sent my message a few days 
ago, but it was still held up in moderation.  I ended up subscribing to 
the list to avoid the delay.

Thanks,

Brent

PS Another quick question.  The FAQ mentions that NFS exports are 
currently extremely slow.  Roughly how slow would it be? Is this only for 
kernel NFS, not user-mode NFS? Our legacy UNIX systems won''t have much 
choice but to use NFS...

On Thu, 8 Dec 2005, Andreas Dilger wrote:
> On Dec 08, 2005  11:59 -0500, Brent A Nelson wrote:
>> Note to moderator: please do not duplicate.
>
> What does that mean, exactly?
>
>> I noticed that the download page mentions that having a client on the
same
>> machine as a Lustre server is not 100% stable.  Could anyone elaborate?
I
>> would think a lot (most?) of the novel uses of Lustre would involve a
>> Lustre server also having access to the Lustre filesystem...
>
> Issue is as follows:
> - node is very short of memory
> - ''client'' is forced by VM to write dirty cache to free
memory
> - ''client'' does RPC to ''server'' (local)
> - ''server'' calls into VM to allocate memory to complete
request
> - VM is blocked, waiting for ''client'' to free memory
>
> We have worked to avoid allocations on the server during writes, but
> this is not entirely possible to fix 100%.  It is also not easily
> possible to set VM flags to avoid this recursion/dependency, since both
> bits of code are running in "filesystem" context so need to
allocate
> with the same GFP_NOFS flags.
>
> That said, we do a lot of our testing with client+server on same node
> without problem.  Warning is just to indicate that this is not yet
> bulletproof.  If this is a desirable operating mode for you I would say
> to give it a try in your environment+workload and it will likely work
> just fine.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.
>
>

On Dec 08, 2005  18:05 -0500, Brent A Nelson wrote:
> The moderator note was due to the fact that I sent my message a few days 
> ago, but it was still held up in moderation.  I ended up subscribing to 
> the list to avoid the delay.
Hmm, I didn''t think the list was moderated at all anymore.  Maybe it
blocks posts from non-subscribers to avoid spam?
> PS Another quick question.  The FAQ mentions that NFS exports are 
> currently extremely slow.  Roughly how slow would it be? Is this only for 
> kernel NFS, not user-mode NFS? Our legacy UNIX systems won''t have
much
> choice but to use NFS...
knfsd export works for 2.4 lustre clients, but hasn''t been ported to
2.6
and is not an officially supported CFS feature.  For 2.6 it is possible
to use user-space NFS servers and Samba export.

As for "slow" that is a relative term.  When we first tested knfsd
export
it was doing about 35MB/s (vs 100MB/s for a native lustre client).  The
problem is that knfsd is opening and closing the file for every NFS RPC.
We''ve worked on a fix for this and knfsd export will likely be
available
in the 1.6 Lustre release.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Note to moderator: please do not duplicate.

I noticed that the download page mentions that having a client on the same 
machine as a Lustre server is not 100% stable.  Could anyone elaborate? I 
would think a lot (most?) of the novel uses of Lustre would involve a 
Lustre server also having access to the Lustre filesystem...

Many thanks,

Brent Nelson
Director of Computing
Dept. of Physics
University of Florida