Hi, I wonder if LNET doesn''t have an atomic operation that unlinks the packet from the delivery process upon receiving a packet. Iirc it does. Why wouldn''t one use that? Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-devel/attachments/20091209/27c0c4d8/attachment.html
2009/12/9 Peter Braam <peter.braam at clusterstor.com>> Hi, > > hi Peter,> I wonder if LNET doesn''t have an atomic operation that unlinks the packet > from the delivery process upon receiving a packet. Iirc it does. Why > wouldn''t one use that? >That is probably because it still may be in use by upper layers (ptlrpc, etc), i.e. it is up to upper layer to decide should its buffers be still linked for access or not. Thanks.> > Peter > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > >-- umka -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-devel/attachments/20091215/fd7d522c/attachment.html
No - buffers can and probably should (because, for example, re-delivery may leaver buffer state undefined) be unlinked from the match list before passing the buffer up to any other layer. Portals/LNET certainly can do this. The question is why isn''t this done, Eric, Isaac, Andreas? Peter On Tue, Dec 15, 2009 at 9:28 AM, Yuriy Umanets < yuriy.umanets at clusterstor.com> wrote:> > > 2009/12/9 Peter Braam <peter.braam at clusterstor.com> > >> Hi, >> >> hi Peter, > > >> I wonder if LNET doesn''t have an atomic operation that unlinks the packet >> from the delivery process upon receiving a packet. Iirc it does. Why >> wouldn''t one use that? >> > That is probably because it still may be in use by upper layers (ptlrpc, > etc), i.e. it is up to upper layer to decide should its buffers be still > linked for access or not. > > Thanks. > >> >> Peter >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> > > > -- > umka >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-devel/attachments/20091215/21a6c9bb/attachment-0001.html
On Tue, Dec 15, 2009 at 07:01:33PM -0700, Peter Braam wrote:> No - buffers can and probably should (because, for example, re-delivery > may leaver buffer state undefined) be unlinked from the match list > before passing the buffer up to any other layer. Portals/LNET > certainly can do this.Hi Peter, LNet does have a MD option to automatically unlink the buffer once it''s been exhausted. Duplicated delivery shouldn''t cause any problem because MD offset is increased locally (i.e. a dup would go into a different offset), unless the MD has enabled peers to manage offset remotely, which is now true for PTLRPC reply buffers (because now early replies and real reply share a same reply buffer). Isaac
On Tue, Dec 15, 2009 at 09:48:18PM -0500, Isaac Huang wrote:> On Tue, Dec 15, 2009 at 07:01:33PM -0700, Peter Braam wrote: > > No - buffers can and probably should (because, for example, re-delivery > > may leaver buffer state undefined) be unlinked from the match list > > before passing the buffer up to any other layer. Portals/LNET > > certainly can do this. > > Hi Peter, LNet does have a MD option to automatically unlink the > buffer once it''s been exhausted. Duplicated delivery shouldn''t cause any > problem because MD offset is increased locally (i.e. a dup would go > into a different offset), unless the MD has enabled peers to manage > offset remotely, which is now true for PTLRPC reply buffers (because > now early replies and real reply share a same reply buffer).There was a recent discussion on this topic, under a similar subject: http://lists.lustre.org/pipermail/lustre-devel/2009-December/003231.html Isaac
Peter,
It depends on how buffers are attached to portal, ptlrpc has three cases
of this:
1. If buffer is for incoming request (not for RDMA, so it''s not unique
match)
the MD is constructed with lnet_md_t::threshold = LNET_MD_THRESH_INF,
so it will be unlinked only for two cases:
a) user call LNetMDUnlink explicitly
b) the buffer is exhausted (lnet_md_exhausted() return 1 in LNet)
Once LNet get a request for that buffer, it will change offset of
the buffer so
it wouldn''t be overwritten by the next match, it''s possible
that upper
layer is handing
the request, and the buffer is still on match list
2. If buffer is for bulk (it''s unique)
the MD is constructed with lnet_md_t::threshold = 1, and
LNetMDAttach(...LNET_UNLINK...)
so LNet will auto-unlink the buffer from match list once it got a
match, I think
this is the case you want
3. If the buffer is for reply (it''s unique as well)
the MD is constructed with lnet_md_t::threshold = LNET_MD_THRESH_INF,
so it''s same as 1), the buffer wouldn''t be unlinked unless
user call
LNetMDUnlink
or LNetMEUnlink, so safety of reply buffer can be guaranteed only if user is
sure the buffer is unlinked.
Regards
Liang
Peter Braam wrote:> No - buffers can and probably should (because, for example,
> re-delivery may leaver buffer state undefined) be unlinked from the
> match list before passing the buffer up to any other layer.
> Portals/LNET certainly can do this.
>
> The question is why isn''t this done, Eric, Isaac, Andreas?
>
> Peter
>
> On Tue, Dec 15, 2009 at 9:28 AM, Yuriy Umanets
> <yuriy.umanets at clusterstor.com <mailto:yuriy.umanets at
clusterstor.com>>
> wrote:
>
>
>
> 2009/12/9 Peter Braam <peter.braam at clusterstor.com
> <mailto:peter.braam at clusterstor.com>>
>
> Hi,
>
> hi Peter,
>
>
> I wonder if LNET doesn''t have an atomic operation that
unlinks
> the packet from the delivery process upon receiving a packet.
> Iirc it does. Why wouldn''t one use that?
>
> That is probably because it still may be in use by upper layers
> (ptlrpc, etc), i.e. it is up to upper layer to decide should its
> buffers be still linked for access or not.
>
> Thanks.
>
>
> Peter
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> <mailto:Lustre-devel at lists.lustre.org>
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
>
>
>
> --
> umka
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>