I have a locking/fsync question.. I have an app that keeps job metadata in an XML file that resides on a Lustre filesystem (I actually just discovered my running system has it on NFS, but I''m seeing an anomaly on Lustre so I''ll keep writing). It uses libxml to read and write the file, and thus has to read the file into memory, make changes, and write back out. The approach I''m taking to this is: open file => fd lock fd (using fcntl F_SETLKW) read from fd ftruncate fd <make modifications> fsync fd unlock fd close fd The lustre system is a 4-OSS system and I''m running the test across 12 compute nodes, all of which have the fs mounted with the flock option (it falls over immediately without flock). I''m at lustre 1.6.6. What I''m seeing is that, occasionally, the file reads will pick up an empty or partial file. This doesn''t seem like it should be the case, but I''m sure I''m missing something. I don''t see any errors showing up on the MDS. thanks, --bob -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100208/b85663bb/attachment.html
Bah, of course there is a write of the new data to the file just before the fsync. The fd is also set for auto-flushing (this is a perl script). On Feb 8, 2010, at 3:45 PM, Robert Olson wrote:> I have a locking/fsync question.. > > I have an app that keeps job metadata in an XML file that resides on > a Lustre filesystem (I actually just discovered my running system > has it on NFS, but I''m seeing an anomaly on Lustre so I''ll keep > writing). > > It uses libxml to read and write the file, and thus has to read the > file into memory, make changes, and write back out. > > The approach I''m taking to this is: > > open file => fd > lock fd (using fcntl F_SETLKW) > read from fd > ftruncate fd > <make modifications> > fsync fd > unlock fd > close fd > > The lustre system is a 4-OSS system and I''m running the test across > 12 compute nodes, all of which have the fs mounted with the flock > option (it falls over immediately without flock). I''m at lustre 1.6.6. > > What I''m seeing is that, occasionally, the file reads will pick up > an empty or partial file. This doesn''t seem like it should be the > case, but I''m sure I''m missing something. I don''t see any errors > showing up on the MDS. > > thanks, > --bob > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100208/6279cf00/attachment.html
>>> On Mon, 8 Feb 2010 15:45:41 -0600, Robert Olson <olson at mcs.anl.gov> said:olson> [ ... ] job metadata in an XML file Sounds like that you are trying to implement a distributed Lustre backed queueing system. Good luck. olson> that resides on a Lustre filesystem (I actually just olson> discovered my running system has it on NFS, but I''m olson> seeing an anomaly on Lustre so I''ll keep writing). [ olson> ... ] all of which have the fs mounted with the flock olson> option (it falls over immediately without flock). I''m at olson> lustre 1.6.6. That mention of "NFS" here is worrying; if you are accessing a Lustre file via an NFS-Lustre proxy server I suspect you should be looking into NFS, and anyhow it may not be that reliable. olson> What I''m seeing is that, occasionally, the file reads olson> will pick up an empty or partial file. [ ... ] That seems to be the case as locking is supported in 1.6.6: http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007365.html http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007366.html
----- "Peter Grandi" <pg_lus at lus.for.sabi.co.UK> wrote:> >>> On Mon, 8 Feb 2010 15:45:41 -0600, Robert Olson > <olson at mcs.anl.gov> said: > > olson> [ ... ] job metadata in an XML file > > Sounds like that you are trying to implement a distributed > Lustre backed queueing system. Good luck.:-) this scheme is actually working very well for us; the current errors cropped up when I changed the job scheduling in a way that many nodes wrote to the metadata at the same time (I''d engineered around that case in the past).> > > That mention of "NFS" here is worrying; if you are accessing a > Lustre file via an NFS-Lustre proxy server I suspect you should > be looking into NFS, and anyhow it may not be that reliable.No - the NFS case is entirely separate and is straight NFS on a Linux server (which is giving me different problems relating to stale locking, etc). What I was describing was using a native Lustre mount. --bob