Lustre discuss - Jan 2010 - [mpich-discuss] adio lustre patch

On Mon, Jan 04, 2010 at 11:09:28AM -0700, Martin Pokorny wrote:
> I took a look at the ADIO Lustre driver last week in an attempt to make
> collective I/O work for my application. I believe that I''ve gotten
the
> driver to do the right thing now. With the following patch all of the
> tests under romio/test pass in my environment. My application, which
> requires reasonably complex file views, also appears to write correct
> files now on a Lustre filesystem using collective I/O.
> Problems in the noncontig and i_noncontig tests are fixed by changes to
> ad_lustre_wrstr.c. Basically, I made changes to that file so that it now
> more closely matches adio/common/ad_write_str.c. The only differences in
> these files should now be in the Lustre-specific parts. Problems in the
> large_array test are fixed in ad_lustre_aggregate.c and
> ad_lustre_wrcoll.c. The major problem was in the handling of the
> "buf_idx" arrays, which were not correctly tracking the data out
of a
> process''s data buffer being sent to multiple other processes. My
tests
> involved four MPI nodes, four Lustre OSTs and both four and eight MPI
> processes.
> I also patched ad_lustre_aggregate in its calculation of
> "avail_cb_nodes" as previously provided by Pascal Deveze. I
don''t know
> whether that has had any effect on my changes, since I did no testing
> without Pascal''s changes.
> There are some other changes, such as formatting and the addition of
> some assertions. They are included in the attached patch, but I could
> take them out if desired.
> The patch should apply cleanly to sources out of a fresh mpich2-1.2.1
> tarball. I''m not an old hand at MPI, so please review my patch
carefully.
Hi, I will review your modifications these next days. On my side, I also
made some
debugging and enhancements. I have some time to help to make the adio
lustre
driver run correctly.

Pascal

Hi,

I have checked the patch proposed  by Martin Pokorny. To me, all seem to
be very judicious. (On my part, I had another solution in test, but I will
adopt
that solution from Martin). For now on, I cannot see any programm that does
not run
correctly.

Now, my two pence contribution one that patch, on the file
ad_lustre_wrstr.c:

- A few lines before the end of the file, just after the call to
ADIO_WriteContig(...),
   if the return code is not success, free the buffer writebuf :

   if (*error_code != MPI_SUCCESS) {
                        ADIOI_Free(writebuf);
                        return;
   }

-  There are two calls to memset() that could be removed or put in comments
    (for performance reason).

Concerning the calculation of "avail_cb_nodes" in
ad_lustre_aggregate.c, I
noticed that my solution (adopted by Martin) and the new patch from LiuYing
are not fully acceptable if the values of nprocs_for_coll and CO increase
(we are
supposed to deal with petaflopic clusters here).
I will propose a new solution in a new mail to the lists, I hope today.

Pascal