Hi folks I followed the directions (http://wiki.lustre.org/index.php/Building_and_Installing_Lustre_from_Source_Code) for building Lustre against the updated SLES11 2.6.27.45-0.1-default kernel, and ran into this error during the make rpms step. /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:114: error: field ?cb_jcb? has incomplete type /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c: In function ?fsfilt_ldiskfs_add_journal_cb?: /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:793: error: implicit declaration of function ?jbd2_journal_callback_set? I am building against this kernel due to some needed fixes in this relative to the downloadable kernel. More full error context below. Is there a workaround for this? This is 1.8.2 Lustre GA. Will the git repository version have fixes for this? Thanks! Joe --- rest of context: Type ''make'' to build Lustre. + make -j 1 -s make[2]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2'' make[3]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2'' Making all in ldiskfs make[4]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' Making all in . make[6]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' make[7]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' Applying patch patches/ext4-wantedi-2.6-sles11.patch Applying patch patches/iopen-sles11.patch Applying patch patches/ext4-map_inode_page-sles11.patch Applying patch patches/export-ext4-2.6-sles11.patch Applying patch patches/ext4-include-fixes-2.6-sles11.patch Applying patch patches/ext4-remove-cond_resched-calls-sles11.patch Applying patch patches/ext4-filterdata-sles11.patch Applying patch patches/ext4-mmp-sles11.patch Applying patch patches/ext4-fiemap-2.6-sles11.patch Applying patch patches/ext4-lookup-dotdot-sles11.patch Applying patch patches/ext4-max-dir-size-sles11.patch Applying patch patches/ext4-print-inum-in-htree-warning-sles11.patch Applying patch patches/ext4-xattr-no-update-ctime-sles11.patch Applying patch patches/ext4-prealloc-sles11.patch Applying patch patches/ext4-mballoc-extra-checks-sles11.patch Applying patch patches/ext4-big-endian-check-2.6-sles11.patch Applying patch patches/ext4-misc-sles11.patch Applying patch patches/ext4-ext_generation-sles11.patch Applying patch patches/ext4-super-warning.patch Applying patch patches/ext4-alloc-policy-2.6-sles11.patch Applying patch patches/ext4-disable-delalloc-sles11.patch Applying patch patches/ext4-lustre-i_version.patch Applying patch patches/ext4-lock-cached_extent.patch Applying patch patches/ext4-convert-group-lock-sles11.patch Applying patch patches/ext4-force_over_8tb-sles11.patch Applying patch patches/ext4-claim_inode-free_inode-race.patch Applying patch patches/ext4_ext_search_right-fix.patch Applying patch patches/ext4-pa_lock-typo.patch Applying patch patches/ext4-pdir-fix.patch Applying patch patches/ext4-osd-iop-common-sles11.patch Applying patch patches/ext4-osd-iam-exports.patch Applying patch patches/ext4-dynlocks-common-sles11.patch Applying patch patches/ext4-dynlocks-2.6-rhel5.patch Applying patch patches/ext4-hash-indexed-dir-dotdot-update.patch Applying patch patches/ext4-disable-write-bar-by-default.patch Now at patch patches/ext4-disable-write-bar-by-default.patch Replacing ''ext4'' with ''ldiskfs'': acl.h ext4_extents.h ext4.h ext4_i.h ext4_jbd2.h ext4_sb.h group.h mballoc.h namei.h xattr.h acl.c balloc.c bitmap.c dir.c ext4_jbd2.c extents.c file.c fsync.c hash.c ialloc.c inode.c ioctl.c mballoc.c migrate.c namei.c resize.c super.c symlink.c xattr.c xattr_security.c xattr_trusted.c xattr_user.c iopen.c iopen.h dynlocks.c fiemap.h ext4_extents.h ext4.h ext4_i.h ext4_jbd2.h ext4_sb.h make[7]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' make[7]: Entering directory `/tmp/linux-2.6.27.45-0.1'' WARNING: Symbol version dump /tmp/linux-2.6.27.45-0.1/Module.symvers is missing; modules will have no dependencies and modversions. Building modules, stage 2. make[7]: Leaving directory `/tmp/linux-2.6.27.45-0.1'' make[6]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' Making all in ldiskfs make[6]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' make[6]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' make[4]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' Making all in . make[4]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2'' make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/lnet'' make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/lnet'' make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/lustre'' make[6]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs'' make[6]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs'' make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/lustre'' make[5]: Entering directory `/tmp/linux-2.6.27.45-0.1'' WARNING: Symbol version dump /tmp/linux-2.6.27.45-0.1/Module.symvers is missing; modules will have no dependencies and modversions. /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:114: error: field ?cb_jcb? has incomplete type /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c: In function ?fsfilt_ldiskfs_add_journal_cb?: /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:793: error: implicit declaration of function ?jbd2_journal_callback_set? make[8]: *** [/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.o] Error 1 make[7]: *** [/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs] Error 2 make[6]: *** [/usr/src/packages/BUILD/lustre-1.8.2/lustre] Error 2 make[5]: *** [_module_/usr/src/packages/BUILD/lustre-1.8.2] Error 2 make[5]: Leaving directory `/tmp/linux-2.6.27.45-0.1'' make[4]: *** [modules] Error 2 make[4]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2'' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2'' make[2]: *** [all] Error 2 make[2]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2'' error: Bad exit status from /var/tmp/rpm-tmp.4639 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.4639 (%build) make[1]: *** [rpms-real] Error 1 make[1]: Leaving directory `/tmp/lustre-1.8.2'' make: *** [rpms] Error 2
On Wed, 2010-03-24 at 13:31 -0400, Joe Landman wrote:> Hi folksHi,> I followed the directions > (http://wiki.lustre.org/index.php/Building_and_Installing_Lustre_from_Source_Code) > for building Lustre against the updated SLES11 2.6.27.45-0.1-default > kernel,But you didn''t check the support status of that kernel (or maybe you did but were just trying your luck with a newer kernel), available in detail in the changelog at http://wiki.lustre.org/index.php/Change_Log_1.8 or on the support matrix at http://wiki.lustre.org/index.php/Lustre_Release_Information#Lustre_Support_Matrix The changelog specifies: Support for kernels: ... 2.6.27.39-0.3.1 (SLES11, i686 & x84_64 only) ... which, as you can see, is older than the kernel you are trying to build for. Maybe you know that. I thought I would just point it out for completeness in any case.> and ran into this error during the > > make rpms > > step. > > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:114: > error: field ?cb_jcb? has incomplete type > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c: In > function ?fsfilt_ldiskfs_add_journal_cb?: > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:793: > error: implicit declaration of function ?jbd2_journal_callback_set?Could be due to the newer kernel you are attempting to build for.> I am building against this kernel due to some needed fixes in this > relative to the downloadable kernel. More full error context below. Is > there a workaround for this? This is 1.8.2 Lustre GA. Will the git > repository version have fixes for this?If the problem is simply too new a kernel that requires some porting on Lustre''s part, perhaps, but not likely. If you can find a bug tracking the effort to support a newer SLES11 kernel, it will tell you if the patches have landed (i.e. are in git) yet. A quick search of BZ yielded bug 21862 which appears to be opened to track the update to 2.6.27.42 although it appears no work has yet been done. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100324/d5678865/attachment.bin
Dardo D Kleiner - CONTRACTOR
2010-Mar-24 20:27 UTC
[Lustre-discuss] SLES11 ''make rpms'' dies
It would appear that the kernel patches necessary to build Lustre server bits have not been applied (and I guess you insisted with --enable-server otherwise it shouldn''t have even tried). jbd2_journal_callback_set is included in: lustre/kernel_patches/patches/jbd2-jcberr-2.6-sles11.patch you would then have the declaration in your kernel source at include/linux/jbd2.h and the implementation in fs/jbd2/transaction.c. Though unsupported, we have built and are running 1.8.2 GA on 2.6.27.45 without problem. Can you send the output of your "quilt push -av" session? - Dardo On 3/24/10 1:31 PM, Joe Landman wrote:> Hi folks > > I followed the directions > (http://wiki.lustre.org/index.php/Building_and_Installing_Lustre_from_Source_Code) > for building Lustre against the updated SLES11 2.6.27.45-0.1-default > kernel, and ran into this error during the > > make rpms > > step. > > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:114: > error: field ?cb_jcb? has incomplete type > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c: In > function ?fsfilt_ldiskfs_add_journal_cb?: > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:793: > error: implicit declaration of function ?jbd2_journal_callback_set? > > > I am building against this kernel due to some needed fixes in this > relative to the downloadable kernel. More full error context below. Is > there a workaround for this? This is 1.8.2 Lustre GA. Will the git > repository version have fixes for this? > > Thanks! > > Joe > > --- > rest of context: > Type ''make'' to build Lustre. > + make -j 1 -s > make[2]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2'' > make[3]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2'' > Making all in ldiskfs > make[4]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > Making all in . > make[6]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > make[7]: Entering directory > `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' > Applying patch patches/ext4-wantedi-2.6-sles11.patch > Applying patch patches/iopen-sles11.patch > Applying patch patches/ext4-map_inode_page-sles11.patch > Applying patch patches/export-ext4-2.6-sles11.patch > Applying patch patches/ext4-include-fixes-2.6-sles11.patch > Applying patch patches/ext4-remove-cond_resched-calls-sles11.patch > Applying patch patches/ext4-filterdata-sles11.patch > Applying patch patches/ext4-mmp-sles11.patch > Applying patch patches/ext4-fiemap-2.6-sles11.patch > Applying patch patches/ext4-lookup-dotdot-sles11.patch > Applying patch patches/ext4-max-dir-size-sles11.patch > Applying patch patches/ext4-print-inum-in-htree-warning-sles11.patch > Applying patch patches/ext4-xattr-no-update-ctime-sles11.patch > Applying patch patches/ext4-prealloc-sles11.patch > Applying patch patches/ext4-mballoc-extra-checks-sles11.patch > Applying patch patches/ext4-big-endian-check-2.6-sles11.patch > Applying patch patches/ext4-misc-sles11.patch > Applying patch patches/ext4-ext_generation-sles11.patch > Applying patch patches/ext4-super-warning.patch > Applying patch patches/ext4-alloc-policy-2.6-sles11.patch > Applying patch patches/ext4-disable-delalloc-sles11.patch > Applying patch patches/ext4-lustre-i_version.patch > Applying patch patches/ext4-lock-cached_extent.patch > Applying patch patches/ext4-convert-group-lock-sles11.patch > Applying patch patches/ext4-force_over_8tb-sles11.patch > Applying patch patches/ext4-claim_inode-free_inode-race.patch > Applying patch patches/ext4_ext_search_right-fix.patch > Applying patch patches/ext4-pa_lock-typo.patch > Applying patch patches/ext4-pdir-fix.patch > Applying patch patches/ext4-osd-iop-common-sles11.patch > Applying patch patches/ext4-osd-iam-exports.patch > Applying patch patches/ext4-dynlocks-common-sles11.patch > Applying patch patches/ext4-dynlocks-2.6-rhel5.patch > Applying patch patches/ext4-hash-indexed-dir-dotdot-update.patch > Applying patch patches/ext4-disable-write-bar-by-default.patch > Now at patch patches/ext4-disable-write-bar-by-default.patch > Replacing ''ext4'' with ''ldiskfs'': acl.h ext4_extents.h ext4.h ext4_i.h > ext4_jbd2.h ext4_sb.h group.h mballoc.h namei.h xattr.h acl.c balloc.c > bitmap.c dir.c ext4_jbd2.c extents.c file.c fsync.c hash.c ialloc.c > inode.c ioctl.c mballoc.c migrate.c namei.c resize.c super.c symlink.c > xattr.c xattr_security.c xattr_trusted.c xattr_user.c iopen.c iopen.h > dynlocks.c fiemap.h ext4_extents.h ext4.h ext4_i.h ext4_jbd2.h ext4_sb.h > make[7]: Leaving directory > `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' > make[7]: Entering directory `/tmp/linux-2.6.27.45-0.1'' > > WARNING: Symbol version dump /tmp/linux-2.6.27.45-0.1/Module.symvers > is missing; modules will have no dependencies and modversions. > > Building modules, stage 2. > make[7]: Leaving directory `/tmp/linux-2.6.27.45-0.1'' > make[6]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > Making all in ldiskfs > make[6]: Entering directory > `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' > make[6]: Leaving directory > `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs/ldiskfs'' > make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > make[4]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > Making all in . > make[4]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2'' > make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/ldiskfs'' > make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/lnet'' > make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/lnet'' > make[5]: Entering directory `/usr/src/packages/BUILD/lustre-1.8.2/lustre'' > make[6]: Entering directory > `/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs'' > make[6]: Leaving directory > `/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs'' > make[5]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2/lustre'' > make[5]: Entering directory `/tmp/linux-2.6.27.45-0.1'' > > WARNING: Symbol version dump /tmp/linux-2.6.27.45-0.1/Module.symvers > is missing; modules will have no dependencies and modversions. > > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:114: > error: field ?cb_jcb? has incomplete type > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c: In > function ?fsfilt_ldiskfs_add_journal_cb?: > /usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.c:793: > error: implicit declaration of function ?jbd2_journal_callback_set? > make[8]: *** > [/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs/fsfilt-ldiskfs.o] Error 1 > make[7]: *** [/usr/src/packages/BUILD/lustre-1.8.2/lustre/lvfs] Error 2 > make[6]: *** [/usr/src/packages/BUILD/lustre-1.8.2/lustre] Error 2 > make[5]: *** [_module_/usr/src/packages/BUILD/lustre-1.8.2] Error 2 > make[5]: Leaving directory `/tmp/linux-2.6.27.45-0.1'' > make[4]: *** [modules] Error 2 > make[4]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2'' > make[3]: *** [all-recursive] Error 1 > make[3]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2'' > make[2]: *** [all] Error 2 > make[2]: Leaving directory `/usr/src/packages/BUILD/lustre-1.8.2'' > error: Bad exit status from /var/tmp/rpm-tmp.4639 (%build) > > > RPM build errors: > Bad exit status from /var/tmp/rpm-tmp.4639 (%build) > make[1]: *** [rpms-real] Error 1 > make[1]: Leaving directory `/tmp/lustre-1.8.2'' > make: *** [rpms] Error 2 > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
Brian J. Murrell wrote:> On Wed, 2010-03-24 at 13:31 -0400, Joe Landman wrote:[...]> which, as you can see, is older than the kernel you are trying to build > for. Maybe you know that. I thought I would just point it out for > completeness in any case.I was and am aware of this. And this is why I had to start working on the build. This kernel (the .39-0.3.1) has several bugs that have been resolved in .42 and .45. Our use case does trigger these bugs, and we can solve our problem by using the .42 or .45 kernels. Anyone with an NFS system will very likely want to migrate to the .42 or .45 kernels due to these issues. Hence we either have to wait for the "official" builds to catch up to the released bits, or perform builds ourselves. [...] n of function ?jbd2_journal_callback_set?> > Could be due to the newer kernel you are attempting to build for.I wouldn''t think they were that much different. [...]> If the problem is simply too new a kernel that requires some porting on > Lustre''s part, perhaps, but not likely. If you can find a bug tracking > the effort to support a newer SLES11 kernel, it will tell you if the > patches have landed (i.e. are in git) yet. > > A quick search of BZ yielded bug 21862 which appears to be opened to > track the update to 2.6.27.42 although it appears no work has yet been > done.I did see this, and this is unfortunate. The other possibility is to backport the needed fixes into the .39 kernel. I will investigate this, as it doesn''t appear that the .42 or .45 will build correctly. -- Joe Landman landman at scalableinformatics.com
Dardo D Kleiner - CONTRACTOR wrote:> It would appear that the kernel patches necessary to build Lustre > server bits have not been applied (and I guess you insisted with > --enable-server otherwise it shouldn''t have even tried).Doesn''t the quilt push -a do the patching? I was following the directions on the web page when it died the first time. I read some more from the various readme''s for SLES in the 1.8.2 distribution. It suggests using --enable-server and several other things. Where I''d like to get to is just building the kernel modules for the "patchless" client on .42 or .45. I didn''t see instructions on that. Any pointers welcomed. regards, -- Joe Landman landman at scalableinformatics.com
Dardo D Kleiner - CONTRACTOR
2010-Mar-24 21:44 UTC
[Lustre-discuss] SLES11 ''make rpms'' dies
On 3/24/10 5:22 PM, Joe Landman wrote:> Dardo D Kleiner - CONTRACTOR wrote: >> It would appear that the kernel patches necessary to build Lustre >> server bits have not been applied (and I guess you insisted with >> --enable-server otherwise it shouldn''t have even tried). > > Doesn''t the > > quilt push -a > > do the patching?It does, and with "v" it should be verbose about it which is why I asked to see that in case it had problems. But this is moot as you are only looking for client bits...> I was following the directions on the web page when it died the first > time. I read some more from the various readme''s for SLES in the 1.8.2 > distribution. It suggests using --enable-server and several other things. > > Where I''d like to get to is just building the kernel modules for the > "patchless" client on .42 or .45. I didn''t see instructions on that. Any > pointers welcomed.The patchless client build should "just work", don''t --enable-server unless you want the server. What are your exact configure options? This is sufficient as a bare minimum test: ./configure --with-linux=/lib/modules/2.6.27.45-0.1-default/source \ --with-linux-obj=/lib/modules/2.6.27.45-0.1-default/build Attach your config.log if possible as well. - Dardo
Dardo D Kleiner - CONTRACTOR wrote:> The patchless client build should "just work", don''t --enable-server unless > you want the server. What are your exact configure options? This is > sufficient as a bare minimum test: > > ./configure --with-linux=/lib/modules/2.6.27.45-0.1-default/source \ > --with-linux-obj=/lib/modules/2.6.27.45-0.1-default/buildThis seems to have worked on an unpatched kernel. Thanks. I presume we need the OFED 1.4.2 stack installed before for o2ib kernel build? -- Joe Landman landman at scalableinformatics.com
On Wed, 2010-03-24 at 18:45 -0400, Joe Landman wrote:> > This seems to have worked on an unpatched kernel. Thanks. I presume we > need the OFED 1.4.2 stack installed before for o2ib kernel build?Specifically, you need kernel-ib-devel or it''s equivalent -- that is, the OFED kernel headers. If they are in a standard OFED kernel-ib-devel and installed at /usr/src/ofa_kernel, then configure will find them, otherwise you need to point to them with --with-o2ib=<path>. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100325/196fe66e/attachment.bin