Lustre discuss - Jul 2009 - lustre-discuss-list - 10 new messages in 5 topics - digest

Hi all,

Regarding to the comparsion of Lustre with GlusterFS, i have the
fallowing together with the attached file.

I used this in a presentation at
http://www.beliefproject.org/events/4th-belief-international-symposium

It can be a start of comparsion. GlusterFS has a lower performance,
but we can make a RAID 10 though the network, in the case that the
glusterfs equivalent of OST are DAS( ie, internal disk, not a SAN
lun). Also,  apparently, there is no deadlock when the glusterfs node
is simultaneously configured to be  client and server.

In my experince, the integration between most recent kernels with
glusterfs and patches of Xen hypervisor works well. The same with
Lustre is harder to do.

Although, there is issues when trying to boot up a virtual machine
image stored in glusterfs mount point.

Also glusterfs has lower performance when overwriting files when
compared with the write process.

Apparently glusterfs does not stripe files between nodes and
apparently there is a single file limit size asspciated to the space
available in the gluster equivalent to OST.

I am waiting for version 2 of Lustre.

Best Regards.
--
Ettore Enrico Delfino Ligorio
ettoredelfinoligorio at gmail.com
55-11-9145-6151



On Fri, Jul 17, 2009 at 4:45 AM, lustre-discuss-list
group<noreply at googlegroups.com> wrote:
>
> lustre-discuss-list
> http://groups.google.com/group/lustre-discuss-list?hl=en
>
> lustre-discuss-list at googlegroups.com
>
> Today''s topics:
>
> * MDT move aka backup w rsync - 3 messages, 2 authors
>
?http://groups.google.com/group/lustre-discuss-list/t/cbab59a18743f8aa?hl=en
> * Lustre compared to Gluster - 1 messages, 1 author
>
?http://groups.google.com/group/lustre-discuss-list/t/18b4c2c838289bed?hl=en
> * One Lustre Client lost One Lustre Disk - 2 messages, 2 authors
>
?http://groups.google.com/group/lustre-discuss-list/t/4c313176573ce691?hl=en
> * One Lustre Client lost One Lustre Disk--solved - 1 messages, 1 author
>
?http://groups.google.com/group/lustre-discuss-list/t/ac24b4ae3b9b8b53?hl=en
> * Hastening lustrefs recovery - 3 messages, 3 authors
>
?http://groups.google.com/group/lustre-discuss-list/t/5ba606574f699a0d?hl=en
>
>
=============================================================================>
TOPIC: MDT move aka backup w rsync
> http://groups.google.com/group/lustre-discuss-list/t/cbab59a18743f8aa?hl=en
>
=============================================================================>
> == 1 of 3 => Date: Wed, Jul 15 2009 10:43?am
> From: Andreas Dilger
>
>
> On Jul 15, 2009 ?18:35 +0200, Thomas Roth wrote:
>> I want to move a MDT from one server to another. After studying some
>> mails concerning MDT backup, I''ve just tried (successfully, it
seems) to
>> do that on a small test system ?with rsync:
>>
>> - Stop Lustre, umount all servers.
>> - Format a suitable disk partition on the new hardware, using the same
>> mkfs-options as for the original MDT.
>> - Mount the original MDT: ? ?mount ? -t ldiskfs ? ? ?/dev/sdb1 ? ?/mnt
>> - Mount the target partition: mount ? -t ldiskfs ? -O ext_attr
>> /dev/sdb1 ? ?/mnt
>> - Copy the data: ?rsync ? -Xav ? oldserver:/mnt/ ? ?newserver:/mnt
>> - Umount partitions, restart MGS
>> - Mount new MDT
>>
>> This procedure was described by Jim Garlick on this list. You might
note
>> that I used the mount option "-O ext_attr" only on the target
machine:
>> my mistake perhaps, but no visible problems. In fact, I
haven''t found
>> this option mentioned in any man page or on the net. Nevertheless, my
>> mount command did not complain about it. So I wonder whether it is
>> necessary at all - I seem to have extracted the attributes from the old
>> MDT all right, without this mount option - ?
>
> If you have verified that the file data is actually present, this should
> work correctly. ?In particular, the critical Lustre information is in the
> "trusted.lov" xattr, so you need to ensure that is present. ?The
MDS will
> "work" without this xattr, but it will assume all of the files
have no
> data.
>
>> I''m investigating this because our production MDT seems to
have a number
>> of problems. In particular the underlying file system is in bad shape,
>> fsck correcting a large number of ext3-errors, incorrect inodes and so
>> forth. We want to verify that it is not a hardware issue - bit-flipping
>> RAID controller, silent "memory corruption", whatever. We
have a
>> DRBD-mirror of this MDT running, but of course DRBD just reproduces all
>> errors on the mirror. ?Copying from one ldiskfs to another should avoid
>> that?
>>
>> The traditional backup method of getting the EAs and tar-ing the MDT
>> doesn''t finish in finite time. It did before, and the
filesystem has
>> since grown by a mere 40GB of data, so it shouldn''t take that
much
>> longer - certainly another indication that there is something wrong.
>> Of course I have yet to see whether "rsync -Xav" does much
better on the
>> full system ;-)
>>
>> The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1
>
> Direct MDT backup has a problem in 1.6.7.1 due to the addition of the
> file size on the MDT inodes. ?If you do "--sparse" backups this
should
> avoid the slowdown. ?You could also try the "dump" program, which
can
> avoid reading the data from sparse files entirely.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
> == 2 of 3 => Date: Thurs, Jul 16 2009 6:43?am
> From: Thomas Roth
>
>
>
> Andreas Dilger wrote:
>> On Jul 15, 2009 ?18:35 +0200, Thomas Roth wrote:
>>>...
>>> The traditional backup method of getting the EAs and tar-ing the
MDT
>>> doesn''t finish in finite time. It did before, and the
filesystem has
>>> since grown by a mere 40GB of data, so it shouldn''t take
that much
>>> longer - certainly another indication that there is something
wrong.
>>> Of course I have yet to see whether "rsync -Xav" does
much better on the
>>> full system ;-)
>>>
>>> The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1
>>
>> Direct MDT backup has a problem in 1.6.7.1 due to the addition of the
>> file size on the MDT inodes. ?If you do "--sparse" backups
this should
>> avoid the slowdown. ?You could also try the "dump" program,
which can
>> avoid reading the data from sparse files entirely.
>
> Is this behavior changed in 1.6.7.2? ?But the size information is still
> there in the inodes, isn''t it?
> Anyhow our backup script already runs tar with the "--sparse"
option.
>
> Regards, Thomas
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
> == 3 of 3 => Date: Thurs, Jul 16 2009 6:51?am
> From: Thomas Roth
>
>
> For the record, I should add that I had forgotten one step which proves
> to be important, also mentioned before on this list:
>
> After copying with rsync, I had to
> cd ? /srv/mdt;
> rm ?CATALOGS ?OBJECTS/*
> on the new MDT partition.
>
> Otherwise the OSTs are kicked out on remount with ?"error looking up
> logfile ...: rc -2" and "OST0000_UUID sync failed -2,
deactivating"
>
> Regards,
> Thomas
>
> Thomas Roth wrote:
>> Hi all,
>>
>> I want to move a MDT from one server to another. After studying some
>> mails concerning MDT backup, I''ve just tried (successfully, it
seems) to
>> do that on a small test system ?with rsync:
>>
>> - Stop Lustre, umount all servers.
>> - Format a suitable disk partition on the new hardware, using the same
>> mkfs-options as for the original MDT.
>> - Mount the original MDT: ? ?mount ? -t ldiskfs ? ? ?/dev/sdb1 ? ?/mnt
>> - Mount the target partition: mount ? -t ldiskfs ? -O ext_attr
>> /dev/sdb1 ? ?/mnt
>> - Copy the data: ?rsync ? -Xav ? oldserver:/mnt/ ? ?newserver:/mnt
>> - Umount partitions, restart MGS
>> - Mount new MDT
>>
>> This procedure was described by Jim Garlick on this list. You might
note
>> that I used the mount option "-O ext_attr" only on the target
machine:
>> my mistake perhaps, but no visible problems. In fact, I
haven''t found
>> this option mentioned in any man page or on the net. Nevertheless, my
>> mount command did not complain about it. So I wonder whether it is
>> necessary at all - I seem to have extracted the attributes from the old
>> MDT all right, without this mount option - ?
>>
>> My main question is whether this is a correct procedure for MDT
backups,
>> or rather copies.
>>
>> I''m investigating this because our production MDT seems to
have a number
>> of problems. In particular the underlying file system is in bad shape,
>> fsck correcting a large number of ext3-errors, incorrect inodes and so
>> forth. We want to verify that it is not a hardware issue - bit-flipping
>> RAID controller, silent "memory corruption", whatever. We
have a
>> DRBD-mirror of this MDT running, but of course DRBD just reproduces all
>> errors on the mirror. ?Copying from one ldiskfs to another should avoid
>> that?
>>
>> The traditional backup method of getting the EAs and tar-ing the MDT
>> doesn''t finish in finite time. It did before, and the
filesystem has
>> since grown by a mere 40GB of data, so it shouldn''t take that
much
>> longer - certainly another indication that there is something wrong.
>> Of course I have yet to see whether "rsync -Xav" does much
better on the
>> full system ;-)
>>
>> Hm, not sure whether this all makes sense.
>>
>> The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1
>>
>> Regards,
>> Thomas
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
> --
> --------------------------------------------------------------------
> Thomas Roth
> Department: Informationstechnologie
> Location: SB3 1.262
> Phone: +49-6159-71 1453 ?Fax: +49-6159-71 2986
>
> GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
> Planckstra?e 1
> D-64291 Darmstadt
> www.gsi.de
>
> Gesellschaft mit beschr?nkter Haftung
> Sitz der Gesellschaft: Darmstadt
> Handelsregister: Amtsgericht Darmstadt, HRB 1528
>
> Gesch?ftsf?hrer: Professor Dr. Horst St?cker
>
> Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph,
> Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
>
>
=============================================================================>
TOPIC: Lustre compared to Gluster
> http://groups.google.com/group/lustre-discuss-list/t/18b4c2c838289bed?hl=en
>
=============================================================================>
> == 1 of 1 => Date: Thurs, Jul 16 2009 5:25?am
> From: Mag Gam
>
>
> We have been hearing a lot of news recently about "Gluster". Does
> anyone know how it compares to Lustre? Can it do the same things as
> Lustre? It seems it has built in SNS. Anyone know?
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
>
>
=============================================================================>
TOPIC: One Lustre Client lost One Lustre Disk
> http://groups.google.com/group/lustre-discuss-list/t/4c313176573ce691?hl=en
>
=============================================================================>
> == 1 of 2 => Date: Thurs, Jul 16 2009 7:40?am
> From: "Ms. Megan Larko"
>
>
> Good Day!
>
> Yesterday evening around 5:30 p.m. local time, one of my lustre client
> systems lost one of its two lustre disks. ?I was not able to remount
> it, even after a reboot of the client. ? The mount command returns the
> following message:
> [root at crew01 ~]# mount /crew2
> mount.lustre: mount ic-mds1 at o2ib:/crew2 at /crew2 failed: Invalid
argument
> This may have multiple causes.
> Is ''crew2'' the correct filesystem name?
> Are the mount options correct?
> Check the syslog for more info.
>
>
> And the /var/log/messages file (CentOS 5.1 system using
?2.6.18-53.1.13.el5):
> Jul 16 10:30:53 crew01 kernel: LustreError: 156-2: The client profile
> ''crew2-client'' could not be read from the MGS. ?Does that
filesystem
> exist?
> Jul 16 10:30:53 crew01 kernel: Lustre: client ffff810188fcfc00 umount
complete
> Jul 16 10:30:53 crew01 kernel: LustreError:
> 26240:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount ?(-22)
>
> The entry in the client /etc/fstab file is unchanged from before:
> ic-mds1 at o2ib:/crew2 ? ? /crew2 ? ? ? ? ?lustre ?nouser_xattr,_netdev ?
?0 0
>
> This same client uses the /etc/fstab entry
> "ic-mds1 at o2ib:/crew8 ? ?/crewdat ? ? ? ?lustre
?nouser_xattr,_netdev ? ?0 0"
> This lustre disk is still mounted and usable:
> ic-mds1 at o2ib:/crew8 ? ?76T ? 30T ? 42T ?42% /crewdat
>
> What is also interesting is that other clients still have access to
> the /crew2 disk, even though this one client does not.
> ?There are no crew2 errors in the MGS/MDS system which serves both of
> the lustre disks.
>
> What has this one particular client lost that prevents it from
> mounting the /crew2 disk to which the other clients still have access?
>
> Any and all suggestions are appreciated.
> megan
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
> == 2 of 2 => Date: Thurs, Jul 16 2009 10:46?pm
> From: Andreas Dilger
>
>
> On Jul 16, 2009 ?10:40 -0400, Ms. Megan Larko wrote:
>> [root at crew01 ~]# mount /crew2
>> mount.lustre: mount ic-mds1 at o2ib:/crew2 at /crew2 failed: Invalid
argument
>> This may have multiple causes.
>> Is ''crew2'' the correct filesystem name?
>> Are the mount options correct?
>> Check the syslog for more info.
>>
>>
>> And the /var/log/messages file (CentOS 5.1 system using
?2.6.18-53.1.13.el5):
>> Jul 16 10:30:53 crew01 kernel: LustreError: 156-2: The client profile
>> ''crew2-client'' could not be read from the MGS. ?Does
that filesystem
>> exist?
>> Jul 16 10:30:53 crew01 kernel: Lustre: client ffff810188fcfc00 umount
complete
>> Jul 16 10:30:53 crew01 kernel: LustreError:
>> 26240:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount ?(-22)
>>
>> The entry in the client /etc/fstab file is unchanged from before:
>> ic-mds1 at o2ib:/crew2 ? /crew2 ? ? ? ? ?lustre ?nouser_xattr,_netdev ?
?0 0
>>
>> This same client uses the /etc/fstab entry
>> ic-mds1 at o2ib:/crew8 ? /crewdat ? ? ? ?lustre ?nouser_xattr,_netdev ?
?0 0
>> This lustre disk is still mounted and usable:
>> ic-mds1 at o2ib:/crew8 ? ?76T ? 30T ? 42T ?42% /crewdat
>>
>> What is also interesting is that other clients still have access to
>> the /crew2 disk, even though this one client does not.
>> ?There are no crew2 errors in the MGS/MDS system which serves both of
>> the lustre disks.
>>
>> What has this one particular client lost that prevents it from
>> mounting the /crew2 disk to which the other clients still have access?
>
> You should also check for messages on the MDS also. ?You can check for
> the config file existing via "debugfs -c -R ''ls -l
CONFIGS'' /dev/{mdsdev}".
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
>
>
=============================================================================>
TOPIC: One Lustre Client lost One Lustre Disk--solved
> http://groups.google.com/group/lustre-discuss-list/t/ac24b4ae3b9b8b53?hl=en
>
=============================================================================>
> == 1 of 1 => Date: Thurs, Jul 16 2009 3:31?pm
> From: "Ms. Megan Larko"
>
>
> Hi,
>
> I fixed the problem of the one Lustre client not mounting one Lustre disk.
>
> Truthfully, the problem expanded slightly. ?When I rebooted another
> client, it also lost contact with this one particular Lustre disk.
> The error messages were exactly the same:
>
> [root at crew01 ~]# mount /crew2
> mount.lustre: mount ic-mds1 at o2ib:/crew2 at /crew2 failed: Invalid
argument
> This may have multiple causes.
> Is ''crew2'' the correct filesystem name?
> Are the mount options correct?
> Check the syslog for more info.
>
> So, I thought something may have become a bit off with the disk
> set-up. ? I had recently upgraded the other MDT disk to a larger
> physical volume. ?This was successfully done following instructions in
> the Lustre Manual. ? So I thought perhaps the MDT that I did not
> change merely needed to be "re-set".
>
> On the MGS, I unmounted the MDT of the problem disk and ran the
> following command:
>>> tunefs.lustre --writeconf --mgs --mdt ?--fsname=crew2
/dev/{sd-whatever}
> I then remounted the MDT (which is also the MGS) successfully.
>
> On the OSS, I first unmounted the OST disks and then I issued the command:
>>> tunefs.lustre --writeconf --ost /dev/{sd-whatever}
> This was issued for each and every OST. ? I mounted my OSTs again
successfully.
>
> On my clients, I issued the mount command for the /crew2 lustre disk
> and it was now successful. ?No more "invalid argument" message.
> One client did give me a "Transport endpoint not connected
message",
> so that client will require a re-boot to remount this lustre disk
> (unless anyone can tell me how to do the re-mount without a reboot of
> this client).
>
> So--- I am guessing that when I did the upgrade in hardware disk size
> on the non-mgs lustre disk a few weeks ago, the other lustre disk,
> which functions as the mgs, was left in a state such that I could not
> pick-up that disk from the clients if I rebooted a client. ?Is this an
> accurate guess? ? ?If it is, then one may with to add to the Lustre
> Manual (Ch. 15 in 1.6.x versions on restoring metadata to an mdt disk)
> that the mgs disk may require an update using tunefs.lustre
> --writeconf even if it was not the disk which was restored.
>
> I may be wrong in my guess, but the above procedure did get my lustre
> disk back onto my clients successfully.
>
> Cheers!
> megan
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
>
>
=============================================================================>
TOPIC: Hastening lustrefs recovery
> http://groups.google.com/group/lustre-discuss-list/t/5ba606574f699a0d?hl=en
>
=============================================================================>
> == 1 of 3 => Date: Thurs, Jul 16 2009 3:59?pm
> From: Josephine Palencia
>
>
>
> OS: Centos5.2 x86_64
> Lustre: 1.6.5.1
> OpenIB: 1.3.1
>
>
> What determines the speed at which a lustre fs will recover? (ex. after a
> crash) ?Can (should) one hasten the recovery by tweaking some parameters?
>
> For 4 OSTS each with 7TB, ~40 connected clients , recovery time
> is 48min. Is that reasonable or is that too long?
>
>
> Thanks,
> josephine
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
> == 2 of 3 => Date: Thurs, Jul 16 2009 5:15?pm
> From: "Brian J. Murrell"
>
>
>
> On Thu, 2009-07-16 at 18:59 -0400, Josephine Palencia wrote:
>>
>> What determines the speed at which a lustre fs will recover? (ex. after
a
>> crash)
>
> How fast all of the clients can reconnect and replay their pending
> transactions.
>
>> Can (should) one hasten the recovery by tweaking some parameters?
>
> There''s not much to tweak. ?Recovery waits for a) all clients to
> reconnect and replay or b) the recovery timer to run out. ?The recovery
> timer is a factor of obd_timeout. ?As you probably know obd_timeout has
> a value below which you will start to see timeouts and evictions --
> which you don''t want of course. ?So you don''t really want
to set it
> below that value.
>
> The first question people tend ask when they discover they need to tune
> their obd_timeout upwards to avoid lock callback timeouts and so forth
> is "why don''t I just set that really high then?". ?The
answer is always
> "because the higher you set it, the longer your recovery process will
> take in the event that not all clients are available to replay.
>
> Of course the bigger your client count the higher the odds that the
> recovery timeout is your deciding factor and not all clients being
> available to connect.
>
> Of interest to all of this is that in 1.8, adaptive timeouts (AT) are
> enabled by default, so obd_timeout should generally always be high
> enough without being too high -- i.e. optimal. ?So if your OSSes and MDS
> are tuned such that they are not overwhelming their disk backend,
> obd_timeout should be reasonable and therefore recovery should be
> reasonable.
>
>> For 4 OSTS each with 7TB, ~40 connected clients , recovery time
>> is 48min. Is that reasonable or is that too long?
>
> Wow. ?That seems long. ?That is recovery of what? ?A single OST or
> single OSS, or something other?
>
> b.
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
> == 3 of 3 => Date: Thurs, Jul 16 2009 5:26?pm
> From: Jeffrey Bennett
>
>
>> > For 4 OSTS each with 7TB, ~40 connected clients , recovery time is
>> > 48min. Is that reasonable or is that too long?
>>
>> Wow. ?That seems long. ?That is recovery of what? ?A single
>> OST or single OSS, or something other?
>>
> Recovery times I have been seeing on similar systems are around 2-3
minutes. That''s what it takes clients to replay their transactions or
time out. This is in the event of failover to a new MDS.
>
> jab
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
>
>
=============================================================================>
> You received this message because you are subscribed to the Google Groups
"lustre-discuss-list"
> group.
>
> To post to this group, send email to lustre-discuss at lists.lustre.org or
visit http://groups.google.com/group/lustre-discuss-list?hl=en
>
> To unsubscribe from this group, send email to
lustre-discuss-list+unsubscribe at googlegroups.com
>
> To change the way you get mail from this group, visit:
> http://groups.google.com/group/lustre-discuss-list/subscribe?hl=en
>
> To report abuse, send email explaining the problem to abuse at
googlegroups.com
>
>
=============================================================================>
Google Groups: http://groups.google.com/?hl=en
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fs_benck_CCE.xls
Type: application/vnd.ms-excel
Size: 17408 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090717/20759328/attachment-0001.xls

Hello!

On Jul 17, 2009, at 2:01 PM, Ettore Enrico Delfino Ligorio
wrote:
> In my experince, the integration between most recent kernels with
> glusterfs and patches of Xen hypervisor works well. The same with
> Lustre is harder to do.
Works out of the box for me both with rhel5 kernels (that contain
xen patches already) and vanilla 2.6.30 (that also happen to contain
domU stuff).
Also some Lustre developers extensively use xen for their testing.
What is the specific problem you are seeing, I wonder?

Bye,
     Oleg

Hi Oleg,

After your message, during the last 3 days i had tried to setup Xen
3.4 and Lustre 2.6.18.8 again.

cd /usr/local/src

wget http://bits.xensource.com/oss-xen/release/3.4.0/xen-3.4.0.tar.gz
tar -zxvf xen-3.4.0-xen.tar.gz

wget http://10.x.x.x/lustre-1.8.0.1.tar.gz
tar -zxvf lustre-1.8.0.1.tar.gz

cd /usr/local/src/xen-3.4.0
make linux-2.6-xen-config CONFIGMODE=menuconfig
KERNELS="linux-2.6-xen"
# I changed only the processor type to EM64T
make -j 8 world
make install

cd /usr/local/src/xen-3.4.0/linux-2.6.18-xen.hg/
cp /boot/config-2.6.18.8-xen .config

ln -s ../../lustre-1.8.0.1/lustre/kernel_patches/patches/ .
ln -s ../../lustre-1.8.0.1/lustre/kernel_patches/series/2.6.18-vanilla.series
series
quilt -av push
rm series ; rm patches;
ln -s
../../lustre-1.8.0.1/ldiskfs/kernel_patches/series/ldiskfs-2.6.18-vanilla.series
series
ln -s ../../lustre-1.8.0.1/ldiskfs/kernel_patches/patches/ .
quilt -av push

make menuconfig
# i marked ext3 and jbd
rm /boot/vmlinuz-2.6.18.8-xen
rm /boot/System.map-2.6.18.8-xen
rm -fr /lib/modules/2.6.18.8-xen
make
make modules
make modules_install install
mkinitramfs -o /boot/initrd.img-2.6.18.8-xen 2.6.18.8-xen

update-grub
# i''m using debian to this setup

This doesn''t worked for me.

I also tried to change the order of application of patches.
  - First ldiskfs then lustre
The Dom0 boots, but there is a problem with ext3 as fallow:

ext3:Unknown symbol c_move_locked
ext3:Unknown symbol d_rehash_cond
modprobe: FATAL: Error inserting ext3
(/lib/modules/2.6.18-8-xen/kernel/fs/ext3/ext3.ko) Unknown symbol in
module or unknow parameter


  - Only Lustre.
I have the fallowing error doing make to compile the kernel

CC      fs/jbd/checkpoint.o
fs/jbd/checkpoint.c: In function ?log_do_checkpoint?:
fs/jbd/checkpoint.c:352: error: too few arguments to function ?__process_buffer?
make[2]: *** [fs/jbd/checkpoint.o] Error 1
make[1]: *** [fs/jbd] Error 2
make: *** [fs] Error 2

The vanilla 2.6.30.2 does not have all patches to enable all features
of xen 3.4 dom0. It should run as a domU, but i haven''t tested it.

The lustre kernel rpm, is working very well in machines with linux
distributions based on RedHat Enterprise 5, but, it does not boot up
as dom0.

Please, can you give any direction to make it work? Does Citrix
XenServer or Sun xVm supports Lustre 1.8?

Thanks in advance.
--
Ettore Enrico Delfino Ligorio
ettoredelfinoligorio at gmail.com
55-11-9145-6151



On Mon, Jul 20, 2009 at 1:25 AM, Oleg Drokin<Oleg.Drokin at sun.com>
wrote:
> Hello!
>
> On Jul 17, 2009, at 2:01 PM, Ettore Enrico Delfino Ligorio wrote:
>>
>> In my experince, the integration between most recent kernels with
>> glusterfs and patches of Xen hypervisor works well. The same with
>> Lustre is harder to do.
>
> Works out of the box for me both with rhel5 kernels (that contain
> xen patches already) and vanilla 2.6.30 (that also happen to contain
> domU stuff).
> Also some Lustre developers extensively use xen for their testing.
> What is the specific problem you are seeing, I wonder?
>
> Bye,
> ? ?Oleg
>

Hello!

On Jul 23, 2009, at 10:45 AM, Ettore Enrico Delfino Ligorio wrote:
> ln -s ../../lustre-1.8.0.1/lustre/kernel_patches/patches/ .
> ln -s ../../lustre-1.8.0.1/lustre/kernel_patches/series/2.6.18- 
> vanilla.series
> series
> quilt -av push
> rm series ; rm patches;
> ln -s ../../lustre-1.8.0.1/ldiskfs/kernel_patches/series/ 
> ldiskfs-2.6.18-vanilla.series
> series
> ln -s ../../lustre-1.8.0.1/ldiskfs/kernel_patches/patches/ .
> quilt -av push
No need to apply ldiskfs patches to the kernel, this is a wrong step  
you need to omit.
> make menuconfig
> # i marked ext3 and jbd
> rm /boot/vmlinuz-2.6.18.8-xen
> rm /boot/System.map-2.6.18.8-xen
> rm -fr /lib/modules/2.6.18.8-xen
> make
> make modules
> make modules_install install
> mkinitramfs -o /boot/initrd.img-2.6.18.8-xen 2.6.18.8-xen
>
> update-grub
> # i''m using debian to this setup
>
> This doesn''t worked for me.
What was the error or how it did not work?
I did not try with xen-distruibutes 2.6.18-8, I only tried with centos  
kernels.
> I also tried to change the order of application of patches.
>  - First ldiskfs then lustre
> The Dom0 boots, but there is a problem with ext3 as fallow:
>
> ext3:Unknown symbol c_move_locked
> ext3:Unknown symbol d_rehash_cond
> modprobe: FATAL: Error inserting ext3
> (/lib/modules/2.6.18-8-xen/kernel/fs/ext3/ext3.ko) Unknown symbol in
> module or unknow parameter
Well, I guess at at any time quilt only applies one series file.
If you apply ldiskfs first, then other symbols are not exported.
>  - Only Lustre.
> I have the fallowing error doing make to compile the kernel
>
> CC      fs/jbd/checkpoint.o
> fs/jbd/checkpoint.c: In function ?log_do_checkpoint?:
> fs/jbd/checkpoint.c:352: error: too few arguments to function  
> ?__process_buffer?
> make[2]: *** [fs/jbd/checkpoint.o] Error 1
> make[1]: *** [fs/jbd] Error 2
> make: *** [fs] Error 2
Hm, I have not looked at them xen-supplied kernel might have  
sufficiently
diverged from mainline (or centos kernel).
> The lustre kernel rpm, is working very well in machines with linux
> distributions based on RedHat Enterprise 5, but, it does not boot up
> as dom0.
Well, we do not build our kernel rpms with XEN dom0 support.
If you need dom0 support, get redhat kernel we support,
patch it, use our supplied config, but enable various XEN features in  
it.
That should work.
> Please, can you give any direction to make it work? Does Citrix
> XenServer or Sun xVm supports Lustre 1.8?
I don''t know. Anything that can work with kernels supported by us
(which includes rebuilding those services from scratch, not necessary  
pre-built
binaries) should work.

Bye,
     Oleg