Lustre discuss - Jul 2009 - Selection of kernel options for the distributed Lustre kernels

Dear all,

what determines the options selected for the distributed Lustre
kernels?

I''m asking as (if I am not mistaken and amongst other things)
iSCSI for instance is enabled for a standard RHEL5 kernel, while
it is disabled for the RHEL5 Lustre kernel.

Thanks,
  Arne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090720/2b9cec8a/attachment.bin

On Mon, 2009-07-20 at 11:53 +0200, Arne Wiebalck wrote:
> Dear all,
Hi,
> what determines the options selected for the distributed Lustre
> kernels?
We try to stick as closely as we can to the vendor''s selected options.
> I''m asking as (if I am not mistaken and amongst other things)
> iSCSI for instance is enabled for a standard RHEL5 kernel, while
> it is disabled for the RHEL5 Lustre kernel.
So I can follow up on this, can I ask how you are determining this?

Which Lustre release and kernel are you referencing?

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090720/6dcb973e/attachment.bin

Hi Brian,
> We try to stick as closely as we can to the vendor''s selected
options.
That''s what I thought :)
>> I''m asking as (if I am not mistaken and amongst other things)
>> iSCSI for instance is enabled for a standard RHEL5 kernel, while
>> it is disabled for the RHEL5 Lustre kernel.
> 
> So I can follow up on this, can I ask how you are determining this?
The Lustre kernel:

-->
[root~]# grep -i iscsi /boot/config-2.6.18-128.1.6.el5_lustre.1.8.0.1smp
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_ISCSI_TCP is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_ISCSI_IBFT_FIND is not set
<--

and (what I believe to be) a standard RHE5 kernel:

-->
[root~]# grep -i iscsi /boot/config-2.6.18-128.el5
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_ISCSI_TCP=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
<--
> Which Lustre release and kernel are you referencing?
lustre 1.8.0.1, RHEL 5, x86_64

Please note that I am not 100% sure that the standard RedHat config
has that enabled. But in case you confirm this one, I have some more
discrepancies ;-)

Thanks for looking into this,
  Arne

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090720/fcba4e70/attachment-0001.bin

On Mon, 2009-07-20 at 17:50 +0200, Arne Wiebalck wrote:
> Hi Brian,
Hi Arne,
> That''s what I thought :)
You have found one of our deviations though.
> The Lustre kernel:
> 
> -->
> [root~]# grep -i iscsi /boot/config-2.6.18-128.1.6.el5_lustre.1.8.0.1smp
> # CONFIG_SCSI_ISCSI_ATTRS is not set
> # CONFIG_ISCSI_TCP is not set
> # CONFIG_SCSI_QLA_ISCSI is not set
> # CONFIG_ISCSI_IBFT_FIND is not set
> <--
Right.

In addition to providing an updated I/B stack from OFED we also provide
the iSCSI stack from OFED as well, as it''s generally "newer"
than what
the vendor provides.  In order to do so and minimize confusion with the
vendor supplied kernel, we disable the vendor''s iSCSI (as well as
infiniband) stacks.

You will find the iSCSI stack in the kernel-ib package we distribute
with our release.  Let me know if that doesn''t pan out.  iSCSI is not
something we routinely test due to lack of hardware but more
importantly, simply lack of desire in our customer base.  iSCSI is just
not a common deployment method within the Lustre customer base.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090720/a2abe029/attachment.bin

Hi Brian,
> In addition to providing an updated I/B stack from OFED we also provide
> the iSCSI stack from OFED as well, as it''s generally
"newer" than what
> the vendor provides.  In order to do so and minimize confusion with the
> vendor supplied kernel, we disable the vendor''s iSCSI (as well as
> infiniband) stacks.
> 
> You will find the iSCSI stack in the kernel-ib package we distribute
> with our release.  Let me know if that doesn''t pan out.  iSCSI is
not
> something we routinely test due to lack of hardware but more
> importantly, simply lack of desire in our customer base.  iSCSI is just
> not a common deployment method within the Lustre customer base.
We plan to use iSCSI arrays over TCP/IP as the shared storage for the 
MDT. So for me it would be of course much more convenient to have the
corresponding support for that in the pre-built rpms as it saves me the 
hassle to re-compile (so much for customer demand ;-).

Also, as you can turn a disk server into an iSCSI target I would think 
that this will more and more become an option to create relatively cheap 
shared storage for the MDT.

Any chance that at least the TCP/IP iSCSI initiator stuff is set back to 
the RHEL defaults?

-->
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_ISCSI_TCP=m
CONFIG_SCSI_QLA_ISCSI=m
<--

Thanks,
  Arne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090721/6e1b43af/attachment.bin

On Tue, 2009-07-21 at 10:59 +0200, Arne Wiebalck wrote:
> Hi Brian,
Hi Arne,
> We plan to use iSCSI arrays over TCP/IP as the shared storage for the 
> MDT. So for me it would be of course much more convenient to have the
> corresponding support for that in the pre-built rpms as it saves me the 
> hassle to re-compile (so much for customer demand ;-).
It is.  As I said in my last e-mail, the iSCSI support is in the
kernel-ib RPM as we build the iSCSI stack from the OFED release.
> Also, as you can turn a disk server into an iSCSI target I would think 
> that this will more and more become an option to create relatively cheap 
> shared storage for the MDT.
Perhaps.  Perhaps not.  I would think that TCP/IP to the storage adds
latencies.  Latencies would be quite bad for the MDT.
> Any chance that at least the TCP/IP iSCSI initiator stuff is set back to 
> the RHEL defaults?
Is the iSCSI stack in the kernel-ib RPM not working for you?  Please
describe if that is the case.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090721/4b6eeafb/attachment.bin

Hi Brian,
> It is.  As I said in my last e-mail, the iSCSI support is in the
> kernel-ib RPM as we build the iSCSI stack from the OFED release.
Ups, sorry, seems I missed that point.
> 
>> Also, as you can turn a disk server into an iSCSI target I would think 
>> that this will more and more become an option to create relatively
cheap
>> shared storage for the MDT.
> 
> Perhaps.  Perhaps not.  I would think that TCP/IP to the storage adds
> latencies.  Latencies would be quite bad for the MDT.
Yes ... perhaps :)
Let''s see.
>> Any chance that at least the TCP/IP iSCSI initiator stuff is set back
to
>> the RHEL defaults?
> 
> Is the iSCSI stack in the kernel-ib RPM not working for you?  Please
> describe if that is the case.
I will try it out and let you know.

Thanks for you help,
  Arne

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090721/a05ce59c/attachment-0001.bin

Hi Brian,
>> Is the iSCSI stack in the kernel-ib RPM not working for you?  Please
>> describe if that is the case.
> 
> I will try it out and let you know.
I reinstalled the pre-built kernel from the rpm as available from the 
Lustre website and installed the kernel-ib package as well:

-->
[root at host ~]# rpm -qa | grep kernel-lustre
kernel-lustre-smp-2.6.18-128.1.6.el5_lustre.1.8.0.1

[root at host ~]# rpm -qa | grep kernel-ib
kernel-ib-1.4.1-2.6.18_128.1.6.el5

[root at host ~]# uname -a
Linux host 2.6.18-128.1.6.el5_lustre.1.8.0.1smp #1 SMP Thu Jun 18 
17:06:38 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux
<--

It seems that the iSCSI modules cannot be loaded:

-->
[root at host ~]# modprobe scsi_transport_iscsi
FATAL: Module scsi_transport_iscsi not found.
<--

That''s probably as modprobe will check in
/lib/modules/2.6.18-128.1.6.el5_lustre.1.8.0.1smp
not in
/lib/modules/2.6.18-128.1.6.el5
where kernel-ib installs its modules

So, insmod maybe?

-->
[root at host ~]# insmod 
/lib/modules/2.6.18-128.1.6.el5/updates/kernel/drivers/scsi/scsi_transport_iscsi.ko
insmod: error inserting 
''/lib/modules/2.6.18-128.1.6.el5/updates/kernel/drivers/scsi/scsi_transport_iscsi.ko'':
-1 Invalid module format
<--

Am I missing anything obvious here?

TIA,
  Arne


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090721/6a646c23/attachment.bin