Lustre discuss - Oct 2009 - strange performance with POSIX file capabilities

Dear list,

with newer vanilla kernels we  saw strange performance
data with iozone on patchless clients: some OSTs had a lower write
bandwith in the iozone benchmark, getting worse with record
sizes below 1024.
After lots of kernel builds, it looks like the kernel config entry
CONFIG_SECURITY_FILE_CAPABILITIES is the one, wich
introduces this problem. If CONFIG_SECURITY_FILE_CAPABILITIES
is not set, iozone data look good, if it''s compiled into the
kernel, we see the problem:
http://www.physik.uni-augsburg.de/~ralfu/LustreTest/Lustre_with_file_caps.html

Any idea, why file capabilities should affect the write
performance on Lustre, and why it should only affect some OSTs?

bye, Ralf
-- 
        Ralf Utermann
_____________________________________________________________________
        Universit?t Augsburg, Institut f?r Physik   --   EDV-Betreuer
        Universit?tsstr.1             
        D-86135 Augsburg                     Phone:  +49-821-598-3231
        SMTP: Ralf.Utermann at Physik.Uni-Augsburg.DE         Fax: -3411

On Oct 06, 2009  15:13 +0200, Ralf Utermann wrote:
> with newer vanilla kernels we  saw strange performance
> data with iozone on patchless clients: some OSTs had a lower write
> bandwith in the iozone benchmark, getting worse with record
> sizes below 1024.
> After lots of kernel builds, it looks like the kernel config entry
> CONFIG_SECURITY_FILE_CAPABILITIES is the one, wich
> introduces this problem. If CONFIG_SECURITY_FILE_CAPABILITIES
> is not set, iozone data look good, if it''s compiled into the
> kernel, we see the problem:
>
http://www.physik.uni-augsburg.de/~ralfu/LustreTest/Lustre_with_file_caps.html
Just to clarify, you are reporting the above config option affects
write performance when changed on the client, correct?  It appears
that this option is off by default in the upstream kernels, so I
suspect it doesn''t get tested much.
> Any idea, why file capabilities should affect the write
> performance on Lustre, and why it should only affect some OSTs?
I can imagine that if this is adding some significant overhead on a
per-system-call basis that it would hurt performance.

It is definitely odd that it would affect the performance of only some
of the OSTs.  I assume they are otherwise identical?  The only thing
I can imagine is that this option is related to SELinux and has some
overhead in getting extended attributes, but even then the xattrs are
only stored on the MDS so this would hurt all OSTs uniformly.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Andreas Dilger wrote:
> On Oct 06, 2009  15:13 +0200, Ralf Utermann wrote:
>> with newer vanilla kernels we  saw strange performance
>> data with iozone on patchless clients: some OSTs had a lower write
>> bandwith in the iozone benchmark, getting worse with record
>> sizes below 1024.
>> After lots of kernel builds, it looks like the kernel config entry
>> CONFIG_SECURITY_FILE_CAPABILITIES is the one, wich
>> introduces this problem. If CONFIG_SECURITY_FILE_CAPABILITIES
>> is not set, iozone data look good, if it''s compiled into the
>> kernel, we see the problem:
>>
http://www.physik.uni-augsburg.de/~ralfu/LustreTest/Lustre_with_file_caps.html
> 
> Just to clarify, you are reporting the above config option affects
> write performance when changed on the client, correct?  It appears
Hi Andreas,

Yes, this option has only been used on the client side. The servers
are running a 2.6.22 kernel and it looks like this option has been
introduced with 2.6.24.
> that this option is off by default in the upstream kernels, so I
> suspect it doesn''t get tested much.
This option is set on by default in the Debian kernels, and that''s the
config
I usually start with. I think, recent fedora kernels would also have this set,
and also RHEL6.
> 
>> Any idea, why file capabilities should affect the write
>> performance on Lustre, and why it should only affect some OSTs?
> 
> I can imagine that if this is adding some significant overhead on a
> per-system-call basis that it would hurt performance.
> 
> It is definitely odd that it would affect the performance of only some
> of the OSTs.  I assume they are otherwise identical?  The only thing
the OSTs are either 4 or 8 data disks on Sun 6140 systems; the 4 with
problems are on 2 OSS, the 3 without problems are on the other 2 OSS.
> I can imagine is that this option is related to SELinux and has some
> overhead in getting extended attributes, but even then the xattrs are
> only stored on the MDS so this would hurt all OSTs uniformly.
As I don''t need this option anyway, I will just build my kernels now
with this option off. Of course an unpleasant feeling remains, not
knowing what really happens ...

As off vanilla kernel 2.6.29 there should be a no_file_caps kernel boot
parameter. I would like to test this setup, but b1_8 only builds fine with
vanilla 2.6.28, I cannot get it running with vanilla 2.6.[29|30] -- but
this should be different thread ...

Bye, Ralf
-- 
        Ralf Utermann
_____________________________________________________________________
        Universit?t Augsburg, Institut f?r Physik   --   EDV-Betreuer
        Universit?tsstr.1             
        D-86135 Augsburg                     Phone:  +49-821-598-3231
        SMTP: Ralf.Utermann at Physik.Uni-Augsburg.DE         Fax: -3411