samba - Aug 2008 - Samba write performance in kernel

hi,

I would like to know is it possible to make writing file to samba completely in
kernel?

I'm using a slow CPU (FA526) , and the memory copy is even slower. The
reading performance is over 7 MB/s, with mmap and sendfile enabled, while
writing is only 4-5 MB/s. Without mmap and sendfile, reading from samba is also
about 4-5 MB/s. I use Oprofile to profile writing file to samba and found that
CPU takes over 30% CPU time on copy_from/to_user, so I think going to user space
and back again is the bottleneck.

Since there is sendfile, why is'nt there counterpart on write path? Is there
some difficalties or what? Is it implementable?

Please give me some advice.


Best Regards,
Mac Lin

_________________________________________________________________
Áo©ú·j´M©MÂsÄýºô¸ôªº§K¶O¤u¨ã¦C ¡X MSN ·j´M¤u¨ã¦C 
http://toolbar.live.com/

Lin Mac wrote:
> hi,
>
> I would like to know is it possible to make writing file to samba
completely in kernel?
>
> I'm using a slow CPU (FA526) , and the memory copy is even slower. The
reading performance is over 7 MB/s, with mmap and sendfile enabled, while
writing is only 4-5 MB/s. Without mmap and sendfile, reading from samba is also
about 4-5 MB/s. I use Oprofile to profile writing file to samba and found that
CPU takes over 30% CPU time on copy_from/to_user, so I think going to user space
and back again is the bottleneck.
>
> Since there is sendfile, why is'nt there counterpart on write path? Is
there some difficalties or what? Is it implementable?
>
> Please give me some advice.
>
>
> Best Regards,
> Mac Lin
>
>   
Are you using DMA, or are you copying byte by byte through the CPU?

On Tue, Aug 26, 2008 at 05:41:51PM +0800, Lin Mac wrote:
> 
> hi,
> 
> I would like to know is it possible to make writing file to samba
completely in kernel?
Yes.
> I'm using a slow CPU (FA526) , and the memory copy is even slower. The
reading performance is over 7 MB/s, with mmap and sendfile enabled, while
writing is only 4-5 MB/s. Without mmap and sendfile, reading from samba is also
about 4-5 MB/s. I use Oprofile to profile writing file to samba and found that
CPU takes over 30% CPU time on copy_from/to_user, so I think going to user space
and back again is the bottleneck.
> 
> Since there is sendfile, why is'nt there counterpart on write path? Is
there some difficalties or what? Is it implementable?
There is a counterpart on the write path, called RECEIVEFILE
in the Samba 3.2.x code. It's already coded up in userspace,
just waiting the kernel counterpart :-). If there's no kernel
receivefile() call then it emulates this in userspace.

Try plumbing in the Linux splice() system call (this is
already coded up in Samba, awaiting working kernel support).

Jeremy.

hi,
> Ok, this is different. I had missed that you are talking
> about a small device with slow memory bandwith. In that
> case, you might certainly gain something by avoiding the
> copies. If you are really memcpy-bound, you should
> definitely make splice work.
yes, but the question is by how much can it improve. We would like to see it
reach 7Mbps in 2 or 3 months, can't just waiting for linux kernel fix the
problem. Is there any individuals/company capable of doing such performance
improvement for charge?

> At the high end, latencies is mostly what kills your
> performance. Mostly you have enough bandwidth, but if you
> just do a simple request->response scheme, you can't get
> beyond a certain overall bandwith that is way below the
> theoretical network bandwith. To fill that, you need to make
> the client issue parallel requests. Multi-threaded windows
> client apps can do it, smbclient from 3.2 does it.
I have tried multiple threads but it doesn't help the performance in my
case. The delay caused by memory copy seems dominate.

Best Regards,
Mac Lin> Date: Fri, 12 Sep 2008 09:45:32 +0200
> From: Volker.Lendecke@SerNet.DE
> To: mkl23@hotmail.com
> CC: jra@samba.org; samba@lists.samba.org
> Subject: Re: [Samba] Samba write performance in kernel
> 
> On Fri, Sep 12, 2008 at 02:43:25PM +0800, Lin Mac wrote:
> > well, in my condition, it might be 30% instead of 5%, IF
> > splice can cover advantage 1 and 2.
> > 1. I'm using a slow CPU (FA526) , and the memory copy is even
slower.
> > 2. The reading performance is over 7 MB/s, with mmap and
> > sendfile enabled, while writing is only 4-5 MB/s. Without
> > mmap and sendfile, reading from samba is also about 4-5
> > MB/s. 
> > 3. I used Oprofile to profile writing file to samba and
> > found that CPU takes over 30% CPU time on
> > copy_from/to_user, so I think going to user space and back
> > again is the bottleneck.
> > 4. My device is only 100Mbps Ethernet
> > 5. I uses Windows client to measure throughput
> 
> Ok, this is different. I had missed that you are talking
> about a small device with slow memory bandwith. In that
> case, you might certainly gain something by avoiding the
> copies. If you are really memcpy-bound, you should
> definitely make splice work.
> 
> > > here, but the network latencies together with non-optimally
> > > queued requests by the client have a MUCH greater influence.
> > 1. If splice works, can memory copy be avoided?
> > 2. Sorry I don't really get what the "non-optimally queued
> > requests" means. And what could I do to make it optimized?
> 
> At the high end, latencies is mostly what kills your
> performance. Mostly you have enough bandwidth, but if you
> just do a simple request->response scheme, you can't get
> beyond a certain overall bandwith that is way below the
> theoretical network bandwith. To fill that, you need to make
> the client issue parallel requests. Multi-threaded windows
> client apps can do it, smbclient from 3.2 does it.
> 
> Volker


_________________________________________________________________
Áo©ú·j´M©MÂsÄýºô¸ôªº§K¶O¤u¨ã¦C ¡X MSN ·j´M¤u¨ã¦C 
http://toolbar.live.com/From dbb at st-andrews.ac.uk  Fri Sep 12 11:20:30 2008
From: dbb at st-andrews.ac.uk (Duncan Brannen)
Date: Fri Sep 12 11:21:55 2008
Subject: [Samba] Solaris nss_ldap vs PADL nss_ldap
In-Reply-To: <89813.57993.qm@web39803.mail.mud.yahoo.com>
References: <89813.57993.qm@web39803.mail.mud.yahoo.com>
Message-ID: <48CA507E.2070101@st-andrews.ac.uk>


Hi Alban,
                You can download padl's nss_ldap library from 
http://www.padl.com/Contents/OpenSourceSoftware.html

If you've already configured solaris for groups and password in LDAP, it 
should just work once you replace the Solaris
nss_ldap with the padl one ( back it up first ;)  and add / configure 
/etc/ldap.conf

mine looks like
> TLS_CACERT /etc/certs/cacert.pem
> TLSCIPHERSUITE TLSv1
> host ldap.st-andrews.ac.uk
> rootbinddn <DN of admin user for doing lookups>
> base ou=People,dc=st-andrews,dc=ac,dc=uk
> ldap_version 3
> nss_base_passwd ou=People,dc=st-andrews,dc=ac,dc=uk?one
> nss_base_shadow ou=People,dc=st-andrews,dc=ac,dc=uk?one
> nss_base_group          ou=Groups,dc=st-andrews,dc=ac,dc=uk?one
> ssl start_tls
> tls_cacertfile /etc/certs/<pem encoded public key of our signing 
> certificate?
> tls_cacertdir /etc/certs
> tls_ciphers TLSv1
With the admin user password in /etc/ldap.secret permission 600.


You could also try  group: compat as suggested by Douglas Engert,  I've 
not managed to get back to trying this yet.
> Have you tried using Solaris version withthis in the nsswitch.conf:
>
>  group: compat
>  group_compat ldap
>
> and adding the + in the /etc/group file.
>
> This appears to work as expected, getting groups info from both
> local and ldap.
>
> Or (I have not tried this):
>
>  group: files [SUCCESS=continue] ldap
Cheers,
          Duncan



albanperso-zatoo@yahoo.com wrote:
> Hi Duncan,
>
> I have the same issue on Solaris and Samba (3.028a and 3.31) that is OK for
primary groups but not for secondaries.
>
> can you describe how do you get / configurePADL's nss_ldap?
>
> Thanks in advance
>
> Regards
>
> Alban
>
>
> ----- Message d'origine ----
>   
>> De : Duncan Brannen <dbb@st-andrews.ac.uk>
>> ? : samba@lists.samba.org
>> Envoy? le : Mercredi, 27 Ao?t 2008, 18h09mn 55s
>> Objet : [Samba] Solaris nss_ldap vs PADL nss_ldap
>>
>>
>>
>> Hi All,
>>           Any thoughts on why, while everything seems ok at the OS
level
>> (getent , id -a ) Samba
>> doesn't pickup any supplementary groups when Solaris is configured
with
>> 'group: files ldap' in
>> nsswitch.conf and using it's own native nss_ldap.so.1 but does when
>> using PADL's nss_ldap?
>> Everything else is equal.
>>
>> Do they use/accept different calls or could it be an openldap vs native
>> ldap incompatibility,
>> Samba being compiled against the openldap libraries.
>>
>> Samba seems not to compile against the native libraries due to a lack
of
>> ldap_start_tls_s
>>
>> Solaris 10 and Samba 3.2.2
>>
>> Cheers,
>>           Duncan
>>
>> -- 
>> The University of St Andrews is a charity registered in Scotland : No
SC013532
>>
>> -- 
>> To unsubscribe from this list go to the following URL and read the
>> instructions:  https://lists.samba.org/mailman/listinfo/samba
>>     
>
>
>
>       
>   

-- 
The University of St Andrews is a charity registered in Scotland : No SC013532

hi,

Just do some debugging, and found something. I hope this do help. Correct me if
anything wrong.

I'm using samba-3.2.2.
For some reason, Linux can not splice a socket and a filefd directly, which
samba does so.

In linux, sys_splice calls do_splice, which would require either in or out fd to
be a PIPE(FIFO).
Both parameter (fromfd, tofd) that  samba sent with splice failed the test
S_ISFIFO() in linux.

Maybe it would work to add an addition FIFO? splice(fromfd, FIFO) and
splice(FIFO, tofd)?

Best Regards,
Mac Lin
> Date: Thu, 18 Sep 2008 16:01:29 +0200
> From: Volker.Lendecke@SerNet.DE
> To: mkl23@hotmail.com
> CC: jra@samba.org; samba@lists.samba.org
> Subject: Re: [Samba] Samba write performance in kernel
> 
> On Thu, Sep 18, 2008 at 09:45:43PM +0800, Lin Mac wrote:
> > > On Mon, Sep 15, 2008 at 02:44:32PM +0800, Lin Mac wrote:
> > > This is a kernel implementation detail. Remember Samba
> > > runs on many other kernels, not just Linux. Linux has
> > > splice, *BSD has receivefile, Solaris probably has
> > > something different.
> 
> > Samba using splice on Linux doesn't work, that is for sure. 
> 
> And there's no way that this can be fixed?
> 
> Volker
_________________________________________________________________
¥Î³¡¸¨®æ¤À¨É·Ó¤ù¡B¼v­µ¡B½ì¨ý¤p¤u¨ã©M³Ì·R²M³æ¡AºÉ±¡¨q¥X§A¦Û¤v ¡X Windows Live
Spaces
http://spaces.live.com/From hpsekhon at googlemail.com  Thu Sep 18 14:37:21 2008
From: hpsekhon at googlemail.com (Hari Sekhon)
Date: Thu Sep 18 14:43:25 2008
Subject: [Samba] Rid generation
In-Reply-To: <200809181521.54035.npf-mlists@eurotux.com>
References: <200809181521.54035.npf-mlists@eurotux.com>
Message-ID: <48D267A1.2090402@googlemail.com>

Nuno Fernandes wrote:
> [root@server samba]# pdbedit -L -v user1
> User SID:             S-1-5-21-1454471165-2146950999-672003340-1416
>
> Where does it get the 1416 Rid from?
>   
Samba does not generate the Rid, that is a Microsoft Domain Controller 
function (allocation of rids is controlled through the Rid master if I 
remember my MCSE days correctly)

Samba is simply querying the domain controller for the Sid. If samba is 
being used as a dc, then I guess it must basically do the same thing as 
Microsoft and generate the rid from 1000 onwards incrementally for each 
user, similar to what happens for unix uids I expect, since rids and 
uids are basically the same idea on different platforms.

-h

-- 
Hari Sekhon