samba - Aug 2017 - extremely low performance on Samba 4.2.14-Debian

Le Fri, 11 Aug 2017 11:55:03 +0100
Rowland Penny via samba <samba at lists.samba.org> écrivait:
> If you are going to upgrade, you might as well go for the highest
> version you can, this will probably mean using Louis's packages.
> 
> Samba is a rapidly changing program, a new minor version is released
> every 6 months (approx) and the changes are significant.
> 
> 4.7.0 is due out at in September and there are going to be a lot of
> improvements in it.
I've upgraded to 4.5.12 : absolutely no difference. smbd consumes 100%
of CPU when writing or reading a paltry 100 MB/s. I'll try 4.6.x but
I'm desperate. I'll probably need to replace the server altogether (and
it's a PITA because it's 600 km away).

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Signature digitale OpenPGP
URL:
<http://lists.samba.org/pipermail/samba/attachments/20170816/9f2359a9/attachment.sig>

Le Wed, 16 Aug 2017 21:55:05 +0200
Emmanuel Florac via samba <samba at lists.samba.org> écrivait:
> > 
> > Samba is a rapidly changing program, a new minor version is released
> > every 6 months (approx) and the changes are significant.
> > 
> > 4.7.0 is due out at in September and there are going to be a lot of
> > improvements in it.  
> 
> I've upgraded to 4.5.12 : absolutely no difference. smbd consumes 100%
> of CPU when writing or reading a paltry 100 MB/s. I'll try 4.6.x but
> I'm desperate. I'll probably need to replace the server altogether
> (and it's a PITA because it's 600 km away).
And here we go: 4.6.7 isn't any better. Unfortunately I'm out of ideas
and out of resources, I'll switch servers for  an old, faithful,
obsolete Opteron-based one.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Signature digitale OpenPGP
URL:
<http://lists.samba.org/pipermail/samba/attachments/20170816/af035037/attachment.sig>

Hai Emmanuel, 

But ive re-read the complete thread.
Its strange that the old machine is so much faster.
In thinging a bit, and it can go two ways.. 
I'm trowing in things i get in my head based on experiance aka wild guesses,
but review it i suggest.

Samba bug/problem in settings combinations ( or wait for multi threaded samba
4.7. and test that)

Hardware/linux (driver/setting) combination, i'll come to that later on.. 

For samba. 
Can try with the following to see if one of these is slowing things down. 
In thinking in (kernel higher = better cifs (smb protocol support), and maybe
improvements in xfs support (i haven't checked that).
And check with modinfo the nic driver. 
I want to know if it possible to see the (by samba) detected interface speed. (
and mtu/tcp size rwin ).

Smb.conf suggestions, and test order.
Change: 
server signing = auto
Test, 
Then again add:
ntlm auth = no
Test,
And im questioning these 2. 
In global you set. 
restrict anonymous 2

In the share.
guest ok = no

The google translater makes crap off:  This parameter nullifies the benefits of
setting restrict anonymous = 2  :-/
Are these to settings conflicting or not.. In not sure here.
>From man smb.conf 
      guest ok (S)

           If this parameter is yes for a service, then no password is required
to connect to the service. Privileges will be those of the guest account.
           This parameter nullifies the benefits of setting restrict anonymous =
2
           See the section below on security for more information about this
option.
           Default: guest ok = no


About the hardware/linux. 

Did you install/setup the hardware, (not judging your knowlidge, my thoughts)
When you installed it, did you check the pci-e bus and the irq relations in the
bios.
It is possible, the 40G nic is shareing the bus with the raid 6 contoller. 
That can slow down things. 
Bios raid/hdds/mainboard up to date, so changelog review cant harm. 
Is the nic using firmware? 
The output of : 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Emmanuel Florac via samba
> Verzonden: woensdag 16 augustus 2017 22:00
> Aan: Emmanuel Florac via samba
> Onderwerp: Re: [Samba] extremely low performance on Samba 
> 4.2.14-Debian
> 
> Le Wed, 16 Aug 2017 21:55:05 +0200
> Emmanuel Florac via samba <samba at lists.samba.org> écrivait:
> 
> > > 
> > > Samba is a rapidly changing program, a new minor version 
> is released 
> > > every 6 months (approx) and the changes are significant.
> > > 
> > > 4.7.0 is due out at in September and there are going to 
> be a lot of 
> > > improvements in it.
> > 
> > I've upgraded to 4.5.12 : absolutely no difference. smbd 
> consumes 100% 
> > of CPU when writing or reading a paltry 100 MB/s. I'll try 
> 4.6.x but 
> > I'm desperate. I'll probably need to replace the server
altogether
> > (and it's a PITA because it's 600 km away).
> 
> And here we go: 4.6.7 isn't any better. Unfortunately I'm out 
> of ideas and out of resources, I'll switch servers for  an 
> old, faithful, obsolete Opteron-based one.
> 
> --
> --------------------------------------------------------------
> ----------
> Emmanuel Florac     |   Direction technique
>                     |   Intellique
>                     |	<eflorac at intellique.com>
>                     |   +33 1 78 94 84 02
> --------------------------------------------------------------
> ----------
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

I HATE these auto send keys.. Did i tell that already..  
( see below ) 
> -----Oorspronkelijk bericht-----
> Van: L.P.H. van Belle [mailto:belle at bazuin.nl] 
> Verzonden: woensdag 16 augustus 2017 23:22
> Aan: 'samba at lists.samba.org'
> Onderwerp: RE: [Samba] extremely low performance on Samba 
> 4.2.14-Debian
> 
> Hai Emmanuel, 
> 
> But ive re-read the complete thread.
> Its strange that the old machine is so much faster.
> In thinging a bit, and it can go two ways.. 
> I'm trowing in things i get in my head based on experiance 
> aka wild guesses, but review it i suggest.
> 
> Samba bug/problem in settings combinations ( or wait for 
> multi threaded samba 4.7. and test that)
> 
> Hardware/linux (driver/setting) combination, i'll come to 
> that later on.. 
> 
> For samba. 
> Can try with the following to see if one of these is slowing 
> things down. 
> In thinking in (kernel higher = better cifs (smb protocol 
> support), and maybe improvements in xfs support (i haven't 
> checked that). 
> And check with modinfo the nic driver. 
> I want to know if it possible to see the (by samba) detected 
> interface speed. ( and mtu/tcp size rwin ).
> 
> Smb.conf suggestions, and test order.
> Change: 
> server signing = auto
> Test,
> Then again add:
> ntlm auth = no
> Test,
> And im questioning these 2. 
> In global you set. 
> restrict anonymous 2
> 
> In the share.
> guest ok = no
> 
> The google translater makes crap off:  This parameter 
> nullifies the benefits of setting restrict anonymous = 2  :-/ 
> Are these to settings conflicting or not.. In not sure here.
> 
> From man smb.conf 
>       guest ok (S)
> 
>            If this parameter is yes for a service, then no 
> password is required to connect to the service. Privileges 
> will be those of the guest account.
>            This parameter nullifies the benefits of setting 
> restrict anonymous = 2
>            See the section below on security for more 
> information about this option.
>            Default: guest ok = no
> 
> 
> About the hardware/linux. 
> 
> Did you install/setup the hardware, (not judging your 
> knowlidge, my thoughts) When you installed it, did you check 
> the pci-e bus and the irq relations in the bios.
> It is possible, the 40G nic is shareing the bus with the raid 
> 6 contoller. 
> That can slow down things. 
> Bios raid/hdds/mainboard up to date, so changelog review cant harm. 
> Is the nic using firmware?
> The output of : 
ethtool -i 

So few thing, i can think of i would check. 
Its a pain, special when its far, the samba options are thing you might be able
to remotely check.
Maybe it helps a bit. 

Greetz, 

Louis

On Wed, 16 Aug 2017 23:28:08 +0200
"L.P.H. van Belle via samba" <samba at lists.samba.org> wrote:
> I HATE these auto send keys.. Did i tell that already..  
> ( see below ) 
> > -----Oorspronkelijk bericht-----
> > Van: L.P.H. van Belle [mailto:belle at bazuin.nl] 
> > Verzonden: woensdag 16 augustus 2017 23:22
> > Aan: 'samba at lists.samba.org'
> > Onderwerp: RE: [Samba] extremely low performance on Samba 
> > 4.2.14-Debian
> > 
> > Hai Emmanuel, 
> > 
> > But ive re-read the complete thread.
> > Its strange that the old machine is so much faster.
> > In thinging a bit, and it can go two ways.. 
> > I'm trowing in things i get in my head based on experiance 
> > aka wild guesses, but review it i suggest.
> > 
> > Samba bug/problem in settings combinations ( or wait for 
> > multi threaded samba 4.7. and test that)
> > 
> > Hardware/linux (driver/setting) combination, i'll come to 
> > that later on.. 
> > 
> > For samba. 
> > Can try with the following to see if one of these is slowing 
> > things down. 
> > In thinking in (kernel higher = better cifs (smb protocol 
> > support), and maybe improvements in xfs support (i haven't 
> > checked that). 
> > And check with modinfo the nic driver. 
> > I want to know if it possible to see the (by samba) detected 
> > interface speed. ( and mtu/tcp size rwin ).
> > 
> > Smb.conf suggestions, and test order.
> > Change: 
> > server signing = auto
> > Test,
> > Then again add:
> > ntlm auth = no
> > Test,
> > And im questioning these 2. 
> > In global you set. 
> > restrict anonymous 2
> > 
> > In the share.
> > guest ok = no
> > 
> > The google translater makes crap off:  This parameter 
> > nullifies the benefits of setting restrict anonymous = 2  :-/ 
> > Are these to settings conflicting or not.. In not sure here.
> > 
> > From man smb.conf 
> >       guest ok (S)
> > 
> >            If this parameter is yes for a service, then no 
> > password is required to connect to the service. Privileges 
> > will be those of the guest account.
> >            This parameter nullifies the benefits of setting 
> > restrict anonymous = 2
> >            See the section below on security for more 
> > information about this option.
> >            Default: guest ok = no
> > 
> > 
> > About the hardware/linux. 
> > 
> > Did you install/setup the hardware, (not judging your 
> > knowlidge, my thoughts) When you installed it, did you check 
> > the pci-e bus and the irq relations in the bios.
> > It is possible, the 40G nic is shareing the bus with the raid 
> > 6 contoller. 
> > That can slow down things. 
> > Bios raid/hdds/mainboard up to date, so changelog review cant harm. 
> > Is the nic using firmware?
> > The output of : 
> ethtool -i 
> 
> So few thing, i can think of i would check. 
> Its a pain, special when its far, the samba options are thing you
> might be able to remotely check. Maybe it helps a bit. 
> 
> Greetz, 
> 
> Louis 
> 
> 
> 
> 
> 
I would suggest that the OP tries this smb.conf:

[global]
    netbios name = storiq-111
    server string = %h server (Samba, Debian)
    security = ADS
    workgroup = TEST
    realm = AD.TEST.COM

    winbind refresh tickets = yes
    winbind use default domain = yes
    winbind cache time = 7200
    winbind offline logon = yes

    idmap config *:backend = tdb
    idmap config *:range = 2000-9999
    idmap config TEST:backend = rid
    idmap config TEST:range = 10000-50000000

    template shell = /bin/bash
    template homedir = /mnt/raid/%u

    log file = /var/log/samba/smbd.log
    max log size = 50

    vfs objects = acl_xattr 
    map acl inherit = yes
    store dos attributes = yes

[test_tr]
    # Set ACLs from Windows
    comment = Utilisateurs de test_tr
    path = /mnt/raid/test_tr
    read only = no

It doesn't have any default settings or any settings that might slow
things down (I also do not understand the reason behind having smb.conf
in three sections.)

Rowland

Le Wed, 16 Aug 2017 23:28:08 +0200
"L.P.H. van Belle via samba" <samba at lists.samba.org>
écrivait:
> > Bios raid/hdds/mainboard up to date, so changelog review cant harm. 
> > Is the nic using firmware?
> > The output of :   
> ethtool -i 
> 
> So few thing, i can think of i would check. 
> Its a pain, special when its far, the samba options are thing you
> might be able to remotely check. Maybe it helps a bit. 
> 
I've checked everything round and round. Speed is normal with FTP, but
not with Samba (I have no other Linux machine on the network to test
NFS). iperf3 proves that the NICs perform normally. I've tested several
kernels, 2 different NICs (Silicom 40G and Myri 10G), same low
samba performance. 

I've built 3 machines on the same configuration (same RAID controller,
Myri 10G nics, Xeon 6420, 64 GB RAM, same mobo, etc) and only this one
is so slow. But only with Samba, everything else performs normally.

It's really crazy.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Signature digitale OpenPGP
URL:
<http://lists.samba.org/pipermail/samba/attachments/20170817/233ca91c/attachment.sig>

Hai, 

Ok, so you have 3 the same machines and only one is slow. Thats important info. 
Did anything special happen with this server. 
For example 2 servers got a new install and this one had an upgrade (or sort of
upgrade) from the "old" server?

Or you configured the new server in an other (ad site) and moved it to the new
location.
In tring to figure out, where what happend, so we can review these steps and
point of pitfalls.

Can you describe the steps you took when you 
- installed a the "problem" new server.
- how you upgraded and/or installed new. 
- was the server move from site to other site ( ! In the AD ) 
- did it have an ip number change when you moved it to the other site.
- did you make backups on the old and restored them on the new? 
( if yes, what and where ) 

You steps, and preffered in order you did them. If you able to recall them all. 
;-)

If you have 3 (exact) the same machines, and all are same hardware. 
Then or one has failing hardware, or something got corrupted. 

So any info can help more. 

Greetz, 

Louis



> -----Oorspronkelijk bericht-----
> Van: Emmanuel Florac [mailto:eflorac at intellique.com] 
> Verzonden: donderdag 17 augustus 2017 14:01
> Aan: L.P.H. van Belle via samba
> CC: L.P.H. van Belle
> Onderwerp: Re: [Samba] extremely low performance on Samba 
> 4.2.14-Debian
> 
> Le Wed, 16 Aug 2017 23:28:08 +0200
> "L.P.H. van Belle via samba" <samba at lists.samba.org>
écrivait:
> 
> > > Bios raid/hdds/mainboard up to date, so changelog review 
> cant harm. 
> > > Is the nic using firmware?
> > > The output of :   
> > ethtool -i
> > 
> > So few thing, i can think of i would check. 
> > Its a pain, special when its far, the samba options are thing you 
> > might be able to remotely check. Maybe it helps a bit.
> > 
> 
> I've checked everything round and round. Speed is normal with 
> FTP, but not with Samba (I have no other Linux machine on the 
> network to test NFS). iperf3 proves that the NICs perform 
> normally. I've tested several kernels, 2 different NICs 
> (Silicom 40G and Myri 10G), same low samba performance. 
> 
> I've built 3 machines on the same configuration (same RAID 
> controller, Myri 10G nics, Xeon 6420, 64 GB RAM, same mobo, 
> etc) and only this one is so slow. But only with Samba, 
> everything else performs normally.
> 
> It's really crazy.
> 
> --
> --------------------------------------------------------------
> ----------
> Emmanuel Florac     |   Direction technique
>                     |   Intellique
>                     |	<eflorac at intellique.com>
>                     |   +33 1 78 94 84 02
> --------------------------------------------------------------
> ----------
>