samba - Jul 2016 - Winbind process stuck at 100% after changing use_mmap to no

Hi Volker,

I apologise if I came across as an a***hole here. Responses below:

On 05/07/16 20:45, Volker Lendecke wrote:
> On Tue, Jul 05, 2016 at 08:12:59PM +0100, Alex Crow wrote:
>>
>> On 05/07/16 19:45, Volker Lendecke wrote:
>>> On Tue, Jul 05, 2016 at 07:21:16PM +0100, Alex Crow wrote:
>>>> I've set up the "DR" side of my cluster to
"use mmap = no" and with
>>>> "private dir" removed from the smb.conf.
>>> Why do you set "use mmap = no"?
>>>
>>>> I have the MooseFS guys on the case as well. Should I put them
in touch
>>>> with you or vice/versa?
>>> MooseFS should not have any business at all in tdb files. If you
still
>>> find tdb files used on MooseFS, you should investigate your
>>> configuration and modify it such that tdb files only exist on local
>>> file systems.
>>>
>>> Volker
>> There are *no* more tdb files on MooseFS, that's what I'm
trying to test.
> Why are you still using "use mmap = no" then?
Because the wiki article does not explicitly state that mmap coherence 
is only needed for areas that store tdb files. I'd perhaps incorrectly 
deduced that as you had explicitly said that "private dir" on the 
cluster FS was a bad idea, that disabling that would stop all tdb files 
being written to the MooseFS FS, and that would be the solution.

My understanding was the "use mmap" affected general FS operations
from
clients, not just tdb files.
>
>>  From the ping_pong wiki page it's instructed to use "use_mmap
= no" if
> As I sad in an earlier mail: The wiki page is wrong. If you don't trust
> one of the primary developers of Samba and to a lesser extent ctdb
> (myself, sorry for bragging) and trust the wiki more, then I am a bit
> lost, sorry.
I am completely aware of who you and Jeremy are and I have the utmost 
respect and thanks for the amazing software you have provided. We're 
planning our migration to Samba 4 AD (from classic) right now.
>
>> Text here:
>>
>> "
>>
>>
>>      Testing mmap coherence
>>
>> If you add the -m switch to ping_pong along with -rw then it will do
the
>> I/O coherence test via mmap. It isn't absolutely essential that a
>                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
> This is the key piece. It is not essential.
Again, I was not clear if it meant that "it's not essential, so if it 
fails, you *must* disable mmap, if it doesn't fail, no worries but 
perhaps do it just in case". However there is no mention anywhere, 
including in the smb.conf man page, that it may lead to huge perfomance 
impact.
>
>> cluster filesystem supports coherent mmap for CTDB/Samba, but it's
nice
>> for bragging points over other cluster filesystems. If your cluster
>> filesystem doesn't pass this test then just use the "use mmap
= no"
>> option in smb.conf. Even if it does pass this test that option may be a
>> good idea on most cluster filesystems."
> This piece is wrong. The wiki should give you hints who wrote this.
> Please contact the author of those lines, refer to the mail thread
> here and ask these lines to be changed or even better -- change these
> lines yourself.
I see what you mean here, but I'd not want to edit a wiki when I have 
less idea of the facts that the person that wrote the original text! I'd 
not want to update it without knowing the precise facts, hence my 
reaching out to the Samba list and yourself.
>
>> So is this documentation wrong as well? It seems to suggest setting it
> Yes, the documentation is wrong.
>
>> anyway, which is clearly not working for us (and now for sure there is
>> no involvement with TDB files as I've deleted the "private
dir" in smb.conf.
>>
>>
>> The only reason I was asking if you'd like to talk to the MooseFS
chaps
>> is because of the failure of the -rw test, which as far as the
>> documentation goes, seems to refer to the shared FS rather than any
>> places where tdb files live.
> If MooseFS does not even survive -rw without asking for mmap coherence,
> just don't use it for Samba. This will eat your data for breakfast.
> Gluster was there loooong ago, and they fixed it before becoming a
> viable storage solution for Samba.
>
> Volker
MooseFS does not ask for mmap coherence. It has inconsistent results on 
a plain -rw test. I think the chaps over there are interested in gaining 
some input from your considerable experience. If they have to pay fees 
via Sernet or we do, I'm not bothered. I already have them going over my 
setup and checking everything, so I'm hoping it's a misconfiguration on 
my part. The other choice was GlusterFS but its small-file performance 
was almost an order of magnitude below MooseFS.

Many thanks

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On Tue, Jul 05, 2016 at 09:06:54PM +0100, Alex Crow
wrote:
> >Why are you still using "use mmap = no" then?
> Because the wiki article does not explicitly state that mmap
> coherence is only needed for areas that store tdb files. I'd perhaps
Maybe that is the omission in the docs. I thought I had mentioned that
"use mmap" is only relevant for tdb files, not for plain exported
shares.
> incorrectly deduced that as you had explicitly said that "private
> dir" on the cluster FS was a bad idea, that disabling that would
> stop all tdb files being written to the MooseFS FS, and that would
> be the solution.
The solution is to use ctdb with the database files on local
directories. That's what ctdb is for.
> My understanding was the "use mmap" affected general FS
operations
> from clients, not just tdb files.
It just affects tdb files.
> >This piece is wrong. The wiki should give you hints who wrote this.
> >Please contact the author of those lines, refer to the mail thread
> >here and ask these lines to be changed or even better -- change these
> >lines yourself.
> I see what you mean here, but I'd not want to edit a wiki when I
> have less idea of the facts that the person that wrote the original
> text! I'd not want to update it without knowing the precise facts,
> hence my reaching out to the Samba list and yourself.
Well, now you should have a better understanding of the issue :-)
> >If MooseFS does not even survive -rw without asking for mmap coherence,
> >just don't use it for Samba. This will eat your data for breakfast.
> >Gluster was there loooong ago, and they fixed it before becoming a
> >viable storage solution for Samba.
> >
> >Volker
> 
> MooseFS does not ask for mmap coherence. It has inconsistent results
> on a plain -rw test.
MooseFS needs to fix that. ping_pong with -rw is pretty basic Posix
stuff. Admittedly it seems to be very hard for cluster file systems to
implement, but it is essential if you want to provide Posix semantics.
Samba and SMB clients depend on this. SMB clients coordinate via byte
range locks, and data coherence coordinated by those locks just basically
assumed.

ping_pong was explicitly designed by Andrew Tridgell as a torture test
case for cluster file systems developers. It does not come with the
huge SMB baggage that is difficult to set up for cluster file systems
developers. The calls it makes are simple enough to be understandable
in file system traces.

Volker

On 05/07/16 21:21, Volker Lendecke wrote:
> On Tue, Jul 05, 2016 at 09:06:54PM +0100, Alex Crow wrote:
>>> Why are you still using "use mmap = no" then?
>> Because the wiki article does not explicitly state that mmap
>> coherence is only needed for areas that store tdb files. I'd
perhaps
> Maybe that is the omission in the docs. I thought I had mentioned that
> "use mmap" is only relevant for tdb files, not for plain exported
> shares.
Perhaps I did not read one of your replies carefully enough, so I
apologise if that is the case.
>
>> incorrectly deduced that as you had explicitly said that "private
>> dir" on the cluster FS was a bad idea, that disabling that would
>> stop all tdb files being written to the MooseFS FS, and that would
>> be the solution.
> The solution is to use ctdb with the database files on local
> directories. That's what ctdb is for.
Yes, I'm clear on that now - I'm just surprised that the "private
dir"
stuff is all over the internet, not just on the Samba wiki. Sometimes
this global sharing community does go a bit wrong!
>
>> My understanding was the "use mmap" affected general FS
operations
>> from clients, not just tdb files.
> It just affects tdb files.
Thanks for the confirmation, I was already suspecting this was the case.
>
>>> This piece is wrong. The wiki should give you hints who wrote this.
>>> Please contact the author of those lines, refer to the mail thread
>>> here and ask these lines to be changed or even better -- change
these
>>> lines yourself.
>> I see what you mean here, but I'd not want to edit a wiki when I
>> have less idea of the facts that the person that wrote the original
>> text! I'd not want to update it without knowing the precise facts,
>> hence my reaching out to the Samba list and yourself.
> Well, now you should have a better understanding of the issue :-)
I do indeed and shall edit the page as much as I feel confident to do so.
>
>>> If MooseFS does not even survive -rw without asking for mmap
coherence,
>>> just don't use it for Samba. This will eat your data for
breakfast.
>>> Gluster was there loooong ago, and they fixed it before becoming a
>>> viable storage solution for Samba.
>>>
>>> Volker
>> MooseFS does not ask for mmap coherence. It has inconsistent results
>> on a plain -rw test.
> MooseFS needs to fix that. ping_pong with -rw is pretty basic Posix
> stuff. Admittedly it seems to be very hard for cluster file systems to
> implement, but it is essential if you want to provide Posix semantics.
> Samba and SMB clients depend on this. SMB clients coordinate via byte
> range locks, and data coherence coordinated by those locks just basically
> assumed.
As I said on another email that has crossed this, I have the devs
probing the problem right now. As it uses an central in-memory metadata
server for all locking, I'm hoping it will be an easy fix for them.
>
> ping_pong was explicitly designed by Andrew Tridgell as a torture test
> case for cluster file systems developers. It does not come with the
> huge SMB baggage that is difficult to set up for cluster file systems
> developers. The calls it makes are simple enough to be understandable
> in file system traces.
Yes, I looked at the code and it is indeed terse but brutal!

Many thanks

Alex


--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).