samba - Jul 2016 - Winbind process stuck at 100% after changing use_mmap to no

On 05/07/16 19:45, Volker Lendecke wrote:
> On Tue, Jul 05, 2016 at 07:21:16PM +0100, Alex Crow wrote:
>> I've set up the "DR" side of my cluster to "use mmap
= no" and with
>> "private dir" removed from the smb.conf.
> Why do you set "use mmap = no"?
>
>> I have the MooseFS guys on the case as well. Should I put them in touch
>> with you or vice/versa?
> MooseFS should not have any business at all in tdb files. If you still
> find tdb files used on MooseFS, you should investigate your
> configuration and modify it such that tdb files only exist on local
> file systems.
>
> Volker
Hi Volker,

There are *no* more tdb files on MooseFS, that's what I'm trying to
test.

From the ping_pong wiki page it's instructed to use "use_mmap =
no" if
the ping_pong test does not work correctly on "ping_ping -m -rw
test.file", which it did, completely.

Text here:

"


    Testing mmap coherence

If you add the -m switch to ping_pong along with -rw then it will do the
I/O coherence test via mmap. It isn't absolutely essential that a
cluster filesystem supports coherent mmap for CTDB/Samba, but it's nice
for bragging points over other cluster filesystems. If your cluster
filesystem doesn't pass this test then just use the "use mmap =
no"
option in smb.conf. Even if it does pass this test that option may be a
good idea on most cluster filesystems."


So is this documentation wrong as well? It seems to suggest setting it
anyway, which is clearly not working for us (and now for sure there is
no involvement with TDB files as I've deleted the "private dir" in
smb.conf.


The only reason I was asking if you'd like to talk to the MooseFS chaps
is because of the failure of the -rw test, which as far as the
documentation goes, seems to refer to the shared FS rather than any
places where tdb files live.

I hope you can see what I'm getting at here!

Many thanks

Alex




--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On Tue, Jul 05, 2016 at 08:12:59PM +0100, Alex Crow
wrote:
> 
> 
> On 05/07/16 19:45, Volker Lendecke wrote:
> > On Tue, Jul 05, 2016 at 07:21:16PM +0100, Alex Crow wrote:
> >> I've set up the "DR" side of my cluster to "use
mmap = no" and with
> >> "private dir" removed from the smb.conf.
> > Why do you set "use mmap = no"?
> >
> >> I have the MooseFS guys on the case as well. Should I put them in
touch
> >> with you or vice/versa?
> > MooseFS should not have any business at all in tdb files. If you still
> > find tdb files used on MooseFS, you should investigate your
> > configuration and modify it such that tdb files only exist on local
> > file systems.
> >
> > Volker
> 
> There are *no* more tdb files on MooseFS, that's what I'm trying to
test.
Why are you still using "use mmap = no" then?
> From the ping_pong wiki page it's instructed to use "use_mmap =
no" if
As I sad in an earlier mail: The wiki page is wrong. If you don't trust
one of the primary developers of Samba and to a lesser extent ctdb
(myself, sorry for bragging) and trust the wiki more, then I am a bit
lost, sorry.
> Text here:
> 
> "
> 
> 
>     Testing mmap coherence
> 
> If you add the -m switch to ping_pong along with -rw then it will do the
> I/O coherence test via mmap. It isn't absolutely essential that a
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
This is the key piece. It is not essential.
> cluster filesystem supports coherent mmap for CTDB/Samba, but it's nice
> for bragging points over other cluster filesystems. If your cluster
> filesystem doesn't pass this test then just use the "use mmap =
no"
> option in smb.conf. Even if it does pass this test that option may be a
> good idea on most cluster filesystems."
This piece is wrong. The wiki should give you hints who wrote this.
Please contact the author of those lines, refer to the mail thread
here and ask these lines to be changed or even better -- change these
lines yourself.
> So is this documentation wrong as well? It seems to suggest setting it
Yes, the documentation is wrong.
> anyway, which is clearly not working for us (and now for sure there is
> no involvement with TDB files as I've deleted the "private
dir" in smb.conf.
> 
> 
> The only reason I was asking if you'd like to talk to the MooseFS chaps
> is because of the failure of the -rw test, which as far as the
> documentation goes, seems to refer to the shared FS rather than any
> places where tdb files live.
If MooseFS does not even survive -rw without asking for mmap coherence,
just don't use it for Samba. This will eat your data for breakfast.
Gluster was there loooong ago, and they fixed it before becoming a
viable storage solution for Samba.

Volker

FYI, by "it did, completely" I meant it failed completely.

Even if the only file we have to have on the cluster FS (which now seems 
to be down solely to the ctdb lock file), I'm still worried what these 
failures mean:

1) does the -rw (without -m) test suggest any problems with the MooseFS 
FS I'm serving files to users from?
2) does the mmap test failure mean I should not even have my CTDB lock 
file on MooseFS?
3) Do the inconsistent reports from a ping_pong -rw test show there is a 
major problem with MooseFS as the underlying FS for Samba exports and 
something needs to be done with the underlying FS?

I'm a paying licensee of MooseFS Pro, so I hope you can understand my 
wishes to get this problem sorted. If contracting some of the Sernet 
chaps is required, we've done it before and might be prepared to do so 
in this case.

Best regards

Alex

On 05/07/16 20:12, Alex Crow wrote:
>
>
> On 05/07/16 19:45, Volker Lendecke wrote:
>> On Tue, Jul 05, 2016 at 07:21:16PM +0100, Alex Crow wrote:
>>> I've set up the "DR" side of my cluster to "use
mmap = no" and with
>>> "private dir" removed from the smb.conf.
>> Why do you set "use mmap = no"?
>>
>>> I have the MooseFS guys on the case as well. Should I put them in
touch
>>> with you or vice/versa?
>> MooseFS should not have any business at all in tdb files. If you still
>> find tdb files used on MooseFS, you should investigate your
>> configuration and modify it such that tdb files only exist on local
>> file systems.
>>
>> Volker
>
> Hi Volker,
>
> There are *no* more tdb files on MooseFS, that's what I'm trying to
test.
>
> From the ping_pong wiki page it's instructed to use "use_mmap =
no" if
> the ping_pong test does not work correctly on "ping_ping -m -rw 
> test.file", which it did, completely.
>
> Text here:
>
> "
>
>
>     Testing mmap coherence
>
> If you add the -m switch to ping_pong along with -rw then it will do 
> the I/O coherence test via mmap. It isn't absolutely essential that a 
> cluster filesystem supports coherent mmap for CTDB/Samba, but it's 
> nice for bragging points over other cluster filesystems. If your 
> cluster filesystem doesn't pass this test then just use the "use
mmap
> = no" option in smb.conf. Even if it does pass this test that option 
> may be a good idea on most cluster filesystems."
>
>
> So is this documentation wrong as well? It seems to suggest setting it 
> anyway, which is clearly not working for us (and now for sure there is 
> no involvement with TDB files as I've deleted the "private
dir" in
> smb.conf.
>
>
> The only reason I was asking if you'd like to talk to the MooseFS 
> chaps is because of the failure of the -rw test, which as far as the 
> documentation goes, seems to refer to the shared FS rather than any 
> places where tdb files live.
>
> I hope you can see what I'm getting at here!
>
> Many thanks
>
> Alex
>
>
>

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On 05/07/16 20:49, Alex Crow wrote:
> FYI, by "it did, completely" I meant it failed completely.
>
> Even if the only file we have to have on the cluster FS (which now 
> seems to be down solely to the ctdb lock file), I'm still worried what 
> these failures mean:
>
> 1) does the -rw (without -m) test suggest any problems with the 
> MooseFS FS I'm serving files to users from?
> 2) does the mmap test failure mean I should not even have my CTDB lock 
> file on MooseFS?
> 3) Do the inconsistent reports from a ping_pong -rw test show there is 
> a major problem with MooseFS as the underlying FS for Samba exports 
> and something needs to be done with the underlying FS?
>
> I'm a paying licensee of MooseFS Pro, so I hope you can understand my 
> wishes to get this problem sorted. If contracting some of the Sernet 
> chaps is required, we've done it before and might be prepared to do so 
> in this case.
>
>
ROFL ROFL

Do you know who Volker is ???

Please listen to him, do what he advises and don't bother contacting 
Sernet, because you already have.

Rowland

Hi Volker,

I apologise if I came across as an a***hole here. Responses below:

On 05/07/16 20:45, Volker Lendecke wrote:
> On Tue, Jul 05, 2016 at 08:12:59PM +0100, Alex Crow wrote:
>>
>> On 05/07/16 19:45, Volker Lendecke wrote:
>>> On Tue, Jul 05, 2016 at 07:21:16PM +0100, Alex Crow wrote:
>>>> I've set up the "DR" side of my cluster to
"use mmap = no" and with
>>>> "private dir" removed from the smb.conf.
>>> Why do you set "use mmap = no"?
>>>
>>>> I have the MooseFS guys on the case as well. Should I put them
in touch
>>>> with you or vice/versa?
>>> MooseFS should not have any business at all in tdb files. If you
still
>>> find tdb files used on MooseFS, you should investigate your
>>> configuration and modify it such that tdb files only exist on local
>>> file systems.
>>>
>>> Volker
>> There are *no* more tdb files on MooseFS, that's what I'm
trying to test.
> Why are you still using "use mmap = no" then?
Because the wiki article does not explicitly state that mmap coherence 
is only needed for areas that store tdb files. I'd perhaps incorrectly 
deduced that as you had explicitly said that "private dir" on the 
cluster FS was a bad idea, that disabling that would stop all tdb files 
being written to the MooseFS FS, and that would be the solution.

My understanding was the "use mmap" affected general FS operations
from
clients, not just tdb files.
>
>>  From the ping_pong wiki page it's instructed to use "use_mmap
= no" if
> As I sad in an earlier mail: The wiki page is wrong. If you don't trust
> one of the primary developers of Samba and to a lesser extent ctdb
> (myself, sorry for bragging) and trust the wiki more, then I am a bit
> lost, sorry.
I am completely aware of who you and Jeremy are and I have the utmost 
respect and thanks for the amazing software you have provided. We're 
planning our migration to Samba 4 AD (from classic) right now.
>
>> Text here:
>>
>> "
>>
>>
>>      Testing mmap coherence
>>
>> If you add the -m switch to ping_pong along with -rw then it will do
the
>> I/O coherence test via mmap. It isn't absolutely essential that a
>                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
> This is the key piece. It is not essential.
Again, I was not clear if it meant that "it's not essential, so if it 
fails, you *must* disable mmap, if it doesn't fail, no worries but 
perhaps do it just in case". However there is no mention anywhere, 
including in the smb.conf man page, that it may lead to huge perfomance 
impact.
>
>> cluster filesystem supports coherent mmap for CTDB/Samba, but it's
nice
>> for bragging points over other cluster filesystems. If your cluster
>> filesystem doesn't pass this test then just use the "use mmap
= no"
>> option in smb.conf. Even if it does pass this test that option may be a
>> good idea on most cluster filesystems."
> This piece is wrong. The wiki should give you hints who wrote this.
> Please contact the author of those lines, refer to the mail thread
> here and ask these lines to be changed or even better -- change these
> lines yourself.
I see what you mean here, but I'd not want to edit a wiki when I have 
less idea of the facts that the person that wrote the original text! I'd 
not want to update it without knowing the precise facts, hence my 
reaching out to the Samba list and yourself.
>
>> So is this documentation wrong as well? It seems to suggest setting it
> Yes, the documentation is wrong.
>
>> anyway, which is clearly not working for us (and now for sure there is
>> no involvement with TDB files as I've deleted the "private
dir" in smb.conf.
>>
>>
>> The only reason I was asking if you'd like to talk to the MooseFS
chaps
>> is because of the failure of the -rw test, which as far as the
>> documentation goes, seems to refer to the shared FS rather than any
>> places where tdb files live.
> If MooseFS does not even survive -rw without asking for mmap coherence,
> just don't use it for Samba. This will eat your data for breakfast.
> Gluster was there loooong ago, and they fixed it before becoming a
> viable storage solution for Samba.
>
> Volker
MooseFS does not ask for mmap coherence. It has inconsistent results on 
a plain -rw test. I think the chaps over there are interested in gaining 
some input from your considerable experience. If they have to pay fees 
via Sernet or we do, I'm not bothered. I already have them going over my 
setup and checking everything, so I'm hoping it's a misconfiguration on 
my part. The other choice was GlusterFS but its small-file performance 
was almost an order of magnitude below MooseFS.

Many thanks

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).