On 03/11/16 19:37, Volker Lendecke wrote:> On Thu, Nov 03, 2016 at 07:11:12PM +0000, Alex Crow via samba wrote: >> >> On 03/11/16 14:28, Ralph Böhme wrote: >>>> Ah, my smb.conf is missing the "ctdb socket" parameter. According to "man >>>> smb.conf" this should be set, but I don't remember seeing it on the wiki >>>> pages. >>> if it's not set, the default is used which works in many cases, but >>> not in mine. If the default wasn't working in your setup, you'd >>> notice, as clients wouldn't be able to talk to ctdb at all. >>> >>> -slow >> I can confirm this made no difference. I see the locking working from >> windows between the first two servers but *not* the third. This seems to >> match what I had with smbclient. Again, could it be that an FS that >> passes ping_pong not be able to work in a CTDB cluster? >> >> The developers of my fs (MooseFS) have said that while file flock() is >> working perfectly there is still no directory flock. Could that be a >> pointer? >> > Does your system have consistent inodes and device ids? Does > > stat /some/shared/file > > give exactly the same information on all nodes, in particular the > Device/Inode fields? > > VolkerThanks Volker, No, it seems it's OK on two servers but not on the third: [root at zalma ~]# stat testfile ... Size: 1286700 Blocks: 2514 IO Block: 65536 regular file Device: 29h/41d Inode: 35820037 Links: 1 Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) Access: 2016-11-03 19:51:46.000000000 +0000 Modify: 2016-11-01 13:06:04.000000000 +0000 Change: 2016-11-01 13:06:04.000000000 +0000 Birth: - [root at zalma ~]# ssh zearing "stat testfile" root at zearing's password: ... Size: 1286700 Blocks: 2514 IO Block: 65536 regular file Device: 29h/41d Inode: 35820037 Links: 1 Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) Access: 2016-11-03 19:51:46.000000000 +0000 Modify: 2016-11-01 13:06:04.000000000 +0000 Change: 2016-11-01 13:06:04.000000000 +0000 Birth: - [root at zalma ~]# ssh metropolis "stat testfile" root at metropolis's password: ... Size: 1286700 Blocks: 2514 IO Block: 65536 regular file Device: 26h/38d Inode: 35820037 Links: 1 Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) Access: 2016-11-03 19:51:46.000000000 +0000 Modify: 2016-11-01 13:06:04.000000000 +0000 Change: 2016-11-01 13:06:04.000000000 +0000 Birth: - NB this is a FUSE mounted FS. What does the Device: XXh/YYd mean? Cheers Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Jeremy Allison
2016-Nov-03 21:26 UTC
[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
On Thu, Nov 03, 2016 at 09:21:46PM +0000, Alex Crow via samba wrote:> > > On 03/11/16 19:37, Volker Lendecke wrote: > > On Thu, Nov 03, 2016 at 07:11:12PM +0000, Alex Crow via samba wrote: > >> > >> On 03/11/16 14:28, Ralph Böhme wrote: > >>>> Ah, my smb.conf is missing the "ctdb socket" parameter. According to "man > >>>> smb.conf" this should be set, but I don't remember seeing it on the wiki > >>>> pages. > >>> if it's not set, the default is used which works in many cases, but > >>> not in mine. If the default wasn't working in your setup, you'd > >>> notice, as clients wouldn't be able to talk to ctdb at all. > >>> > >>> -slow > >> I can confirm this made no difference. I see the locking working from > >> windows between the first two servers but *not* the third. This seems to > >> match what I had with smbclient. Again, could it be that an FS that > >> passes ping_pong not be able to work in a CTDB cluster? > >> > >> The developers of my fs (MooseFS) have said that while file flock() is > >> working perfectly there is still no directory flock. Could that be a > >> pointer? > >> > > Does your system have consistent inodes and device ids? Does > > > > stat /some/shared/file > > > > give exactly the same information on all nodes, in particular the > > Device/Inode fields? > > > > Volker > Thanks Volker, > > No, it seems it's OK on two servers but not on the third:Ah, well that explains a lot :-). Samba uses the device/ino pair to determine if two files are the same. If they're different - to smbd then they're different files. That's why the locking doesn't work.> [root at zalma ~]# stat testfile > ... > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > Device: 29h/41d Inode: 35820037 Links: 1 > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > Access: 2016-11-03 19:51:46.000000000 +0000 > Modify: 2016-11-01 13:06:04.000000000 +0000 > Change: 2016-11-01 13:06:04.000000000 +0000 > Birth: - > [root at zalma ~]# ssh zearing "stat testfile" > root at zearing's password: > ... > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > Device: 29h/41d Inode: 35820037 Links: 1 > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > Access: 2016-11-03 19:51:46.000000000 +0000 > Modify: 2016-11-01 13:06:04.000000000 +0000 > Change: 2016-11-01 13:06:04.000000000 +0000 > Birth: - > [root at zalma ~]# ssh metropolis "stat testfile" > root at metropolis's password: > ... > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > Device: 26h/38d Inode: 35820037 Links: 1 > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > Access: 2016-11-03 19:51:46.000000000 +0000 > Modify: 2016-11-01 13:06:04.000000000 +0000 > Change: 2016-11-01 13:06:04.000000000 +0000 > Birth: - > > NB this is a FUSE mounted FS. > > What does the Device: XXh/YYd mean? > > Cheers > > Alex > -- > This message is intended only for the addressee and may contain > confidential information. Unless you are that person, you may not > disclose its contents or use it in any way and are requested to delete > the message along with any attachments and notify us immediately. > This email is not intended to, nor should it be taken to, constitute advice. > The information provided is correct to our knowledge & belief and must not > be used as a substitute for obtaining tax, regulatory, investment, legal or > any other appropriate advice. > > "Transact" is operated by Integrated Financial Arrangements Ltd. > 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. > (Registered office: as above; Registered in England and Wales under > number: 3727592). Authorised and regulated by the Financial Conduct > Authority (entered on the Financial Services Register; no. 190856). > > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba
Absolutely fantastic Jeremy. That's exactly what I needed and with the MooseFS chaps help this should be an easy fix. Your and Volker's insight much appreciated. Thanks to Ralph and everyone else who's piled into this! Goodnight, Alex On 3 November 2016 21:26:58 GMT+00:00, Jeremy Allison <jra at samba.org> wrote:>On Thu, Nov 03, 2016 at 09:21:46PM +0000, Alex Crow via samba wrote: >> >> >> On 03/11/16 19:37, Volker Lendecke wrote: >> > On Thu, Nov 03, 2016 at 07:11:12PM +0000, Alex Crow via samba >wrote: >> >> >> >> On 03/11/16 14:28, Ralph Böhme wrote: >> >>>> Ah, my smb.conf is missing the "ctdb socket" parameter. >According to "man >> >>>> smb.conf" this should be set, but I don't remember seeing it on >the wiki >> >>>> pages. >> >>> if it's not set, the default is used which works in many cases, >but >> >>> not in mine. If the default wasn't working in your setup, you'd >> >>> notice, as clients wouldn't be able to talk to ctdb at all. >> >>> >> >>> -slow >> >> I can confirm this made no difference. I see the locking working >from >> >> windows between the first two servers but *not* the third. This >seems to >> >> match what I had with smbclient. Again, could it be that an FS >that >> >> passes ping_pong not be able to work in a CTDB cluster? >> >> >> >> The developers of my fs (MooseFS) have said that while file >flock() is >> >> working perfectly there is still no directory flock. Could that be >a >> >> pointer? >> >> >> > Does your system have consistent inodes and device ids? Does >> > >> > stat /some/shared/file >> > >> > give exactly the same information on all nodes, in particular the >> > Device/Inode fields? >> > >> > Volker >> Thanks Volker, >> >> No, it seems it's OK on two servers but not on the third: > >Ah, well that explains a lot :-). > >Samba uses the device/ino pair to determine if two files >are the same. If they're different - to smbd then they're >different files. > >That's why the locking doesn't work. > >> [root at zalma ~]# stat testfile >> ... >> Size: 1286700 Blocks: 2514 IO Block: 65536 regular >file >> Device: 29h/41d Inode: 35820037 Links: 1 >> Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain >Users) >> Access: 2016-11-03 19:51:46.000000000 +0000 >> Modify: 2016-11-01 13:06:04.000000000 +0000 >> Change: 2016-11-01 13:06:04.000000000 +0000 >> Birth: - >> [root at zalma ~]# ssh zearing "stat testfile" >> root at zearing's password: >> ... >> Size: 1286700 Blocks: 2514 IO Block: 65536 regular >file >> Device: 29h/41d Inode: 35820037 Links: 1 >> Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain >Users) >> Access: 2016-11-03 19:51:46.000000000 +0000 >> Modify: 2016-11-01 13:06:04.000000000 +0000 >> Change: 2016-11-01 13:06:04.000000000 +0000 >> Birth: - >> [root at zalma ~]# ssh metropolis "stat testfile" >> root at metropolis's password: >> ... >> Size: 1286700 Blocks: 2514 IO Block: 65536 regular >file >> Device: 26h/38d Inode: 35820037 Links: 1 >> Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain >Users) >> Access: 2016-11-03 19:51:46.000000000 +0000 >> Modify: 2016-11-01 13:06:04.000000000 +0000 >> Change: 2016-11-01 13:06:04.000000000 +0000 >> Birth: - >> >> NB this is a FUSE mounted FS. >> >> What does the Device: XXh/YYd mean? >> >> Cheers >> >> Alex >> -- >> This message is intended only for the addressee and may contain >> confidential information. Unless you are that person, you may not >> disclose its contents or use it in any way and are requested to >delete >> the message along with any attachments and notify us immediately. >> This email is not intended to, nor should it be taken to, constitute >advice. >> The information provided is correct to our knowledge & belief and >must not >> be used as a substitute for obtaining tax, regulatory, investment, >legal or >> any other appropriate advice. >> >> "Transact" is operated by Integrated Financial Arrangements Ltd. >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) >7608 5300. >> (Registered office: as above; Registered in England and Wales under >> number: 3727592). Authorised and regulated by the Financial Conduct >> Authority (entered on the Financial Services Register; no. 190856). >> >> -- >> To unsubscribe from this list go to the following URL and read the >> instructions: https://lists.samba.org/mailman/options/samba-- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Volker Lendecke
2016-Nov-04 07:12 UTC
[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
On Thu, Nov 03, 2016 at 09:21:46PM +0000, Alex Crow via samba wrote:> [root at zalma ~]# stat testfile > ... > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > Device: 29h/41d Inode: 35820037 Links: 1 > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > Access: 2016-11-03 19:51:46.000000000 +0000 > Modify: 2016-11-01 13:06:04.000000000 +0000 > Change: 2016-11-01 13:06:04.000000000 +0000 > Birth: - > [root at zalma ~]# ssh zearing "stat testfile" > root at zearing's password: > ... > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > Device: 29h/41d Inode: 35820037 Links: 1 > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > Access: 2016-11-03 19:51:46.000000000 +0000 > Modify: 2016-11-01 13:06:04.000000000 +0000 > Change: 2016-11-01 13:06:04.000000000 +0000 > Birth: - > [root at zalma ~]# ssh metropolis "stat testfile" > root at metropolis's password: > ... > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > Device: 26h/38d Inode: 35820037 Links: 1 > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > Access: 2016-11-03 19:51:46.000000000 +0000 > Modify: 2016-11-01 13:06:04.000000000 +0000 > Change: 2016-11-01 13:06:04.000000000 +0000 > Birth: - > > NB this is a FUSE mounted FS. > > What does the Device: XXh/YYd mean?That's the so-called device number. Unix concept from the early days. You can use vfs objects = fileid and fileid:mapping = fsname to fix this issue. Volker
On 04/11/16 07:12, Volker Lendecke wrote:> On Thu, Nov 03, 2016 at 09:21:46PM +0000, Alex Crow via samba wrote: >> [root at zalma ~]# stat testfile >> ... >> Size: 1286700 Blocks: 2514 IO Block: 65536 regular file >> Device: 29h/41d Inode: 35820037 Links: 1 >> Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) >> Access: 2016-11-03 19:51:46.000000000 +0000 >> Modify: 2016-11-01 13:06:04.000000000 +0000 >> Change: 2016-11-01 13:06:04.000000000 +0000 >> Birth: - >> [root at zalma ~]# ssh zearing "stat testfile" >> root at zearing's password: >> ... >> Size: 1286700 Blocks: 2514 IO Block: 65536 regular file >> Device: 29h/41d Inode: 35820037 Links: 1 >> Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) >> Access: 2016-11-03 19:51:46.000000000 +0000 >> Modify: 2016-11-01 13:06:04.000000000 +0000 >> Change: 2016-11-01 13:06:04.000000000 +0000 >> Birth: - >> [root at zalma ~]# ssh metropolis "stat testfile" >> root at metropolis's password: >> ... >> Size: 1286700 Blocks: 2514 IO Block: 65536 regular file >> Device: 26h/38d Inode: 35820037 Links: 1 >> Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) >> Access: 2016-11-03 19:51:46.000000000 +0000 >> Modify: 2016-11-01 13:06:04.000000000 +0000 >> Change: 2016-11-01 13:06:04.000000000 +0000 >> Birth: - >> >> NB this is a FUSE mounted FS. >> >> What does the Device: XXh/YYd mean? > That's the so-called device number. Unix concept from the early days. > > You can use > > vfs objects = fileid > > and > > fileid:mapping = fsname > > to fix this issue. > > VolkerCan that go in the global section so it applies to all shares? And this should fix the CTDB locking problem? If so, brilliant! Cheers Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Jeremy Allison
2016-Nov-04 16:22 UTC
[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
On Fri, Nov 04, 2016 at 08:12:02AM +0100, Volker Lendecke via samba wrote:> On Thu, Nov 03, 2016 at 09:21:46PM +0000, Alex Crow via samba wrote: > > [root at zalma ~]# stat testfile > > ... > > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > > Device: 29h/41d Inode: 35820037 Links: 1 > > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > > Access: 2016-11-03 19:51:46.000000000 +0000 > > Modify: 2016-11-01 13:06:04.000000000 +0000 > > Change: 2016-11-01 13:06:04.000000000 +0000 > > Birth: - > > [root at zalma ~]# ssh zearing "stat testfile" > > root at zearing's password: > > ... > > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > > Device: 29h/41d Inode: 35820037 Links: 1 > > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > > Access: 2016-11-03 19:51:46.000000000 +0000 > > Modify: 2016-11-01 13:06:04.000000000 +0000 > > Change: 2016-11-01 13:06:04.000000000 +0000 > > Birth: - > > [root at zalma ~]# ssh metropolis "stat testfile" > > root at metropolis's password: > > ... > > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > > Device: 26h/38d Inode: 35820037 Links: 1 > > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > > Access: 2016-11-03 19:51:46.000000000 +0000 > > Modify: 2016-11-01 13:06:04.000000000 +0000 > > Change: 2016-11-01 13:06:04.000000000 +0000 > > Birth: - > > > > NB this is a FUSE mounted FS. > > > > What does the Device: XXh/YYd mean? > > That's the so-called device number. Unix concept from the early days. > > You can use > > vfs objects = fileid > > and > > fileid:mapping = fsname > > to fix this issue.Well that's an interesting little vfs module I don't remember existing ! You learn something new every day :-). I still think your filesystem should try and get dev/ino pairs the same on different nodes though in the long run.
Martin Schwenke
2016-Nov-07 05:42 UTC
[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
Hi Volker, On Fri, 4 Nov 2016 08:12:02 +0100, Volker Lendecke via samba <samba at lists.samba.org> wrote:> On Thu, Nov 03, 2016 at 09:21:46PM +0000, Alex Crow via samba wrote: > > [root at zalma ~]# stat testfile > > ... > > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > > Device: 29h/41d Inode: 35820037 Links: 1 > > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > > Access: 2016-11-03 19:51:46.000000000 +0000 > > Modify: 2016-11-01 13:06:04.000000000 +0000 > > Change: 2016-11-01 13:06:04.000000000 +0000 > > Birth: - > > [root at zalma ~]# ssh zearing "stat testfile" > > root at zearing's password: > > ... > > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > > Device: 29h/41d Inode: 35820037 Links: 1 > > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > > Access: 2016-11-03 19:51:46.000000000 +0000 > > Modify: 2016-11-01 13:06:04.000000000 +0000 > > Change: 2016-11-01 13:06:04.000000000 +0000 > > Birth: - > > [root at zalma ~]# ssh metropolis "stat testfile" > > root at metropolis's password: > > ... > > Size: 1286700 Blocks: 2514 IO Block: 65536 regular file > > Device: 26h/38d Inode: 35820037 Links: 1 > > Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users) > > Access: 2016-11-03 19:51:46.000000000 +0000 > > Modify: 2016-11-01 13:06:04.000000000 +0000 > > Change: 2016-11-01 13:06:04.000000000 +0000 > > Birth: - > > > > NB this is a FUSE mounted FS. > > > > What does the Device: XXh/YYd mean? > > That's the so-called device number. Unix concept from the early days. > > You can use > > vfs objects = fileid > > and > > fileid:mapping = fsname > > to fix this issue.I'm going to document this in the wiki (at current non-existent https://wiki.samba.org/index.php?title=Setting_up_a_cluster_filesystem&action=edit&redlink=1). It is already mentioned under OCFS2 section (https://wiki.samba.org/index.php/CTDB_Setup#OCFS2) without explanation. According to https://www.samba.org/samba/docs/man/manpages/vfs_fileid.8.html which I will point to from the wiki, the modern way seems to be to use fileid:algorithm = fsname instead of fileid:mapping = fsname Just a few quick questions: * This is only done at connect time so there's no performance penalty? That's my reading of the code but I didn't want to turn this into a "let's learn all about the VFS" exercise. :-) * If the device name is different across nodes then this doesn't help. I was wondering why there isn't an choice to hash m->mnt_dir since that will always be the same on each node (if you want a consistent share definition across the cluster). Perhaps there's no cluster filesystem so far that has needed this but has had inconsistent device names? * Is there anything similar for inconsistent/unstable inode numbers? That would clearly come with a performance penalty... unless caching, which is always an easy problem to solve... :-) Thanks... peace & happiness, martin