Hi Ralph, I applied the patch to a new build of Sernet 4.4.6 SRPM. I could still open the file from both hosts, and in fact this time I could actually open it on the same host without any problems: host 1: smbclient -U ajc //172.31.0.120/ifa_v ... smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609 213_ ISL Balance sheet variance analysis.xls" open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609 213_ ISL Balance sheet variance analysis.xls: for read/write fnum 56287 host 2: smbclient -U ajc //172.31.0.120/ifa_v ... smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609 213_ ISL Balance sheet variance analysis.xls" open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609 213_ ISL Balance sheet variance analysis.xls: for read/write fnum 17669 But this: # smbstatus -L Locked files: Pid Uid DenyMode Access R/W Oplock SharePath Name Time -------------------------------------------------------------------------------------------------- 1:19650 1046 DENY_NONE 0x83 RDWR NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls Sat Oct 22 19:26:06 2016 0:13503 1046 DENY_NONE 0x83 RDWR NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls Sat Oct 22 18:58:06 2016 Is odd as the patch was applied fine during the RPM build process. Is there something I need to do when building from sRPMs to make this stick? Cheers Alex On 21/10/16 18:06, Ralph Böhme wrote:> diff --git a/source3/client/client.c b/source3/client/client.c > index 831b9bc..28e98af 100644 > --- a/source3/client/client.c > +++ b/source3/client/client.c > @@ -2498,12 +2498,12 @@ static int cmd_open(void) > > status = cli_ntcreate(targetcli, targetname, 0, > FILE_READ_DATA|FILE_WRITE_DATA, 0, > - FILE_SHARE_READ|FILE_SHARE_WRITE, FILE_OPEN, > + 0, FILE_OPEN, > 0x0, 0x0, &fnum, NULL); > if (!NT_STATUS_IS_OK(status)) { > status = cli_ntcreate(targetcli, targetname, 0, > FILE_READ_DATA, 0, > - FILE_SHARE_READ|FILE_SHARE_WRITE, FILE_OPEN, > + 0, FILE_OPEN, > 0x0, 0x0, &fnum, NULL); > if (NT_STATUS_IS_OK(status)) { > d_printf("open file %s: for read/write fnum %d\n", targetname, fnum); > -- 2.7.4-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Ralph Böhme
2016-Oct-23 09:00 UTC
[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
Alex, On Sat, Oct 22, 2016 at 07:30:01PM +0100, Alex Crow wrote:> I applied the patch to a new build of Sernet 4.4.6 SRPM. I could still > open the file from both hosts, and in fact this time I could actually > open it on the same host without any problems:I don't see where you're opening the file twice from one smbclient below. Am I missing something? And then there's no need to use an xls file, simply create an empty file in the root of the share with `touch file` and use that.> > host 1: > > smbclient -U ajc //172.31.0.120/ifa_v > > ... > > smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609 > 213_ ISL Balance sheet variance analysis.xls" > open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609 > 213_ ISL Balance sheet variance analysis.xls: for read/write fnum 56287 > > host 2: > > smbclient -U ajc //172.31.0.120/ifa_v > > ... > > smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609 > 213_ ISL Balance sheet variance analysis.xls" > open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609 > 213_ ISL Balance sheet variance analysis.xls: for read/write fnum 17669 > > But this: > > # smbstatus -L > Locked files: > Pid Uid DenyMode Access R/W > Oplock SharePath Name Time > -------------------------------------------------------------------------------------------------- > 1:19650 1046 DENY_NONE 0x83 RDWR > NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd > Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls > Sat Oct 22 19:26:06 2016 > 0:13503 1046 DENY_NONE 0x83 RDWR > NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd > Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls > Sat Oct 22 18:58:06 2016Is this from an open with the patched smbclient or from Excel?> Is odd as the patch was applied fine during the RPM build process. > > Is there something I need to do when building from sRPMs to make > this stick?You could double check in ~/rpmbuild/BUILD/ if the patch was really applied. If it was, an smbclient open should result in DENY_ALL denymode in smbstatus -L, eg: $ ./bin/smbclient -Uslow%x -msmb3 //10.10.11.100/test Domain=[SLOWSERVER] OS=[] Server=[] smb: \> open test open file \test: for read/write fnum 1 $ sudo ./bin/smbstatus -L Locked files: Pid Uid DenyMode Access R/W Oplock SharePath Name Time -------------------------------------------------------------------------------------------------- 12757 1000 DENY_ALL 0x3 RDWR NONE /shares/test test Sun Oct 23 10:39:25 2016 Cheerio! -slow
On 23/10/16 10:00, Ralph Böhme wrote:> Alex, > > On Sat, Oct 22, 2016 at 07:30:01PM +0100, Alex Crow wrote: >> I applied the patch to a new build of Sernet 4.4.6 SRPM. I could still >> open the file from both hosts, and in fact this time I could actually >> open it on the same host without any problems: > I don't see where you're opening the file twice from one smbclient > below. Am I missing something?Not the same client session, but two sessions connecting to the same host.> > And then there's no need to use an xls file, simply create an empty > file in the root of the share with `touch file` and use that.Yes, it's just I had been testing with that file the whole time so I had it in my clipboard...> >> host 1: >> >> smbclient -U ajc //172.31.0.120/ifa_v >> >> ... >> >> smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609 >> 213_ ISL Balance sheet variance analysis.xls" >> open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609 >> 213_ ISL Balance sheet variance analysis.xls: for read/write fnum 56287 >> >> host 2: >> >> smbclient -U ajc //172.31.0.120/ifa_v >> >> ... >> >> smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609 >> 213_ ISL Balance sheet variance analysis.xls" >> open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609 >> 213_ ISL Balance sheet variance analysis.xls: for read/write fnum 17669 >> >> But this: >> >> # smbstatus -L >> Locked files: >> Pid Uid DenyMode Access R/W >> Oplock SharePath Name Time >> -------------------------------------------------------------------------------------------------- >> 1:19650 1046 DENY_NONE 0x83 RDWR >> NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd >> Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls >> Sat Oct 22 19:26:06 2016 >> 0:13503 1046 DENY_NONE 0x83 RDWR >> NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd >> Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls >> Sat Oct 22 18:58:06 2016 > Is this from an open with the patched smbclient or from Excel?smbclient.> >> Is odd as the patch was applied fine during the RPM build process. >> >> Is there something I need to do when building from sRPMs to make >> this stick? > You could double check in ~/rpmbuild/BUILD/ if the patch was really > applied. If it was, an smbclient open should result in DENY_ALL > denymode in smbstatus -L, eg: > > $ ./bin/smbclient -Uslow%x -msmb3 //10.10.11.100/test > Domain=[SLOWSERVER] OS=[] Server=[] > smb: \> open test > open file \test: for read/write fnum 1 > > $ sudo ./bin/smbstatus -L > Locked files: > Pid Uid DenyMode Access R/W Oplock SharePath Name Time > -------------------------------------------------------------------------------------------------- > 12757 1000 DENY_ALL 0x3 RDWR NONE /shares/test test Sun Oct 23 10:39:25 2016 > > Cheerio! > -slowDoes look like it didn't get applied. Odd, as it was in the specfile and the SOURCES dir and the build finished. I'll check, Cheers Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
On 02/11/16 07:29, Ralph Böhme wrote:> On Tue, Nov 01, 2016 at 09:56:06PM +0000, Alex Crow wrote: >>> well, yes and no. I did find some time to look into those logs and >>> those just tell use, that the second smbd on the second node that >>> fetches the lock record from the (clustered) locking db doesn't see >>> the record from the first node. ctdb debug logs might be telling, so >>> posting a full set of smbd and ctdb logs to the mailing list would be >>> the next step. >>> >>> Currently I have no idea how this can happen, getting to the bottom of >>> this is certainly fun and entertaining, but also very time >>> consuming. At the moment I don't have the time to do this in my free >>> time, maybe someone else from the mailing list has. >>> >>> -slow >> Hi Ralph, >> >> Thanks for that. Can I post the above back to the ML with links to >> appropriate logs? > sure. > > Btw, I forgot to mention that I did test this on my own devel cluster, > both development version from git master as well as production 4.4, > and could not reproduce the issue. > > -slowIs anyone able to assist with this issue? I have produced level 10 logs but not sure how to debug CTDB. As a quick reminder the problem is that locks on one server are not being honoured when a client connects to another server in the cluster. Cheers Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
On 02/11/16 11:29, Alex Crow via samba wrote:> On 02/11/16 07:29, Ralph Böhme wrote: > >> On Tue, Nov 01, 2016 at 09:56:06PM +0000, Alex Crow wrote: >>>> well, yes and no. I did find some time to look into those logs and >>>> those just tell use, that the second smbd on the second node that >>>> fetches the lock record from the (clustered) locking db doesn't see >>>> the record from the first node. ctdb debug logs might be telling, so >>>> posting a full set of smbd and ctdb logs to the mailing list would be >>>> the next step. >>>> >>>> Currently I have no idea how this can happen, getting to the bottom of >>>> this is certainly fun and entertaining, but also very time >>>> consuming. At the moment I don't have the time to do this in my free >>>> time, maybe someone else from the mailing list has. >>>> >>>> -slow >>> Hi Ralph, >>> >>> Thanks for that. Can I post the above back to the ML with links to >>> appropriate logs? >> sure. >> >> Btw, I forgot to mention that I did test this on my own devel cluster, >> both development version from git master as well as production 4.4, >> and could not reproduce the issue. >> >> -slow > > Is anyone able to assist with this issue? I have produced level 10 > logs but not sure how to debug CTDB. > > As a quick reminder the problem is that locks on one server are not > being honoured when a client connects to another server in the cluster. > > Cheers > > Alex >FYI When I reset my cluster for two nodes and upgraded from 4,4,6 to 4.4.7, restarted all services the locking worked between two clients accessing different servers. So my next step was to add back the 3rd server. At this point, clients would see locking working OK on the 1st and 2nd servers but opening the same file from the 3rd server from the 2nd client allowed all access. It's almost as if if you establish a cluster with 2 CTDB servers everything works OK but it all goes to hell when you add a 3rd member., In all cases the file was initially opened R/W from the 1st client to the 1st server. To clarify: by "Nth server" I mean the virtual IPs set up by CTDB in numerical order. I would hope this helps anyone that's looking at this. Best regards Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Martin Schwenke
2016-Nov-03 06:34 UTC
[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
Hi Alex, On Wed, 2 Nov 2016 11:29:13 +0000, Alex Crow via samba <samba at lists.samba.org> wrote:> On 02/11/16 07:29, Ralph Böhme wrote: > > > On Tue, Nov 01, 2016 at 09:56:06PM +0000, Alex Crow wrote: > [...] > >> Hi Ralph, > >> > >> Thanks for that. Can I post the above back to the ML with links to > >> appropriate logs? > > sure. > > > > Btw, I forgot to mention that I did test this on my own devel cluster, > > both development version from git master as well as production 4.4, > > and could not reproduce the issue. > > > > -slow > > Is anyone able to assist with this issue? I have produced level 10 logs > but not sure how to debug CTDB.If you lock the file from a client attached to 1 node and "smbstatus -L" shows the lock from all the nodes then CTDB is doing what it is meant to. That is, it makes sure that Samba processes see the same TDB records on all nodes. Not sure what's going on with the rest of it. Doing a bit of testing... peace & happiness, martin