samba - Feb 2018 - Upgrading a ctdb cluster: samba not listening on TCP port 445

Am Fr 02.02.2018 um 10:27 schrieb Martin Schwenke:
> On Thu, 1 Feb 2018 13:11:58 +0100, Nicolas Zuber via samba
> <samba at lists.samba.org> wrote:
> 
>> after starting the ctdb process and setting the log level to 5. If I
set
>> the log level to 2 it shows that winbind and nmbd starts successfully
>> and no complains about a failed smbd process (see log below).
> 
> Now the guesses get more esoteric.  :-)
> 
> What do you see if you run "ss -tlnp src :445" as root?
> 
> I see smbd listening on both IPv4 and IPv6:
> 
> LISTEN     0      50
> *:445                                  *:*
> users:(("smbd",pid=2315,fd=36)) LISTEN     0
> 50                      :::445                                 :::*
> users:(("smbd",pid=2315,fd=34))
> 
> If you see smbd listening on port 445 then what happens if you run:
> 
> # ctdb checktcpport 445
> Failed to bind to TCP port 445
> 
> It should fail to bind, which is what CTDB's monitor event looks for
> when testing to see smbd is listing on the port.
> 
> peace & happiness,
> martin
> 
Hello Martin,

Thank you very much for your help. When using the commands you 
suggested, it showed that samba was not listing on port 445 and ctdb 
checktcpport 445 did not fail. Somehow the upgrade process to 4.7.1 
worked after several tries and I tried to figure out why. During the 
investigation I found out that the smbd process failed after one minute 
(before that I was probably not patient enough to see smbd fail), 
reporting that samba 4.7.1 was running and it refused to start 4.7.4 (at 
this time I tried to directly upgrade to 4.7.4):

[2018/02/04 20:18:16.968393,  0] 
../source3/smbd/server.c:1532(smbd_claim_version)
Feb  4 20:18:16 fileb smbd[5835]:  smbd_claim_version: smbd 4.7.1 
already running, refusing to start version 4.7.4

The next thing was to stop ctdb on both file servers to do an offline 
upgrade. This worked fine without any problems. So it seems that a 
rolling upgrade from 4.7.1 to 4.7.4 did not work in my case in contrast 
to the information on the samba wiki:"For releases within a minor 
version (i.e. X.Y.Z to X.Y.Z'), rolling upgrades will work unless 
otherwise stated.". I could not find any information in the release 
notes that rolling upgrades do not work, but next time I will try to do 
an offline upgrade first.

Thank you for your time and help.

peace & happiness,
Nicolas

Hi Nicolas,

On Mon, 5 Feb 2018 13:43:33 +0100, Nicolas Zuber
<n.zuber at physik.uni-stuttgart.de> wrote:
> Am Fr 02.02.2018 um 10:27 schrieb Martin Schwenke:
> > On Thu, 1 Feb 2018 13:11:58 +0100, Nicolas Zuber via samba
> > <samba at lists.samba.org> wrote:
> >   
> >> after starting the ctdb process and setting the log level to 5. If
I set
> >> the log level to 2 it shows that winbind and nmbd starts
successfully
> >> and no complains about a failed smbd process (see log below).  
> > 
> > Now the guesses get more esoteric.  :-)
> > 
> > What do you see if you run "ss -tlnp src :445" as root?
> > 
> > I see smbd listening on both IPv4 and IPv6:
> > 
> > LISTEN     0      50
> > *:445                                  *:*
> > users:(("smbd",pid=2315,fd=36)) LISTEN     0
> > 50                      :::445                                 :::*
> > users:(("smbd",pid=2315,fd=34))
> > 
> > If you see smbd listening on port 445 then what happens if you run:
> > 
> > # ctdb checktcpport 445
> > Failed to bind to TCP port 445
> > 
> > It should fail to bind, which is what CTDB's monitor event looks
for
> > when testing to see smbd is listing on the port.
> > 
> > peace & happiness,
> > martin
> >   
> 
> Hello Martin,
> 
> Thank you very much for your help. When using the commands you 
> suggested, it showed that samba was not listing on port 445 and ctdb 
> checktcpport 445 did not fail. Somehow the upgrade process to 4.7.1 
> worked after several tries and I tried to figure out why. During the 
> investigation I found out that the smbd process failed after one minute 
> (before that I was probably not patient enough to see smbd fail), 
> reporting that samba 4.7.1 was running and it refused to start 4.7.4 (at 
> this time I tried to directly upgrade to 4.7.4):
> 
> [2018/02/04 20:18:16.968393,  0] 
> ../source3/smbd/server.c:1532(smbd_claim_version)
> Feb  4 20:18:16 fileb smbd[5835]:  smbd_claim_version: smbd 4.7.1 
> already running, refusing to start version 4.7.4
> 
> The next thing was to stop ctdb on both file servers to do an offline 
> upgrade. This worked fine without any problems. So it seems that a 
> rolling upgrade from 4.7.1 to 4.7.4 did not work in my case in contrast 
> to the information on the samba wiki:"For releases within a minor 
> version (i.e. X.Y.Z to X.Y.Z'), rolling upgrades will work unless 
> otherwise stated.". I could not find any information in the release 
> notes that rolling upgrades do not work, but next time I will try to do 
> an offline upgrade first.
> 
> Thank you for your time and help.
I wasn't aware that the the code stopping rolling upgrades had actually
gone upstream and into a release.  :-(

I'll ask some questions and fix the documentation on the wiki. 

peace & happiness,
martin

On Tue, 6 Feb 2018 16:07:16 +1100, Martin Schwenke via samba
<samba at lists.samba.org> wrote:
> Hi Nicolas,
> 
> On Mon, 5 Feb 2018 13:43:33 +0100, Nicolas Zuber
> <n.zuber at physik.uni-stuttgart.de> wrote:
> 
> > Am Fr 02.02.2018 um 10:27 schrieb Martin Schwenke:  
> > > On Thu, 1 Feb 2018 13:11:58 +0100, Nicolas Zuber via samba
> > > <samba at lists.samba.org> wrote:
> > >     
>  [...]  
> > > 
> > > Now the guesses get more esoteric.  :-)
> > > 
> > > What do you see if you run "ss -tlnp src :445" as root?
> > > 
> > > I see smbd listening on both IPv4 and IPv6:
> > > 
> > > LISTEN     0      50
> > > *:445                                  *:*
> > > users:(("smbd",pid=2315,fd=36)) LISTEN     0
> > > 50                      :::445                                
:::*
> > > users:(("smbd",pid=2315,fd=34))
> > > 
> > > If you see smbd listening on port 445 then what happens if you
run:
> > > 
> > > # ctdb checktcpport 445
> > > Failed to bind to TCP port 445
> > > 
> > > It should fail to bind, which is what CTDB's monitor event
looks for
> > > when testing to see smbd is listing on the port.
> > > 
> > > peace & happiness,
> > > martin
> > >     
> > 
> > Hello Martin,
> > 
> > Thank you very much for your help. When using the commands you 
> > suggested, it showed that samba was not listing on port 445 and ctdb 
> > checktcpport 445 did not fail. Somehow the upgrade process to 4.7.1 
> > worked after several tries and I tried to figure out why. During the 
> > investigation I found out that the smbd process failed after one
minute
> > (before that I was probably not patient enough to see smbd fail), 
> > reporting that samba 4.7.1 was running and it refused to start 4.7.4
(at
> > this time I tried to directly upgrade to 4.7.4):
> > 
> > [2018/02/04 20:18:16.968393,  0] 
> > ../source3/smbd/server.c:1532(smbd_claim_version)
> > Feb  4 20:18:16 fileb smbd[5835]:  smbd_claim_version: smbd 4.7.1 
> > already running, refusing to start version 4.7.4
> > 
> > The next thing was to stop ctdb on both file servers to do an offline 
> > upgrade. This worked fine without any problems. So it seems that a 
> > rolling upgrade from 4.7.1 to 4.7.4 did not work in my case in
contrast
> > to the information on the samba wiki:"For releases within a minor
> > version (i.e. X.Y.Z to X.Y.Z'), rolling upgrades will work unless 
> > otherwise stated.". I could not find any information in the
release
> > notes that rolling upgrades do not work, but next time I will try to
do
> > an offline upgrade first.
> > 
> > Thank you for your time and help.  
> 
> I wasn't aware that the the code stopping rolling upgrades had actually
> gone upstream and into a release.  :-(
> 
> I'll ask some questions and fix the documentation on the wiki. 
I have updated the "Policy" section of

  https://wiki.samba.org/index.php/Upgrading_a_CTDB_cluster

Sorry that this was out of date for a significant amount of time.

Although I wrote it, I think the "Suggestions" section of that page is
useful, especially this:

  You should probably schedule a maintenance window for an upgrade just
  in case things go wrong - if you have to do that then you might as
  well briefly take down the whole cluster

:-)

peace & happiness,
martin