thr3ads.net - samba - [Samba] CTDB and nfs-ganesha [Oct 2019]

If this information is useful, please help other people find it:
Share via:

Max DiOrio

2019-Oct-01 18:57 UTC

[Samba] CTDB and nfs-ganesha

Hi there ? I seem to be having trouble wrapping my brain about the CTDB and
ganesha configuration.  I thought I had it figured out, but it doesn?t seem to
be doing any checking of the nfs-ganesha service.

I put nfs-ganesha-callout as executable in /etc/ctdb
I create nfs-checks-ganesha.d folder in /etc/ctdb and in there I have
20.nfs_ganesha.check

In my ctdbd.conf file I have:

# Options to ctdbd, read by ctdbd_wrapper(1)
#
# See ctdbd.conf(5) for more information about CTDB configuration variables.

# Shared recovery lock file to avoid split brain.  No default.
#
# Do NOT run CTDB without a recovery lock file unless you know exactly
# what you are doing.
CTDB_RECOVERY_LOCK=/run/gluster/shared_storage/.CTDB-lockfile

# List of nodes in the cluster.  Default is below.
CTDB_NODES=/etc/ctdb/nodes

# List of public addresses for providing NAS services.  No default.
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses

# What services should CTDB manage?  Default is none.
# CTDB_MANAGES_SAMBA=yes
# CTDB_MANAGES_WINBIND=yes
CTDB_MANAGES_NFS=yes

# Raise the file descriptor limit for CTDB?
# CTDB_MAX_OPEN_FILES=10000

# Default is to use the log file below instead of syslog.
CTDB_LOGGING=file:/var/log/log.ctdb

# Default log level is NOTICE.  Want less logging?
CTDB_DEBUGLEVEL=DEBUG

# Set some CTDB tunable variables during CTDB startup?
# CTDB_SET_TDBMutexEnabled=0

CTDB_NFS_CALLOUT=/etc/ctdb/nfs-ganesha-callout
CTDB_NFS_CHECKS_DIR=/etc/ctdb/nfs-checks-ganesha.d
CTBS_NFS_STATE_FS_TYPE=glusterfs
CTDB_NFS_STATE_MNT=/run/gluster/shared_storage
CTDB_NFS_SKIP_SHARE_CHECK=yes
NFS_HOSTNAME=hq-6pnfs



But in the logs, I see nothing relating to nfs-ganesha, and when the ganesha
service fails, the IP?s don?t get migrated and/or the service doesn?t get
restarted.

Any ideas?

Max DiOrio
IT Architect

Martin Schwenke

2019-Oct-01 21:46 UTC

head link

[Samba] CTDB and nfs-ganesha

Hi Max,

On Tue, 1 Oct 2019 18:57:43 +0000, Max DiOrio via samba
<samba at lists.samba.org> wrote:
> Hi there ? I seem to be having trouble wrapping my brain about the
> CTDB and ganesha configuration.  I thought I had it figured out, but
> it doesn?t seem to be doing any checking of the nfs-ganesha service.
> I put nfs-ganesha-callout as executable in /etc/ctdb
> I create nfs-checks-ganesha.d folder in /etc/ctdb and in there I have
> 20.nfs_ganesha.check
You may want to symlink some of the others across, such as
00.portmapper.check.
> In my ctdbd.conf file I have:
Aha!  In >=4.9 nothing looks at that file anymore.  If you run

 
https://git.samba.org/?p=samba.git;a=blob_plain;f=ctdb/doc/examples/config_migrate.sh;h=8479aeb39f383bf9d3a05d79b9357a7e07a6a836;hb=refs/heads/v4-9-stable

on it then you'll get some useful suggestions...  but keep reading
below...
> # Options to ctdbd, read by ctdbd_wrapper(1)
> #
> # See ctdbd.conf(5) for more information about CTDB configuration
variables.
> 
> # Shared recovery lock file to avoid split brain.  No default.
> #
> # Do NOT run CTDB without a recovery lock file unless you know exactly
> # what you are doing.
> CTDB_RECOVERY_LOCK=/run/gluster/shared_storage/.CTDB-lockfile
This should be in ctdb.conf:

[cluster]

  recovery lock = /run/gluster/shared_storage/.CTDB-lockfile
> # List of nodes in the cluster.  Default is below.
> CTDB_NODES=/etc/ctdb/nodes
> 
> # List of public addresses for providing NAS services.  No default.
> CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
These are no longer used.  The above defaults in /etc/ctdb/ are now
hardwired.  Symlinks can be used if necessary.
> # What services should CTDB manage?  Default is none.
> # CTDB_MANAGES_SAMBA=yes
> # CTDB_MANAGES_WINBIND=yes
> CTDB_MANAGES_NFS=yes
Gone.  Now just enable the event scripts.
> # Raise the file descriptor limit for CTDB?
> # CTDB_MAX_OPEN_FILES=10000
Gone.  Either do the right thing in the systemd unit file or put a
ulimit command in /etc/sysconfig/ctdb or /etc/default/ctdb (depending
on distro).
> # Default is to use the log file below instead of syslog.
> CTDB_LOGGING=file:/var/log/log.ctdb
ctdb.conf:

[logging]
  location = file:/var/log/log.ctdb

However, this is the default so you can just omit it.

If you can use syslog instead, then I strongly suggest you do that
(see ctdb.conf(5) manpage).  CTDB's file logging has no useful way of
rotating the logs. There's an open bug, there are plans, but it is
complicated.
> # Default log level is NOTICE.  Want less logging?
> CTDB_DEBUGLEVEL=DEBUG
ctdb.conf:

[logging]
  log level = DEBUG
> CTDB_NFS_CALLOUT=/etc/ctdb/nfs-ganesha-callout
> CTDB_NFS_CHECKS_DIR=/etc/ctdb/nfs-checks-ganesha.d
> CTBS_NFS_STATE_FS_TYPE=glusterfs
> CTDB_NFS_STATE_MNT=/run/gluster/shared_storage
> CTDB_NFS_SKIP_SHARE_CHECK=yes
> NFS_HOSTNAME=hq-6pnfs
Move all of these to /etc/ctdb/script.options.
> But in the logs, I see nothing relating to nfs-ganesha, and when the
> ganesha service fails, the IP?s don?t get migrated and/or the service
> doesn?t get restarted.
> Any ideas?
Yeah, complete rewrite of configuration handling.  :-)

At some stage where will be another change some time in the future when:

* service event scripts go to /etc/ctdb/events/service/ - not many
  related options to change here...

* failover event scripts go to /etc/ctdb/events/failover/ and most of
  the failover-related tunables become script options

* ...

After that, things should be very easy to understand...  :-)

peace & happiness,
martin

Max DiOrio

2019-Oct-02 14:30 UTC

head link

[Samba] CTDB and nfs-ganesha

Martin - thank you for this.  I don't know why I couldn't find any of
this information anywhere.  How long has this change been in place, every
website I see about configuring nfs-ganesha with ctdb shows the old information,
not the new.

Do I need to enable the legacy 06.nfs 60.nfs files when using ganesha?  I assume
no.



?On 10/1/19, 5:46 PM, "Martin Schwenke" <martin at meltin.net>
wrote:

    NOTE: This email originated from outside of the organization.
    
    
    Hi Max,
    
    On Tue, 1 Oct 2019 18:57:43 +0000, Max DiOrio via samba
    <samba at lists.samba.org> wrote:
    
    > Hi there ? I seem to be having trouble wrapping my brain about the
    > CTDB and ganesha configuration.  I thought I had it figured out, but
    > it doesn?t seem to be doing any checking of the nfs-ganesha service.
    
    > I put nfs-ganesha-callout as executable in /etc/ctdb
    
    > I create nfs-checks-ganesha.d folder in /etc/ctdb and in there I have
    > 20.nfs_ganesha.check
    
    You may want to symlink some of the others across, such as
    00.portmapper.check.
    
    > In my ctdbd.conf file I have:
    
    Aha!  In >=4.9 nothing looks at that file anymore.  If you run
    
     
https://git.samba.org/?p=samba.git;a=blob_plain;f=ctdb/doc/examples/config_migrate.sh;h=8479aeb39f383bf9d3a05d79b9357a7e07a6a836;hb=refs/heads/v4-9-stable
    
    on it then you'll get some useful suggestions...  but keep reading
    below...
    
    > # Options to ctdbd, read by ctdbd_wrapper(1)
    > #
    > # See ctdbd.conf(5) for more information about CTDB configuration
variables.
    >
    > # Shared recovery lock file to avoid split brain.  No default.
    > #
    > # Do NOT run CTDB without a recovery lock file unless you know exactly
    > # what you are doing.
    > CTDB_RECOVERY_LOCK=/run/gluster/shared_storage/.CTDB-lockfile
    
    This should be in ctdb.conf:
    
    [cluster]
    
      recovery lock = /run/gluster/shared_storage/.CTDB-lockfile
    
    > # List of nodes in the cluster.  Default is below.
    > CTDB_NODES=/etc/ctdb/nodes
    >
    > # List of public addresses for providing NAS services.  No default.
    > CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
    
    These are no longer used.  The above defaults in /etc/ctdb/ are now
    hardwired.  Symlinks can be used if necessary.
    
    > # What services should CTDB manage?  Default is none.
    > # CTDB_MANAGES_SAMBA=yes
    > # CTDB_MANAGES_WINBIND=yes
    > CTDB_MANAGES_NFS=yes
    
    Gone.  Now just enable the event scripts.
    
    > # Raise the file descriptor limit for CTDB?
    > # CTDB_MAX_OPEN_FILES=10000
    
    Gone.  Either do the right thing in the systemd unit file or put a
    ulimit command in /etc/sysconfig/ctdb or /etc/default/ctdb (depending
    on distro).
    
    > # Default is to use the log file below instead of syslog.
    > CTDB_LOGGING=file:/var/log/log.ctdb
    
    ctdb.conf:
    
    [logging]
      location = file:/var/log/log.ctdb
    
    However, this is the default so you can just omit it.
    
    If you can use syslog instead, then I strongly suggest you do that
    (see ctdb.conf(5) manpage).  CTDB's file logging has no useful way of
    rotating the logs. There's an open bug, there are plans, but it is
    complicated.
    
    > # Default log level is NOTICE.  Want less logging?
    > CTDB_DEBUGLEVEL=DEBUG
    
    ctdb.conf:
    
    [logging]
      log level = DEBUG
    
    > CTDB_NFS_CALLOUT=/etc/ctdb/nfs-ganesha-callout
    > CTDB_NFS_CHECKS_DIR=/etc/ctdb/nfs-checks-ganesha.d
    > CTBS_NFS_STATE_FS_TYPE=glusterfs
    > CTDB_NFS_STATE_MNT=/run/gluster/shared_storage
    > CTDB_NFS_SKIP_SHARE_CHECK=yes
    > NFS_HOSTNAME=hq-6pnfs
    
    Move all of these to /etc/ctdb/script.options.
    
    > But in the logs, I see nothing relating to nfs-ganesha, and when the
    > ganesha service fails, the IP?s don?t get migrated and/or the service
    > doesn?t get restarted.
    
    > Any ideas?
    
    Yeah, complete rewrite of configuration handling.  :-)
    
    At some stage where will be another change some time in the future when:
    
    * service event scripts go to /etc/ctdb/events/service/ - not many
      related options to change here...
    
    * failover event scripts go to /etc/ctdb/events/failover/ and most of
      the failover-related tunables become script options
    
    * ...
    
    After that, things should be very easy to understand...  :-)
    
    peace & happiness,
    martin

Max DiOrio

2019-Oct-02 14:51 UTC

head link

[Samba] CTDB and nfs-ganesha

Hi Marin - again thank you for the help.  I can't believe I coundn't
find any info about this big configuration change.  Even the Samba WIKI
doesn't really spell this out at all in instructs you to use ctdbd.conf.

Do I need to enable the 20.nfs_ganesha.check script file at all, or will the
config itself take care of that?  Also, are there any recommendations on which
nfs-checks.d checks should still be used in the nfs-checks-ganesha?

Gluster, nfs-ganesha, ctdb seems like a great combination that's sorely
lacking on proper documentation.  I know there's storhaug that is supposed
to tie this together, but that flat out didn't work for me at all, has poor
documentation and needed manual edits just to get it to work.



?On 10/1/19, 5:46 PM, "Martin Schwenke" <martin at meltin.net>
wrote:

    NOTE: This email originated from outside of the organization.
    
    
    Hi Max,
    
    On Tue, 1 Oct 2019 18:57:43 +0000, Max DiOrio via samba
    <samba at lists.samba.org> wrote:
    
    > Hi there ? I seem to be having trouble wrapping my brain about the
    > CTDB and ganesha configuration.  I thought I had it figured out, but
    > it doesn?t seem to be doing any checking of the nfs-ganesha service.
    
    > I put nfs-ganesha-callout as executable in /etc/ctdb
    
    > I create nfs-checks-ganesha.d folder in /etc/ctdb and in there I have
    > 20.nfs_ganesha.check
    
    You may want to symlink some of the others across, such as
    00.portmapper.check.
    
    > In my ctdbd.conf file I have:
    
    Aha!  In >=4.9 nothing looks at that file anymore.  If you run
    
     
https://git.samba.org/?p=samba.git;a=blob_plain;f=ctdb/doc/examples/config_migrate.sh;h=8479aeb39f383bf9d3a05d79b9357a7e07a6a836;hb=refs/heads/v4-9-stable
    
    on it then you'll get some useful suggestions...  but keep reading
    below...
    
    > # Options to ctdbd, read by ctdbd_wrapper(1)
    > #
    > # See ctdbd.conf(5) for more information about CTDB configuration
variables.
    >
    > # Shared recovery lock file to avoid split brain.  No default.
    > #
    > # Do NOT run CTDB without a recovery lock file unless you know exactly
    > # what you are doing.
    > CTDB_RECOVERY_LOCK=/run/gluster/shared_storage/.CTDB-lockfile
    
    This should be in ctdb.conf:
    
    [cluster]
    
      recovery lock = /run/gluster/shared_storage/.CTDB-lockfile
    
    > # List of nodes in the cluster.  Default is below.
    > CTDB_NODES=/etc/ctdb/nodes
    >
    > # List of public addresses for providing NAS services.  No default.
    > CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
    
    These are no longer used.  The above defaults in /etc/ctdb/ are now
    hardwired.  Symlinks can be used if necessary.
    
    > # What services should CTDB manage?  Default is none.
    > # CTDB_MANAGES_SAMBA=yes
    > # CTDB_MANAGES_WINBIND=yes
    > CTDB_MANAGES_NFS=yes
    
    Gone.  Now just enable the event scripts.
    
    > # Raise the file descriptor limit for CTDB?
    > # CTDB_MAX_OPEN_FILES=10000
    
    Gone.  Either do the right thing in the systemd unit file or put a
    ulimit command in /etc/sysconfig/ctdb or /etc/default/ctdb (depending
    on distro).
    
    > # Default is to use the log file below instead of syslog.
    > CTDB_LOGGING=file:/var/log/log.ctdb
    
    ctdb.conf:
    
    [logging]
      location = file:/var/log/log.ctdb
    
    However, this is the default so you can just omit it.
    
    If you can use syslog instead, then I strongly suggest you do that
    (see ctdb.conf(5) manpage).  CTDB's file logging has no useful way of
    rotating the logs. There's an open bug, there are plans, but it is
    complicated.
    
    > # Default log level is NOTICE.  Want less logging?
    > CTDB_DEBUGLEVEL=DEBUG
    
    ctdb.conf:
    
    [logging]
      log level = DEBUG
    
    > CTDB_NFS_CALLOUT=/etc/ctdb/nfs-ganesha-callout
    > CTDB_NFS_CHECKS_DIR=/etc/ctdb/nfs-checks-ganesha.d
    > CTBS_NFS_STATE_FS_TYPE=glusterfs
    > CTDB_NFS_STATE_MNT=/run/gluster/shared_storage
    > CTDB_NFS_SKIP_SHARE_CHECK=yes
    > NFS_HOSTNAME=hq-6pnfs
    
    Move all of these to /etc/ctdb/script.options.
    
    > But in the logs, I see nothing relating to nfs-ganesha, and when the
    > ganesha service fails, the IP?s don?t get migrated and/or the service
    > doesn?t get restarted.
    
    > Any ideas?
    
    Yeah, complete rewrite of configuration handling.  :-)
    
    At some stage where will be another change some time in the future when:
    
    * service event scripts go to /etc/ctdb/events/service/ - not many
      related options to change here...
    
    * failover event scripts go to /etc/ctdb/events/failover/ and most of
      the failover-related tunables become script options
    
    * ...
    
    After that, things should be very easy to understand...  :-)
    
    peace & happiness,
    martin

Possibly Parallel Threads

Search for more apparently analagous threads

samba - Oct 2019 - CTDB and nfs-ganesha

[Samba] CTDB and nfs-ganesha

[Samba] CTDB and nfs-ganesha

[Samba] CTDB and nfs-ganesha

[Samba] CTDB and nfs-ganesha

Possibly Parallel Threads