freebsd stable - Nov 2019 - Long-shot: repeatable macOS samba share unmounting during Lightroom import

Hi all,

This is a long-shot question, because it involves a lot of moving
pieces, most of which are opaque commercial, poorly documented
things.  Never the less, it does involve FreeBSD-stable as one of
the players, and my experience over the years has been that FreeBSD
folk are both knowledgeable and helpful, so here's hoping.  Herwith
my tale of computer-induced irritation:

The story takes place at home, where the FreeBSD system in question
is a local network file server.  The FreeBSD tracks -STABLE every
week.  It boots from ZFS on NVM flash and has four 4TB Hitachi ATA
drives in a RAIDZ.  The current motherboard has a Ryzen 7 1700
8-core locked at 3GHz by the bios to avoid a problem of going to
sleep permanently by failing to come out of some sort of low-power
state.  It has 32G RAM.  It has intel "PRO/1000 PCI-Express Network
Driver" network connected to a simple gigabit switch, with both
IPv4 and IPv6 configured and working.

The other protagonist in this tale, also connected to the gigabit
LAN, is an iMac running current-Catalina on APFS flash, mounting
three filesystems over SMB, from Samba 4.10.10.  After appropriate
Samba tweaking this seems to be at least as reliable as it ever was
with netatalk or NFS, and apparently better supported by Apple.

I keep my Lightroom Classic catalogue on the mac's local (flash)
drive, but the photo storage is on the network.  The Import Backups
directory is also on (a different) network drive.  I use Lightroom's
Import function to copy photos off SD cards using the mac's built-in
SD card reader and register with the catalogue.  So far so normal,
I think.

The problem arose about ?three or four? months ago: could be
coincident with OS or Lightroom upgrades, I can't remember, but I
haven't changed anything about the setup, configuration or workflow.
Now, every single time Lightroom does an import, while it's doing
the first scan of the SD card to identify photos that it's seen
before, all three of the Samba filesystems unmount from the mac,
silently.  I can find no record of error in any of the logs,
suggesting that the system thinks that it happened deliberately.
Needless to say, this throws out the import workflow, although it
manages to pick itself up OK if I just re-mount everything.

Anyone have any similar experiences?  Any thoughts of where I could
poke it to find out why this might be happening?  It feels like a
time-out bug somewhere, but (a) there is no complaint, and (b) the
network traffic is light at the time.  Needless to say Apple
documentation is useless.

Probably another good reason to find an alternative to Lightroom...

Cheers,

Andrew Reilly
M: 0409-824-272
areilly at bigpond.net.au

On Sat, Nov 23, 2019 at 9:34 PM Andrew Reilly <areilly at bigpond.net.au>
wrote:
>
> The other protagonist in this tale, also connected to the gigabit
> LAN, is an iMac running current-Catalina on APFS flash, mounting
> three filesystems over SMB, from Samba 4.10.10.  After appropriate
> Samba tweaking this seems to be at least as reliable as it ever was
> with netatalk or NFS, and apparently better supported by Apple.

Considering all of the other bugs and instability introduced (or
reintroduced) in Catalina: did you have this same Lightroom import workflow
configured in Mojave (or whichever other previous macOS version you were
using), and if so, were you encountering the same issue?


Thanks,
?
Matt Garber

On 2019-11-23 21:33, Andrew Reilly wrote:
> It feels like a
> time-out bug somewhere, but (a) there is no complaint, and (b) the
> network traffic is light at the time.  Needless to say Apple
> documentation is useless.
Maybe there is a way to reproduce the Lightroom's file access pattern 
from a shell script in a way that reproduces the problem. Then the same 
access pattern could be done from a FreeBSD client to see whether it 
happens there as well.

Apple's SMB client appears to be FreeBSD's smbfs, however extended by 
Apple to support protocol version 3.
https://opensource.apple.com/source/smb/smb-759.40.1/kernel/smbfs/smbfs.h.auto.html
(Off-topic: I wonder if anyone has looked at porting this back to FreeBSD?)

Maybe the differences are too great for the bug to appear on both systems.

I have a very similar setup to you for serving files to my Mac from a 
FreeBSD server. I haven't seen the unmount problem, but I di have a few 
oddities until I added the 'fruit' module on the Samba side, which helps
with compatbiloty with the Mac. The appropriate bit of my config looks 
like this:

    vfs objects = fruit streams_xattr zfsacl
    fruit:resource = xattr
    fruit:encoding = private

Don't ask me what they do anymore, I added them ages ago, but it does 
work very nicely for me. You may already have this of course, but worth 
pointing out just in case as it took me a few years to discover it!

As someone else has said though, this may well be a Catalina bug. I am 
not running that (MacBook too old, and not buying another until the new 
keyboards are avilable n the replacement I want).

-pete.

Hi,
> On 24 Nov 2019, at 02:33, Andrew Reilly <areilly at bigpond.net.au>
wrote:
[tale of woe trimmed]
>   I use Lightroom's
> Import function to copy photos off SD cards using the mac's built-in
> SD card reader and register with the catalogue.
[etc]

Long shot, but try using an external SD card reader.

--
Bob Bishop
rb at gid.co.uk

WARNING:  Mostly deviating from a FreeBSD specific discussion.
> On Nov 23, 2019, at 9:33 PM, Andrew Reilly <areilly at
bigpond.net.au> wrote:
> 
> Hi all,
> 
> This is a long-shot question, because it involves a lot of moving
> pieces, most of which are opaque commercial, poorly documented
> things.  Never the less, it does involve FreeBSD-stable as one of
> the players, and my experience over the years has been that FreeBSD
> folk are both knowledgeable and helpful, so here's hoping.  Herwith
> my tale of computer-induced irritation:
> 
> The story takes place at home, where the FreeBSD system in question
> is a local network file server.  The FreeBSD tracks -STABLE every
> week.  It boots from ZFS on NVM flash and has four 4TB Hitachi ATA
> drives in a RAIDZ.  The current motherboard has a Ryzen 7 1700
> 8-core locked at 3GHz by the bios to avoid a problem of going to
> sleep permanently by failing to come out of some sort of low-power
> state.  It has 32G RAM.  It has intel "PRO/1000 PCI-Express Network
> Driver" network connected to a simple gigabit switch, with both
> IPv4 and IPv6 configured and working.
> 
> The other protagonist in this tale, also connected to the gigabit
> LAN, is an iMac running current-Catalina on APFS flash, mounting
> three filesystems over SMB, from Samba 4.10.10.  After appropriate
> Samba tweaking this seems to be at least as reliable as it ever was
> with netatalk or NFS, and apparently better supported by Apple.
> 
> I keep my Lightroom Classic catalogue on the mac's local (flash)
> drive, but the photo storage is on the network.  The Import Backups
> directory is also on (a different) network drive.  I use Lightroom's
> Import function to copy photos off SD cards using the mac's built-in
> SD card reader and register with the catalogue.  So far so normal,
> I think.
> 
> The problem arose about ?three or four? months ago: could be
> coincident with OS or Lightroom upgrades, I can't remember, but I
> haven't changed anything about the setup, configuration or workflow.
> Now, every single time Lightroom does an import, while it's doing
> the first scan of the SD card to identify photos that it's seen
> before, all three of the Samba filesystems unmount from the mac,
> silently.  I can find no record of error in any of the logs,
> suggesting that the system thinks that it happened deliberately.
> Needless to say, this throws out the import workflow, although it
> manages to pick itself up OK if I just re-mount everything.
> 
> Anyone have any similar experiences?  Any thoughts of where I could
> poke it to find out why this might be happening?  It feels like a
> time-out bug somewhere, but (a) there is no complaint, and (b) the
> network traffic is light at the time.  Needless to say Apple
> documentation is useless.
> 
> Probably another good reason to find an alternative to Lightroom...
Other than the hardware specifics, I have the same exact workflow and same
players involved.

Maybe an odd question, but how are you mounting the SMB shares?

Since updating to Catalina, I've found lots of problems dealing with SMB
using the Finder window and the items under the Locations side bar.  For
instance:
- Mount a share.  At some point overnight when nothing is using it, the share is
unmounted.  I can't find anything in the logs to say why, when, what, etc. 
Just unmounted.
- With a fresh start of the Finder process and I can access the SMB
server/shares.  After some time, activity, something, I only get
"Connection Failed" and can't access anything.  What's really
crazy, is that I can't even unmount mounted shares from under Locations when
it gets in this state (I can unmount via right click on the desktop item, CLI,
etc).  I see the share with the eject button, but just get useless error message
(if anything).
- Killing (killall -HUP Finder) makes everything work again for a short bit.

If I mount the share via the Finder menubar (Go -> Connect to Server)
everything works and is rock solid.  Mounts with no problems, no mystery
unmount, etc.  I did test a Lightroom import and had no SMB issues when the
shared where mounted via the menu bar.

I also store all of my music on the server and access it via SMB mounts. 
I've noticed that the Apple Music app will automatically mount the SMB share
when I hit play.  Unlike iTunes, I assume it's more aware of the filesystem
and mounts.  Even with the Finder window off in la-la land, the auto-mount by
Music works fine. Now if this behavior could only be exposed to every other
app....

My completely un-researched guess is that when they removed the NetBIOS support
(one of the changes in Catalina), some bug was introduced or uncovered causing
the problematic behavior.  My guess ins based on the assumption that the Finder
window (not sure what to call it) displays all of the network browsing and
discovery and  is the code path that was impacted by the NetBIOS change.  Any
mount tied in with this path gets impacted, and when this goes south, it takes
the mounts with it.  Mounting via other means doesn't bring in the
associated path.

I will check each new OS X update to see if this issue goes away.  Until then, I
just mount my shares via "Go- > 'Connect to Server'".  
Also note that you can re-enable NetBIOS support in Catalina
(https://medium.com/@gobinathm/how-to-access-smb-printer-shares-in-macos-catalina-10-15-17ea91d2c10b).
I've not done nor tested that.

Hopefully a minor tweak to how you mount the SMB share will save you the huge
hassle of changing out tools.

Hope that helps.

Chris

HI again,

You might remember my Samba-related cry for help, last November.  I'm
writing again to report that the problem has been resolved on my systems, as
near as I can tell.  Perhaps my experience will help others.

The "fix" has come as a result of some changes to my filesystem
structure that I made as a result of a recent drive replacement (the 1.8T of
photos from my last trip ran me out of storage).  The upshot of that is that I
don't know whether my previous troubles were misconfiguration on my part, or
some sort of subtle bug in any of the players at the table: Samba on FreeBSD on
ZFS or macOS Catalina's smbfs.  After some experimentation I was able to
conclude that Lightroom itself was unlikely to be the root problem: failure was
easily repeatable with a find in a macOS terminal session.

Quick summary of the problem: macOS Catalina would silently and
seemingly-spontaneously disconnect from my Samba filesystems, exported from my
FreeBSD-12-STABLE server when doing any activity that listed or stat-ed all of
the files in a directory of some thousands.  To make matters worse, before the
macOS 10.15.3 release, this disconnection would leave the smbfs driver, or some
other macOS kernel state broken, and it would not be possible to re-mount or
properly eject the failed mount without a reboot.  Since I was rebooting a lot,
Apple were getting a lot of crash dumps from me, and their 10.15.3 release seems
to have fixed whatever was stuck in the kernel - reboots were no longer required
- but the disconnection problem persisted.

I have a couple of users on the file server, with the usual home directories in
/home/<uname> and these were exported using Samba's [homes]
configuration stanza, in the usual fashion.  I also had a shared directory, at
/home/us, which was exported by a separate Samba stanza with an explicit path =
/home/us, because "us" isn't a user.

When the drives filled up and I installed a new set, since I was going to
restore from backups I took the opportunity to create a new zfs file system for
"us" (/tank/us), and tweaked the smb4.conf accordingly.

That seems to have returned the file systems/server/macOS combination to robust
good health.  I have not experienced a spontaneous-unmount since that change,
and nothing else about the configuration has changed.

My guess: SMB2 enhanced security rules, which Samba enforces, insist that each
host can only log-in/mount a given share once: any secondary mount attempt
should cause the server to drop all mounts.  (Sorry, I can't remember which
reference site pointed that out to me.)  Perhaps Samba is interpreting the rule
on the basis of underlying host-side filesystem structure?  That would put the
old /home/us and /home/uname shares on the same filesystem, and trigger the
security rule?  Well, I always had both "mounted", but perhaps there
was a particular pattern of heavy or slow access that triggered something? 
Anyway, having all of the filesystems exported by Samba coming from unique zfs
filesystems on the FreeBSD side has made everything happy.  Yay?

FWIW I'm running the latest Samba-4.10.13 from ports on FreeBSD 12.1-STABLE
r358963 GENERIC  amd64 on a AMD Ryzen 7 1700 Eight-Core Processor with hardware
threads turned on.

By the way, the current Samba stable release is 4.12.0 (one week old) and 4.11.0
was released last September.  Is there some particular incompatibility with
FreeBSD that is keeping these out of Ports?  I got quite close to trying to
build the newer version myself, but then the problem went away...

Cheers,

Andrew Reilly
M: 0409-824-272
areilly at bigpond.net.au


> On 24 Nov 2019, at 13:46 , Matt Garber <matt.garber at gmail.com>
wrote:
> 
> On Sat, Nov 23, 2019 at 9:34 PM Andrew Reilly <areilly at bigpond.net.au
<mailto:areilly at bigpond.net.au>> wrote:
> 
> The other protagonist in this tale, also connected to the gigabit
> LAN, is an iMac running current-Catalina on APFS flash, mounting
> three filesystems over SMB, from Samba 4.10.10.  After appropriate
> Samba tweaking this seems to be at least as reliable as it ever was
> with netatalk or NFS, and apparently better supported by Apple.
> 
> Considering all of the other bugs and instability introduced (or
reintroduced) in Catalina: did you have this same Lightroom import workflow
configured in Mojave (or whichever other previous macOS version you were using),
and if so, were you encountering the same issue?
> 
> 
> Thanks,
> ?
> Matt Garber
>