samba - Jun 2020 - Apparent large memory leak with encryption + SMB3_00 or SMB3_02

Hello,

I've recently set up Samba (4.12.3) on Arch Linux as the target for Time
Machine backups for a couple of Macs. Shortly thereafter I started seeing
OOMs whenever a backup would start. I stumbled upon disabling encryption on
the server (i.e. changing "smb encrypt" from "required" to
"off") to
prevent this issue.

After further digging, I'm able to reproduce this issue using smbclient on
the server machine with either SMB3_00 or SMB3_02 as the max protocol with
the server configured to require encryption. Uploading a file increases the
RSS of the smbd process by roughly the size of the uploaded file.

My minimal smb.conf and relevant smbstatus output are below. Have I missed
something in the configuration? Or is this an issue with the AES-128-CCM
encryption? I noticed that SMB3_10 or SMB3_11 do not suffer from the memory
leak and use AES-128-GCM. Am happy to troubleshoot further as I would like
to re-enable encryption if possible.

Thank you,
Andrew

[global]
   smb encrypt = required

[data]
   path = /mnt/data
   writable = yes


Samba version 4.12.3
PID     Username     Group        Machine
     Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
837051  anichols     anichols     10.0.1.201 (ipv4:10.0.1.201:44812)
     SMB3_02           AES-128-CCM          AES-128-CMAC

Service      pid     Machine       Connected at
Encryption   Signing
---------------------------------------------------------------------------------------------
data         837051  10.0.1.201    Fri Jun 19 03:58:16 PM 2020 BST
AES-128-CCM  AES-128-CMAC

No locked files

On Fri, Jun 19, 2020 at 04:07:03PM +0100, Andrew Nicholson via samba
wrote:
> Hello,
> 
> I've recently set up Samba (4.12.3) on Arch Linux as the target for
Time
> Machine backups for a couple of Macs. Shortly thereafter I started seeing
> OOMs whenever a backup would start. I stumbled upon disabling encryption on
> the server (i.e. changing "smb encrypt" from "required"
to "off") to
> prevent this issue.
> 
> After further digging, I'm able to reproduce this issue using smbclient
on
> the server machine with either SMB3_00 or SMB3_02 as the max protocol with
> the server configured to require encryption. Uploading a file increases the
> RSS of the smbd process by roughly the size of the uploaded file.
> 
> My minimal smb.conf and relevant smbstatus output are below. Have I missed
> something in the configuration? Or is this an issue with the AES-128-CCM
> encryption? I noticed that SMB3_10 or SMB3_11 do not suffer from the memory
> leak and use AES-128-GCM. Am happy to troubleshoot further as I would like
> to re-enable encryption if possible.
I believe this is already known. It's a bug in gnutls
which we started using for (most) of our encryption.

I believe it's already been fixed upstream, but the
folks involved should be able to comment more.