samba - Apr 2018 - Samba 4 - ISO-8859-15 filenames not listed while browsing

Hello,

I have an issue with a german application that generates files with
iso-8859-15 encoded filenames.

I have 2 Linux Suse servers, both connected to the same NAS filesystem, and
sharing it via Samba.

"unix charset" in smb.conf set to default utf-8
LC_CTYPE & LANG locale are set to en_US.UTF-8


- First server is Samba-3.6.3 (SLES11 SP4, kernel 3.0.101-107-default) :
those files are accessible from windows/smbclient, with UCS2 characters (ie
"\344" for "ä") being replaced with a "_"

- Second one is Samba-4.6.9 (SLES12 SP2, kernel 4.4.103-6.38-default) :
those files simply don't show up while browsing.


In any case, log.smdb complains about :
convert_string_internal: Conversion error: Illegal multibyte sequence

If I change the  "unix charset = iso-8859-15" in the config for
Samba4, it
shows the file while browsing, but then users are reporting that access is
now very slow (bench method is by compiling a program whose sources are on
the share).

I've tried various things already, but to no avail.

Any ideas or help is much appreciated.
Regards

On Tue, Apr 24, 2018 at 06:53:52PM +0200, Ali SIMON via samba
wrote:
>  Hello,
> 
> I have an issue with a german application that generates files with
> iso-8859-15 encoded filenames.
> 
> I have 2 Linux Suse servers, both connected to the same NAS filesystem, and
> sharing it via Samba.
> 
> "unix charset" in smb.conf set to default utf-8
> LC_CTYPE & LANG locale are set to en_US.UTF-8
> 
> 
> - First server is Samba-3.6.3 (SLES11 SP4, kernel 3.0.101-107-default) :
> those files are accessible from windows/smbclient, with UCS2 characters (ie
> "\344" for "ä") being replaced with a "_"
> 
> - Second one is Samba-4.6.9 (SLES12 SP2, kernel 4.4.103-6.38-default) :
> those files simply don't show up while browsing.
> 
> 
> In any case, log.smdb complains about :
> convert_string_internal: Conversion error: Illegal multibyte sequence
> 
> If I change the  "unix charset = iso-8859-15" in the config for
Samba4, it
> shows the file while browsing, but then users are reporting that access is
> now very slow (bench method is by compiling a program whose sources are on
> the share).
> 
> I've tried various things already, but to no avail.
Well you already enumerated your choices. The filesystem
is reporting 8859-15 encoded filenames, and Samba configured
by default expects utf-8.

Can you add a step that runs iconv over the generated
filenames that converts them to utf-8 ?

If not, setting unix charset = iso-8859-15 works, as
you've found, but smbd must convert the characters.

I doubt that is making your access slow though, as
filename conversion is pretty fast and not usually
something affecting the fast-path.

Thanks for your reply.



> Can you add a step that runs iconv over the generated
> filenames that converts them to utf-8 ?
>

Indeed my choices are limited as I don't have the hand on the application,
and the devs won't hear anything, so adding a conversion step would not be
possible. I know the problem come from that, but my hands are tied.


The problem was always there, it's just that it was 'masked' before,
and
the customer is used to filenames not displaying correctly, but obviously
missing files is more of a problem.


If not, setting unix charset = iso-8859-15 works, as
> you've found, but smbd must convert the characters.
I doubt that is making your access slow though, as
> filename conversion is pretty fast and not usually
> something affecting the fast-path.
>
Is there a way for me to pinpoint where the slowness issue can come from?
Or tune some parameters that might help in this matter?
I can clearly see the difference in performance between setting the 'unix
charset' to 'iso8859' and leaving it to its default 'utf8'
value.


I was hoping to find a way for the samba4 server to behave like samba3 (as
in ignoring bad characters and replacing them with an underscore
character), which in the end resulted in the best compromise.

Any advices are welcomed.
Thank you for your time and help.

Regards.