samba - Nov 2016 - slow directory access, convert_string_internal: Conversion error: Incomplete multibyte sequence

There are definitely some files with some weird names-   in an ssh session they
don't even have regular characters.

e.g
-rw-rw----   1 xxx   xxx     114985112 Oct 31 14:39 ▒^t


Not sure if that is related to problems though.        

The top command shows

Memory: 12G phys mem, 343M free mem, 2048M total swap, 2048M free swap

This is in the evening so should not be much load but I think it was about the
same earlier.  Although I don't have a lot of free physical memory I am not
eating into my swap file.

Thanks


-----Original Message-----
From: Jeremy Allison [mailto:jra at samba.org] 
Sent: Tuesday, November 29, 2016 7:09 PM
To: gaiseric.vandal at gmail.com
Cc: Samba <samba at lists.samba.org>
Subject: Re: [Samba] slow directory access, convert_string_internal: Conversion
error: Incomplete multibyte sequence

On Tue, Nov 29, 2016 at 06:36:02PM -0500, Gaiseric Vandal via samba
wrote:
> Mail file is server is a member server is Samba 3.6.26 on Solaris
> 11.    It had been a BDC but a few week ago a switched it back to a
> member server.  The 2 current DC's had been running Samba 3.6.25 but 
> switched to Samba 4.4.7.  This is  a classic domain.
> 
> 
> 
> Some Windows users reported that browsing folders shared from the 
> member server have become very slow over the last few weeks. Which may 
> correlate to either changing server role on the main file server or 
> upgrading the DC's to Samba 4.
> 
> 
> I do see this issue intermittently.  I open one file network folder 
> with no problems.  I try another, and the progress bar progresses 
> really slowly and it may time out.
> 
> 
> Some client machine logs shows things like:
> 
> 
>           smbd_dirptr_get_entry mask=[*] found
>        somedirectory/somedirectory/somefile/@ ! P fname=@ ! P (@ ! P)
>        [2016/11/29 18:23:02.279482,  3]
>        lib/charcnv.c:537(convert_string_talloc)
>           convert_string_talloc: Conversion error: Illegal multibyte
>        sequence( ! P)
>        [2016/11/29 18:23:02.279560,  3]
>        lib/charcnv.c:528(convert_string_talloc)
>           convert_string_talloc: Conversion error: Incomplete multibyte
>        sequence( P)
>        [2016/11/29 18:23:02.279628,  3]
>        lib/charcnv.c:161(convert_string_internal)
>           convert_string_internal: Conversion error: Illegal multibyte
>        sequence( ! P)
>        [2016/11/29 18:23:02.279689,  3]
>        lib/charcnv.c:140(convert_string_internal)
>           convert_string_internal: Conversion error: Incomplete
>        multibyte sequence( P)
>        [2016/11/29 18:23:02.279845,  3]
>        smbd/dir.c:1049(smbd_dirptr_get_entry)
> 
> 
> 
> which may correlate to directories with files with invalid names
> 
> 
> I don't seem to see entries correlating to directories I was trying to 
> open .
> 
> testparm shows
> 
>         dos charset = CP850
>         unix charset = UTF8
My guess is you have filenames on that share that are not utf8. Look inside:

somedirectory/somedirectory/somefile/

On Tue, Nov 29, 2016 at 08:28:45PM -0500, Gaiseric Vandal
wrote:
> There are definitely some files with some weird names-   in an ssh session
they don't even have regular characters.
> 
> e.g
> -rw-rw----   1 xxx   xxx     114985112 Oct 31 14:39 ▒^t
> 
> 
> Not sure if that is related to problems though.        
Well that's the source of your debug messages. I'd fix
those if I were you.

I think the issue may be related to the broken trusts.        First of 
all only two people had specifically reported issues which meant for 
most people it was acceptable.   Then I noticed that some directories 
seemed slower than others.

If I ssh'd into the server and checked permissions, listing permissions 
on most directories was quick (under 2 second.)  On directories that 
allowed access to trusted domain members (e.g. "Research") listing 
permissions might take 30 seconds.     The unix uid numbers   10157 and 
10172  shd resolve to users from the trusted domain but can't.

Solaris ZFS file systems supports ACL's  similar to windows in the sense 
that not limited to a single user and single group.



# ls -lvd /Pool1/Department/Sales
drwxrwx---+ 13 jsmith      sales            48 Nov 29 18:58 
/Pool1/Department/Sales
      0:user:nobody:read_xattr/read_attributes/read_acl:allow
      1:user:root:read_xattr/read_attributes/read_acl:allow
2:group@:list_directory/read_data/add_file/write_data/add_subdirectory
/append_data/read_xattr/write_xattr/execute/delete_child
/read_attributes/write_attributes/delete/read_acl/write_acl
          /write_owner/synchronize:allow
3:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
/append_data/read_xattr/write_xattr/execute/delete_child
/read_attributes/write_attributes/delete/read_acl/write_acl
          /write_owner/synchronize:allow

# ls -lvd  /DataPool1/Dept/Research/
drwxr-xr-x+  4 jsmith  research       5 Jun  6 15:05 
/Pool1/Department/Research
0:everyone@:list_directory/read_data/read_xattr/execute/read_attributes
          /read_acl:allow
1:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
/append_data/read_xattr/write_xattr/execute/delete_child
/read_attributes/write_attributes/delete/read_acl/write_acl
          /write_owner:file_inherit/dir_inherit:allow
2:group@:list_directory/read_data/read_xattr/execute/read_attributes
          /read_acl:file_inherit/dir_inherit:allow
3:group:group_leaders:list_directory/read_data/read_xattr/execute
/read_attributes/read_acl:file_inherit/dir_inherit:allow
4:user:10157:list_directory/read_data/read_xattr/execute/read_attributes
          /read_acl:file_inherit/dir_inherit/no_propagate:allow
      5:user:10162:list_directory/read_data/add_file/write_data
/add_subdirectory/append_data/read_xattr/write_xattr/execute
/delete_child/read_attributes/write_attributes/delete/read_acl
          :file_inherit/dir_inherit/no_propagate:allow
      6:group:15001:list_directory/read_data/read_xattr/execute
/read_attributes/read_acl:file_inherit/dir_inherit:allow




Clearing the invalid users seems to help.     Although even problem 
directories, access on a 2nd try with in a minute would go faster.



Thanks for the help.





On 11/30/16 11:50, Jeremy Allison wrote:
> On Tue, Nov 29, 2016 at 08:28:45PM -0500, Gaiseric Vandal wrote:
>> There are definitely some files with some weird names-   in an ssh
session they don't even have regular characters.
>>
>> e.g
>> -rw-rw----   1 xxx   xxx     114985112 Oct 31 14:39 ▒^t
>>
>>
>> Not sure if that is related to problems though.
> Well that's the source of your debug messages. I'd fix
> those if I were you.