On Thu, 08 Nov 2018 22:08:51 +1300 Andrew Bartlett <abartlet at samba.org> wrote:> On Thu, 2018-11-08 at 08:48 +0000, Rowland Penny via samba wrote: > > > > Yes, but it isn't doing a read, it is trying to do a write and then > > being cancelled. > > Rowland, > > Can you point us as the evidence that leads you to that conclusion?OK, the OP posted lines like these: 04:11:08 AM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command 04:11:28 AM 0 832 0.00 0.20 0.20 0 /usr/sbin/smbd -D And from 'man pidstat' kB_rd/s Number of kilobytes the task has caused to be read from disk per second. kB_wr/s Number of kilobytes the task has caused, or shall cause to be written to disk per second. kB_ccwr/s Number of kilobytes whose writing to disk has been cancelled by the task. This may occur when the task truncates some dirty pagecache. In this case, some IO which another task has been accounted for will not be happening. kB_rd/s is 0.00 kB_wr/s and kB_ccwr/s are both 0.20 So, from my reading, nothing is being read, but something is trying to write, but being cancelled. I could of course be totally wrong, but if I am the 'pidstat' manpage needs rewriting.> > I think you are extrapolating too much from an smb.conf line, even > when serving read only shares, there are a lot of things in Samba > that can case both reads and writes.Not really extrapolating anything from smb.conf, I only pointed out that 'read only = no' and 'writable = no' are exact opposites of each other and shouldn't be in the same share together, or do you disagree ? Rowland
On Thu, 2018-11-08 at 09:29 +0000, Rowland Penny wrote:> On Thu, 08 Nov 2018 22:08:51 +1300 > Andrew Bartlett <abartlet at samba.org> wrote: > > > On Thu, 2018-11-08 at 08:48 +0000, Rowland Penny via samba wrote: > > > > > > Yes, but it isn't doing a read, it is trying to do a write and then > > > being cancelled. > > > > Rowland, > > > > Can you point us as the evidence that leads you to that conclusion? > > OK, the OP posted lines like these: > > 04:11:08 AM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command > 04:11:28 AM 0 832 0.00 0.20 0.20 0 /usr/sbin/smbd -D > > And from 'man pidstat' > > kB_rd/s > Number of kilobytes the task has caused to be read from disk per second. > > kB_wr/s > Number of kilobytes the task has caused, or shall cause to be written to disk per second. > > kB_ccwr/s > Number of kilobytes whose writing to disk has been cancelled by the task. > This may occur when the task truncates some dirty pagecache. > In this case, some IO which another task has been accounted for will not be > happening. > > kB_rd/s is 0.00 > kB_wr/s and kB_ccwr/s are both 0.20 > > So, from my reading, nothing is being read, but something is trying to > write, but being cancelled. > > I could of course be totally wrong, but if I am the 'pidstat' manpage > needs rewriting.It is speculation, but this might be due to a mutex-enabled TDB. Each read from a tdb would now be a write while the mutex is set. This might not have been seen in the past when fcntl() was in use. The cancellation might be the mutex being unlocked again.> > > > I think you are extrapolating too much from an smb.conf line, even > > when serving read only shares, there are a lot of things in Samba > > that can case both reads and writes. > > Not really extrapolating anything from smb.conf, I only pointed out > that 'read only = no' and 'writable = no' are exact opposites of each > other and shouldn't be in the same share together, or do you disagree ?OK. Either way I don't think it is relevant to the diagnosis. The last statement unambiguously wins. Thanks, Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
On Fri, 09 Nov 2018 07:40:23 +1300 Andrew Bartlett <abartlet at samba.org> wrote:> On Thu, 2018-11-08 at 09:29 +0000, Rowland Penny wrote: > > On Thu, 08 Nov 2018 22:08:51 +1300 > > Andrew Bartlett <abartlet at samba.org> wrote: > > > > > On Thu, 2018-11-08 at 08:48 +0000, Rowland Penny via samba wrote: > > > > > > > > Yes, but it isn't doing a read, it is trying to do a write and > > > > then being cancelled. > > > > > > Rowland, > > > > > > Can you point us as the evidence that leads you to that > > > conclusion? > > > > OK, the OP posted lines like these: > > > > 04:11:08 AM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay > > Command 04:11:28 AM 0 832 0.00 0.20 0.20 > > 0 /usr/sbin/smbd -D > > > > And from 'man pidstat' > > > > kB_rd/s > > Number of kilobytes the task has caused to be read from disk > > per second. > > > > kB_wr/s > > Number of kilobytes the task has caused, or shall cause to be > > written to disk per second. > > > > kB_ccwr/s > > Number of kilobytes whose writing to disk has been cancelled > > by the task. This may occur when the task truncates some dirty > > pagecache. In this case, some IO which another task has been > > accounted for will not be happening. > > > > kB_rd/s is 0.00 > > kB_wr/s and kB_ccwr/s are both 0.20 > > > > So, from my reading, nothing is being read, but something is trying > > to write, but being cancelled. > > > > I could of course be totally wrong, but if I am the 'pidstat' > > manpage needs rewriting. > > It is speculation, but this might be due to a mutex-enabled TDB. Each > read from a tdb would now be a write while the mutex is set. This > might not have been seen in the past when fcntl() was in use.Yes, but surely the 'read' would be shown as a 'read' as well as a write> > The cancellation might be the mutex being unlocked again.The OP actually has some of the locks turned off: locking = no strict locking = no> > > > > > > I think you are extrapolating too much from an smb.conf line, even > > > when serving read only shares, there are a lot of things in Samba > > > that can case both reads and writes. > > > > Not really extrapolating anything from smb.conf, I only pointed out > > that 'read only = no' and 'writable = no' are exact opposites of > > each other and shouldn't be in the same share together, or do you > > disagree ? > > OK. Either way I don't think it is relevant to the diagnosis. The > last statement unambiguously wins. >That is what I thought. I have had 'pidstat' running on a Unix domain member, Samba 4.6.15-Debian, for quite sometime and I got two lines like this: 15:13:57 0 3571 -1.00 -1.00 -1.00 1 /usr/sbin/winbindd and this twice: 11:38:17 10000 27033 -1.00 -1.00 -1.00 108 /usr/sbin/smbd -D Start time: 10:10:37 End time: 18:47:37 I have also run 'pidstat' on a Raspberrypi, Samba 4.5.12-Debian and got similar lines to the above, just slightly more often, but no where near as often as the OP. If it helps, I can set up a standalone server on the rpi tomorrow and see what I get. Rowland
On Thu, Nov 08, 2018 at 09:29:26AM +0000, Rowland Penny via samba wrote:> On Thu, 08 Nov 2018 22:08:51 +1300 > Andrew Bartlett <abartlet at samba.org> wrote: > > > On Thu, 2018-11-08 at 08:48 +0000, Rowland Penny via samba wrote: > > > > > > Yes, but it isn't doing a read, it is trying to do a write and then > > > being cancelled. > > > > Rowland, > > > > Can you point us as the evidence that leads you to that conclusion? > > OK, the OP posted lines like these: > > 04:11:08 AM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command > 04:11:28 AM 0 832 0.00 0.20 0.20 0 /usr/sbin/smbd -D > > And from 'man pidstat' > > kB_rd/s > Number of kilobytes the task has caused to be read from disk per second. > > kB_wr/s > Number of kilobytes the task has caused, or shall cause to be written to disk per second. > > kB_ccwr/s > Number of kilobytes whose writing to disk has been cancelled by the task. > This may occur when the task truncates some dirty pagecache. > In this case, some IO which another task has been accounted for will not be > happening. > > kB_rd/s is 0.00 > kB_wr/s and kB_ccwr/s are both 0.20 > > So, from my reading, nothing is being read, but something is trying to > write, but being cancelled. > > I could of course be totally wrong, but if I am the 'pidstat' manpage > needs rewriting.The housekeeping is doing an fstat(fd,..) on a file descriptor that is attached to a log file. That is a meta-data fetch that may wake up the disk. That's why syscall strace would be conclusive here.