Hello list, I setup Iometer to test AIO for 100% random read. If "Transfer Request Size" is more than or equal to 256 kilobytes,in the beginning the transmission is good. But 3~5 seconds later,the throughput will drop to zero. Server OS: Ubuntu Server 14.04.1 LTS Samba: Version 4.1.6-Ubuntu Dialect: SMB 2.0 AIO settings : aio read size = 1 aio write size = 1 vfs objects = aio_pthread Client OS: Win 8.1 professional 64Bit Iometer Setting: Iometer 2006.07.27 Worker : 8 Max Disk Size (Sectors) : 16777216 Outstanding I/Os : 128 Starting Disk Sector : 0 100% Random read Any insight into why this is so and any other tuning suggestion is appreciated. Thanks. Regards, Mike -- View this message in context: http://samba.2283325.n4.nabble.com/Problem-with-AIO-random-read-tp4677191.html Sent from the Samba - General mailing list archive at Nabble.com.
On Wed, Dec 03, 2014 at 03:45:24AM -0800, mikeliu wrote:> Hello list, > > I setup Iometer to test AIO for 100% random read. > If "Transfer Request Size" is more than or equal to 256 kilobytes,in the > beginning the transmission is good. > But 3~5 seconds later,the throughput will drop to zero. > > Server OS: > Ubuntu Server 14.04.1 LTS > > Samba: > Version 4.1.6-Ubuntu > > Dialect: > SMB 2.0 > > AIO settings : > aio read size = 1 > aio write size = 1 > vfs objects = aio_pthreadPlease don't use this module. pthread aio is built in for 4.1. aio_pthread is only used for creating files asynchronously, read/write is done without any special module. Removing that module won't help you probably though. We need to really take a look at what's happening, this should just work fine. When the system is stalled, how many threads does the smbd have active? /proc/pid/tasks will tell you. Also, can you attach to smbd with gdb and show a backtrace? Maybe it does not happen anymore under strace, but could you strace -f that smbd while its running into the issue? Ah, btw, you don't happen to have sendfile/recvfile active? I think those don't play well with aio. With best regards, Volker Lendecke -- SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen phone: +49-551-370000-0, fax: +49-551-370000-9 AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen http://www.sernet.de, mailto:kontakt at sernet.de
Hello Volker, Thanks for kindly feedback :)> When the system is stalled, how many threads does the smbd >have active? /proc/pid/tasks will tell you.At the beginning ... pidstat -t|grep smbd 11:41:16 AM 0 5388 - 0.00 0.00 0.00 0.00 1 smbd 11:41:16 AM 0 - 5388 0.00 0.00 0.00 0.00 1 |__smbd 11:41:16 AM 0 5390 - 0.00 0.00 0.00 0.00 1 smbd 11:41:16 AM 0 - 5390 0.00 0.00 0.00 0.00 1 |__smbd 11:41:16 AM 65534 5423 - 0.03 0.26 0.00 0.28 1 smbd 11:41:16 AM 65534 - 5423 0.02 0.20 0.00 0.22 1 |__smbd 11:41:16 AM 65534 - 5593 0.00 0.00 0.00 0.00 1 |__smbd 11:41:16 AM 65534 - 5594 0.00 0.00 0.00 0.01 1 |__smbd 11:41:16 AM 65534 - 5595 0.00 0.00 0.00 0.00 1 |__smbd 11:41:16 AM 65534 - 5599 0.00 0.00 0.00 0.00 1 |__smbd When the throughput drop to zero ... pidstat -t|grep smbd 02:37:45 PM 0 5388 - 0.00 0.00 0.00 0.00 1 smbd 02:37:45 PM 0 - 5388 0.00 0.00 0.00 0.00 1 |__smbd 02:37:45 PM 0 5390 - 0.00 0.00 0.00 0.00 0 smbd 02:37:45 PM 0 - 5390 0.00 0.00 0.00 0.00 0 |__smbd 02:37:45 PM 0 5423 - 0.03 0.34 0.00 0.38 0 smbd 02:37:45 PM 0 - 5423 0.03 0.26 0.00 0.29 0 |__smbd>Maybe it does not happen anymore under strace, but could you >strace -f that smbd while its running into the issue?When I executed strace on smbd,the error did not occur.(Timing ?) For log.smbd,the highest debug level I could reproduce this error is log level 5.> Ah, btw, you don't happen to have sendfile/recvfile active? > I think those don't play well with aio.# testparm -sv 2>/dev/null|grep -E 'receivefile|sendfile' min receivefile size = 0 use sendfile = No This symptom did not occur in Samba-4.0.13,but it occur after upgrading to Samba-4.0.14. So I use Samba-4.0.13 and patch this one "bug #10298: Reduce smb2_server processing overhead". This symptom can reproduce again. Regards, Mike -- View this message in context: http://samba.2283325.n4.nabble.com/Problem-with-AIO-random-read-tp4677191p4677287.html Sent from the Samba - General mailing list archive at Nabble.com.