Andy Liebman
2009-Nov-05 03:01 UTC
[Samba] SMBD CPU climbs sky high when writing DPX files
Hello, I am experiencing a strange problem when writing (capturing) DPX video files to a Linux/Samba share. Basically, I'm seeing seeing a single smbd process go from 9 percent CPU utilization to 100 percent CPU utilization over the course of about 40 minutes. When smbd reaches 100 percent, the capture stops (drops frames). I have tested this with three different filesystem formats (XFS, ext3 and ext4) with similar -- although slightly different -- results. My test systems have been running the following: 2.6.31.4 kernel with Mandriva 2009.1 with Samba 3.3.2 2.6.28-11 Ubuntu 9.04 with Samba 3.3.2 2.6.27.38 kernel with Samba 3.0.23d 2.6.20.15 kernel with Samba 3.0.23d For those of you who may not be familiar with the DPX format, unlike QuickTime or other "streaming" media formats, DPX is "one file per frame". So, a 20 minute DPX movie in the US "NTSC TV" format would consist of 30 fps x 60 secs x 20 minutes = 36000 individual files all written into one directory. I have a very capable networked storage system. The underlying hardware RAID is capable of reading and writing at least 700 MB/sec sustained. The storage system is running on a 3.2 Ghz Intel 5482 Quad Core CPU with 4 or 6 GBs of RAM (depending on the system). I don't have any problem writing uncompressed high-definition QuickTime files to Samba shares at 160 MB/sec over 10 Gigabit Ethernet. However, trying to write standard definition DPX files over Samba -- only 40 MB/sec -- fails consistently after about 60,000 files have been written, plus or minus about 5000 files. In the screenshots shown in the link below, you can this happening when I was writing to a 6 TB ext4 filesystem. This was a real shocker. http://sites.google.com/site/andrewl733info/ext4_and_samba_2 On the above system (which had 6 GB of RAM), it seems that most cpu activity was restricted to a single core of a Quad Core CPU. SMBD cpu utilization climbed steadily throughout the capture. CPU began at 7 percent for one core and by the time I got to 66000 frames, I was at 100 percent cpu for one core. I started taking screen shots a third of the way through. Then I went back and started again. I also tested XFS as the filesystem. In fact, that's my preferred filesystem. I formatted the same 6 TB volume using the suggestions of experienced XFS developers (specifying "lazy-count=1" for the "log" and properly specifying the "sunit" and "swidth" for the data portion) and I mounted it with the important flags "-o,noatime,nobarrier,inode64". Here's a record of what happened under these circumstances. http://sites.google.com/site/andrewl733info/xfs_and_dpx-1 Depending on the kernel tested (and in some cases whether an irqbalance daemon was running), either most activity was concentrated on a single core, or it was spread out evenly among all 4 cores. But in either case, smbd cpu utilization would climb steadily from under 10 percent at the beginning to 10 times that at the end. And when "top" showed that my single active smbd was at 100 percent, the capture would fail. (To clarify this last point, when cpu activity is spread evenly among all cores, top still shows the total activity for smbd. When it says 100 percent, that would be about 25 percent on EACH core, or 100 percent out of 400 percent total.) My questions are: 1) why is smbd cpu utilization climbing so high in just 40 minutes? 2) could the cause be related to creating, opening and closing so many files so quickly? After all, we are creating, opening and closing 30 files each second. 3) is there anything I can do about it (besides give up)? Samba doesn't seem to be running out of memory. There is still 3-cores worth of CPU being unused at the time the capture fails. I have collected Level 10 logs of the capture to the XFS volume. Please feel free to look at them. I'm sorry if the format isn't totally convenient. The actual log file turned out to be 1.6 GB. I made a single bz2 file out of this -- 27 MB, but Google Sites only lets me upload in 11 MB pieces, so I split up the bz2 file into 3 pieces with the "split" program. I believe you will have to download all of them, and then put them back together with something like: "cat sambalogpart_a* sambalog.bz2" http://sites.google.com/site/andrewl733info/files/sambalogpart_aa?attredirects=0&d=1 http://sites.google.com/site/andrewl733info/files/sambalogpart_ab?attredirects=0&d=1 http://sites.google.com/site/andrewl733info/files/sambalogpart_ac?attredirects=0&d=1 Andy
Volker Lendecke
2009-Nov-05 11:27 UTC
[Samba] SMBD CPU climbs sky high when writing DPX files
On Wed, Nov 04, 2009 at 10:01:46PM -0500, Andy Liebman wrote:> 1) why is smbd cpu utilization climbing so high in just 40 minutes? > 2) could the cause be related to creating, opening and closing so many > files so quickly? After all, we are creating, opening and closing 30 > files each second. > 3) is there anything I can do about it (besides give up)? Samba doesn't > seem to be running out of memory. There is still 3-cores worth of CPU > being unused at the time the capture fails.Very likely you're hitting the case [in]sensivity issue. Try to set preserve case = no short preserve case = no default case = lower case sensitive = yes This assumes that all files on your files system are all lower-case, all new files will also be created that way, regardless of what the Windows clients request. Volker -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: <http://lists.samba.org/pipermail/samba/attachments/20091105/19ae1ac8/attachment.pgp>
Andy Liebman
2009-Nov-05 15:02 UTC
[Samba] SMBD CPU climbs sky high when writing DPX files
Hello, I'm sorry of this gets posted twice. I submitted the same post about 12 hours ago and it seems it didn't go through. I am experiencing a strange problem when writing (capturing) DPX video files to a Linux/Samba share. Basically, I'm seeing seeing a single smbd process go from 9 percent CPU utilization to 100 percent CPU utilization over the course of about 40 minutes. When smbd reaches 100 percent, the capture stops (drops frames). I have tested this with three different filesystem formats (XFS, ext3 and ext4) with similar -- although slightly different -- results. My test systems have been running the following: 2.6.31.4 kernel with Mandriva 2009.1 with Samba 3.3.2 2.6.28-11 Ubuntu 9.04 with Samba 3.3.2 2.6.27.38 kernel with Samba 3.0.23d 2.6.20.15 kernel with Samba 3.0.23d For those of you who may not be familiar with the DPX format, unlike QuickTime or other "streaming" media formats, DPX is "one file per frame". So, a 20 minute DPX movie in the US "NTSC TV" format would consist of 30 fps x 60 secs x 20 minutes = 36000 individual files all written into one directory. I have a very capable networked storage system. The underlying hardware RAID is capable of reading and writing at least 700 MB/sec sustained. The storage system is running on a 3.2 Ghz Intel 5482 Quad Core CPU with 4 or 6 GBs of RAM (depending on the system). I don't have any problem writing uncompressed high-definition QuickTime files to Samba shares at 160 MB/sec over 10 Gigabit Ethernet. However, trying to write standard definition DPX files over Samba -- only 40 MB/sec -- fails consistently after about 60,000 files have been written, plus or minus about 5000 files. In the screenshots shown in the link below, you can this happening when I was writing to a 6 TB ext4 filesystem. This was a real shocker. http://sites.google.com/site/andrewl733info/ext4_and_samba_2 On the above system (which had 6 GB of RAM), it seems that most cpu activity was restricted to a single core of a Quad Core CPU. SMBD cpu utilization climbed steadily throughout the capture. CPU began at 7 percent for one core and by the time I got to 66000 frames, I was at 100 percent cpu for one core. I started taking screen shots a third of the way through. Then I went back and started again. I also tested XFS as the filesystem. In fact, that's my preferred filesystem. I formatted the same 6 TB volume using the suggestions of experienced XFS developers (specifying "lazy-count=1" for the "log" and properly specifying the "sunit" and "swidth" for the data portion) and I mounted it with the important flags "-o,noatime,nobarrier,inode64". Here's a record of what happened under these circumstances. http://sites.google.com/site/andrewl733info/xfs_and_dpx-1 Depending on the kernel tested (and in some cases whether an irqbalance daemon was running), either most activity was concentrated on a single core, or it was spread out evenly among all 4 cores. But in either case, smbd cpu utilization would climb steadily from under 10 percent at the beginning to 10 times that at the end. And when "top" showed that my single active smbd was at 100 percent, the capture would fail. (To clarify this last point, when cpu activity is spread evenly among all cores, top still shows the total activity for smbd. When it says 100 percent, that would be about 25 percent on EACH core, or 100 percent out of 400 percent total.) My questions are: 1) why is smbd cpu utilization climbing so high in just 40 minutes? 2) could the cause be related to creating, opening and closing so many files so quickly? After all, we are creating, opening and closing 30 files each second. 3) is there anything I can do about it (besides give up)? Samba doesn't seem to be running out of memory. There is still 3-cores worth of CPU being unused at the time the capture fails. I have collected Level 10 logs of the capture to the XFS volume. Please feel free to look at them. I'm sorry if the format isn't totally convenient. The actual log file turned out to be 1.6 GB. I made a single bz2 file out of this -- 27 MB, but Google Sites only lets me upload in 11 MB pieces, so I split up the bz2 file into 3 pieces with the "split" program. I believe you will have to download all of them, and then put them back together with something like: "cat sambalogpart_a* sambalog.bz2" http://sites.google.com/site/andrewl733info/files/sambalogpart_aa?attredirects=0&d=1 http://sites.google.com/site/andrewl733info/files/sambalogpart_ab?attredirects=0&d=1 http://sites.google.com/site/andrewl733info/files/sambalogpart_ac?attredirects=0&d=1 Andy