Andy Liebman
2009-Nov-05 03:01 UTC
[Samba] SMBD CPU climbs sky high when writing DPX files
Hello,
I am experiencing a strange problem when writing (capturing) DPX video
files to a Linux/Samba share. Basically, I'm seeing seeing a single smbd
process go from 9 percent CPU utilization to 100 percent CPU utilization
over the course of about 40 minutes. When smbd reaches 100 percent, the
capture stops (drops frames). I have tested this with three different
filesystem formats (XFS, ext3 and ext4) with similar -- although
slightly different -- results.
My test systems have been running the following:
2.6.31.4 kernel with Mandriva 2009.1 with Samba 3.3.2
2.6.28-11 Ubuntu 9.04 with Samba 3.3.2
2.6.27.38 kernel with Samba 3.0.23d
2.6.20.15 kernel with Samba 3.0.23d
For those of you who may not be familiar with the DPX format, unlike
QuickTime or other "streaming" media formats, DPX is "one file
per
frame". So, a 20 minute DPX movie in the US "NTSC TV" format
would
consist of 30 fps x 60 secs x 20 minutes = 36000 individual files all
written into one directory.
I have a very capable networked storage system. The underlying hardware
RAID is capable of reading and writing at least 700 MB/sec sustained.
The storage system is running on a 3.2 Ghz Intel 5482 Quad Core CPU with
4 or 6 GBs of RAM (depending on the system). I don't have any problem
writing uncompressed high-definition QuickTime files to Samba shares at
160 MB/sec over 10 Gigabit Ethernet. However, trying to write standard
definition DPX files over Samba -- only 40 MB/sec -- fails consistently
after about 60,000 files have been written, plus or minus about 5000 files.
In the screenshots shown in the link below, you can this happening when
I was writing to a 6 TB ext4 filesystem. This was a real shocker.
http://sites.google.com/site/andrewl733info/ext4_and_samba_2
On the above system (which had 6 GB of RAM), it seems that most cpu
activity was restricted to a single core of a Quad Core CPU. SMBD cpu
utilization climbed steadily throughout the capture. CPU began at 7
percent for one core and by the time I got to 66000 frames, I was at 100
percent cpu for one core. I started taking screen shots a third of the
way through. Then I went back and started again.
I also tested XFS as the filesystem. In fact, that's my preferred
filesystem. I formatted the same 6 TB volume using the suggestions of
experienced XFS developers (specifying "lazy-count=1" for the
"log" and
properly specifying the "sunit" and "swidth" for the data
portion) and I
mounted it with the important flags "-o,noatime,nobarrier,inode64".
Here's a record of what happened under these circumstances.
http://sites.google.com/site/andrewl733info/xfs_and_dpx-1
Depending on the kernel tested (and in some cases whether an irqbalance
daemon was running), either most activity was concentrated on a single
core, or it was spread out evenly among all 4 cores. But in either
case, smbd cpu utilization would climb steadily from under 10 percent at
the beginning to 10 times that at the end. And when "top" showed that
my
single active smbd was at 100 percent, the capture would fail. (To
clarify this last point, when cpu activity is spread evenly among all
cores, top still shows the total activity for smbd. When it says 100
percent, that would be about 25 percent on EACH core, or 100 percent out
of 400 percent total.)
My questions are:
1) why is smbd cpu utilization climbing so high in just 40 minutes?
2) could the cause be related to creating, opening and closing so many
files so quickly? After all, we are creating, opening and closing 30
files each second.
3) is there anything I can do about it (besides give up)? Samba doesn't
seem to be running out of memory. There is still 3-cores worth of CPU
being unused at the time the capture fails.
I have collected Level 10 logs of the capture to the XFS volume. Please
feel free to look at them. I'm sorry if the format isn't totally
convenient. The actual log file turned out to be 1.6 GB. I made a
single bz2 file out of this -- 27 MB, but Google Sites only lets me
upload in 11 MB pieces, so I split up the bz2 file into 3 pieces with
the "split" program. I believe you will have to download all of them,
and then put them back together with something like:
"cat sambalogpart_a* sambalog.bz2"
http://sites.google.com/site/andrewl733info/files/sambalogpart_aa?attredirects=0&d=1
http://sites.google.com/site/andrewl733info/files/sambalogpart_ab?attredirects=0&d=1
http://sites.google.com/site/andrewl733info/files/sambalogpart_ac?attredirects=0&d=1
Andy
Volker Lendecke
2009-Nov-05 11:27 UTC
[Samba] SMBD CPU climbs sky high when writing DPX files
On Wed, Nov 04, 2009 at 10:01:46PM -0500, Andy Liebman wrote:> 1) why is smbd cpu utilization climbing so high in just 40 minutes? > 2) could the cause be related to creating, opening and closing so many > files so quickly? After all, we are creating, opening and closing 30 > files each second. > 3) is there anything I can do about it (besides give up)? Samba doesn't > seem to be running out of memory. There is still 3-cores worth of CPU > being unused at the time the capture fails.Very likely you're hitting the case [in]sensivity issue. Try to set preserve case = no short preserve case = no default case = lower case sensitive = yes This assumes that all files on your files system are all lower-case, all new files will also be created that way, regardless of what the Windows clients request. Volker -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: <http://lists.samba.org/pipermail/samba/attachments/20091105/19ae1ac8/attachment.pgp>
Andy Liebman
2009-Nov-05 15:02 UTC
[Samba] SMBD CPU climbs sky high when writing DPX files
Hello,
I'm sorry of this gets posted twice. I submitted the same post about 12
hours ago and it seems it didn't go through.
I am experiencing a strange problem when writing (capturing) DPX video
files to a Linux/Samba share. Basically, I'm seeing seeing a single smbd
process go from 9 percent CPU utilization to 100 percent CPU utilization
over the course of about 40 minutes. When smbd reaches 100 percent, the
capture stops (drops frames). I have tested this with three different
filesystem formats (XFS, ext3 and ext4) with similar -- although
slightly different -- results.
My test systems have been running the following:
2.6.31.4 kernel with Mandriva 2009.1 with Samba 3.3.2
2.6.28-11 Ubuntu 9.04 with Samba 3.3.2
2.6.27.38 kernel with Samba 3.0.23d
2.6.20.15 kernel with Samba 3.0.23d
For those of you who may not be familiar with the DPX format, unlike
QuickTime or other "streaming" media formats, DPX is "one file
per
frame". So, a 20 minute DPX movie in the US "NTSC TV" format
would
consist of 30 fps x 60 secs x 20 minutes = 36000 individual files all
written into one directory.
I have a very capable networked storage system. The underlying hardware
RAID is capable of reading and writing at least 700 MB/sec sustained.
The storage system is running on a 3.2 Ghz Intel 5482 Quad Core CPU with
4 or 6 GBs of RAM (depending on the system). I don't have any problem
writing uncompressed high-definition QuickTime files to Samba shares at
160 MB/sec over 10 Gigabit Ethernet. However, trying to write standard
definition DPX files over Samba -- only 40 MB/sec -- fails consistently
after about 60,000 files have been written, plus or minus about 5000 files.
In the screenshots shown in the link below, you can this happening when
I was writing to a 6 TB ext4 filesystem. This was a real shocker.
http://sites.google.com/site/andrewl733info/ext4_and_samba_2
On the above system (which had 6 GB of RAM), it seems that most cpu
activity was restricted to a single core of a Quad Core CPU. SMBD cpu
utilization climbed steadily throughout the capture. CPU began at 7
percent for one core and by the time I got to 66000 frames, I was at 100
percent cpu for one core. I started taking screen shots a third of the
way through. Then I went back and started again.
I also tested XFS as the filesystem. In fact, that's my preferred
filesystem. I formatted the same 6 TB volume using the suggestions of
experienced XFS developers (specifying "lazy-count=1" for the
"log" and
properly specifying the "sunit" and "swidth" for the data
portion) and I
mounted it with the important flags "-o,noatime,nobarrier,inode64".
Here's a record of what happened under these circumstances.
http://sites.google.com/site/andrewl733info/xfs_and_dpx-1
Depending on the kernel tested (and in some cases whether an irqbalance
daemon was running), either most activity was concentrated on a single
core, or it was spread out evenly among all 4 cores. But in either
case, smbd cpu utilization would climb steadily from under 10 percent at
the beginning to 10 times that at the end. And when "top" showed that
my
single active smbd was at 100 percent, the capture would fail. (To
clarify this last point, when cpu activity is spread evenly among all
cores, top still shows the total activity for smbd. When it says 100
percent, that would be about 25 percent on EACH core, or 100 percent out
of 400 percent total.)
My questions are:
1) why is smbd cpu utilization climbing so high in just 40 minutes?
2) could the cause be related to creating, opening and closing so many
files so quickly? After all, we are creating, opening and closing 30
files each second.
3) is there anything I can do about it (besides give up)? Samba doesn't
seem to be running out of memory. There is still 3-cores worth of CPU
being unused at the time the capture fails.
I have collected Level 10 logs of the capture to the XFS volume. Please
feel free to look at them. I'm sorry if the format isn't totally
convenient. The actual log file turned out to be 1.6 GB. I made a
single bz2 file out of this -- 27 MB, but Google Sites only lets me
upload in 11 MB pieces, so I split up the bz2 file into 3 pieces with
the "split" program. I believe you will have to download all of them,
and then put them back together with something like:
"cat sambalogpart_a* sambalog.bz2"
http://sites.google.com/site/andrewl733info/files/sambalogpart_aa?attredirects=0&d=1
http://sites.google.com/site/andrewl733info/files/sambalogpart_ab?attredirects=0&d=1
http://sites.google.com/site/andrewl733info/files/sambalogpart_ac?attredirects=0&d=1
Andy