zfs discuss - Jul 2008 - Streaming video and audio over CIFS lags.

I built a NAS with three 750 SATA disks in RAIDZ configuration and I''ve
exported some filesystems using the Solaris kernel CIFS.

Streaming video or even audio from the exported shares to windows xp gives a
laggy performance. Seeking the video can take ages, audio (playing mp3 with
winamp from the cifs share) stops from time to time and also the video playback
lags and pauses from time to time. The videos work just fine when I play them
from my local computer, or stream them from another windows computer via CIFS.

I''ve searched this forums and found some other users reporting same
problems, but not any good answers. I''ve tested network performance
with iperf, which reports network speed to be about 50 Mt/sec (1Gbps network, so
it should work much faster).

This http://www.opensolaris.org/jive/thread.jspa?messageID=232250&#232250
posting suggests trying to set zfetch_block_cap to 16..32, but I didn''t
found any ways how to set it. Any ideas? How I could improve my NAS to allow me
to stream HD video from it?

 - Juho M?kinen
 
 
This message posted from opensolaris.org

On Tue, Jul 1, 2008 at 14:47, Juho M?kinen <juho.makinen at gmail.com>
wrote:
> Streaming video or even audio from the exported shares to windows xp gives
a laggy performance. Seeking the video can take ages, audio (playing mp3 with
winamp from the cifs share) stops from time to time and also the video playback
lags and pauses from time to time. The videos work just fine when I play them
from my local computer, or stream them from another windows computer via CIFS.
What does local disk performance look like?  Try bonnie++
(http://will.incorrige.us/solaris-packages/CSEEbonnie++.pkg.gz or
build it yourself, it''s straightforward).  Also, what ethernet card is
this?  What does "dladm show-dev" report?  How does copying a large
file behave?  Do you get bursty transfers of a higher rate, or a
steady 50 megabits (look at Windows Task Manager''s various tabs)?

Will

Here''s bonnie++ output with default settings:
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
sonas         4100M 138010  74 144083  33 76546  19 138071  90 185735  15 464.7 
1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24307  99 +++++ +++ +++++ +++ 24438  99 +++++ +++ +++++ +++
sonas,4100M,138010,74,144083,33,76546,19,138071,90,185735,15,464.7,1,16,24307,99,+++++,+++,+++++,+++,24438,99,+++++,+++,+++++,+++

root at sonas:/etc# dladm show-dev
LINK            STATE  SPEED    DUPLEX
rge0            up     1000Mb   full

A bit speed testing with dd:
root at sonas:/etc# dd if=/dev/zero of=/storagepool/users/test.dump bs=128k
2035941376 bytes (2.0 GB) copied, 23.6579 seconds, 86.1 MB/s

The network rate is about 35-45 MB/s when copying large files from Windows XP
over CIFS to the ZFS share. At the same time "zpool iostat 5" shows
write speed which is jumping in the following manner (there''s actual
sequent numbers for write speed) 50.9M, 43.0M, 7.88M, 50.9M, 50.9M, 18.6M,
32.3M, 50.9M

Monitoring "iostat -xnz 1" shows the following trend (while copying
large files from windows):

three seconds of writing 
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  477.1    0.0 57656.7  2.8 31.6    5.9   66.3  97 100 c3t0d0
    0.0  469.4    0.0 56671.8  2.8 31.6    6.0   67.4  97 100 c3t1d0
    0.0  482.6    0.0 58648.3  2.8 31.6    5.8   65.5  97 100 c3t2d0

then three seconds without any writing. Then again three seconds of writing
following with three seconds without writing.

My machine is based on GIGABYTE GA-P35-DS3P motherboard with Intel Code 2 Duo
processor, 2GB fast 800Mhz DDR2 ram in dual-channel configuration and three
Seagate 7200.11 750GB disks connected to the motherboard SATA controler running
in AHCI mode. The motherboard uses Realtek 8111B network chipset to provide the
gigabit ethernet.

Does that info help anything?

 - Juho M?kinen
 
 
This message posted from opensolaris.org

On Tue, Jul 1, 2008 at 16:34, Juho M?kinen <juho.makinen at gmail.com>
wrote:
> Here''s bonnie++ output with default settings:
> Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
>                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
> sonas         4100M 138010  74 144083  33 76546  19 138071  90 185735  15
464.7   1
These are exactly the numbers I''d expect to see.  The disks
aren''t the problem.
> 2035941376 bytes (2.0 GB) copied, 23.6579 seconds, 86.1 MB/s
and neither is the filesystem.
> The network rate is about 35-45 MB/s when copying large files from Windows
XP over CIFS to the ZFS share.
This is the most surprising part of your findings, honestly.  I would
have expected lousy write rates over CIFS as well.

A few things to try: put in a different ethernet card if you have one,
on one or more ends.  Realtek works, but I''ve been unimpressed with
their performance in the past.  An Intel x1 pci express card will only
run you around $40, and I''ve seen much better results with them.  Is
hardware checksum offloading enabled on either end?  How does another
application (e.g., scp) behave between the two machines?
> Monitoring "iostat -xnz 1" shows the following trend (while
copying large files from windows):
> three seconds of writing
> then three seconds without any writing. Then again three seconds of writing
following with three seconds without writing.
If you watch network traffic during this time (e.g., with Wireshark or
snoop) do you see anything suspicious?
> Does that info help anything?
Yes, at least by ruling out ZFS itself as causing the problem.

Will

I mentioned this too, but on the performance forum:
http://www.opensolaris.org/jive/thread.jspa?threadID=64907&tstart=0

Unfortunately the performance forum has tumbleweeds blowing through it, so that
was probably the wrong place to complain.  Not that people don''t care
about performance, but the forum is dead.
 
 
This message posted from opensolaris.org

MC wrote:
> I mentioned this too, but on the performance forum:
http://www.opensolaris.org/jive/thread.jspa?threadID=64907&tstart=0
>
> Unfortunately the performance forum has tumbleweeds blowing through it, so
that was probably the wrong place to complain.  Not that people don''t
care about performance, but the forum is dead.
>   
It looks pretty lively from my browser :-)
http://www.opensolaris.org/jive/forum.jspa?forumID=26&start=0

But it may be that you are experiencing network performance issues
specifically, which pretty quickly descends into hardware+driver
details.  It is unusual for general performance group to know much
detail about more than a few such combinations.

Note: NIC device drivers can vary widely in their designs and
optimizations.  For large systems, we tend to tune for large-scale
efficiency which de-tunes for small-scale rapid response.  For
example, interrupt coalescing is important for big machines
handling many requests, but it works against rapid response for
a small workload.
 -- richard

> It looks pretty lively from my browser :-)
Now that you showed up ;)

In my case it is OpenSolaris in VirtualBox so I was expecting more cooperation,
or at least people striving to make them cooperate.

But like you said, this is likely just a case of OpenSolaris being optimized for
big iron and not home computers.  Which is a shame because OpenSolaris is for
home computers..
 
 
This message posted from opensolaris.org

> A few things to try: put in a different ethernet card
> if you have one,
> on one or more ends.  Realtek works, but I''ve been
> unimpressed with
> their performance in the past.  An Intel x1 pci
> express card will only
> run you around $40, and I''ve seen much better results
> with them.  
I first replaced my cheapest-from-the-shop-cheap 1Gbps network switch with a
much better HP unmanaged switch, which didn''t solve the problem.

Then I went and bought an Intel PCI Gigabit Ethernet card for 25? which seems to
have solved the problem. I still need to do some testing though to verify.
> Is hardware checksum offloading enabled on either end?
>  How does another
> pplication (e.g., scp) behave between the two
> machines?
What''s this?

So far thanks for the responses so far, hopefully my Intel network card solved
my problems =)

 - Juho M?kinen
 
 
This message posted from opensolaris.org

On Wed, Jul 2, 2008 at 13:16, Juho M?kinen <juho.makinen at gmail.com>
wrote:
> Then I went and bought an Intel PCI Gigabit Ethernet card for 25? which
seems to have solved the problem. I still need to do some testing though to
verify.
Glad to hear it.
>> Is hardware checksum offloading enabled on either end?
>>  How does another
>> pplication (e.g., scp) behave between the two
>> machines?
> What''s this?
Hardware checksum offloading is when the card calculates the checksum
of the outgoing packets in hardware rather than letting the OS do it.
It can give better performance than software (depending on
configuration) but some cards do the checksums improperly, which can
lead to poor performance since bad packets are dropped.  On Solaris
you can try the program here:
http://www.opensolaris.org/jive/thread.jspa?threadID=31058&tstart=150
to see if it''s enabled; on Linux run "ethtool -k
<ifname>", or
"netstat -ant" on Windows.

scp is secure copy; it transfers files from one machine to another
over an ssh tunnel.  It may be processor-bound, especially since its
encryption is single-threaded, but it''s a good thing to compare to
when CIFS is misbehaving.  It''s included with Linux and Solaris, and
you can try WinSCP on Windows.

That said, if it''s working with the replacement card I
wouldn''t worry
about it too much ;)
> So far thanks for the responses so far, hopefully my Intel network card
solved my problems =)
You''re welcome, and I hope it keeps working for you.

Will

> Then I went and bought an Intel PCI Gigabit Ethernet card for 25? which
seems to have solved the problem.
Is this really the case?  If so that is an important clue to finding out why
virtualized opensolaris performance is so poor.  I tried every network adapter
in virtualbox and vmware and performance always sucked, but maybe it is still
narrowed down to a networking configuration problem.
 
 
This message posted from opensolaris.org

It solved my problems, the difference was really huge. The onboard realtek 8111B
chip could do about 40Mt/sec file transfer over CIFS which looked good, but in
reality the io speed was very bad: backups to the CIFS share took ages, copying
files to using ftp and unzipping files located in the cifs share to the same
folder on cifs share.

The new intel network card solved all this and more: backups are going in a
fraction of time, raw network speed doubled from 400Mbps to 800Mbps and video
streaming doesnt lag anymore at all. It really feels now like a local harddisk.

 - Juho M?kinen
 
 
This message posted from opensolaris.org