Here is one iperf log for dom0 => nas (LRO=1) case.
It seems that the bench sucks at low BW while some occasional peak
shows up, raising the overall BW to the level of 300Mbps.
# iperf -c 192.168.1.9 -f m -i 1 -w 256k -l 256k -t 20
------------------------------------------------------------
Client connecting to 192.168.1.9, TCP port 5001
TCP window size: 0.25 MByte (WARNING: requested 0.25 MByte)
------------------------------------------------------------
[ 3] local 192.168.1.6 port 39519 connected with 192.168.1.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 160 MBytes 1340 Mbits/sec
[ 3] 1.0- 2.0 sec 5.75 MBytes 48.2 Mbits/sec
[ 3] 2.0- 3.0 sec 9.00 MBytes 75.5 Mbits/sec
[ 3] 3.0- 4.0 sec 6.00 MBytes 50.3 Mbits/sec
[ 3] 4.0- 5.0 sec 6.00 MBytes 50.3 Mbits/sec
[ 3] 5.0- 6.0 sec 8.25 MBytes 69.2 Mbits/sec
[ 3] 6.0- 7.0 sec 9.25 MBytes 77.6 Mbits/sec
[ 3] 7.0- 8.0 sec 161 MBytes 1351 Mbits/sec
[ 3] 8.0- 9.0 sec 5.75 MBytes 48.2 Mbits/sec
[ 3] 9.0-10.0 sec 6.00 MBytes 50.3 Mbits/sec
[ 3] 10.0-11.0 sec 8.75 MBytes 73.4 Mbits/sec
[ 3] 11.0-12.0 sec 56.0 MBytes 470 Mbits/sec
[ 3] 12.0-13.0 sec 108 MBytes 906 Mbits/sec
[ 3] 13.0-14.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 14.0-15.0 sec 6.00 MBytes 50.3 Mbits/sec
[ 3] 15.0-16.0 sec 9.25 MBytes 77.6 Mbits/sec
[ 3] 16.0-17.0 sec 5.75 MBytes 48.2 Mbits/sec
[ 3] 17.0-18.0 sec 152 MBytes 1275 Mbits/sec
[ 3] 18.0-19.0 sec 6.00 MBytes 50.3 Mbits/sec
[ 3] 19.0-20.0 sec 6.25 MBytes 52.4 Mbits/sec
[ 3] 0.0-20.1 sec 746 MBytes 312 Mbits/sec
On Thu, Apr 4, 2013 at 8:00 PM, G.R. <firemeteor@users.sourceforge.net>
wrote:> Hi guys,
> I''m running a freenas domU (FreeBSD 8.3 based, ZFS v28, 2 vcpus
mapped to
> the same HT capable core) to serve storage for all purpose including other
> domUs running on the same host. I did some study to understand how well it
> works and the result is kind of confusing.
>
> In summary, the network performance between domains on the same host is
> worse than expected. And NFS service to other domains on the same host is
> not reliable. The last issue make it impossible to serve VM disk image from
> the nas. Anybody has met the same issue?
>
> Benchmark details below:
>
> 1. iperf bench against external host (through physical network).
> The bench can fully utilize the Giga-bit network.
> ext => nas ~930Mbps
> nas => ext ~ 900Mbps
> TSO / LRO config has no impact on the bench result.
>
> 2. iperf bench against dom0 (through virtual bridge, handled fully by
driver
> stack)
> The bench result highly depends on TSO / LRO config.
> dom0 => nas (LRO=1) 60Mbps ~ 300Mbps (vary a lot time to time, xentop
> reports low cpu usage in both domain (varies corresponding to BPS, ~ 20%)).
> dom0 => nas (LRO=0) 10900Mbps (stable, high cpu usage ~190%)
>
> nas => dom0 (TSO=1) 470Mbps (stable result, low cpu usage ~40%)
> nas => dom0 (TSO=0) 860Mbps (stable result, high cpu usage ~170%. some
> times can stable at ~2000Mbps, not sure how to reproduce)
>
> Similar result can be observed when talking to another linux domU. However,
> linux domU can generally achieve better performance than dom0.
>
> 3. As a comparasion, here is a iperf test between linux domU and dom0.
> dom0 => lin domU ~10300Mbps
> lin domU => dom0 ~13500Mbps
>
> 4. file transfer from external host to nas
> PS: Here the bottleneck is the network rather than the disk.
> NFS: 50MB/s
> FTP: 102MB/s
> I''m not quite sure about how much perf to expect from NFS. But it
appears to
> be a bottleneck.
>
> 5. file transfer from dom0 to nas
> copy to the NFS mount served by the nas domU can cause hang. Almost 100%
> reproduce.
> Typically the hang is limited to the nas and dom0 processes that accesses
> the mount.
> "umount -f" can resolve the hang in such case. But in some cases,
all ssh
> sessions to dom0 hang too. Not sure if this means a dom0 network hang or a
> full freeze.
>
> Tcpdump seems to show that some traffic is lost. But there is no package
> drop statistics shown in both end from the netstat / ifconfig command.
>
> Thanks,
> Timothy