Hi all, I''m not sure if the problem is with FreeBSD or ZFS or both so I cross-post (I known it''s bad). Well I''ve server running FreeBSD 9.0 with (don''t count / on differents disks) zfs pool with 36 disk. The performance is very very good on the server. I''ve one NFS client running FreeBSD 8.3 and the performance over NFS is very good : For example : Read from the client and write over NFS to ZFS: [root@ .tmp]# time tar xf /tmp/linux-3.7.5.tar real 1m7.244s user 0m0.921s sys 0m8.990s this client is on 1Gbits/s network cable and same network switch as the server. I''ve a second NFS client running FreeBSD 9.1-Stable, and on this second client the performance is catastrophic. After 1 hour the tar isn''t finish. OK this second client is connect with 100Mbit/s and not on the same switch. But well from 2 min --> ~ 90 min ...:-( I''ve try for this second client to change on the ZFS-NFS server the zfs set sync=disabled and that change nothing. On a third NFS client linux (recent Ubuntu) I got the almost same catastrophic performance. With or without sync=disabled. Those three NFS client use TCP. If I do a classic scp I got normal speed ~9-10 Mbytes/s so the network is not the problem. I try to something like (find with google): net.inet.tcp.sendbuf_max: 2097152 -> 16777216 net.inet.tcp.recvbuf_max: 2097152 -> 16777216 net.inet.tcp.sendspace: 32768 -> 262144 net.inet.tcp.recvspace: 65536 -> 262144 net.inet.tcp.mssdflt: 536 -> 1452 net.inet.udp.recvspace: 42080 -> 65535 net.inet.udp.maxdgram: 9216 -> 65535 net.local.stream.recvspace: 8192 -> 65535 net.local.stream.sendspace: 8192 -> 65535 and that change nothing either. Anyone have any idea ? Regards. JAS -- Albert SHIH DIO b?timent 15 Observatoire de Paris 5 Place Jules Janssen 92195 Meudon Cedex T?l?phone : 01 45 07 76 26/06 86 69 95 71 xmpp: jas at obspm.fr Heure local/Local time: jeu 31 jan 2013 23:04:47 CET
On Jan 31, 2013, at 5:16 PM, Albert Shih wrote:> Well I''ve server running FreeBSD 9.0 with (don''t count / on differents > disks) zfs pool with 36 disk. > > The performance is very very good on the server. > > I''ve one NFS client running FreeBSD 8.3 and the performance over NFS is > very good : > > For example : Read from the client and write over NFS to ZFS: > > [root@ .tmp]# time tar xf /tmp/linux-3.7.5.tar > > real 1m7.244s > user 0m0.921s > sys 0m8.990s > > this client is on 1Gbits/s network cable and same network switch as the > server. > > I''ve a second NFS client running FreeBSD 9.1-Stable, and on this second > client the performance is catastrophic. After 1 hour the tar isn''t finish. > OK this second client is connect with 100Mbit/s and not on the same switch. > But well from 2 min --> ~ 90 min ...:-( > > I''ve try for this second client to change on the ZFS-NFS server the > > zfs set sync=disabled > > and that change nothing.I have been using FreeBSD 9 with ZFS and NFS to a couple Mac OS X (10.6.8 Snow Leopard) boxes and I get between 40 and 50 MB/sec throughput on a Gigabit ethernet link. Since you have already ruled out the known sync issue with ZFS and no SSD-based write cache, then perhaps you are running into an NFS 3 vs. NFS 4 issue. I am not sure if Mac OS X is using NFS 3 or NFS 4. -- Paul Kraus Deputy Technical Director, LoneStarCon 3 Sound Coordinator, Schenectady Light Opera Company
Le 04/02/2013 ? 11:21:12-0500, Paul Kraus a ?crit> On Jan 31, 2013, at 5:16 PM, Albert Shih wrote: > > > Well I''ve server running FreeBSD 9.0 with (don''t count / on > > differents disks) zfs pool with 36 disk. > > > > The performance is very very good on the server. > > > > I''ve one NFS client running FreeBSD 8.3 and the performance over NFS > > is very good : > > > > For example : Read from the client and write over NFS to ZFS: > > > > [root@ .tmp]# time tar xf /tmp/linux-3.7.5.tar > > > > real 1m7.244s user 0m0.921s sys 0m8.990s > > > > this client is on 1Gbits/s network cable and same network switch as > > the server. > > > > I''ve a second NFS client running FreeBSD 9.1-Stable, and on this > > second client the performance is catastrophic. After 1 hour the tar > > isn''t finish. OK this second client is connect with 100Mbit/s and > > not on the same switch. But well from 2 min --> ~ 90 min ...:-( > > > > I''ve try for this second client to change on the ZFS-NFS server the > > > > zfs set sync=disabled > > > > and that change nothing. > > I have been using FreeBSD 9 with ZFS and NFS to a couple Mac OS X > (10.6.8 Snow Leopard) boxes and I get between 40 and 50 MB/secThanks for your answer. Can you give me the average ping time between you''r client and NFS server ? Regards. JAS -- Albert SHIH DIO b?timent 15 Observatoire de Paris 5 Place Jules Janssen 92195 Meudon Cedex T?l?phone : 01 45 07 76 26/06 86 69 95 71 xmpp: jas at obspm.fr Heure local/Local time: mar 5 f?v 2013 16:15:11 CET
Sašo Kiselkov
2013-Feb-05 16:04 UTC
[zfs-discuss] zfs + NFS + FreeBSD with performance prob
On 01/31/2013 11:16 PM, Albert Shih wrote:> Hi all, > > I''m not sure if the problem is with FreeBSD or ZFS or both so I cross-post > (I known it''s bad). > > Well I''ve server running FreeBSD 9.0 with (don''t count / on differents > disks) zfs pool with 36 disk. > > The performance is very very good on the server. > > I''ve one NFS client running FreeBSD 8.3 and the performance over NFS is > very good : > > For example : Read from the client and write over NFS to ZFS: > > [root@ .tmp]# time tar xf /tmp/linux-3.7.5.tar > > real 1m7.244s > user 0m0.921s > sys 0m8.990s > > this client is on 1Gbits/s network cable and same network switch as the > server. > > I''ve a second NFS client running FreeBSD 9.1-Stable, and on this second > client the performance is catastrophic. After 1 hour the tar isn''t finish. > OK this second client is connect with 100Mbit/s and not on the same switch. > But well from 2 min --> ~ 90 min ...:-( > > I''ve try for this second client to change on the ZFS-NFS server the > > zfs set sync=disabled > > and that change nothing. > > On a third NFS client linux (recent Ubuntu) I got the almost same catastrophic > performance. With or without sync=disabled. > > Those three NFS client use TCP. > > If I do a classic scp I got normal speed ~9-10 Mbytes/s so the network is > not the problem. > > I try to something like (find with google): > > net.inet.tcp.sendbuf_max: 2097152 -> 16777216 > net.inet.tcp.recvbuf_max: 2097152 -> 16777216 > net.inet.tcp.sendspace: 32768 -> 262144 > net.inet.tcp.recvspace: 65536 -> 262144 > net.inet.tcp.mssdflt: 536 -> 1452 > net.inet.udp.recvspace: 42080 -> 65535 > net.inet.udp.maxdgram: 9216 -> 65535 > net.local.stream.recvspace: 8192 -> 65535 > net.local.stream.sendspace: 8192 -> 65535 > > > and that change nothing either. > > Anyone have any idea ?What you describe sounds like a bad networking issue. Check your network via the usual tools like ping, mtr, netperf, etc. Verify cabling and interface counters on your machines too, for stuff like CRC errors or jabbers - a few of those and the throughput of a TCP link goes down the drain. Cheers, -- Saso
Sašo Kiselkov
2013-Feb-05 16:08 UTC
[zfs-discuss] zfs + NFS + FreeBSD with performance prob
On 02/05/2013 05:04 PM, Sa?o Kiselkov wrote:> On 01/31/2013 11:16 PM, Albert Shih wrote: >> Hi all, >> >> I''m not sure if the problem is with FreeBSD or ZFS or both so I cross-post >> (I known it''s bad). >> >> Well I''ve server running FreeBSD 9.0 with (don''t count / on differents >> disks) zfs pool with 36 disk. >> >> The performance is very very good on the server. >> >> I''ve one NFS client running FreeBSD 8.3 and the performance over NFS is >> very good : >> >> For example : Read from the client and write over NFS to ZFS: >> >> [root@ .tmp]# time tar xf /tmp/linux-3.7.5.tar >> >> real 1m7.244s >> user 0m0.921s >> sys 0m8.990s >> >> this client is on 1Gbits/s network cable and same network switch as the >> server. >> >> I''ve a second NFS client running FreeBSD 9.1-Stable, and on this second >> client the performance is catastrophic. After 1 hour the tar isn''t finish. >> OK this second client is connect with 100Mbit/s and not on the same switch. >> But well from 2 min --> ~ 90 min ...:-( >> >> I''ve try for this second client to change on the ZFS-NFS server the >> >> zfs set sync=disabled >> >> and that change nothing. >> >> On a third NFS client linux (recent Ubuntu) I got the almost same catastrophic >> performance. With or without sync=disabled. >> >> Those three NFS client use TCP. >> >> If I do a classic scp I got normal speed ~9-10 Mbytes/s so the network is >> not the problem. >> >> I try to something like (find with google): >> >> net.inet.tcp.sendbuf_max: 2097152 -> 16777216 >> net.inet.tcp.recvbuf_max: 2097152 -> 16777216 >> net.inet.tcp.sendspace: 32768 -> 262144 >> net.inet.tcp.recvspace: 65536 -> 262144 >> net.inet.tcp.mssdflt: 536 -> 1452 >> net.inet.udp.recvspace: 42080 -> 65535 >> net.inet.udp.maxdgram: 9216 -> 65535 >> net.local.stream.recvspace: 8192 -> 65535 >> net.local.stream.sendspace: 8192 -> 65535 >> >> >> and that change nothing either. >> >> Anyone have any idea ? > > What you describe sounds like a bad networking issue. Check your network > via the usual tools like ping, mtr, netperf, etc. Verify cabling and > interface counters on your machines too, for stuff like CRC errors or > jabbers - a few of those and the throughput of a TCP link goes down the > drain.Just one more thing: simply doing SCP need not show a problem. SCP is very uni-directional, and you may be hitting an issue in the opposite direction (I''ve seen TP cables where one pair was fine and the other was giving bad data). Also check for dropped packets on your source and target machines via tools like DTrace. Cheers, -- Saso