Elliott Freis
2004-Jan-17 14:26 UTC
Odd lag/hanging issue with production ftp server - Please help AS AP!
I am forwarding this to this list per recommendation.
Note on recent test: Trying to do a transfer on the public LAN off the ftp
server, I am only able to get a bursty 20k/s! down from the ftp server.
This was with nfs completely unmounted, ftping from a local drive so its not
an NFS problem. It should get a full 2-3mb a sec at least. It used to. Up
is still full speed. The machine has a Intel pro NIC, direct to a Cisco
ArrowPoint. Both are set to 100 FD. ifconfig_fxp2="inet 66.151.XXX.XXX
netmask 255.255.255.224 media 100baseTX"
Any help GREATLY appreciated, I have exhausted all avenues I can think of,
including hardware swaps. Here is some diag:
last pid: 49903; load averages: 0.12, 0.20, 0.25
up 0+15:19:19 14:19:00
73 processes: 1 running, 72 sleeping
CPU states: 1.4% user, 0.0% nice, 17.1% system, 2.3% interrupt, 79.2%
idle
Mem: 24M Active, 1233M Inact, 178M Wired, 68M Cache, 163M Buf, 3600K Free
Swap: 1024M Total, 12K Used, 1024M Free
353/17120/262144 mbufs in use (current/peak/max):
271 mbufs allocated to data
82 mbufs allocated to packet headers
213/16806/65536 mbuf clusters in use (current/peak/max)
37892 Kbytes allocated to network (19% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs
Coll
fxp2 1500 <Link#3> 00:02:a5:13:fc:b5 30214691 0 22672636 0
589096
fxp2 1500 66.151.XXX.XXX ftp 30150589 - 22683851
- -
fxp2 1500 fe80:3::202 fe80:3::202:a5ff: 0 - 0 -
-
> -----Original Message-----
> From: Elliott Freis
> Sent: Friday, January 09, 2004 3:28 PM
> To: 'freebsd-questions@FreeBSD.ORG'
> Subject: Odd lag/hanging issue with production ftp server - Please
> help ASAP!
>
> I have been struggling with this one for over a month now. Here is a
> quick layout of my setup:
>
> Primary FTP server:
> Compaq DL380 1.5gb ram
> FreeBSD 4.5
> 3x36gb RAID 5 drives as local boot/storage
> ProFTPd
>
> Array server for FTP:
> AMD Athlon 2200 512mb ram
> FreeBSD 4.8
> 7x36gb Fiber channel drives, RAID 5 via Vinum.
>
> Both machines are connected via a cross-over cable, that has been tested
> good and swapped just in case. The primary storage for FTP is done on the
> Fiber drives via nfs from FTP to Array server. NFS options are -U -3.
>
> My problem is this. As more users connect and store files, the primary
> FTP machine becomes increasingly unresponsive. Currently, I max at about
> 350 concurrent FTP connections. The most basic test I have been doing is
> just holding down enter on an SSH session. As you hold enter down, you
> see it visually just hang for a second or more (up to about 5 seconds
> depending on the load). It is even worse if I spam "df -k" for
example.
> For part of the time, it responds fine, though its randomly a second to
> multiple seconds. In other words, it is randomly responsive and not
> responsive every 5 seconds or so. During the "hanging" time, ftp
sessions
> are also hung. So you see very bursty data transfers. Now thankfully, no
> ftp sessions drop, so we do get the data we need. But this is a terrible
> thing to be happening to a production server.
>
> One other thing of note, this happened to me about 4-5 months ago, but a
> reboot fixed it for some reason. So I concluded it was just a hiccup.
> But it has returned after a different reboot, and won't go away.
>
> Any help in troubleshooting this is very appreciated! Happy new year,
>
> -Elliott
>
> Example of "enter" latency (this is from a LAN connection):
> At shell prompt ">"
> >
> >
>
>
>
>
>
>
>
>
>
>
>
>
> >
> >
> >
> >
> >
>
> >
>
> >
> >
> >
>
> >
> >
> >
>
> >
> >
>
>
>
> >
> >
> >
Brad Davis
2004-Jan-17 14:49 UTC
Odd lag/hanging issue with production ftp server - Please help AS AP!
Elliott, You might try downloading a file from the internet to locate where the problem is. Or a different FTP server. Etc. Keep trying to narrow down where the problem is. Regards, Brad On Sat, Jan 17, 2004 at 02:24:51PM -0800, Elliott Freis wrote:> I am forwarding this to this list per recommendation. > > > Note on recent test: Trying to do a transfer on the public LAN off the ftp > server, I am only able to get a bursty 20k/s! down from the ftp server. > This was with nfs completely unmounted, ftping from a local drive so its not > an NFS problem. It should get a full 2-3mb a sec at least. It used to. Up > is still full speed. The machine has a Intel pro NIC, direct to a Cisco > ArrowPoint. Both are set to 100 FD. ifconfig_fxp2="inet 66.151.XXX.XXX > netmask 255.255.255.224 media 100baseTX" > > Any help GREATLY appreciated, I have exhausted all avenues I can think of, > including hardware swaps. Here is some diag: > > last pid: 49903; load averages: 0.12, 0.20, 0.25 > up 0+15:19:19 14:19:00 > 73 processes: 1 running, 72 sleeping > CPU states: 1.4% user, 0.0% nice, 17.1% system, 2.3% interrupt, 79.2% > idle > Mem: 24M Active, 1233M Inact, 178M Wired, 68M Cache, 163M Buf, 3600K Free > Swap: 1024M Total, 12K Used, 1024M Free > > 353/17120/262144 mbufs in use (current/peak/max): > 271 mbufs allocated to data > 82 mbufs allocated to packet headers > 213/16806/65536 mbuf clusters in use (current/peak/max) > 37892 Kbytes allocated to network (19% of mb_map in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > > Name Mtu Network Address Ipkts Ierrs Opkts Oerrs > Coll > fxp2 1500 <Link#3> 00:02:a5:13:fc:b5 30214691 0 22672636 0 > 589096 > fxp2 1500 66.151.XXX.XXX ftp 30150589 - 22683851 > - - > fxp2 1500 fe80:3::202 fe80:3::202:a5ff: 0 - 0 - > - > > > -----Original Message----- > > From: Elliott Freis > > Sent: Friday, January 09, 2004 3:28 PM > > To: 'freebsd-questions@FreeBSD.ORG' > > Subject: Odd lag/hanging issue with production ftp server - Please > > help ASAP! > > > > I have been struggling with this one for over a month now. Here is a > > quick layout of my setup: > > > > Primary FTP server: > > Compaq DL380 1.5gb ram > > FreeBSD 4.5 > > 3x36gb RAID 5 drives as local boot/storage > > ProFTPd > > > > Array server for FTP: > > AMD Athlon 2200 512mb ram > > FreeBSD 4.8 > > 7x36gb Fiber channel drives, RAID 5 via Vinum. > > > > Both machines are connected via a cross-over cable, that has been tested > > good and swapped just in case. The primary storage for FTP is done on the > > Fiber drives via nfs from FTP to Array server. NFS options are -U -3. > > > > My problem is this. As more users connect and store files, the primary > > FTP machine becomes increasingly unresponsive. Currently, I max at about > > 350 concurrent FTP connections. The most basic test I have been doing is > > just holding down enter on an SSH session. As you hold enter down, you > > see it visually just hang for a second or more (up to about 5 seconds > > depending on the load). It is even worse if I spam "df -k" for example. > > For part of the time, it responds fine, though its randomly a second to > > multiple seconds. In other words, it is randomly responsive and not > > responsive every 5 seconds or so. During the "hanging" time, ftp sessions > > are also hung. So you see very bursty data transfers. Now thankfully, no > > ftp sessions drop, so we do get the data we need. But this is a terrible > > thing to be happening to a production server. > > > > One other thing of note, this happened to me about 4-5 months ago, but a > > reboot fixed it for some reason. So I concluded it was just a hiccup. > > But it has returned after a different reboot, and won't go away. > > > > Any help in troubleshooting this is very appreciated! Happy new year, > > > > -Elliott > > > > Example of "enter" latency (this is from a LAN connection): > > At shell prompt ">" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Matt Douhan
2004-Jan-17 15:48 UTC
Odd lag/hanging issue with production ftp server - Please help AS AP!
> > Note on recent test: Trying to do a transfer on the public LAN off the ftp > server, I am only able to get a bursty 20k/s! down from the ftp server. > This was with nfs completely unmounted, ftping from a local drive so its > not an NFS problem. It should get a full 2-3mb a sec at least. It used > to. Up is still full speed. The machine has a Intel pro NIC, direct to a > Cisco ArrowPoint. Both are set to 100 FD. ifconfig_fxp2="inet > 66.151.XXX.XXX netmask 255.255.255.224 media 100baseTX"If you use mediaopt on your client you will need to also hardcode the settings on the arrowpoint or you may end up in trouble there with the negotiation, and also if you only set the speed and not the duplex hardcoded you will have troubles as well. your options are, hardcode neither speed nor duplex, or hardcode both speed and duplex, you seem to have a mix of both and that is never a good idea. -- Matt Douhan www.fruitsalad.org matt@fruitsalad.org ping elvis elvis is alive -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20040118/84ae2ad3/attachment.bin