On Fri, Oct 21, 2016 at 11:21 AM, <m.roth at 5-cent.us> wrote:> Larry Martell wrote: >> We have 1 system ruining Centos7 that is the NFS server. There are 50 >> external machines that FTP files to this server fairly continuously. >> >> We have another system running Centos6 that mounts the partition the files >> are FTP-ed to using NFS. > <snip> > What filesystem?Sorry for being dense, but I am not a sys admin, I am programmer and we have no sys admin. I don't know what you mean by your question. I am NFS mounting to what ever the default filesystem would be on a CentOS6 system.
Larry Martell wrote:> On Fri, Oct 21, 2016 at 11:21 AM, <m.roth at 5-cent.us> wrote: >> Larry Martell wrote: >>> We have 1 system ruining Centos7 that is the NFS server. There are 50 >>> external machines that FTP files to this server fairly continuously. >>> >>> We have another system running Centos6 that mounts the partition the >>> files >>> are FTP-ed to using NFS. >> <snip> >> What filesystem? > > Sorry for being dense, but I am not a sys admin, I am programmer and > we have no sys admin. I don't know what you mean by your question. I > am NFS mounting to what ever the default filesystem would be on a > CentOS6 system.This *is* a sysadmin issue. Each partition is formatted as a specific type of filesystem. The standard Linux filesystems for Upsteam-descended have been ext3, then ext4, and now xfs. Tools to manipulate xfs will not work with extx, and vice versa. cat /etc/fstab on the systems, and see what they are. If either is xfs, and assuming that the systems are on UPSes, then the fstab which controls drive mounting on a system should have, instead of "defaults", nobarrier,inode64. Note that the inode64 is relevant if the filesystem is > 2TB. The reason I say this is that we we started rolling out CentOS 7, we tried to put one of our user's home directory on one, and it was a disaster. 100% repeatedly, untarring a 100M tarfile onto an nfs-mounted drive took seven minutes, where before, it had taken 30 seconds. Timed. It took us months to discover that NFS 4 tries to make transactions atomic, which is fine if you're worrying about losing power or connectivity. If you're on a UPS, and hardwired, adding the nobarrier immediately brought it down to 40 seconds or so. mark
On Fri, Oct 21, 2016 at 11:42 AM, <m.roth at 5-cent.us> wrote:> Larry Martell wrote: >> On Fri, Oct 21, 2016 at 11:21 AM, <m.roth at 5-cent.us> wrote: >>> Larry Martell wrote: >>>> We have 1 system ruining Centos7 that is the NFS server. There are 50 >>>> external machines that FTP files to this server fairly continuously. >>>> >>>> We have another system running Centos6 that mounts the partition the >>>> files >>>> are FTP-ed to using NFS. >>> <snip> >>> What filesystem? >> >> Sorry for being dense, but I am not a sys admin, I am programmer and >> we have no sys admin. I don't know what you mean by your question. I >> am NFS mounting to what ever the default filesystem would be on a >> CentOS6 system. > > This *is* a sysadmin issue.I know it's a sysadmin issue. I wish we had one, but we don't and I am the one being asked to fix things.> Each partition is formatted as a specific type > of filesystem. The standard Linux filesystems for Upsteam-descended have > been ext3, then ext4, and now xfs. Tools to manipulate xfs will not work > with extx, and vice versa. > > cat /etc/fstab on the systems, and see what they are. If either is xfs, > and assuming that the systems are on UPSes, then the fstab which controls > drive mounting on a system should have, instead of "defaults", > nobarrier,inode64. > > Note that the inode64 is relevant if the filesystem is > 2TB. > > The reason I say this is that we we started rolling out CentOS 7, we tried > to put one of our user's home directory on one, and it was a disaster. > 100% repeatedly, untarring a 100M tarfile onto an nfs-mounted drive took > seven minutes, where before, it had taken 30 seconds. Timed. It took us > months to discover that NFS 4 tries to make transactions atomic, which is > fine if you're worrying about losing power or connectivity. If you're on a > UPS, and hardwired, adding the nobarrier immediately brought it down to 40 > seconds or so.I have no remote access to this system, only on site, so it will have to wait until Monday for me to check. (The system is in Japan and I traveled from NY to Japan, where I am now, just to fix this issue.) Thanks for the reply.
On Fri, Oct 21, 2016 at 11:42 AM, <m.roth at 5-cent.us> wrote:> Larry Martell wrote: >> On Fri, Oct 21, 2016 at 11:21 AM, <m.roth at 5-cent.us> wrote: >>> Larry Martell wrote: >>>> We have 1 system ruining Centos7 that is the NFS server. There are 50 >>>> external machines that FTP files to this server fairly continuously. >>>> >>>> We have another system running Centos6 that mounts the partition the >>>> files >>>> are FTP-ed to using NFS. >>> <snip> >>> What filesystem? >> >> Sorry for being dense, but I am not a sys admin, I am programmer and >> we have no sys admin. I don't know what you mean by your question. I >> am NFS mounting to what ever the default filesystem would be on a >> CentOS6 system. > > This *is* a sysadmin issue. Each partition is formatted as a specific type > of filesystem. The standard Linux filesystems for Upsteam-descended have > been ext3, then ext4, and now xfs. Tools to manipulate xfs will not work > with extx, and vice versa. > > cat /etc/fstab on the systems, and see what they are. If either is xfs, > and assuming that the systems are on UPSes, then the fstab which controls > drive mounting on a system should have, instead of "defaults", > nobarrier,inode64.The server is xfs (the client is nfs). The server does have inode64 specified, but not nobarrier.> Note that the inode64 is relevant if the filesystem is > 2TB.The file system is 51TB.> The reason I say this is that we we started rolling out CentOS 7, we tried > to put one of our user's home directory on one, and it was a disaster. > 100% repeatedly, untarring a 100M tarfile onto an nfs-mounted drive took > seven minutes, where before, it had taken 30 seconds. Timed. It took us > months to discover that NFS 4 tries to make transactions atomic, which is > fine if you're worrying about losing power or connectivity. If you're on a > UPS, and hardwired, adding the nobarrier immediately brought it down to 40 > seconds or so.We are not seeing a performance issue - do you think nobarrier would help with our lock up issue? I wanted to try it but my client did not want me to make any changes until we got the bad disk replaced. Unfortunately that will not happen until Wednesday.