On Mon, Oct 24, 2016 at 5:25 PM, Matt Garman <matthew.garman at gmail.com> wrote:> On Mon, Oct 24, 2016 at 2:42 PM, Larry Martell <larry.martell at gmail.com> wrote: >>> At any rate, what I was looking at was seeing if there was any way to >>> simplify this process, and cut NFS out of the picture. If you need >>> only to push these files around, what about rsync? >> >> It's not just moving files around. The files are read, and their >> contents are loaded into a MySQL database. > > On what server does the MySQL database live?The C6 host, same one that the script runs on. We can of course access the MySQL server from the C7 host, assuming the needed packages are there.>> This site is not in any way connected to the internet, and you cannot >> bring in any computers, phones, or media of any kind. There is a >> process to get machines or files in, but it is onerous and time >> consuming. This system was set up and configured off site and then >> brought on site. > > But clearly you have a means to log in to both the C6 and C7 servers, > right? Otherwise, how would be able to see these errors, check > top/sar/free/iostat/etc? > > And if you are logging in to both of these boxes, I assume you are > doing so via ssh? > > Or are you actually physically sitting in front of these machines?The machines are on a local network. I access them with putty from a windows machine, but I have to be at the site to do that.> If you have ssh access to these machines, then you can trivially copy > files to/from them. If ssh is installed and working, then scp should > also be installed and working. Even if you don't have scp, you can > use tar over ssh to the same effect. It's ugly, but doable, and there > are examples online for how to do it. > > Also: you made a couple comments about these machines, it looks like > the C7 box (FTP server + NFS server) is running bare metal (i.e. not a > virtual machine). The C6 instance (NFS client) is virtualized.Correct.> What host is the C6 instance? > > Is the C6 instance running under the C7 instance? I.e., are both > machines on the same physical hardware? If that is true, then your > "network" (at least the one between C7 and C6) is basically virtual, > and to have issues like this on the same physical box is certainly > indicative of a mis-configuration.Yes, the C6 instance is running on the C7 machine. What could be mis-configured? What would I check to find out?>> To run the script on the C7 NFS server instead of the C6 NFS client >> many python libs will have to installed. I do have someone off site >> working on setting up a local yum repo with what I need, and then we >> are going to see if we can zip and email the repo and get it on site. >> But none of us are sys admins and we don't really know what we're >> doing so we may not succeed and it may take longer then I will be here >> in Japan (I am scheduled to leave Saturday). > > Right, but my point is you can write your own custom script(s) to copy > files from C7 to C6 (based on rsync or ssh), do the processing on C6 > (DB loading, whatever other processing), then move back to C7 if > necessary. You said yourself you are a programmer not a sysadmin, so > change the nature of the problem from a sysadmin problem to a > programming problem.Yes, that is potential solution I had not thought of. The issue with this is that we have the same system installed at many, many sites, and they all work fine. It is only this site that is having an issue. We really do not want to have different SW running at just this one site. Running the script on the C7 host is a change, but at least it will be the same software as every place else.> I'm certain I'm missing something, but the fundamental architecture > doesn't make sense to me given what I understand of the process flow. > > Were you able to run some basic network testing tools between the C6 > and C7 machines? I'm interested specifically in netperf, which does > round trip packet testing, both TCP and UDP. I would look for packet > drops with UDP, and/or major performance outliers with TCP, and/or any > kind of timeouts with either protocol.netperf is not installed.> How is name resolution working on both machines? Do you address > machines by hostname (e.g., "my_c6_server"), or explicitly by IP > address? Are you using DNS or are the IPs hard-coded in /etc/hosts?Everything is by ip address.> To me it still "smells" like a networking issue...
On Mon, Oct 24, 2016 at 6:09 PM, Larry Martell <larry.martell at gmail.com> wrote:> The machines are on a local network. I access them with putty from a > windows machine, but I have to be at the site to do that.So that means when you are offsite there is no way to access either machine? Does anyone have a means to access these machines from offsite?> Yes, the C6 instance is running on the C7 machine. What could be > mis-configured? What would I check to find out?OK, so these two machines are actually the same physical hardware, correct? Do you know, is the networking between the two machines "soft", as in done locally on the machine (typically through NAT or briding)? Or is it "hard", in that you have a dedicated NIC for the host and a separate dedicated NIC for the guest, and actual cables going out of each interface and connected to a switch/hub/router? I would expect the former... If it truly is a "soft" network between the machines, then that is more evidence of a configuration error. Now, unfortunately, with what to look for: I have virtually no experience setting up C6 guests on a C7 host; at least not enough to help you troubleshoot the issue. But in general, you should be able to hit up a web search and look for howtos and other documents on setting up networking between a C7 host and its guests. That will allow you to (1) understand how it's currently setup, (2) verify if there is any misconfig, and (3) correct or change if needed.> Yes, that is potential solution I had not thought of. The issue with > this is that we have the same system installed at many, many sites, > and they all work fine. It is only this site that is having an issue. > We really do not want to have different SW running at just this one > site. Running the script on the C7 host is a change, but at least it > will be the same software as every place else.IIRC, you said this is the only C7 instance? That would mean it is already not the same as every other site. It may be conceptually the same, but "under the hood", there are a tremendous number of changes between C6 and C7. Effectively every single package is different, from the kernel all the way to trivial userspace tools.> netperf is not installed.Again, if you can use putty (which is ssh) to access these systems, you implicitly have the ability to upload files (i.e. packages) to the systems. A simple tool like netperf should have few (if any) dependencies, so you don't have to mess with mirroring the whole centos repo. Just grab the netperf rpm file from wherever, then use scp (I believe it's called pscp when part of the Putty package) to copy to your servers, yum install and start testing.
On Tue, Oct 25, 2016 at 12:48 PM, Matt Garman <matthew.garman at gmail.com> wrote:> On Mon, Oct 24, 2016 at 6:09 PM, Larry Martell <larry.martell at gmail.com> wrote: >> The machines are on a local network. I access them with putty from a >> windows machine, but I have to be at the site to do that. > > So that means when you are offsite there is no way to access either > machine? Does anyone have a means to access these machines from > offsite? > >> Yes, the C6 instance is running on the C7 machine. What could be >> mis-configured? What would I check to find out? > > OK, so these two machines are actually the same physical hardware, correct?Yes.> > Do you know, is the networking between the two machines "soft", as in > done locally on the machine (typically through NAT or briding)? Or is > it "hard", in that you have a dedicated NIC for the host and a > separate dedicated NIC for the guest, and actual cables going out of > each interface and connected to a switch/hub/router? I would expect > the former...I don't know, but would also guess the former.> If it truly is a "soft" network between the machines, then that is > more evidence of a configuration error. Now, unfortunately, with what > to look for: I have virtually no experience setting up C6 guests on a > C7 host; at least not enough to help you troubleshoot the issue. But > in general, you should be able to hit up a web search and look for > howtos and other documents on setting up networking between a C7 host > and its guests. That will allow you to (1) understand how it's > currently setup, (2) verify if there is any misconfig, and (3) correct > or change if needed. > >> Yes, that is potential solution I had not thought of. The issue with >> this is that we have the same system installed at many, many sites, >> and they all work fine. It is only this site that is having an issue. >> We really do not want to have different SW running at just this one >> site. Running the script on the C7 host is a change, but at least it >> will be the same software as every place else. > > IIRC, you said this is the only C7 instance? That would mean it is > already not the same as every other site. It may be conceptually the > same, but "under the hood", there are a tremendous number of changes > between C6 and C7. Effectively every single package is different, > from the kernel all the way to trivial userspace tools.Yes, of course it's different at that level. But I was talking about our application software and set up. It is that that I want to keep consistent across deployments.>> netperf is not installed. > > Again, if you can use putty (which is ssh) to access these systems, > you implicitly have the ability to upload files (i.e. packages) to the > systems. A simple tool like netperf should have few (if any) > dependencies, so you don't have to mess with mirroring the whole > centos repo. Just grab the netperf rpm file from wherever, then use > scp (I believe it's called pscp when part of the Putty package) to > copy to your servers, yum install and start testing.Again, no machine on the internal network that my 2 CentOS hosts are on are connected to the internet. I have no way to download anything., There is an onerous and protracted process to get files into the internal network and I will see if I can get netperf in.