CentOS - Nov 2012 - Slow login to system without internet connection

Hi.

For a long time I am puzzled why, when internet connection is gone 
(beyond first router, outside of subnet) SSH (and telnet?) connection is 
drastically slowed.

Same behavior, but less impacted is observed when system boots without 
working internet connection, and I am not sure, but I think even access 
to SAMBA fileshares can be/is affected. Affected systems are mostly 
production servers, and too spaced apart in time to be able to point a 
finger in something I am doing wrong.

I have not had time to gather more facts, I always forget/don;t have 
time, so I will only focus on definite SSH issue on CentOS 6.3, although 
I have seen same with 5.x.

1 server in question is set in following maner:
[root at chiron ~]# cat /etc/networks
default 0.0.0.0
loopback 127.0.0.0
link-local 169.254.0.0

[root at chiron ~]# cat /etc/sysconfig/network
NETWORKING=yes
NTPSERVERARGS=iburst
HOSTNAME=chiron.example.com

[root at chiron ~]# cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 
localhost4.localdomain4 chiron.example.com chiron
::1	localhost localhost.localdomain localhost6 localhost6.localdomain6 
chiron.example.com chiron

Does it have anything to do with /etc/hosts file not having row with 
used IP address, or is problem somewhere else?

Searching the net gives too much irrelevant links, and all I need it to 
be pointed in the right direction.


-- 

Ljubomir Ljubojevic
(Love is in the Air)
PL Computers
Serbia, Europe

Google is the Mother, Google is the Father, and traceroute is your
trusty Spiderman...
StarOS, Mikrotik and CentOS/RHEL/Linux consultant

Ljubomir Ljubojevic wrote:
> Hi.
>
> For a long time I am puzzled why, when internet connection is gone
> (beyond first router, outside of subnet) SSH (and telnet?) connection is
> drastically slowed.
>
> Same behavior, but less impacted is observed when system boots without
> working internet connection, and I am not sure, but I think even access
> to SAMBA fileshares can be/is affected. Affected systems are mostly
> production servers, and too spaced apart in time to be able to point a
> finger in something I am doing wrong.
>
> I have not had time to gather more facts, I always forget/don;t have
> time, so I will only focus on definite SSH issue on CentOS 6.3, although
> I have seen same with 5.x.
<snip>
Have you tried traceroute to a well-known location, like google.com, and
seen where the delay is?

     mark

Ljubomir Ljubojevic wrote:
> Hi.
>
> For a long time I am puzzled why, when internet connection is gone
> (beyond first router, outside of subnet) SSH (and telnet?) connection is
> drastically slowed.
>
> Same behavior, but less impacted is observed when system boots without
> working internet connection, and I am not sure, but I think even access
> to SAMBA fileshares can be/is affected. Affected systems are mostly
> production servers, and too spaced apart in time to be able to point a
> finger in something I am doing wrong.
>
> I have not had time to gather more facts, I always forget/don;t have
> time, so I will only focus on definite SSH issue on CentOS 6.3, although
> I have seen same with 5.x.
>
> 1 server in question is set in following maner:
> [root at chiron ~]# cat /etc/networks
> default 0.0.0.0
> loopback 127.0.0.0
> link-local 169.254.0.0
>
> [root at chiron ~]# cat /etc/sysconfig/network
> NETWORKING=yes
> NTPSERVERARGS=iburst
> HOSTNAME=chiron.example.com
>
> [root at chiron ~]# cat /etc/hosts
> 127.0.0.1	localhost localhost.localdomain localhost4
> localhost4.localdomain4 chiron.example.com chiron
> ::1	localhost localhost.localdomain localhost6 localhost6.localdomain6
> chiron.example.com chiron
>
> Does it have anything to do with /etc/hosts file not having row with
> used IP address, or is problem somewhere else?
>
> Searching the net gives too much irrelevant links, and all I need it to
> be pointed in the right direction.
perhaps due to DNS being down?

try starting sshd in debug mode (-d up to -ddd) on the server and 
connecting while the internet connection is down and see what is being 
logged.

On 11/20/2012 09:25 PM, Les Mikesell wrote:
> On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic <office at
plnet.rs> wrote:
>>
>> Now, when there was power failure (works on the transformer there), I
>> lost my internet connection, but router and wireless routers for the
>> rest of my towers were accessible. SSH logins to any of them is
instant.
>>
>> But when I tried to login to my server, it was not instantenous, and I
>> think it was 15+, maybe even 30+ seconds (I forgot to time it) from
>> start of ssh command to password prompt. It is in-house connection, so
>> there is nothing to traceroute.
>
> Most server apps will do a reverse-DNS lookup, if only to log the name
> for the connection, some will try an ident query for the user at the
> other end of the socket.   A 30+ second delay is a pretty sure sign
> that one or more of the DNS servers in your resolv.conf did not
> respond.  Running a local nameserver with a dummy local domain is one
> way to fix it, but just putting all your local systems in the
> /etc/hosts file will work too.
>
OK, that is what crossed my mind, but what I was hopping for is some 
elegant solution that would decrease the timeout. My server already has 
DNS server running and "nameserver 127.0.0.1" as first in
/etc/resolv.conf.

So the question is: "is there a setting that will reduce that DNS 
timeout for all running services, maybe like a ping-watchdog that would 
recognize the problem and skip the reverse-DNS lookup if DNS servers are 
not reachable?"

Adding and maintaining 30+ subnets in /etc/hosts is not really a good 
solution, and booting the server without reachable DNS server in some 
cases can be really frustrating, like if I boot Lap-top on the silo when 
internet connection is down (It was happening to me when I ran RHEL 6 
beta I think on each opening of the terminal, but I can not say I have 
seen this lately).

-- 

Ljubomir Ljubojevic
(Love is in the Air)
PL Computers
Serbia, Europe

Google is the Mother, Google is the Father, and traceroute is your
trusty Spiderman...
StarOS, Mikrotik and CentOS/RHEL/Linux consultant

Ljubomir Ljubojevic wrote:
> On 11/20/2012 09:25 PM, Les Mikesell wrote:
>> On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic <office at
plnet.rs>
>> wrote:
<snip>
>>> But when I tried to login to my server, it was not instantenous,
and I
>>> think it was 15+, maybe even 30+ seconds (I forgot to time it) from
>>> start of ssh command to password prompt. It is in-house connection,
so
>>> there is nothing to traceroute.
>>
>> Most server apps will do a reverse-DNS lookup, if only to log the name
>> for the connection, some will try an ident query for the user at the
>> other end of the socket.   A 30+ second delay is a pretty sure sign
>> that one or more of the DNS servers in your resolv.conf did not
>> respond.  Running a local nameserver with a dummy local domain is one
>> way to fix it, but just putting all your local systems in the
>> /etc/hosts file will work too.
<snip>
> So the question is: "is there a setting that will reduce that DNS
> timeout for all running services, maybe like a ping-watchdog that would
> recognize the problem and skip the reverse-DNS lookup if DNS servers are
> not reachable?"
<snip>
What does it say in /etc/nsswitch: is it dns files, or files dns?

       mark

On 11/20/2012 09:47 PM, m.roth at 5-cent.us wrote:
> Ljubomir Ljubojevic wrote:
>> On 11/20/2012 09:25 PM, Les Mikesell wrote:
>>> On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic <office at
plnet.rs>
>>> wrote:
> <snip>
>>>> But when I tried to login to my server, it was not
instantenous, and I
>>>> think it was 15+, maybe even 30+ seconds (I forgot to time it)
from
>>>> start of ssh command to password prompt. It is in-house
connection, so
>>>> there is nothing to traceroute.
>>>
>>> Most server apps will do a reverse-DNS lookup, if only to log the
name
>>> for the connection, some will try an ident query for the user at
the
>>> other end of the socket.   A 30+ second delay is a pretty sure sign
>>> that one or more of the DNS servers in your resolv.conf did not
>>> respond.  Running a local nameserver with a dummy local domain is
one
>>> way to fix it, but just putting all your local systems in the
>>> /etc/hosts file will work too.
> <snip>
>> So the question is: "is there a setting that will reduce that DNS
>> timeout for all running services, maybe like a ping-watchdog that would
>> recognize the problem and skip the reverse-DNS lookup if DNS servers
are
>> not reachable?"
> <snip>
> What does it say in /etc/nsswitch: is it dns files, or files dns?
"files dns"
-- 

Ljubomir Ljubojevic
(Love is in the Air)
PL Computers
Serbia, Europe

Google is the Mother, Google is the Father, and traceroute is your
trusty Spiderman...
StarOS, Mikrotik and CentOS/RHEL/Linux consultant