openssh bugs - Sep 2007 - [Bug 1357] New: SOCKS proxy attempts fail to some servers due to DNS timeouts

http://bugzilla.mindrot.org/show_bug.cgi?id=1357

           Summary: SOCKS proxy attempts fail to some servers due to DNS
                    timeouts
           Product: Portable OpenSSH
           Version: 4.6p1
          Platform: ix86
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: sshd
        AssignedTo: bitbucket at mindrot.org
        ReportedBy: sbbvai2hwkmh at jampee.com


Created an attachment (id=1345)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=1345)
proposed patch

---PROBLEM DESCRIPTION---

I use ssh as a SOCKS 5 proxy for Firefox, and I have configured firefox
to perform remote DNS lookups. That is, the SOCKS request contains the
hostname rather than the IP address of the host I want to connect to.

For the vast majority of sites I connect to, this works great. However,
for a few hosts, including www.etrade.com and www.vanguard.com, the
connection hangs for several seconds, then times out.

Although I think it's irrelevant, my SSH client is OpenSSH 4.6p1 on
MacOS 10.4.

My server is OpenSSH 4.6p1 on Linux 2.6.12.5.


---INVESTIGATION---

I ran strace on the sshd and saw that the DNS lookup of
www.vanguard.com was hanging (the DNS server took a long time to
respond, much more than 5 seconds). I decoded the DNS request and saw
that it is requesting QTYPE  28, which is the DNS AAAA record. This is
the request for the IPv6 address.

Next I tried this DNS lookup with dig. I ran "dig -t aaaa
www.vanguard.com", and it hung for about 20 seconds before finally
returning. I ran "dig -t aaaa www.yahoo.com", and it returned
immediately.

I ran these same dig tests on a different machine, serviced by a
different ISP and DNS servers, and got the same results.

My conclusion is that an AAAA lookup on some hosts will hang for a long
time.

Next I downloaded portable OpenSSH, compiled my own sshd, and found the
function connect_to() in channels.c. Note that the call to
getaddrinfo() is passing in a hints structure consisting of
ai_family=IPv4or6 and ai_socktype=SOCK_STREAM. The hints parameter is
optional, and if it is not specified it still allows either IPv4 or
IPv6 results. I replaced hints with NULL and recompiled. My problem
went away.

---RECOMMENDATION---

I recommend that the hints parameter be omitted, as this seems to fix
the hanging behavior while still working correctly on all sites I try
to connect to.


-- 
Configure bugmail: http://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

http://bugzilla.mindrot.org/show_bug.cgi?id=1357


Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org




--- Comment #1 from Damien Miller <djm at mindrot.org>  2007-09-04
11:44:03 ---
Could you try setting "AddressFamily inet" in your
/etc/ssh/sshd_config
instead?

The fix is not correct and will, among other things, break the
AddressFamily option.


-- 
Configure bugmail: http://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

http://bugzilla.mindrot.org/show_bug.cgi?id=1357





--- Comment #2 from Jamie Nicolson <sbbvai2hwkmh at jampee.com> 
2007-09-04 11:55:05 ---
Yes, that fixed it. *sigh*

Would you not agree that "AddressFamily=any" is still broken in the
common case (where IPv6 is not used)? It should not hang like it does.


-- 
Configure bugmail: http://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

http://bugzilla.mindrot.org/show_bug.cgi?id=1357


Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au




--- Comment #3 from Darren Tucker <dtucker at zip.com.au>  2007-09-04
12:03:55 ---
(In reply to comment #2)
> Yes, that fixed it. *sigh*
> 
> Would you not agree that "AddressFamily=any" is still broken in
the
> common case (where IPv6 is not used)? It should not hang like it does.
I think the brokenness is in the DNS infrastructure in question.

Quoth RFC4074 (ftp://ftp.rfc-editor.org/in-notes/rfc4074.txt):

"4.  Problematic Behaviors

   There are some known cases at authoritative servers that do not
   conform to the expected behavior.  This section describes those
   problematic cases.

  4.1.  Ignore Queries for AAAA

   Some authoritative servers seem to ignore queries for an AAAA RR,
   causing a delay at the stub resolver to fall back to a query for an
A
   RR.  This behavior may cause a fatal timeout at the resolver or at
   the application that calls the resolver.  Even if the resolver
   eventually falls back, the result can be an unacceptable delay for
   the application user, especially with interactive applications like
   web browsing."


-- 
Configure bugmail: http://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

http://bugzilla.mindrot.org/show_bug.cgi?id=1357





--- Comment #4 from Damien Miller <djm at mindrot.org>  2007-09-04
12:06:24 ---
Your platform's resolver should "do the right thing" when
AddressFamily=any is in use, as this sets hints.ai_family to be
AF_UNSPEC which should be equivalent to not setting a hints (for the
common case at least). 

Perhaps your resolver is defaulting to IPv4-only lookups when no hints
is specified, but doing IPv6-then-IPv4 when hints.ai_family==AF_UNSPEC.
IMO this would be rather silly behaviour, but I have seen libc authors
do dumber things...

IIRC we need to fill out hints for either SRV RR or SCTP support, but
my memory is hazy.


-- 
Configure bugmail: http://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

http://bugzilla.mindrot.org/show_bug.cgi?id=1357





--- Comment #5 from Jamie Nicolson <sbbvai2hwkmh at jampee.com> 
2007-09-04 12:18:54 ---
Wow, great research on the AAAA problem.

So it's pretty clear that the DNS server is misbehaving. My getaddrinfo
implementation (glibc 2.5 I think) is also doing something I don't
understand, which may be wrong. Maybe the behavior has changed in later
versions of getaddrinfo(). I'll try to explore that more.

Does this warrant a note in the FAQ or other documentation? Maybe it's
rare problem. "If you use SOCKS proxying with remote DNS lookup and
connections to some hosts timeout, and you don't need IPv6 support, try
setting 'AddressFamily inet' in your sshd config." Unfortunately
many
users won't have access to their server's sshd config. Would changing
it on the client side would work?


-- 
Configure bugmail: http://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.