openssh unix dev - Oct 2018 - SSH framing problem? Bad packet length 1397966893

I am trying to troubleshoot an SSH connectivity problem (client v6.6.1p1,
server v6.7p1) that also involves a custom packet forwarding service (so,
it may not actually be an SSH problem).

The issue is intermittent, and, when present, manifests itself with the
client error message "Bad packet length 1397966893. Disconnecting: Packet
corrupt." Based on my limited understanding of the SSH Transport Layer
Protocol, I gathered that this essentially signifies that the first four
bytes of the initial protocol identification packet ("SSH-") are being
misinterpreted as the packet length header (per RFC 4253 section 6).
When analyzing the traffic with Wireshark, I noticed that the SSH server
(or the packet forwarder) combines the initial protocol identification and
the Key Exchange Init message into a single TCP packet. For example, I
captured a 991-byte packet that consisted of the 39-byte string
"SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1<CR><LF>", followed
by what appears
to be a 948-byte Key Exchange Init message with 4 bytes of padding, all in
a single TCP packet. The client then sends a Key Exchange Init packet and
an Elliptic Curve Diffie-Hellman Key Exchange Init packet in quick
succession. However, now, instead of sending its own ECDH Key Exchange Init
packet the server apparently starts over and resends another initial
protocol identification packet. This packet is then interpreted by the
client per RFC 4253 section 6, which, of course, makes no sense, and the
client sends a disconnect.

My main question is: does the SSH Transport Layer Protocol expect the
various messages always to be sent as individual packets, or does it
support stream-based connections where messages can be reframed into more
or fewer packets?
I looked at RFC 4253 for clues, but I couldn't find any definitive
statement on the particular framing requirements of the protocol.

Wireshark summary is below.

Thanks for any pointers how to further troubleshoot this.

Mirko

No. Time     srcport dstport Protocol Length Payload Info
4   0.000457  58438   23619  SSHv2    112    44      Client: Protocol
(SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.10)
6   0.323053  23619   58438  SSHv2    1059   991     Server: Protocol
(SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u1), Key Exchange Init
8   0.325071  58438   23619  SSHv2    2036   1968    Client: Key Exchange
Init
10  0.331070  58438   23619  SSHv2    116    48      Client: Elliptic Curve
Diffie-Hellman Key Exchange Init
12  1.063073  23619   58438  TCP      107    39      23619 ? 58438 [PSH,
ACK] Seq=992 Ack=2061 Win=174720 Len=39 TSval=3778028126 TSecr=3778027943
[TCP segment of a reassembled PDU]
13  1.063248  58438   23619  SSHv2    108    40      Client: Disconnect

On 9 October 2018 at 04:58, Mirko Raner <mirko at raner.ws>
wrote:
> I am trying to troubleshoot an SSH connectivity problem (client v6.6.1p1,
> server v6.7p1) that also involves a custom packet forwarding service (so,
> it may not actually be an SSH problem).
>
> The issue is intermittent, and, when present, manifests itself with the
> client error message "Bad packet length 1397966893. Disconnecting:
Packet
> corrupt." Based on my limited understanding of the SSH Transport Layer
> Protocol, I gathered that this essentially signifies that the first four
> bytes of the initial protocol identification packet ("SSH-") are
being
> misinterpreted as the packet length header (per RFC 4253 section 6).
That seems like a plausible explanation.

$ perl -le 'print pack("N1", "1397966893")'
SSH-
> My main question is: does the SSH Transport Layer Protocol expect the
> various messages always to be sent as individual packets, or does it
> support stream-based connections where messages can be reframed into more
> or fewer packets?
SSH assumes only an 8 bit clean bidirectional byte stream and should
not be affected as long as the byte stream remains intact and in the
correct order.  Quoting RFC4253:

   SSH works over any 8-bit clean, binary-transparent transport.  The
   underlying transport SHOULD protect against transmission errors, as
   such errors cause the SSH connection to terminate.

I'd guess your packet forwarding thing is either delivering the
initial packet containing the SSH banner twice or mixing up two
different streams.  I don't think it's delivering the stream out of
order because that would not get past the initial exchange. I'd look
for duplicate packets first because that seems like the most plausible
one to me.

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.