openssh unix dev - Feb 2016 - [Bug] Regression problem in transfer.sh for OpenSSH 7.1 P2 on HPE NSE above dd-size 32k

On February 8, 2016 3:30 PM, Darren Tucker wrote:
> To: Randall S. Becker <rsbecker at nexbridge.com>
> Cc: OpenSSH Devel List <openssh-unix-dev at mindrot.org>
> Subject: Re: [Bug] Regression problem in transfer.sh for OpenSSH 7.1 P2 on
> HPE NSE above dd-size 32k
> 
> On Tue, Feb 9, 2016 at 6:36 AM, Randall S. Becker
> <rsbecker at nexbridge.com> wrote:
> [...]
> > Getting detailed debug logs of the regression test would also really
> > help here. Anyone have any pointers on where I can look to try to
resolve
> this?
> 
> The logs should be in the regress directory.  There should be ssh.log and
> ssh.log for the currently running test and failed-ssh.log and
failed-sshd.log
> containing the concatenation of the logs from any failed tests.  You can
run
> "make t-exec LTESTS=transfer" to run just that single test while
you're
playing
> around to save some time.
> 
> I have seen something similar on old FreeBSD systems when the test was run
> on an NFS mount.  I never figured out why this was, but running the test
on
> local disk worked in that case.
I've got those logs. Unfortunately, hoping for more details. The logs do
show the test failure, but no details in failed-sshd.log indicating a
specific problem. Some of the formats are a bit wonky but I can live with
those. Since you mentioned NFS, which could have been constrained by UDP
packet sizes, and this platform sometimes cannot read beyond 56Kb off disk
(in some situations), it may be the read of regress/data that is failing.
Any pointers where that is hiding so that I can verify that data is actually
getting into sshd from the disk?

Cheers,
Randall

On Tue, Feb 9, 2016 at 8:51 AM, Randall S. Becker
<rsbecker at nexbridge.com> wrote:
[...]
> I've got those logs. Unfortunately, hoping for more details.
Me too.  For example, a copy of those logs :-)
> The logs do
> show the test failure, but no details in failed-sshd.log indicating a
> specific problem. Some of the formats are a bit wonky but I can live with
> those. Since you mentioned NFS, which could have been constrained by UDP
> packet sizes, and this platform sometimes cannot read beyond 56Kb off disk
> (in some situations), it may be the read of regress/data that is failing.
> Any pointers where that is hiding so that I can verify that data is
actually
> getting into sshd from the disk?
Unfortunately debugging test failures is a pain that I don't have a
good answer too.

One thing I do is edit regress/test-exec.sh to add a "set -x" at the
top and "exit 1" into the fail() function.  Now when you run the test
at the point it fails you'll have the exact command that failed and
exits without cleaning up, leaving the keys and configs in place so
you can rerun the command in you shell, playing around with
truss/strace/whatever your platform has.

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On February 8, 2016 5:05 PM, Darren Tucker wrote:
> On Tue, Feb 9, 2016 at 8:51 AM, Randall S. Becker
> <rsbecker at nexbridge.com> wrote:
> [...]
> > I've got those logs. Unfortunately, hoping for more details.
> 
> Me too.  For example, a copy of those logs :-)
> 
> > The logs do
> > show the test failure, but no details in failed-sshd.log indicating a
> > specific problem. Some of the formats are a bit wonky but I can live
> > with those. Since you mentioned NFS, which could have been constrained
> > by UDP packet sizes, and this platform sometimes cannot read beyond
> > 56Kb off disk (in some situations), it may be the read of regress/data
that is
> failing.
> > Any pointers where that is hiding so that I can verify that data is
> > actually getting into sshd from the disk?
> 
> Unfortunately debugging test failures is a pain that I don't have a
good
> answer too.
> 
> One thing I do is edit regress/test-exec.sh to add a "set -x" at
the top and
> "exit 1" into the fail() function.  Now when you run the test at
the point it fails
> you'll have the exact command that failed and exits without cleaning
up,
> leaving the keys and configs in place so you can rerun the command in you
> shell, playing around with truss/strace/whatever your platform has.
Tracked down to a bad 'dd' implementation. I'm working on fixing
that. Likely of this problem being OpenSSH just dropped to under 5% ;)

Thanks Darren.

Cheers,
Randall