openssh unix dev - Jan 2019 - [Regression] OpenSSH 7.7p1 no longer tests on NonStop Port

Hi All,

I finally got around to trying to test the 7.7p1 release on the HPE NonStop
Platform. 7.6p1 worked just fine - no issues at all.

In 7.7p1, when the agent test ran, the following happened (prior tests
worked perfectly):

	run test agent.sh ...
	Couldn't open logfile /home/git/openssh-portable/regress/ssh.log:
Permission denied
	agent fwd failed (exit code 1)
	failed simple agent test

This seems to be entirely correct, in that the log file is secured 600 by
the root user (we run sudo make tests), but the agent ssh is running as my
user. It is unable to open the ssh.log file and happily dies, killing the
rest of the test suite. The weird part is, this seems entirely correct. The
log file is open as root as mode 600, which my own user cannot possible
read, so the error message appears correct. The problem is that this tosses
the remaining tests.

I was wondering whether this is expected behaviour and if so, how to get
around it. The release otherwise seems to work, but I can't release it to my
community without it passing regression tests. The error report is really
early in the agent's stack trace:

#0  0x700e84a0:0 in log_redirect_stderr_to (
    logfile=0x8017e32 "/home/git/openssh-portable/regress/ssh.log") at
    /home/git/openssh-portable/log.c:378
#1  0x700504e0:0 in main (ac=2, av=<optimized out>) at
    /home/git/openssh-portable/ssh.c:1075
#2  0x7004a9c0:0 in _MAIN () at \ATOM.$RLSE.T8432ACH.CPLMAINC:52

This is a variant of commit a0349a1c with our platform mods applied, but
none of the mods were hit at this point in the agent.

As a side note: we are that weird platform where root is 65535 and the
logged off user is 0, but that's not the case here. We have a rather large
set of changes that could be useful to make the user checks handle this,
which I can put into a Pull Request, and yet, we have this issue.

Any help would be appreciated. 

Thanks,
Randall

-- Brief whoami:
 NonStop developer since approximately 211288444200000000
 UNIX developer since approximately 421664400
-- In my real life, I talk too much.

On Tue, 22 Jan 2019 at 12:01, Randall S. Becker <rsbecker at
nexbridge.com>
wrote:
> Hi All,
>
> I finally got around to trying to test the 7.7p1 release on the HPE NonStop
> Platform. 7.6p1 worked just fine - no issues at all.
>
> In 7.7p1, when the agent test ran, the following happened (prior tests
> worked perfectly):
>
>         run test agent.sh ...
>         Couldn't open logfile
/home/git/openssh-portable/regress/ssh.log:
> Permission denied
>         agent fwd failed (exit code 1)
>         failed simple agent test
>
My guess is that this is due to a previous test creating ssh.log as root,
however offhand I can't think of any changes in that time frame that would
cause it.

Do you see the problem if you remove the log file and run the test in
isolation?

$ SUDO=sudo make tests LTESTS=agent SKIP_UNIT=1

[...]
> We have a rather large set of changes
are these changes publicly available?

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On Mon, Jan 21, 2019 at 05:52:39PM -0500, Randall S. Becker
wrote:
> In 7.7p1, when the agent test ran, the following happened (prior tests
> worked perfectly):
> 
> 	run test agent.sh ...
> 	Couldn't open logfile /home/git/openssh-portable/regress/ssh.log:
> Permission denied
> 	agent fwd failed (exit code 1)
> 	failed simple agent test
> 
> This seems to be entirely correct, in that the log file is secured 600 by
> the root user (we run sudo make tests),
Can I ask before anything else why you're doing "sudo make tests"?
The
regress documentation instead suggests running just "make tests" but
with SUDO=sudo set in the environment; you might find that approach
works better.

-- 
Colin Watson                                       [cjwatson at debian.org]

On January 21, 2019 19:00, Darren Tucker wrote:
> On Tue, 22 Jan 2019 at 12:01, Randall S. Becker <rsbecker at
nexbridge.com>
> wrote:
> 
> > Hi All,
> >
> > I finally got around to trying to test the 7.7p1 release on the HPE
> > NonStop Platform. 7.6p1 worked just fine - no issues at all.
> >
> > In 7.7p1, when the agent test ran, the following happened (prior tests
> > worked perfectly):
> >
> >         run test agent.sh ...
> >         Couldn't open logfile
/home/git/openssh-portable/regress/ssh.log:
> > Permission denied
> >         agent fwd failed (exit code 1)
> >         failed simple agent test
> >
> 
> My guess is that this is due to a previous test creating ssh.log as root,
> however offhand I can't think of any changes in that time frame that
would
> cause it.
> 
> Do you see the problem if you remove the log file and run the test in
> isolation?
> 
> $ SUDO=sudo make tests LTESTS=agent SKIP_UNIT=1
<snip>
run test agent.sh ...
ok simple agent test
make[1]: Leaving directory '/home/git/openssh-portable/regress'
all tests passed

So that's a good thing. Operator error. When running the full test:

key option expiry-time valid
failed key options
Makefile:206: recipe for target 't-exec' failed

/home/git/openssh-portable/regress: cat failed-regress.log, failed-ssh.log #
are the same
trace: key option pty default
FAIL: key option failed default

trace: key option pty restrict,pty
FAIL: key option failed restrict,pty

/home/git/openssh-portable/regress: tail -7 failed-sshd.log
debug3: do_exec_no_pty: fork() in child
debug3: send packet: type 99
debug3: do_child: entering do_child()
debug1: permanently_set_uid: 25603/100
FAIL: key option failed restrict,pty
debug3: do_child: Shell path=/bin/ksh

So, better, but this is a new break for us.
> 
> [...]
> > We have a rather large set of changes
> 
> are these changes publicly available?
My GitHub fork for this is https://github.com/rsbeckerca/openssh-portable on
the Randall branch, which is work in progress. The 7.6p1 port changes are at
2abb363, representing the bulk of the work. 7.7p1 is from there upward to
814c8f2.

Regards,
Randall

On January 22, 2019 10:45, I wrote:
> On January 21, 2019 19:00, Darren Tucker wrote:
> > On Tue, 22 Jan 2019 at 12:01, Randall S. Becker
> > <rsbecker at nexbridge.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I finally got around to trying to test the 7.7p1 release on the
HPE
> > > NonStop Platform. 7.6p1 worked just fine - no issues at all.
> > >
> > > In 7.7p1, when the agent test ran, the following happened (prior
> > > tests worked perfectly):
> > >
> > >         run test agent.sh ...
> > >         Couldn't open logfile
/home/git/openssh-portable/regress/ssh.log:
> > > Permission denied
> > >         agent fwd failed (exit code 1)
> > >         failed simple agent test
> > >
> >
> > My guess is that this is due to a previous test creating ssh.log as
> > root, however offhand I can't think of any changes in that time
frame
> > that would cause it.
> >
> > Do you see the problem if you remove the log file and run the test in
> > isolation?
> >
> > $ SUDO=sudo make tests LTESTS=agent SKIP_UNIT=1
> 
> <snip>
> run test agent.sh ...
> ok simple agent test
> make[1]: Leaving directory '/home/git/openssh-portable/regress'
> all tests passed
> 
> So that's a good thing. Operator error. When running the full test:
> 
> key option expiry-time valid
> failed key options
> Makefile:206: recipe for target 't-exec' failed
> 
> /home/git/openssh-portable/regress: cat failed-regress.log, failed-ssh.log
#
> are the same
> trace: key option pty default
> FAIL: key option failed default
> 
> trace: key option pty restrict,pty
> FAIL: key option failed restrict,pty
> 
> /home/git/openssh-portable/regress: tail -7 failed-sshd.log
> debug3: do_exec_no_pty: fork() in child
> debug3: send packet: type 99
> debug3: do_child: entering do_child()
> debug1: permanently_set_uid: 25603/100
> FAIL: key option failed restrict,pty
> debug3: do_child: Shell path=/bin/ksh
> 
> So, better, but this is a new break for us.
> 
> >
> > [...]
> > > We have a rather large set of changes
> >
> > are these changes publicly available?
> 
> My GitHub fork for this is https://github.com/rsbeckerca/openssh-portable
> on the Randall branch, which is work in progress. The 7.6p1 port changes
are
> at 2abb363, representing the bulk of the work. 7.7p1 is from there upward
to
> 814c8f2.
Here's where we are in debugging. I've set +x and fail in test-exec.sh,
here's the log. I am having difficulty interpreting the output or where to
look next. To the uninitiated, this does not look bad. 

+ SSH_KEYTYPES=rsa ed25519
+ trace generate keys
+ [ ! -f /home/git/openssh-portable/regress/rsa ]
+ [ /home/git/openssh-portable/ssh-keygen -nt
/home/git/openssh-portable/regress/rsa ]
+ 1>> /home/git/openssh-portable/regress/known_hosts
+ printf localhost-with-alias,127.0.0.1,::1
+ cat /home/git/openssh-portable/regress/rsa.pub
+ cat /home/git/openssh-portable/regress/rsa.pub
+ 1>> /home/git/openssh-portable/regress/authorized_keys_randall
+ echo IdentityFile /home/git/openssh-portable/regress/rsa
+ 1>> /home/git/openssh-portable/regress/ssh_config
+ sudo cp /home/git/openssh-portable/regress/rsa
/home/git/openssh-portable/regress/host.rsa
+ echo HostKey /home/git/openssh-portable/regress/host.rsa
+ 1>> /home/git/openssh-portable/regress/sshd_config
+ echo HostKey /home/git/openssh-portable/regress/rsa
+ 1>> /home/git/openssh-portable/regress/sshd_proxy
+ [ ! -f /home/git/openssh-portable/regress/ed25519 ]
+ [ /home/git/openssh-portable/ssh-keygen -nt
/home/git/openssh-portable/regress/ed25519 ]
+ 1>> /home/git/openssh-portable/regress/known_hosts
+ printf localhost-with-alias,127.0.0.1,::1
+ cat /home/git/openssh-portable/regress/ed25519.pub
+ cat /home/git/openssh-portable/regress/ed25519.pub
+ 1>> /home/git/openssh-portable/regress/authorized_keys_randall
+ echo IdentityFile /home/git/openssh-portable/regress/ed25519
+ 1>> /home/git/openssh-portable/regress/ssh_config
+ sudo cp /home/git/openssh-portable/regress/ed25519
/home/git/openssh-portable/regress/host.ed25519
+ echo HostKey /home/git/openssh-portable/regress/host.ed25519
+ 1>> /home/git/openssh-portable/regress/sshd_config
+ echo HostKey /home/git/openssh-portable/regress/ed25519
+ 1>> /home/git/openssh-portable/regress/sshd_proxy
+ chmod 644 /home/git/openssh-portable/regress/authorized_keys_randall
+ REGRESS_INTEROP_CONCH=no
+ test -x
+ REGRESS_INTEROP_PUTTY=no
+ test -x  -a -x
+ REGRESS_INTEROP_PUTTY=no
+ test no = yes
+ 1> /home/git/openssh-portable/regress/ssh_proxy
+ cat /home/git/openssh-portable/regress/ssh_config
+ echo proxycommand sudo sh
/home/git/openssh-portable/regress/sshd-log-wrapper.sh
/home/git/openssh-portable/regress/sshd.log /home/git/openssh-portable/sshd
-i -f /home/git/openssh-portable/regress/sshd_proxy
+ /home/git/openssh-portable/sshd -t -f
/home/git/openssh-portable/regress/sshd_proxy
+ . /home/git/openssh-portable/regress/key-options.sh
+ tid=key options
+ origkeys=/home/git/openssh-portable/regress/authkeys_orig
+ authkeys=/home/git/openssh-portable/regress/authorized_keys_randall
+ cp /home/git/openssh-portable/regress/authorized_keys_randall
/home/git/openssh-portable/regress/authkeys_orig
+ sed s/.*/command="echo bar" &/
/home/git/openssh-portable/regress/authkeys_orig
+ 1> /home/git/openssh-portable/regress/authorized_keys_randall
+ verbose key option command="echo bar"
key option command="echo bar"
+ + /home/git/openssh-portable/regress/ssh-log-wrapper.sh -q -F
/home/git/openssh-portable/regress/ssh_proxy somehost echo foo
r=bar
+ [ bar = foo ]
+ [ bar != bar ]
+ sed s/.*/no-pty,command="echo bar" &/
/home/git/openssh-portable/regress/authkeys_orig
+ 1> /home/git/openssh-portable/regress/authorized_keys_randall
+ verbose key option no-pty,command="echo bar"
key option no-pty,command="echo bar"
+ + /home/git/openssh-portable/regress/ssh-log-wrapper.sh -q -F
/home/git/openssh-portable/regress/ssh_proxy somehost echo foo
r=bar
+ [ bar = foo ]
+ [ bar != bar ]
+ expect_pty_succeed default
key option pty default
key option failed default
Makefile:206: recipe for target 't-exec' failed
make[1]: *** [t-exec] Error 1