openssh unix dev - Jul 2001 - Debugging ssh-keygen dsa on Solaris8

Hello,
>From the response to my original post regarding openssh, obviously the
only way to resolve the problem of getting openssh to work properly
under Solaris 8 with openssl 0.9.6b was to compile with debugging an
analyze to core file. Let me state that I am not a developer, but maybe
the following will help. I have compiled both openssh2.96p2 and openssl
0.9.6b with debugging using the Sun Forte Compiler update 2. I compiled
each piece of software without any additional options, the only thing I
did was change the --prefix. Below is some output from dbx: (The core I
am examining here comes from the make install...when it trys to create
the private/public dsa keys.)

(If there is anything else I can do to help let me know. The system is
5.8 Generic_108528-08 with the recommended patch cluster from July 11th.
This is an Ultra10 workstation) I also have the same problem using gcc
2.95.3 on Solaris 8 and Solaris 2.6.


Reading ssh-keygen
core file header read successfully
Reading ld.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libgen.so.1
Reading libc.so.1
Reading libdl.so.1
Reading libmp.so.2
Reading libc_psr.so.1
program terminated by signal BUS (invalid address alignment)
Current function is DSA_new_method (optimized)
  127   	ret->flags=ret->meth->flags;
(/opt/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx)
where                           
=>[1] DSA_new_method(meth = ???) (optimized), at 0x4b6b8 (line ~127) in
"dsa_lib.c"
  [2] DSA_new() (optimized), at 0x4b580 (line ~86) in "dsa_lib.c"
  [3] DSA_generate_parameters(bits = ???, seed_in = ???, seed_len = ???,
counter_ret = ???, h_ret = ???, callback = ???, cb_arg = ???)
(optimized), at 0x4ace4 (line ~115) in "dsa_gen.c"
  [4] dsa_generate_private_key(bits = 1024U), line 560 in "key.c"
  [5] key_generate(type = 2, bits = 1024U), line 576 in "key.c"
  [6] main(ac = 7, av = 0xffbefc74), line 781 in "ssh-keygen.c"

--On Tuesday, July 24, 2001 2:58 PM -0500 Scott Burch <somar at tiny.net> 
wrote:
> analyze to core file. Let me state that I am not a developer, but maybe
> the following will help. I have compiled both openssh2.96p2 and openssl
> 0.9.6b with debugging using the Sun Forte Compiler update 2. I compiled
Update 2? Isn't that beta? Or did it just ship?
> where
> =>[1] DSA_new_method(meth = ???) (optimized), at 0x4b6b8 (line ~127) in
> "dsa_lib.c"
Please disable optimization, at least for dsa_lib.c


-- 
Carson Gaspar - carson at taltos.org
Queen trapped in a butch body

On Tue, Jul 24, 2001 at 02:58:31PM -0500, Scott Burch
wrote:
> (If there is anything else I can do to help let me know. The system is
> 5.8 Generic_108528-08 with the recommended patch cluster from July 11th.
> This is an Ultra10 workstation) I also have the same problem using gcc
> 2.95.3 on Solaris 8 and Solaris 2.6.
You are receiving a BUS error, which means that something is not properly
aligned (e.g. omething is on a "odd" 4byte boundary while it should be
on a
8 byte boundary).
Please understand that I don't have Solaris around, so I can only give
you a wild guess. I would think, that the OpenSSL library was compiled
with some "64bit-alignment flag" (or maybe for some 64bit processor),
while
OpenSSH was compiled without this flag. Hence the members of the structure
are not properly aligned and it will later fail in the OpenSSL library.
> Reading ssh-keygen
...
> program terminated by signal BUS (invalid address alignment)
> Current function is DSA_new_method (optimized)
>   127   	ret->flags=ret->meth->flags;
> (/opt/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx)
> where                           
> =>[1] DSA_new_method(meth = ???) (optimized), at 0x4b6b8 (line ~127) in
> "dsa_lib.c"
At this point a member of a structure is accessed. malloc() always tends
to return data aligned for the worst case, so it only fails within
a structure, which is not properly aligned.

In any case: if the alignement (and hence the position) of members in 
a structure is wrong, the program must fail anyway, as the routines
accessing the members will pick up wrong data.

Best regards,
	Lutz
-- 
Lutz Jaenicke                             Lutz.Jaenicke at aet.TU-Cottbus.DE
BTU Cottbus               http://www.aet.TU-Cottbus.DE/personen/jaenicke/
Lehrstuhl Allgemeine Elektrotechnik                  Tel. +49 355 69-4129
Universitaetsplatz 3-4, D-03044 Cottbus              Fax. +49 355 69-4153

Lutz,

Responding to some of your statements:

1) Both openssl 0.9.6b and openssh 2.9p2 were compiled by myself on
Solaris 8 as 32 bit binaries, I did not compile them for 64 bit.
(The exact same compiler and compiler flags were used...the only extra
flag I used was -g so I could use dbx).
2) If I use 0.9.5a of openssl I do not have any of these problems with
the exact same compiler and options. (This is why I think some code
change is causing this problem.
3) I can't get this work on any of our machines, however we do use
custom jumpstart for all of our systems....and maybe something in our
jumpstart image is causing this problem. I may have to build a new
jumpstart image from scratch from the April 2001 Solaris 8 release and
see if that makes a difference. This also happens on our 2.6 machines
(which are also built from jumpstart).

Lukas Ruf reported similar problems. He has a better situation than me,
however....he says compiles work fine on every system except one...in my
case I can't get the DSA stuff to work on any of the machines I am
working with unless I use openssl 0.9.5a. Unless anyone else has any
other ideas..I will try a new jumpstart image and see if that makes a
difference. I would ultimately like to figure out what in Solaris is
causing this not too work....it's starting to seem like some patch on
our systems is causing the problem, but I'm at a loss as to what it
might be...the system I am building on has all the recommended patches
from July 11th and is based on the January 2001 Solaris 8 release.

-Scott

On Tue, Jul 24, 2001 at 02:58:31PM -0500, Scott Burch
wrote:
> (If there is anything else I can do to help let me know. The system is
> 5.8 Generic_108528-08 with the recommended patch cluster from July 11th.
> This is an Ultra10 workstation) I also have the same problem using gcc
> 2.95.3 on Solaris 8 and Solaris 2.6.
You are receiving a BUS error, which means that something is not
properly
aligned (e.g. omething is on a "odd" 4byte boundary while it should be
on a
8 byte boundary).
Please understand that I don't have Solaris around, so I can only give
you a wild guess. I would think, that the OpenSSL library was compiled
with some "64bit-alignment flag" (or maybe for some 64bit processor),
while
OpenSSH was compiled without this flag. Hence the members of the
structure
are not properly aligned and it will later fail in the OpenSSL library.
> Reading ssh-keygen
...
> program terminated by signal BUS (invalid address alignment)
> Current function is DSA_new_method (optimized)
>   127         ret->flags=ret->meth->flags;
> (/opt/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx)
> where                           
> =>[1] DSA_new_method(meth = ???) (optimized), at 0x4b6b8 (line ~127) in
> "dsa_lib.c"
At this point a member of a structure is accessed. malloc() always tends
to return data aligned for the worst case, so it only fails within
a structure, which is not properly aligned.

In any case: if the alignement (and hence the position) of members in 
a structure is wrong, the program must fail anyway, as the routines
accessing the members will pick up wrong data.

Best regards,
        Lutz

On Wed, Jul 25, 2001 at 10:01:05AM -0500, Scott Burch
wrote:
> 2) If I use 0.9.5a of openssl I do not have any of these problems with
> the exact same compiler and options. (This is why I think some code
> change is causing this problem.
I do not see any significant code changes in the dsa-code between
0.9.5a and 0.9.6b...
Please change line 69 in openssl-0.9.6b/crypto/dsa/dsa_lib.c from
  static DSA_METHOD *default_DSA_method;
to 
  static DSA_METHOD *default_DSA_method = NULL;

(Actually, if this helps it means that the bug was already in 0.9.5a
but it was not triggered for some reason. The initialization of things
may be compiler and optimization dependant anyway.)

Hmm. Strange. But there is also another report I had about strange problems
generating DSA keys with ssh-keygen under Solaris...
	Lutz
-- 
Lutz Jaenicke                             Lutz.Jaenicke at aet.TU-Cottbus.DE
BTU Cottbus               http://www.aet.TU-Cottbus.DE/personen/jaenicke/
Lehrstuhl Allgemeine Elektrotechnik                  Tel. +49 355 69-4129
Universitaetsplatz 3-4, D-03044 Cottbus              Fax. +49 355 69-4153