Hi, We create virtual machine image templates by doing automated minimal installations of different Linux distributions (via preseed/kickstarter/autoyast). At the end of the installation, we remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host keys will be generated on the first boot of the image instances. This is done by adding a "dpkg-reconfigure openssh-server" call in /etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the init script of sshd on the other distributions. This leads to working SSH server running on the virtual machines most of the times, but sometimes the SSH connection fails with "connection reset by peer". The investigation of Debian 7 "wheezy" images showed that these faulty machines have empty (zero byte) SSH host key files. These files do not exist before the machines are started, but they do exist before "dpkg-reconfigure openssh-server" is called. So it seems that some process creates these empty SSH host key files. Can you help to further debugging this strange behavior? Does sshd create SSH host keys? -- Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: benjamin.drung at profitbricks.com URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Gesch?ftsf?hrer: Andreas Gauger, Achim Weiss.
On Wed, 6 Jan 2016, Benjamin Drung wrote:> Hi, > > We create virtual machine image templates by doing automated minimal > installations of different Linux distributions (via > preseed/kickstarter/autoyast). At the end of the installation, we > remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host > keys will be generated on the first boot of the image instances. This > is done by adding a "dpkg-reconfigure openssh-server" call in > /etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the init > script of sshd on the other distributions. > > This leads to working SSH server running on the virtual machines most > of the times, but sometimes the SSH connection fails with "connection > reset by peer". The investigation of Debian 7 "wheezy" images showed > that these faulty machines have empty (zero byte) SSH host key files. > These files do not exist before the machines are started, but they do > exist before "dpkg-reconfigure openssh-server" is called. > > So it seems that some process creates these empty SSH host key files. > Can you help to further debugging this strange behavior? Does sshd > create SSH host keys?No, sshd only reads and never writes host keys. It's possible that either ssh-keygen is failing during writing the keys out or there is some bug in the init script that is calling it. -d
On Wed, Jan 6, 2016 at 1:08 PM, Benjamin Drung <benjamin.drung at profitbricks.com> wrote:> Hi, > > We create virtual machine image templates by doing automated minimal > installations of different Linux distributions (via > preseed/kickstarter/autoyast). At the end of the installation, we > remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host > keys will be generated on the first boot of the image instances. This > is done by adding a "dpkg-reconfigure openssh-server" call in > /etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the init > script of sshd on the other distributions. > > This leads to working SSH server running on the virtual machines most > of the times, but sometimes the SSH connection fails with "connection > reset by peer". The investigation of Debian 7 "wheezy" images showed > that these faulty machines have empty (zero byte) SSH host key files. > These files do not exist before the machines are started, but they do > exist before "dpkg-reconfigure openssh-server" is called. > > So it seems that some process creates these empty SSH host key files. > Can you help to further debugging this strange behavior? Does sshd > create SSH host keys?It could be a filesystem issue in your image generation process. Are you doing a "sync" before recording your image? And are you using one of the more fragile, "optimized" filesystems like ReiserFS, which is infamous for zeroing files? Until you discover the issue, you *could* edit the OpenSSH init script, or /etc/sysconfig/sshd, to clear the hostkeys if they're empty and before new keys would be auto-generated.
Am Donnerstag, den 07.01.2016, 10:09 +1100 schrieb Damien Miller:> On Wed, 6 Jan 2016, Benjamin Drung wrote: > > > Hi, > > > > We create virtual machine image templates by doing automated > > minimal > > installations of different Linux distributions (via > > preseed/kickstarter/autoyast). At the end of the installation, we > > remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host > > keys will be generated on the first boot of the image instances. > > This > > is done by adding a "dpkg-reconfigure openssh-server" call in > > /etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the > > init > > script of sshd on the other distributions. > > > > This leads to working SSH server running on the virtual machines > > most > > of the times, but sometimes the SSH connection fails with > > "connection > > reset by peer". The investigation of Debian 7 "wheezy" images > > showed > > that these faulty machines have empty (zero byte) SSH host key > > files. > > These files do not exist before the machines are started, but they > > do > > exist before "dpkg-reconfigure openssh-server" is called. > > > > So it seems that some process creates these empty SSH host key > > files. > > Can you help to further debugging this strange behavior? Does sshd > > create SSH host keys? > > No, sshd only reads and never writes host keys.Thanks for confirming it.> It's possible that > either ssh-keygen is failing during writing the keys out or there > is some bug in the init script that is calling it.The strange thing is that the empty files appear before ssh-keygen is called (via our "dpkg-reconfigure openssh-server" call in /etc/rc.local). I am not aware of any other ssh-keygen calls besides our one. Who creates these empty SSH host key files? I also checked the sysvinit script of ssh on Debian 7 "wheezy". It only checks/creates /var/run/sshd before calling sshd. There is not host key handling in the init script of ssh. -- Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: benjamin.drung at profitbricks.com URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Gesch?ftsf?hrer: Andreas Gauger, Achim Weiss.
Am Donnerstag, den 07.01.2016, 03:18 -0500 schrieb Nico Kadel-Garcia:> On Wed, Jan 6, 2016 at 1:08 PM, Benjamin Drung > <benjamin.drung at profitbricks.com> wrote: > > Hi, > > > > We create virtual machine image templates by doing automated > > minimal > > installations of different Linux distributions (via > > preseed/kickstarter/autoyast). At the end of the installation, we > > remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host > > keys will be generated on the first boot of the image instances. > > This > > is done by adding a "dpkg-reconfigure openssh-server" call in > > /etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the > > init > > script of sshd on the other distributions. > > > > This leads to working SSH server running on the virtual machines > > most > > of the times, but sometimes the SSH connection fails with > > "connection > > reset by peer". The investigation of Debian 7 "wheezy" images > > showed > > that these faulty machines have empty (zero byte) SSH host key > > files. > > These files do not exist before the machines are started, but they > > do > > exist before "dpkg-reconfigure openssh-server" is called. > > > > So it seems that some process creates these empty SSH host key > > files. > > Can you help to further debugging this strange behavior? Does sshd > > create SSH host keys? > > It could be a filesystem issue in your image generation process. Are > you doing a "sync" before recording your image? And are you using one > of the more fragile, "optimized" filesystems like ReiserFS, which is > infamous for zeroing files?All images use ext4 as filesystem. The image creation works as following: 1) Create a raw HDD image (2 GiB in size) 2) Launch qemu to do a preseed/kickstarter/autoyast installation 2b) as part of the installation remove the SSH host keys (via a late_command or post command) 2c) wait for the installation to finish / exit of qemu 3) mount the raw image 4) copy the installer logs and remove them 5) unmout the raw image 6) run zerofree on the partition 7) convert the raw image to qcow2 and upload/distribute it When an instance is created from this template, following steps are done: 1) create a block device (could be larger than the image) 2) copy the template to the block device 3) resize the root partition and add a tmpfs partition 4) mount the resized partition 5) (optionally) inject a authorized_key file 6) set the root password 7) verify that no SSH host keys are on this partition 8) unmount the partition 9) launch qemu> Until you discover the issue, you *could* edit the OpenSSH init > script, or /etc/sysconfig/sshd, to clear the hostkeys if they're > empty > and before new keys would be auto-generated.That's exactly the workaround that we are currently testing. -- Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: benjamin.drung at profitbricks.com URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Gesch?ftsf?hrer: Andreas Gauger, Achim Weiss.