Stephan Seitz
2007-Oct-10 20:00 UTC
[Xen-devel] dom0 and domU /dev/urandom generating too less entropy
Hi there, I''ve recently seen problems after migrating physical servers into paravirtualized domU''s. The migrated systems vary from debian woody, sarge, ubuntu >=breezy, each system with it''s own but manageable problems. On thing in common is: /dev/urandom generates too less entropy for e.g. ssh-keygen. Last days, i found even sshd itself dying by too less entropy: sshd[26134]: fatal: Couldn''t obtain random bytes (error 604389476) We''re using an own build derived from the 3.1.0 tarball, but without any substantial changes to the code: The currently used kernel has been heavily patched, but this issue doesn''t seem to be kernel-specific. We tried the 2.6.18 (xensource 3.1.0) as well as different distro kernels. host : release : 2.6.20-100-server version : #2 SMP Sat Jun 2 12:18:40 UTC 2007 machine : i686 nr_cpus : 4 nr_nodes : 1 sockets_per_node : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2394 hw_caps : bfebfbff:20100000:00000000:00000140:0000e3bd:00000000:00000001 total_memory : 8190 free_memory : 1 xen_major : 3 xen_minor : 1 xen_extra : .0 xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4) cc_compile_by : root cc_compile_domain : halo.local cc_compile_date : Wed May 23 02:33:53 CEST 2007 xend_config_format : 4 Do you know about a workaround, or maybe the possibility for another (xen-specific) RNG besides of /dev/urandom ? Thanks in advance! -- Stephan Seitz Senior System Administrator *netz-haut* e.K. multimediale kommunikation zweierweg 22 97074 würzburg fon: +49 931 2876247 fax: +49 931 2876248 web: www.netz-haut.de <http://www.netz-haut.de/> registriergericht: amtsgericht würzburg, hra 5054 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Oct-11 07:27 UTC
Re: [Xen-devel] dom0 and domU /dev/urandom generating too less entropy
On 10/10/07 21:00, "Stephan Seitz" <s.seitz@netz-haut.de> wrote:> Do you know about a workaround, or maybe the possibility for another > (xen-specific) RNG > besides of /dev/urandom ?I''m surprised you see failures. By my understanding, /dev/urandom is always supposed to return the request number of bytes, but their randomness depends on the amount of entropy currently in the pool. Perhaps sshd explicitly interrogates urandom to find out how much entropy it has gathered? Anyway, the domU kernel gathers entropy from the interrupt delivery times of the netfront and blkfront drivers. This is similar to what a native kernel does. It''s not clear how we can easily improve on that without e.g., plumbing through a hardware RNG to domUs. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Robbie Dinn
2007-Oct-11 11:44 UTC
Re: [Xen-users] Re: [Xen-devel] dom0 and domU /dev/urandom generating too less entropy
Keir Fraser wrote:> On 10/10/07 21:00, "Stephan Seitz" <s.seitz@netz-haut.de> wrote: > >> Do you know about a workaround, or maybe the possibility for another >> (xen-specific) RNG >> besides of /dev/urandom ? > > I''m surprised you see failures. By my understanding, /dev/urandom is always > supposed to return the request number of bytes, but their randomness depends > on the amount of entropy currently in the pool. Perhaps sshd explicitly > interrogates urandom to find out how much entropy it has gathered?I haven''t checked (I am too laxy to strace it) but I believe that sshd is using /dev/random not /dev/urandom. You can see how much entropy is available by cat''ing /proc/sys/kernel/random/entropy_avail .> > Anyway, the domU kernel gathers entropy from the interrupt delivery times of > the netfront and blkfront drivers. This is similar to what a native kernel > does. It''s not clear how we can easily improve on that without e.g., > plumbing through a hardware RNG to domUs.I had a similar problem on a mail server providing a pop3 service. Every time a client machine connected to the pop3 daemon (cyrus imap actually), it consumed entropy. More entropy was consumed for each connection than was provided by the packets arriving. The machine ran of entropy and stopped providing bytes via /dev/random. The pop3 daemon ground to a halt because it was waiting to read bytes from /dev/random. The work around was to feed entropy into the random number generator. There is a user space tool to do this called ''rngd''. The correct way to do this would be, as you say, to get the the entropy from outside the domU. I used a dirty hack instead, I ran /sbin/rngd --rng-device=/dev/urandom Yes is wrong and evil but it got me up and running again. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Robbie Dinn
2007-Oct-11 15:28 UTC
Re: [Xen-users] Re: [Xen-devel] dom0 and domU /dev/urandom generating too less entropy
Robbie Dinn wrote:> Keir Fraser wrote: >> On 10/10/07 21:00, "Stephan Seitz" <s.seitz@netz-haut.de> wrote: >> >>> Do you know about a workaround, or maybe the possibility for another >>> (xen-specific) RNG >>> besides of /dev/urandom ? >> I''m surprised you see failures. By my understanding, /dev/urandom is always >> supposed to return the request number of bytes, but their randomness depends >> on the amount of entropy currently in the pool. Perhaps sshd explicitly >> interrogates urandom to find out how much entropy it has gathered? > I haven''t checked (I am too laxy to strace it) but I believe that sshd > is using /dev/random not /dev/urandom. You can see how much entropy is > available by cat''ing /proc/sys/kernel/random/entropy_avail .No I am wrong. I did an strace of sshd and it does read /dev/urandom not /dev/random as I claimed. read(5, "\0\0\0\1\212\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"..., 752) = 752 open("/dev/urandom", O_RDONLY|O_NONBLOCK|O_NOCTTY) = 3 fstat64(3, {st_mode=S_IFCHR|0644, st_rdev=makedev(1, 9), ...}) = 0 poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 10) = 1 read(3, "p:\326\232y\326f\364<#L\204(<\244\"\275,r\263\r%Z\f\304"..., 32) = 32 close(3) = 0 Sorry for the noise.>> Anyway, the domU kernel gathers entropy from the interrupt delivery times of >> the netfront and blkfront drivers. This is similar to what a native kernel >> does. It''s not clear how we can easily improve on that without e.g., >> plumbing through a hardware RNG to domUs. > > I had a similar problem on a mail server providing a pop3 service. Every > time a client machine connected to the pop3 daemon (cyrus imap actually), > it consumed entropy. More entropy was consumed for each connection > than was provided by the packets arriving. The machine ran of entropys/machine ran of entropy/machine ran out of entropy/> and stopped providing bytes via /dev/random. The pop3 daemon ground > to a halt because it was waiting to read bytes from /dev/random. > > The work around was to feed entropy into the random number generator. > There is a user space tool to do this called ''rngd''. > > The correct way to do this would be, as you say, to get the the entropy > from outside the domU. I used a dirty hack instead, I ran > > /sbin/rngd --rng-device=/dev/urandom > > Yes is wrong and evil but it got me up and running again.s/Yes is wrong/Yes, it is wrong/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stephan Seitz
2007-Oct-11 16:10 UTC
Re: [Xen-users] Re: [Xen-devel] dom0 and domU /dev/urandom generating too less entropy
> Robbie Dinn wrote: >> Keir Fraser wrote:Thanks for your suggestions Keir and Robbie, but it looks like /dev/urandom is used by sshd, but somewhere inside of sshd the stream is checked against a minimum entropy. I don''t think this is a an issue by getting blocked via /dev/random. I''ll try the userspace rngd, but - as you said - this shouldn''t be the preferred method of getting more entropy into the system. Another thing i''ll try (later at night, as the domU''s are productive) is, to configure the domU''s using the same pCPU''s targetting that the pCPU''s cache is filled only sub-optimal and the latency is higher ... (maybe ;) ) I''ll report here later. Greetings Stephan -- Stephan Seitz Senior System Administrator *netz-haut* e.K. multimediale kommunikation zweierweg 22 97074 würzburg fon: +49 931 2876247 fax: +49 931 2876248 web: www.netz-haut.de <http://www.netz-haut.de/> registriergericht: amtsgericht würzburg, hra 5054 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel