John McDermott CIV
2012-Oct-04 18:16 UTC
Serial Console Not Connecting to Dom0 on Ubuntu Server 12 Running Xen 4.1.3
Xen-users, I have what I think is a bug in the serial console interaction of Xen and Ubuntu. Essentially, the serial console appears to work just fine until it is passed off to dom 0 from Xen. Then it stops communicating with dom0, but still communicates with Xen, if I use the control-A escape sequence. Google will not show me anything related to this. I have local RS-232 serial console working correctly, when the host is Fedora instead of Ubuntu, so I know that the client side is just fine. I have tried 2 different clients, Linux Mint and Mac OS X, with the identical problematic results. Serial connection to bare metal Ubuntu 12 works just fine. The serial console shows all of the GRUB and Xen boot messages but then seems to disconnect from dom 0. Serial I/O to the firmware works fine; I can stop the boot and change the BIOS. Serial I/O to GRUB works fine, I can work with GRUB. However when booting from Xen into dom 0, the serial I/O seems to be only going to the hypervisor. I have tried with and without an hvc0.cfg file in /etc/init but it makes no difference. The xm dmesg does not give me much help: ---- root@xenpro3:~# xm dmesg | grep serial (XEN) ******* that all output is synchronously delivered on the serial line. ---- Serial console output server side, failure at last line (mountall) ---- Loading the saved-state of the serial devices... /dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A * Stopping System V initialisation compatibility [ OK ] * Starting System V runlevel compatibility [ OK ] * Starting save kernel messages [ OK ] * Starting KVM [ OK ] * Starting automatic crash report generation [ OK ] * Starting ACPI daemon [ OK ] * Starting regular background program processing daemon [ OK ] * Starting deferred execution scheduler [ OK ] * Starting CPU interrupts balancing daemon [ OK ] * Starting crash report submission daemon [ OK ] * Starting Block the mounting event for NFS filesytems until statd is ru[ OK ] * Stopping Block the mounting event for NFS filesytems until statd is ru[ OK ] * Starting libvirt daemon [ OK ] * Exporting directories for NFS kernel daemon... [ OK ] * Stopping save kernel messages [ OK ] * Starting configure network device security [ OK ] * Starting configure network device [ OK ] * Starting NFS kernel daemon [ OK ] SSL tunnels disabled, see /etc/default/stunnel4 * Starting Block the mounting event for NFS filesytems until statd is ru[ OK ] * Stopping Block the mounting event for NFS filesytems until statd is ru[ OK ] [ 18.782960] XENBUS: Unable to read cpu state [ 18.787858] XENBUS: Unable to read cpu state [ 18.792665] XENBUS: Unable to read cpu state [ 18.797492] XENBUS: Unable to read cpu state * Starting Xen daemons [ OK ] * Stopping System V runlevel compatibility [ OK ] mountall: Disconnected from Plymouth ---- If I press Control-A 3 times, I get ---- [ 18.797492] XENBUS: Unable to read cpu state * Starting Xen daemons [ OK ] * Stopping System V runlevel compatibility [ OK ] mountall: Disconnected from Plymouth (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) ---- /etc/default/grub ---- # If you change this file, run ''update-grub'' afterwards to update # /boot/grub/grub.cfg. # For full documentation of the options in this file, see: # info -f grub -n ''Simple configuration'' GRUB_TERMINAL=serial GRUB_SERIAL_COMMAND="serial --port=0x3f8 --unit=0 --speed=115200 --word=8 --parity=no --stop=1" GRUB_DEFAULT="Xen" #GRUB_HIDDEN_TIMEOUT=0 GRUB_HIDDEN_TIMEOUT_QUIET=true GRUB_TIMEOUT=32 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_XEN="loglvl=all com1=115200,8n1 console=com1,vga console_to_ring, sync_console" GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen" # Uncomment to enable BadRAM filtering, modify to suit your needs # This works with Linux (no patch required) and with any kernel that obtains # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" # Uncomment to disable graphical terminal (grub-pc only) #GRUB_TERMINAL=console # The resolution used on graphical terminal # note that you can use only modes which your graphic card supports via VBE # you can see them in real GRUB with the command `vbeinfo'' #GRUB_GFXMODE=640x480 ---- ---- root@xenpro3:~# xm info host : xenpro3 release : 3.2.0-30-generic version : #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 machine : x86_64 nr_cpus : 4 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3392 hw_caps : bfebfbff:28100800:00000000:00003f40:13bae3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 16108 free_memory : 1427 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .3 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Thu Sep 13 09:24:16 2012 -0400 54:b522cd61dce2 xen_commandline : placeholder loglvl=all com1=115200,8n1 console=com1,vga console_to_ring, sync_console cc_compiler : gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) cc_compile_by : root cc_compile_domain : fw5540.net cc_compile_date : Fri Sep 14 09:05:11 EDT 2012 xend_config_format : 4 ---- ---- What is the formal meaning of the one-line program #include "/dev/tty" J.P. McDermott building 12 Code 5542 john.mcdermott@nrl.navy.mil Naval Research Laboratory voice: +1 202.404.8301 Washington, DC 20375, US fax: +1 202.404.7942
John McDermott CIV
2012-Oct-04 18:31 UTC
Correction: Serial Console Not Connecting to Dom0 on Ubuntu Server 12 Running Xen 4.1.3
> Xen-users, > > I have what I think is a bug in the serial console interaction of Xen and Ubuntu. Essentially, the serial console appears to work just fine until it is passed off to dom 0 from Xen. Then it stops communicating with dom0, but still communicates with Xen, if I use the control-A escape sequence. Google will not show me anything related to this. > > I have local RS-232 serial console working correctly, when the host is Fedora instead of Ubuntu, so I know that the client side is just fine. I have tried 2 different clients, Linux Mint and Mac OS X, with the identical problematic results. > > Serial connection to bare metal Ubuntu 12 works just fine. > > The serial console shows all of the GRUB and Xen boot messages but then seems to disconnect from dom 0. Serial I/O to the firmware works fine; I can stop the boot and change the BIOS. Serial I/O to GRUB works fine, I can work with GRUB. However when booting from Xen into dom 0, the serial I/O seems to be only going to the hypervisor. > > I have tried with and without an hvc0.cfg file in /etc/init but it makes no difference. > > The xm dmesg does not give me much help: > > ---- > root@xenpro3:~# xm dmesg | grep serial > (XEN) ******* that all output is synchronously delivered on the serial line. > ---- > > Serial console output server side, failure at last line (mountall) > ---- > Loading the saved-state of the serial devices... > /dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A > * Stopping System V initialisation compatibility [ OK ] > * Starting System V runlevel compatibility [ OK ] > * Starting save kernel messages [ OK ] > * Starting KVM [ OK ] > * Starting automatic crash report generation [ OK ] > * Starting ACPI daemon [ OK ] > * Starting regular background program processing daemon [ OK ] > * Starting deferred execution scheduler [ OK ] > * Starting CPU interrupts balancing daemon [ OK ] > * Starting crash report submission daemon [ OK ] > * Starting Block the mounting event for NFS filesytems until statd is ru[ OK ] > * Stopping Block the mounting event for NFS filesytems until statd is ru[ OK ] > * Starting libvirt daemon [ OK ] > * Exporting directories for NFS kernel daemon... [ OK ] > * Stopping save kernel messages [ OK ] > * Starting configure network device security [ OK ] > * Starting configure network device [ OK ] > * Starting NFS kernel daemon [ OK ] > SSL tunnels disabled, see /etc/default/stunnel4 > * Starting Block the mounting event for NFS filesytems until statd is ru[ OK ] > * Stopping Block the mounting event for NFS filesytems until statd is ru[ OK ] > [ 18.782960] XENBUS: Unable to read cpu state > [ 18.787858] XENBUS: Unable to read cpu state > [ 18.792665] XENBUS: Unable to read cpu state > [ 18.797492] XENBUS: Unable to read cpu state > * Starting Xen daemons [ OK ] > * Stopping System V runlevel compatibility [ OK ] > mountall: Disconnected from Plymouth > > ---- > > If I press Control-A 3 times, I get > ---- > [ 18.797492] XENBUS: Unable to read cpu state > * Starting Xen daemons [ OK ] > * Stopping System V runlevel compatibility [ OK ] > mountall: Disconnected from Plymouth > (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) > ---- > > /etc/default/grub > ---- > > # If you change this file, run ''update-grub'' afterwards to update > # /boot/grub/grub.cfg. > # For full documentation of the options in this file, see: > # info -f grub -n ''Simple configuration'' > > GRUB_TERMINAL=serial > GRUB_SERIAL_COMMAND="serial --port=0x3f8 --unit=0 --speed=115200 --word=8 --parity=no --stop=1" > > GRUB_DEFAULT="Xen" > #GRUB_HIDDEN_TIMEOUT=0 > GRUB_HIDDEN_TIMEOUT_QUIET=true > GRUB_TIMEOUT=32 > GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` > GRUB_CMDLINE_XEN="loglvl=all com1=115200,8n1 console=com1,vga console_to_ring, sync_console" > GRUB_CMDLINE_LINUX_DEFAULT="" > GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen" > > > # Uncomment to enable BadRAM filtering, modify to suit your needs > # This works with Linux (no patch required) and with any kernel that obtains > # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) > #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" > > # Uncomment to disable graphical terminal (grub-pc only) > #GRUB_TERMINAL=console > > # The resolution used on graphical terminal > # note that you can use only modes which your graphic card supports via VBE > # you can see them in real GRUB with the command `vbeinfo'' > #GRUB_GFXMODE=640x480 > ----Correction, wrong hypervisor. Same problem with the binary from Canonical, so its not likely caused by the hypervisor per se: root@xenpro3:~# xm info host : xenpro3 release : 3.2.0-30-generic version : #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 machine : x86_64 nr_cpus : 4 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3392 hw_caps : bfebfbff:28100800:00000000:00003f40:13bae3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 16108 free_memory : 1427 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : placeholder loglvl=all com1=115200,8n1 console=com1,vga console_to_ring, sync_console cc_compiler : gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) cc_compile_by : stefan.bader cc_compile_domain : canonical.com cc_compile_date : Sat Jul 21 09:01:19 UTC 2012 xend_config_format : 4 ---- What is the formal meaning of the one-line program #include "/dev/tty" J.P. McDermott building 12 Code 5542 john.mcdermott@nrl.navy.mil Naval Research Laboratory voice: +1 202.404.8301 Washington, DC 20375, US fax: +1 202.404.7942
Alexandre Kouznetsov
2012-Oct-04 20:58 UTC
Re: Serial Console Not Connecting to Dom0 on Ubuntu Server 12 Running Xen 4.1.3
Hello. El 04/10/12 13:16, John McDermott CIV escribió:> I have what I think is a bug in the serial console interaction of Xen and Ubuntu. Essentially, the serial console appears to work just fine until it is passed off to dom 0 from Xen. Then it stops communicating with dom0, but still communicates with Xen, if I use the control-A escape sequence. Google will not show me anything related to this.In order to get the serial console working and useful with Xen, several things are needed: Make Grub talk serial. The Grub config you pasted looks fine. I guess it''s working and allows you to interact with the boot loader. I use it like this: GRUB_TERMINAL="console serial" GRUB_SERIAL_COMMAND="serial --unit=1 --speed=115200 \ --word=8 --parity=no --stop=1" Make Xen talk serial. Th config you pastes looks fine on that part too. Personally, I do not enable the serial console for the hypervisor itself. It happened to me that it broke the boot process. In case of remote troubleshooting and since I have access to Grub, i can enable it manually any time. If I was enabling it, it would look like this: GRUB_CMDLINE_XEN="${GRUB_CMDLINE_XEN} com2=115200,8n1 console=com2,vga" Make kernel output to serial. That seems to be disbeled. I use it like this: GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,115200 console=tty0" Finally, getty shall be instructed to spawn a "login" on the serial interface: # grep 115200 /etc/inittab T1:2345:respawn:/sbin/getty -L ttyS1 115200 vt100 Note that I always use the second serial port in my examples, probably it''s not your case. -- Alexandre Kouznetsov
John McDermott CIV
2012-Nov-08 18:22 UTC
Solved: Serial Console Not Connecting to Dom0 on Ubuntu Server 12 Running Xen 4.1.3
Begin forwarded message:> From: John McDermott CIV <john.mcdermott@nrl.navy.mil> > Subject: Correction: Serial Console Not Connecting to Dom0 on Ubuntu Server 12 Running Xen 4.1.3 > Date: October 4, 2012 2:31:29 PM EDT > To: xen-users <xen-users@lists.xensource.com> > > > >> Xen-users, >> >> I have what I think is a bug in the serial console interaction of Xen and Ubuntu. Essentially, the serial console appears to work just fine until it is passed off to dom 0 from Xen. Then it stops communicating with dom0, but still communicates with Xen, if I use the control-A escape sequence. Google will not show me anything related to this. >> >> I have local RS-232 serial console working correctly, when the host is Fedora instead of Ubuntu, so I know that the client side is just fine. I have tried 2 different clients, Linux Mint and Mac OS X, with the identical problematic results. >> >> Serial connection to bare metal Ubuntu 12 works just fine. >> >> The serial console shows all of the GRUB and Xen boot messages but then seems to disconnect from dom 0. Serial I/O to the firmware works fine; I can stop the boot and change the BIOS. Serial I/O to GRUB works fine, I can work with GRUB. However when booting from Xen into dom 0, the serial I/O seems to be only going to the hypervisor. >> >> I have tried with and without an hvc0.cfg file in /etc/init but it makes no difference. >> >> The xm dmesg does not give me much help: >> >> ---- >> root@xenpro3:~# xm dmesg | grep serial >> (XEN) ******* that all output is synchronously delivered on the serial line. >> ---- >> >> Serial console output server side, failure at last line (mountall) >> ---- >> Loading the saved-state of the serial devices... >> /dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A >> * Stopping System V initialisation compatibility [ OK ] >> * Starting System V runlevel compatibility [ OK ] >> * Starting save kernel messages [ OK ] >> * Starting KVM [ OK ] >> * Starting automatic crash report generation [ OK ] >> * Starting ACPI daemon [ OK ] >> * Starting regular background program processing daemon [ OK ] >> * Starting deferred execution scheduler [ OK ] >> * Starting CPU interrupts balancing daemon [ OK ] >> * Starting crash report submission daemon [ OK ] >> * Starting Block the mounting event for NFS filesytems until statd is ru[ OK ] >> * Stopping Block the mounting event for NFS filesytems until statd is ru[ OK ] >> * Starting libvirt daemon [ OK ] >> * Exporting directories for NFS kernel daemon... [ OK ] >> * Stopping save kernel messages [ OK ] >> * Starting configure network device security [ OK ] >> * Starting configure network device [ OK ] >> * Starting NFS kernel daemon [ OK ] >> SSL tunnels disabled, see /etc/default/stunnel4 >> * Starting Block the mounting event for NFS filesytems until statd is ru[ OK ] >> * Stopping Block the mounting event for NFS filesytems until statd is ru[ OK ] >> [ 18.782960] XENBUS: Unable to read cpu state >> [ 18.787858] XENBUS: Unable to read cpu state >> [ 18.792665] XENBUS: Unable to read cpu state >> [ 18.797492] XENBUS: Unable to read cpu state >> * Starting Xen daemons [ OK ] >> * Stopping System V runlevel compatibility [ OK ] >> mountall: Disconnected from Plymouth >> >> ---- >> >> If I press Control-A 3 times, I get >> ---- >> [ 18.797492] XENBUS: Unable to read cpu state >> * Starting Xen daemons [ OK ] >> * Stopping System V runlevel compatibility [ OK ] >> mountall: Disconnected from Plymouth >> (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) >> ---- >> >> /etc/default/grub >> ---- >> >> # If you change this file, run ''update-grub'' afterwards to update >> # /boot/grub/grub.cfg. >> # For full documentation of the options in this file, see: >> # info -f grub -n ''Simple configuration'' >> >> GRUB_TERMINAL=serial >> GRUB_SERIAL_COMMAND="serial --port=0x3f8 --unit=0 --speed=115200 --word=8 --parity=no --stop=1" >> >> GRUB_DEFAULT="Xen" >> #GRUB_HIDDEN_TIMEOUT=0 >> GRUB_HIDDEN_TIMEOUT_QUIET=true >> GRUB_TIMEOUT=32 >> GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` >> GRUB_CMDLINE_XEN="loglvl=all com1=115200,8n1 console=com1,vga console_to_ring, sync_console" >> GRUB_CMDLINE_LINUX_DEFAULT="" >> GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen" >> >> >> # Uncomment to enable BadRAM filtering, modify to suit your needs >> # This works with Linux (no patch required) and with any kernel that obtains >> # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) >> #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" >> >> # Uncomment to disable graphical terminal (grub-pc only) >> #GRUB_TERMINAL=console >> >> # The resolution used on graphical terminal >> # note that you can use only modes which your graphic card supports via VBE >> # you can see them in real GRUB with the command `vbeinfo'' >> #GRUB_GFXMODE=640x480 >> ---- > > Correction, wrong hypervisor. Same problem with the binary from Canonical, so its not likely caused by the hypervisor per se: > > root@xenpro3:~# xm info > host : xenpro3 > release : 3.2.0-30-generic > version : #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 > machine : x86_64 > nr_cpus : 4 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 3392 > hw_caps : bfebfbff:28100800:00000000:00003f40:13bae3ff:00000000:00000001:00000000 > virt_caps : hvm hvm_directio > total_memory : 16108 > free_memory : 1427 > free_cpus : 0 > xen_major : 4 > xen_minor : 1 > xen_extra : .2 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > xen_commandline : placeholder loglvl=all com1=115200,8n1 console=com1,vga console_to_ring, sync_console > cc_compiler : gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) > cc_compile_by : stefan.bader > cc_compile_domain : canonical.com > cc_compile_date : Sat Jul 21 09:01:19 UTC 2012 > xend_config_format : 4 > > ---- > What is the formal meaning of the one-line program > #include "/dev/tty" > > J.P. McDermott building 12 > Code 5542 john.mcdermott@nrl.navy.mil > Naval Research Laboratory voice: +1 202.404.8301 > Washington, DC 20375, US fax: +1 202.404.7942 > > > > > > > > > >The problem is no hvc0 configuration file in /etc/init. (Ubuntu does not use inittab.) Copy /etc/init/tty1 to /etc/init/hvc0 and edit to replace "tty1" with "hvc0". Sincerely, John ---- What is the formal meaning of the one-line program #include "/dev/tty" J.P. McDermott building 12 Code 5542 john.mcdermott@nrl.navy.mil Naval Research Laboratory voice: +1 202.404.8301 Washington, DC 20375, US fax: +1 202.404.7942