Xen devel - Apr 2011 - on starting HVM-domU whole system freezes with "soft lockup - CPU X stuck for XXs! [qemu-dm:...]" [xen 4.0, xen 4.1]

Hi xen-devel,

---
have tried now with xen 4.1 from debian sid repository (still 2.6.32 kernel from
debian squeeze, I have only installed xen-hypervisor, xen-utils and
xen-utils-common). same problem, machine stucks on starting any HVM domU.
I''ve also tried now without CPU powersaving-features, no change. Have
attached new xm_dmesg, also now a fresh dmesg from dom0.
Hopefully anybody can tell me what else I can try to get this running.
Thank you.
----
I already posted this on xen-users - Todd Deshane asked me to post this
here for feedback.
He also asked if I can test xen 4.1 - I am looking if I can find and
install backports but if not I''m afraid I think I''m not able
to
compiling on myself.
----

I am trying since a few days my first steps with xen.
Hardware: Core2 T7200, Intel 945GME, 2GB RAM
Software: debian squeeze, i686, with debian delivered xen 4.0.1
CPU and BIOS support hardware virtualisation: (XEN) HVM: VMX enabled

I can successfully boot with xen my dom0 with squeeze-delivered i686 kernel.
As soon as I start a HVM (paravirtualisation works without problems) my
dom0 stucks immediately.
I can see about 10 seconds after starting domU "nothing", then the
first
messages like "hrtimer: interrupt took 1739955444 ns" messages appear,
then my disk gets timeouts. then kernel panics like below are starting -
normally they are not written to messages-log, one time i had "luck":

kernel: : [ 4815.144473] saa7146 (0) vpeirq: used 3 times>80% of buffer
(1049604 bytes now)
kernel: : [ 4815.144473] Modules linked in: tun xt_physdev loop
ipt_REJECT ip6table_filter ip6_tables ebtable_nat ebtables bridge stp
xen_evtchn xenfs nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs
xt_recent ipt_MASQUERADE xt_tcpudp xt_state iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables
x_tables fuse ext4 jbd2 crc16 it87 hwmon_vid coretemp tda10021
snd_hda_codec_via budget_av snd_hda_intel snd_hda_codec saa7146_vv
snd_hwdep videodev v4l1_compat snd_pcm_oss snd_mixer_oss videobuf_dma_sg
videobuf_core snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event
snd_seq budget_core i915 drm_kms_helper dvb_core snd_timer saa7146
snd_seq_device ttpci_eeprom drm rng_core pcspkr evdev i2c_i801
i2c_algo_bit snd i2c_core soundcore video output button snd_page_alloc
processor acpi_processor ext3 jbd mbcache dm_mod sd_mod crc_t10dif
ata_generic uhci_hcd ata_piix fan ehci_hcd libata scsi_mod e1000e
usbcore nls_base thermal thermal_sys [last unloaded: scsi_wait_scan]
kernel: : [ 4815.144473]
kernel: : [ 4815.324579] saa7146 (0) saa7146_i2c_writeout [irq]: timed
out waiting for end of xfer
kernel: : [ 4815.324722] ata1: lost interrupt (Status 0x50)
kernel: : [ 4815.324772] sd 0:0:0:0: [sda] Unhandled error code
kernel: : [ 4815.324775] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK
driverbyte=DRIVER_TIMEOUT
kernel: : [ 4815.324780] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 d3
41 9f 00 00 28 00
kernel: : [ 4815.324806] lost page write due to I/O error on sda1
kernel: : [ 4815.324817] lost page write due to I/O error on sda1
kernel: : [ 4815.324826] lost page write due to I/O error on sda1
kernel: : [ 4815.324834] lost page write due to I/O error on sda1
kernel: : [ 4815.324843] lost page write due to I/O error on sda1
kernel: : [ 4815.450483] Pid: 1337, comm: qemu-dm Not tainted
(2.6.32-5-xen-686 #1) 945GM/E-ITE8712
kernel: : [ 4815.450483] EIP: 0061:[<c1002227>] EFLAGS: 00200246 CPU: 0
kernel: : [ 4815.450483] EIP is at hypercall_page+0x227/0x1001
kernel: : [ 4815.450483] EAX: 00040000 EBX: 00000000 ECX: 00000000 EDX:
c357a7b4
kernel: : [ 4815.450483] ESI: 00000009 EDI: 00000028 EBP: c13959e4 ESP:
ddb6defc
kernel: : [ 4815.450483]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
kernel: : [ 4815.450483] CR0: 8005003b CR2: 0807f9d0 CR3: 1da42000 CR4:
00002660
kernel: : [ 4815.450483] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
kernel: : [ 4815.450483] DR6: ffff0ff0 DR7: 00000400
kernel: : [ 4815.450483] Call Trace:
kernel: : [ 4815.450483]  [<c1006048>] ?
xen_force_evtchn_callback+0xc/0x10
kernel: : [ 4815.450483]  [<c1006778>] ? check_events+0x8/0xc
kernel: : [ 4815.450483]  [<c1006737>] ? xen_irq_enable_direct_end+0x0/0x1
kernel: : [ 4815.450483]  [<c103c80b>] ? __do_softirq+0x4b/0x156
kernel: : [ 4815.450483]  [<c103c947>] ? do_softirq+0x31/0x3c
kernel: : [ 4815.450483]  [<c103ca21>] ? irq_exit+0x26/0x58
kernel: : [ 4815.450483]  [<c1199a16>] ? xen_evtchn_do_upcall+0x22/0x2c
kernel: : [ 4815.653736]  [<c1009b5f>] ? xen_do_upcall+0x7/0xc
kernel: : [ 4815.653736]  [<c104a74c>] ? sys_clock_gettime+0x46/0x7e
kernel: : [ 4815.653736]  [<c1008f9c>] ? syscall_call+0x7/0xb
kernel: : [ 4815.676006] saa7146 (0) vpeirq: used 1 times>80% of buffer
(1300396 bytes now)

on monitor I can see frequently messages which look like:
soft lockup - CPU X stuck for XXs! [qemu-dm:...]
always i saw qemu-dm is listed with PID in this message. these messages
I can never find in messages-log, i think the machine is too dead to
write them down onto disc.

the only way to get the machine back is to be fast enough after starting
(means less than about 10 seconds) do a xm destroy<name>. otherwise the
system needs to be powercycled, rarely ctrl-alt-delete on console
initiates a reboot, most time this also does not work.

can anybody tell me where I can search for the issue?
currently I tried to change architecture to amd64, but as it is not only
the kernel and xen itself this is not a "fast try", so hopefully
somebody can help me here.
attached: xm dmesg (from i686)
in there I''ve limited memory of dom0 to 1GB because I tought maybe
balooning causes the issue. my HVM machines never had more than 512MB
configured RAM, but nevertheless the problem also occurs with 128 or
256MB configured.

thank you for your investigations.

best regards
Alois






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 13/04/2011 19:20, "MadLoisae@gmx.net" <MadLoisae@gmx.net>
wrote:
> can anybody tell me where I can search for the issue?
> currently I tried to change architecture to amd64, but as it is not only
> the kernel and xen itself this is not a "fast try", so hopefully
> somebody can help me here.
You only need to update Xen itself to x86_64. 64-bit Xen can run 32-bit
kernels just fine. And 64-bit Xen gets *loads* more testing than 32-bit Xen.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello Keir,

thanks for your idea.
Have installed xen 4.1 amd64 packages, boots correctly but sadly the 
same issue.
I''ve attached xm dmesg and dmesg from dom0 (and also attached this time
missing logs from my last mail)

Thanks in advance for further hints.

Alois

On 04/13/2011 08:44 PM, Keir Fraser wrote:
> On 13/04/2011 19:20, "MadLoisae@gmx.net"<MadLoisae@gmx.net>
wrote:
>
>    
>> can anybody tell me where I can search for the issue?
>> currently I tried to change architecture to amd64, but as it is not
only
>> the kernel and xen itself this is not a "fast try", so
hopefully
>> somebody can help me here.
>>      
> You only need to update Xen itself to x86_64. 64-bit Xen can run 32-bit
> kernels just fine. And 64-bit Xen gets *loads* more testing than 32-bit
Xen.
>
>   -- Keir
>
>
>
>    






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 13/04/2011 22:38, "MadLoisae@gmx.net" <MadLoisae@gmx.net>
wrote:
> Hello Keir,
> 
> thanks for your idea.
> Have installed xen 4.1 amd64 packages, boots correctly but sadly the
> same issue.
> I''ve attached xm dmesg and dmesg from dom0 (and also attached this
time
> missing logs from my last mail)
> 
> Thanks in advance for further hints.
The logs don''t show much, except that dom0 seems to be losing
interrupts
after the HVM guest starts. Do you get more from ''xm dmesg'' if
you boot Xen
with the extra boot parameters "loglvl=all guest_loglvl=all"? What is
your
HVM domain config?

 -- Keir
> Alois
> 
> On 04/13/2011 08:44 PM, Keir Fraser wrote:
>> On 13/04/2011 19:20,
"MadLoisae@gmx.net"<MadLoisae@gmx.net>  wrote:
>> 
>>    
>>> can anybody tell me where I can search for the issue?
>>> currently I tried to change architecture to amd64, but as it is not
only
>>> the kernel and xen itself this is not a "fast try", so
hopefully
>>> somebody can help me here.
>>>      
>> You only need to update Xen itself to x86_64. 64-bit Xen can run 32-bit
>> kernels just fine. And 64-bit Xen gets *loads* more testing than 32-bit
Xen.
>> 
>>   -- Keir
>> 
>> 
>> 
>>    
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Apr 13, 2011 at 11:04:04PM +0100, Keir Fraser
wrote:
> On 13/04/2011 22:38, "MadLoisae@gmx.net"
<MadLoisae@gmx.net> wrote:
> 
> > Hello Keir,
> > 
> > thanks for your idea.
> > Have installed xen 4.1 amd64 packages, boots correctly but sadly the
> > same issue.
> > I''ve attached xm dmesg and dmesg from dom0 (and also attached
this time
> > missing logs from my last mail)
> > 
> > Thanks in advance for further hints.
> 
> The logs don''t show much, except that dom0 seems to be losing
interrupts
> after the HVM guest starts. Do you get more from ''xm
dmesg'' if you boot Xen
> with the extra boot parameters "loglvl=all guest_loglvl=all"?
What is your
> HVM domain config?
Keir, didn''t we have a problem with hvmloader where it would call
populate_physmap
and take a very long time when moving the PFNs around? I can''t remember
the details though :-(

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I just saw that I made two threads instead of one ... hopefully this 
email now receives in the correct one. ;)

---
Hi Keir, Hi Konrad,

thanks for your time.
at first, Keir: I''ve added the lines to xen boot, attached a new xm 
dmesg. until the line "(XEN) HVM1: HVM Loader" all logs were generated
from boot, the logs afterwards from starting/killing the domU-machine. 
also attached a dmesg from dom0. My config is still xen-4.1-amd64 with 
dom0 debian squeeze kernel i686.
my hvm domain config is also attached here.
I am sure that to this domU I''ve configured 512MB ram, my dom0 is 
limited to 1024MB ram, so there should be no bottleneck (host with 
2048MB ram, if I do not limit dom0 memory with xen I can allocate there 
about 1800MB). nevertheless the issue also occurs with a 256MB ram domU.

Konrad, I do not know if this is happening also to other machines, first 
time for me with xen, have never tried or seen on other hardware - or do 
you mean other domUs on my host? if last then yes, this happens if I use 
HVM, as soon as I try paravirt it works flawless. I have tried about 10 
or 15 different configurations of HVM domUs, always the same problem.
I ran your code, the output is attached. also xen ran this time with 
loglvl=all.
If I was running only dom0 the first 4 lines in the output of your code 
always repeated, the two big output blocks with 8 lines were generated 
in the about 15 to 20 seconds stucking, afterwards the 4 lines like on 
beginning repeated again.

hopefully this helps, just contact me if you need more information.

Alois
---


On 04/14/2011 03:23 PM, Konrad Rzeszutek Wilk wrote:
> On Wed, Apr 13, 2011 at 11:04:04PM +0100, Keir Fraser wrote:
>    
>> On 13/04/2011 22:38,
"MadLoisae@gmx.net"<MadLoisae@gmx.net>  wrote:
>>
>>      
>>> Hello Keir,
>>>
>>> thanks for your idea.
>>> Have installed xen 4.1 amd64 packages, boots correctly but sadly
the
>>> same issue.
>>> I''ve attached xm dmesg and dmesg from dom0 (and also
attached this time
>>> missing logs from my last mail)
>>>
>>> Thanks in advance for further hints.
>>>        
>> The logs don''t show much, except that dom0 seems to be losing
interrupts
>> after the HVM guest starts. Do you get more from ''xm
dmesg'' if you boot Xen
>> with the extra boot parameters "loglvl=all guest_loglvl=all"?
What is your
>> HVM domain config?
>>      
> Keir, didn''t we have a problem with hvmloader where it would call
populate_physmap
> and take a very long time when moving the PFNs around? I can''t
remember the details though :-(
>
>    





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, Apr 16, 2011 at 12:11:34AM +0200, MadLoisae@gmx.net
wrote:
> I just saw that I made two threads instead of one ... hopefully this
> email now receives in the correct one. ;)
The log file is a bit stange. The HVM says:

(XEN) HVM1: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (   0 MBytes)

Is your /home/guest1.dsk really 0 megabytes? Have you
tried to use a physical device (say LVM logical volume) just to eliminate
it being the blktap at fault here? (there was a bug in the blktap
driver that Daniel posted recently here).

Instead of using the configuration with the sxp expression can you just
post the .xm one please? They are much easier to read.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Konrad,

thank you.
I''ve read in a small howto that touching the virtual disk is enough. So
it isn''t, I''ve filled it not with 2048MBytes.
I have now attached the virt-install command (i hope this is what you 
expect by ".xm").
Also I discovered now a behaviour: only initializing the vm needs so 
much time and stalls the machine, as soon as I can see i.e. the 
boot-screen from the iso the dom0 is back. The initialisation needs 
nearly 5 minutes (so wairing and hoping this 5 minutes ;) ).
I can see in "Bios"-Bootscreen and also in xm dmesg log IDE timeouts. 
Maybe they point to the reason?
I''ve attached xm dmesg and dom0 dmesg again after creating and booting 
the machine with the attached command and waiting for the iso-bootprompt.

Alois

On 04/18/2011 11:02 PM, Konrad Rzeszutek Wilk wrote:
> On Sat, Apr 16, 2011 at 12:11:34AM +0200, MadLoisae@gmx.net wrote:
>    
>> I just saw that I made two threads instead of one ... hopefully this
>> email now receives in the correct one. ;)
>>      
> The log file is a bit stange. The HVM says:
>
> (XEN) HVM1: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (   0 MBytes)
>
> Is your /home/guest1.dsk really 0 megabytes? Have you
> tried to use a physical device (say LVM logical volume) just to eliminate
> it being the blktap at fault here? (there was a bug in the blktap
> driver that Daniel posted recently here).
>
> Instead of using the configuration with the sxp expression can you just
> post the .xm one please? They are much easier to read.
>
>
>    




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I have to revise my last email.
Not only the initialisation is slow, also further I/O stalls the dom0.
This is an Intel 945GM/E chipset, with an IDE-CF-Card as / (ext3) 
filesystem and an IDE-2.5" hardisk (/home, ext4). IDE-CF-card is slower, 
only able to read/write about 35MBytes/second, the 2.5" HDD is at about 
55MBytes/sec.
Both, the ISO-image and the virtual harddisk are on the /home directory 
and therefore on the harddisk.
Is there maybe an I/O problem? Dom0 is able to do everysth. like usual 
on all filesystems, both disks are without errors.

Alois

On 04/18/2011 11:02 PM, Konrad Rzeszutek Wilk wrote:
> On Sat, Apr 16, 2011 at 12:11:34AM +0200, MadLoisae@gmx.net wrote:
>    
>> I just saw that I made two threads instead of one ... hopefully this
>> email now receives in the correct one. ;)
>>      
> The log file is a bit stange. The HVM says:
>
> (XEN) HVM1: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (   0 MBytes)
>
> Is your /home/guest1.dsk really 0 megabytes? Have you
> tried to use a physical device (say LVM logical volume) just to eliminate
> it being the blktap at fault here? (there was a bug in the blktap
> driver that Daniel posted recently here).
>
> Instead of using the configuration with the sxp expression can you just
> post the .xm one please? They are much easier to read.
>
>
>    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Apr 19, 2011 at 12:35:33AM +0200, MadLoisae@gmx.net
wrote:
> I have to revise my last email.
> Not only the initialisation is slow, also further I/O stalls the dom0.
> This is an Intel 945GM/E chipset, with an IDE-CF-Card as / (ext3)
> filesystem and an IDE-2.5" hardisk (/home, ext4). IDE-CF-card is
> slower, only able to read/write about 35MBytes/second, the 2.5" HDD
> is at about 55MBytes/sec.
> Both, the ISO-image and the virtual harddisk are on the /home
> directory and therefore on the harddisk.
> Is there maybe an I/O problem? Dom0 is able to do everysth. like
> usual on all filesystems, both disks are without errors.
Perhaps? Not sure. Can you put your guest image on an LVM (logical
volume) or a disk partition instead of a file?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Konrad,

have tried now - but not with lvm (actually I have no lvm group here and 
I do not want to lvm one complete drive) but with direct assigning a 
partition. /dev/sdb is 80GB big.
two tries:
1st one:
virt-install --prompt
Would you like a fully virtualized guest (yes or no)? This will allow 
you to run unmodified operating systems. yes
  What is the name of your virtual machine? test
  How much RAM should be allocated (in megabytes)? 256
  What would you like to use as the disk (file path)? /dev/sdb3
  What is the install CD-ROM/ISO or URL? 
/home/GWAY-4.2.15-037-BarracudaNG.iso

Machine stuck, in qemu I could see after recognizing ata0 as 80GB qemu 
harddisc, after some time the ata1 as virtual DVD. again after some 
while I could see the iso bootscreen and machine was back until pressing 
enter for further booting.

2nd one: to avoid file-command for iso-file:
virt-install --prompt
Would you like a fully virtualized guest (yes or no)? This will allow 
you to run unmodified operating systems. yes
  What is the name of your virtual machine? test
  How much RAM should be allocated (in megabytes)? 256
  What would you like to use as the disk (file path)? /dev/sdb3
  What is the install CD-ROM/ISO or URL? 
http://mirrors.kernel.org/centos/5.6/os/x86_64/

After downloading boot-images machine stuck, in qemu I could see again 
ata0 as 80GB harddisc, after some time it tried to boot but could not 
find any bootable devices.

I''ve attached again xm dmesg and dom0 dmesg.

Thanks for your effords.

best regards
Alois

On 04/19/2011 03:28 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Apr 19, 2011 at 12:35:33AM +0200, MadLoisae@gmx.net wrote:
>    
>> I have to revise my last email.
>> Not only the initialisation is slow, also further I/O stalls the dom0.
>> This is an Intel 945GM/E chipset, with an IDE-CF-Card as / (ext3)
>> filesystem and an IDE-2.5" hardisk (/home, ext4). IDE-CF-card is
>> slower, only able to read/write about 35MBytes/second, the 2.5"
HDD
>> is at about 55MBytes/sec.
>> Both, the ISO-image and the virtual harddisk are on the /home
>> directory and therefore on the harddisk.
>> Is there maybe an I/O problem? Dom0 is able to do everysth. like
>> usual on all filesystems, both disks are without errors.
>>      
> Perhaps? Not sure. Can you put your guest image on an LVM (logical
> volume) or a disk partition instead of a file?
>
>    



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel