Xen devel - Feb 2009 - stubdom fs-backend issue

[This email is either empty or too large to be displayed at this time]

I initially encountered this issue testing live migration with
stubdoms. The first 19 live migrations work but the domU and domU-dm
freeze on the 20th attempt. The same behavior is observed when doing
save/restore cycles.

The issue seems to be related to fs-backend. After fs-backend has
launched 20 stubdoms it is no longer able to restore a domU.

Restarting fs-backend after 19 uses avoids this issue but orphans any
running stubdoms, causing them to lock up.

I have tried various flavors of linux and windows in domU. On dom0 I
have tried xen 3.3.0, 3.3.1, 3.3.2-rc1-pre on the official 2.6.18.8
and the latest lenny dom0 kernel. I tried 32bit and amd64. The issue
exists across the board.

To help debug the problem I am attaching several logs. Here are the
steps I took:
strace fs-backend 1>fs-backend-strace-1 2>fs-backend-strace-2
xm create win3-stub
xm save win3 win3.save; sleep 2; xm restore win3.save   (did this 19 times)
xm save win3 win3.save
xm restore win3.save
At this point the domU and domU-dm are not functioning:
xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     2     r-----    254.6
win3                                        41   512     1     ------      0.0
win3-dm                                     42    32     1     -b----      0.0

destroyed win3 and killed fs-backend

I have uploaded the last several qemu logs that show the domUs working
and the last one that didn''t work, xend.log (starting when I booted
the dom0 machine), domain-builder-ng.log and the output of stracing
fs-backend:

https://svn.8svn.com/fs-backend-error-00.tgz (3.5 MB)
sha1sum: 1614629183c8677016fe7e3f24e15c3a22ef7310  fs-backend-error-00.tgz

qemu-dm-win3-dm.log
qemu-dm-win3-dm.log.1
qemu-dm-win3-dm.log.2
qemu-dm-win3-dm.log.3
qemu-dm-win3.log
qemu-dm-win3.log.1
qemu-dm-win3.log.2
qemu-dm-win3.log.3
xend.log
domain-builder-ng.log
fs-backend-strace-1 (~90MB uncompressed)
fs-backend-strace-2

I hope someone can take a few minutes to reproduce this issue. I
mentioned it on the users list but got no responses and haven''t been
able to find anyone successfully using stubdoms that has time to test.
When stubdoms are working they work so well that I don''t want to give
up.

Thanks,
Keith Coleman

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keith Coleman wrote:
> I initially encountered this issue testing live migration with
> stubdoms. The first 19 live migrations work but the domU and domU-dm
> freeze on the 20th attempt. The same behavior is observed when doing
> save/restore cycles.

I am able to reproduce the issue, I''ll try to come up with a fix.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 20, 2009 at 12:05 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> Keith Coleman wrote:
>
>> I initially encountered this issue testing live migration with
>> stubdoms. The first 19 live migrations work but the domU and domU-dm
>> freeze on the 20th attempt. The same behavior is observed when doing
>> save/restore cycles.
>
>
> I am able to reproduce the issue, I''ll try to come up with a fix.
>
>
Do we need to file a bug report for this issue?

Keith Coleman

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keith Coleman wrote:
> On Fri, Feb 20, 2009 at 12:05 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>> Keith Coleman wrote:
>>
>>> I initially encountered this issue testing live migration with
>>> stubdoms. The first 19 live migrations work but the domU and
domU-dm
>>> freeze on the 20th attempt. The same behavior is observed when
doing
>>> save/restore cycles.
>>
>> I am able to reproduce the issue, I''ll try to come up with a
fix.
>>
>>
> 
> Do we need to file a bug report for this issue?
> 
No need: I am fixing this issue right now.
Unfortunately fs-backend has more problems than I though, so the fix is
taking time.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel