Xen users - Apr 2010 - trouble with xenserver and xfs (soft lockup - CPU#0 stuck for 61s!)

Hi everybody,

i hope that i'm on the right mailing list. I'm having some trouble with 
xfs and xen. We are running the newest xenserver version with ha on dell 
servers and a dell equalogic.

I set up a fileserver (debian lenny 2.6.29-xs5.5.0.17) with one ext3 
partiton for the os and one xfs partition for the data. Two webserver 
are using this fileserver with nfs.

Now i run the second time into following problem when rsync was running 
for backup:

[212521.428003] BUG: soft lockup - CPU#0 stuck for 61s! [rsync:29921]
[212521.428003] Modules linked in: ipv6 nfsd nfs lockd nfs_acl 
auth_rpcgss sunrpc xenfs xfs exportfs loop evdev xen_netfront pcspkr 
ext3 jbd mbcache xen_blkfront thermal_sys
[212521.428003]
[212521.428003] Pid: 29921, comm: rsync Tainted: G      D 
(2.6.29-xs5.5.0.17 #1)
[212521.428003] EIP: 0061:[<c02f2221>] EFLAGS: 00000287 CPU: 0
[212521.428003] EIP is at __write_lock_failed+0x9/0x1c
[212521.428003] EAX: e1875cbc EBX: c1351e1a ECX: e2407880 EDX: dd004c50
[212521.428003] ESI: 00000000 EDI: c536fd40 EBP: e1875cbc ESP: c1351dac
[212521.428003]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[212521.428003] CR0: 8005003b CR2: b806e348 CR3: 20d99000 CR4: 00002620
[212521.428003] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[212521.428003] DR6: ffff0ff0 DR7: 00000400
[212521.428003] Call Trace:
[212521.428003]  [<c02f23a3>] ? _write_lock+0xe/0xf
[212521.428003]  [<e33f7409>] ? xfs_iget+0x328/0x44f [xfs]
[212521.428003]  [<e340e925>] ? xfs_lookup+0x69/0x97 [xfs]
[212521.428003]  [<e341638b>] ? xfs_vn_lookup+0x36/0x6e [xfs]
[212521.428003]  [<c0194ee1>] ? do_lookup+0xa6/0x116
[212521.428003]  [<c019584d>] ? __link_path_walk+0x524/0x631
[212521.428003]  [<c01057fc>] ? xen_force_evtchn_callback+0xc/0x10
[212521.428003]  [<c0195d5c>] ? path_walk+0x4f/0xa3
[212521.428003]  [<c0196aa0>] ? do_path_lookup+0x132/0x178
[212521.428003]  [<c019732e>] ? getname+0x5e/0xb0
[212521.428003]  [<c0197b05>] ? user_path_at+0x37/0x5f
[212521.428003]  [<c0198e80>] ? filldir64+0x0/0xc5
[212521.428003]  [<c019187f>] ? vfs_lstat_fd+0x12/0x3
[212455.928003]  [<c0191912>] ? sys_lstat64+0xf/0x23
[212455.928003]  [<c0199124>] ? vfs_readdir+0x7c/0x8c
[212455.928003]  [<c0198e80>] ? filldir64+0x0/0xc5
[212455.928003]  [<c01991cf>] ? sys_getdents64+0x9b/0xa5
[212455.928003]  [<c0107f23>] ? sysenter_past_esp+0x3c/0x62
[212455.928003]  [<c0107f5b>] ? sysenter_do_call+0x12/0x2f

I have to force reboot to bring the vm back online. Are there any known 
problems with xfs and xen? Because i know from another deployment that 
they are having nearly the same issues with xfs. There are no problems 
with the ext3 filesystem.

Thank you for your help!

-- 
Kind regards,

--sg


Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht
gestattet.

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error)
please notify
the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Sebastian Gosenheimer a écrit :
> Hi everybody,
>
> i hope that i''m on the right mailing list. I''m having
some trouble with
> xfs and xen. We are running the newest xenserver version with ha on dell 
> servers and a dell equalogic.
>
> I set up a fileserver (debian lenny 2.6.29-xs5.5.0.17) with one ext3 
> partiton for the os and one xfs partition for the data. Two webserver 
> are using this fileserver with nfs.
>
> Now i run the second time into following problem when rsync was running 
> for backup:
>
> [212521.428003] BUG: soft lockup - CPU#0 stuck for 61s! [rsync:29921]
> [212521.428003] Modules linked in: ipv6 nfsd nfs lockd nfs_acl 
> auth_rpcgss sunrpc xenfs xfs exportfs loop evdev xen_netfront pcspkr 
> ext3 jbd mbcache xen_blkfront thermal_sys
> [212521.428003]
> [212521.428003] Pid: 29921, comm: rsync Tainted: G      D 
> (2.6.29-xs5.5.0.17 #1)
> [212521.428003] EIP: 0061:[<c02f2221>] EFLAGS: 00000287 CPU: 0
> [212521.428003] EIP is at __write_lock_failed+0x9/0x1c
> [212521.428003] EAX: e1875cbc EBX: c1351e1a ECX: e2407880 EDX: dd004c50
> [212521.428003] ESI: 00000000 EDI: c536fd40 EBP: e1875cbc ESP: c1351dac
> [212521.428003]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
> [212521.428003] CR0: 8005003b CR2: b806e348 CR3: 20d99000 CR4: 00002620
> [212521.428003] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> [212521.428003] DR6: ffff0ff0 DR7: 00000400
> [212521.428003] Call Trace:
> [212521.428003]  [<c02f23a3>] ? _write_lock+0xe/0xf
> [212521.428003]  [<e33f7409>] ? xfs_iget+0x328/0x44f [xfs]
> [212521.428003]  [<e340e925>] ? xfs_lookup+0x69/0x97 [xfs]
> [212521.428003]  [<e341638b>] ? xfs_vn_lookup+0x36/0x6e [xfs]
> [212521.428003]  [<c0194ee1>] ? do_lookup+0xa6/0x116
> [212521.428003]  [<c019584d>] ? __link_path_walk+0x524/0x631
> [212521.428003]  [<c01057fc>] ? xen_force_evtchn_callback+0xc/0x10
> [212521.428003]  [<c0195d5c>] ? path_walk+0x4f/0xa3
> [212521.428003]  [<c0196aa0>] ? do_path_lookup+0x132/0x178
> [212521.428003]  [<c019732e>] ? getname+0x5e/0xb0
> [212521.428003]  [<c0197b05>] ? user_path_at+0x37/0x5f
> [212521.428003]  [<c0198e80>] ? filldir64+0x0/0xc5
> [212521.428003]  [<c019187f>] ? vfs_lstat_fd+0x12/0x3
> [212455.928003]  [<c0191912>] ? sys_lstat64+0xf/0x23
> [212455.928003]  [<c0199124>] ? vfs_readdir+0x7c/0x8c
> [212455.928003]  [<c0198e80>] ? filldir64+0x0/0xc5
> [212455.928003]  [<c01991cf>] ? sys_getdents64+0x9b/0xa5
> [212455.928003]  [<c0107f23>] ? sysenter_past_esp+0x3c/0x62
> [212455.928003]  [<c0107f5b>] ? sysenter_do_call+0x12/0x2f
>
> I have to force reboot to bring the vm back online. Are there any known 
> problems with xfs and xen? Because i know from another deployment that 
> they are having nearly the same issues with xfs. There are no problems 
> with the ext3 filesystem.
>
> Thank you for your help!
>
>   
Hi Sebastian,

I''m having similar issues with two Lenny domU (amd64 and i386),
both with XFS + quite heavy load (I should say normal load for
what they do : database server and fileserver).
They are running the standard Lenny Xen kernel (2.6.26-2-xen-*)

After searching a little, I found this thread :
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=28968&start=0&tstart=0

Not sure that it is connected to my (or your) problem, but the domUs
I have problems with both have XFS log version 2, i have others domUs
with XFS filesystems and quite heavy charge (mail servers basically),
but they are still in Etch and XFS log version 1, no problem with them.

So I think I will give the log version workaround a try.

Perhaps could you give it a try to ?

In any case, if you find something about that problem, I would be happy
to have feedback from you.

Cheers,

-- 
Yann Cézard - Administrateur Systèmes Serveurs


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Am 27.04.2010 10:51, schrieb Yann Cezard:
> Sebastian Gosenheimer a écrit :
>> Hi everybody,
>>
>> i hope that i'm on the right mailing list. I'm having some
trouble with
>> xfs and xen. We are running the newest xenserver version with ha on
dell
>> servers and a dell equalogic.
>>
>> I set up a fileserver (debian lenny 2.6.29-xs5.5.0.17) with one ext3
>> partiton for the os and one xfs partition for the data. Two webserver
>> are using this fileserver with nfs.
>>
>> Now i run the second time into following problem when rsync was running
>> for backup:
>>
>> [212521.428003] BUG: soft lockup - CPU#0 stuck for 61s! [rsync:29921]
>> [212521.428003] Modules linked in: ipv6 nfsd nfs lockd nfs_acl
>> auth_rpcgss sunrpc xenfs xfs exportfs loop evdev xen_netfront pcspkr
>> ext3 jbd mbcache xen_blkfront thermal_sys
>> [212521.428003]
>> [212521.428003] Pid: 29921, comm: rsync Tainted: G      D
>> (2.6.29-xs5.5.0.17 #1)
>> [212521.428003] EIP: 0061:[<c02f2221>] EFLAGS: 00000287 CPU: 0
>> [212521.428003] EIP is at __write_lock_failed+0x9/0x1c
>> [212521.428003] EAX: e1875cbc EBX: c1351e1a ECX: e2407880 EDX: dd004c50
>> [212521.428003] ESI: 00000000 EDI: c536fd40 EBP: e1875cbc ESP: c1351dac
>> [212521.428003]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
>> [212521.428003] CR0: 8005003b CR2: b806e348 CR3: 20d99000 CR4: 00002620
>> [212521.428003] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
>> [212521.428003] DR6: ffff0ff0 DR7: 00000400
>> [212521.428003] Call Trace:
>> [212521.428003]  [<c02f23a3>] ? _write_lock+0xe/0xf
>> [212521.428003]  [<e33f7409>] ? xfs_iget+0x328/0x44f [xfs]
>> [212521.428003]  [<e340e925>] ? xfs_lookup+0x69/0x97 [xfs]
>> [212521.428003]  [<e341638b>] ? xfs_vn_lookup+0x36/0x6e [xfs]
>> [212521.428003]  [<c0194ee1>] ? do_lookup+0xa6/0x116
>> [212521.428003]  [<c019584d>] ? __link_path_walk+0x524/0x631
>> [212521.428003]  [<c01057fc>] ?
xen_force_evtchn_callback+0xc/0x10
>> [212521.428003]  [<c0195d5c>] ? path_walk+0x4f/0xa3
>> [212521.428003]  [<c0196aa0>] ? do_path_lookup+0x132/0x178
>> [212521.428003]  [<c019732e>] ? getname+0x5e/0xb0
>> [212521.428003]  [<c0197b05>] ? user_path_at+0x37/0x5f
>> [212521.428003]  [<c0198e80>] ? filldir64+0x0/0xc5
>> [212521.428003]  [<c019187f>] ? vfs_lstat_fd+0x12/0x3
>> [212455.928003]  [<c0191912>] ? sys_lstat64+0xf/0x23
>> [212455.928003]  [<c0199124>] ? vfs_readdir+0x7c/0x8c
>> [212455.928003]  [<c0198e80>] ? filldir64+0x0/0xc5
>> [212455.928003]  [<c01991cf>] ? sys_getdents64+0x9b/0xa5
>> [212455.928003]  [<c0107f23>] ? sysenter_past_esp+0x3c/0x62
>> [212455.928003]  [<c0107f5b>] ? sysenter_do_call+0x12/0x2f
>>
>> I have to force reboot to bring the vm back online. Are there any known
>> problems with xfs and xen? Because i know from another deployment that
>> they are having nearly the same issues with xfs. There are no problems
>> with the ext3 filesystem.
>>
>> Thank you for your help!
>>
>>
> Hi Sebastian,
>
> I'm having similar issues with two Lenny domU (amd64 and i386),
> both with XFS + quite heavy load (I should say normal load for
> what they do : database server and fileserver).
> They are running the standard Lenny Xen kernel (2.6.26-2-xen-*)
>
> After searching a little, I found this thread :
>
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=28968&start=0&tstart=0
>
> Not sure that it is connected to my (or your) problem, but the domUs
> I have problems with both have XFS log version 2, i have others domUs
> with XFS filesystems and quite heavy charge (mail servers basically),
> but they are still in Etch and XFS log version 1, no problem with them.
>
> So I think I will give the log version workaround a try.
>
> Perhaps could you give it a try to ?
>
> In any case, if you find something about that problem, I would be happy
> to have feedback from you.
>
> Cheers,
>
Hi Yann,

sorry for the late answer. I was messing arround the week with some 
other things.

I have now since four days up and running a new kernel on the fileserver 
(2.6.32.12-xs5.5.0.17). Since this time everything seems to be fine.

Right now i don't know if there is a xfs or nfs problem in the kernel 
2.6.26-2-xen-*. I will update all our machines to a newer kernel know 
which are using nfs and/or xfs.

 From my opionion the thread you sent isn't connected to my problem.

Did the workaround work for you or do you still have problems.

--sg


Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht
gestattet.

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error)
please notify
the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users