Xen users - Mar 2013 - Experiencing kernel panic with SSH + LVM + Xen + significant amounts of data

Hi all,

 I''m running into a kernel panic with Xen, under fairly specific
 circumstances. I''ll post it here first, and follow up on xen-devel if
 it looks like it''s a Xen bug.

 After some experimentation, it seems like the panic occurs under the
 following circumstances:

 1) A significant amount of data is being moved

 2) ...Onto (or off of?) an LVM partition

 3) ...With SSH being involved somehow?

 4) ...On Xen.

 Originally, I was trying to move a filesystem over the network via "dd
 if=... | ssh dd of=...", onto an LVM partition. The filesystem was
 20G, and it repeatably crashed, usually after >25% transferred.

 I''ve also tried scp''ing files, using dd to transfer files,
 transferring the 20G FS to /dev/null, moving data OFF the host in
 question, etc. None of these triggered a panic.

 Moving large files on the host itself doesn''t trigger a
 crash. However, piping data to an LVM partition (and in one case,
 piping from LVM to /dev/null) has triggered a crash, though the panic
 seems to occur simultaneously with an SSH login when it''s
 happened. Minus an SSH login, I''ve successfully and repeatably moved
 data from LVM => LVM, LVM => null, yes => LVM, without a crash (logged
 in via SSH the whole time, I should add...there seems to be something
 about logins specifically)

 The system has ended up in a bad state more than once, too, which
 complicates things. It''s possible that the crash triggered by moving
 OFF of LVM to /dev/null was the result of a bad state triggered
 earlier, because I wasn''t able to reproduce it.

 No such issues occur if I''m not running Xen, which seems to rule out
 HW failure. To be clear, the host in question is the hypervisor, and
 the panics occur regardless of whether VMs are running.

 Version info:

 Essentially, it''s a fully up-to-date Ubuntu 12.04 system.

 HW - HP ProLiant DL145 G2/K85NL (rackmount)

 Xen - 4.1.2-2ubuntu2.6

 LVM:
   LVM version:     2.02.66(2) (2010-05-20)
   Library version: 1.02.48 (2010-05-20)
   Driver version:  4.22.0

 SSH:
   OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012

 Kernel - 3.2.0-39-generic

 I can provide more info on request.

 I''ve also attached a sample kernel panic. This one''s a little
old, but
 the crashes don''t always dump to syslog, and the symptoms
haven''t
 changed.

 Thanks!

 Peter

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

I can''t help with the Kernel Panic but I can say that I too used to
copy
LV''s around with  "dd if=... | ssh dd of=...", a much more
elegeant and
fault tolerant way to do these copies is to export an LV as an iSCSI
Target, attach the Target on the remote machine and then dd locally.

Aaron Cossey
aaron.cossey@gmail.com


On Fri, Mar 22, 2013 at 1:24 AM, Peter Selby <peter@selby.ca> wrote:
> Hi all,
>
> I''m running into a kernel panic with Xen, under fairly specific
> circumstances. I''ll post it here first, and follow up on xen-devel
if
> it looks like it''s a Xen bug.
>
> After some experimentation, it seems like the panic occurs under the
> following circumstances:
>
> 1) A significant amount of data is being moved
>
> 2) ...Onto (or off of?) an LVM partition
>
> 3) ...With SSH being involved somehow?
>
> 4) ...On Xen.
>
> Originally, I was trying to move a filesystem over the network via "dd
> if=... | ssh dd of=...", onto an LVM partition. The filesystem was
> 20G, and it repeatably crashed, usually after >25% transferred.
>
> I''ve also tried scp''ing files, using dd to transfer
files,
> transferring the 20G FS to /dev/null, moving data OFF the host in
> question, etc. None of these triggered a panic.
>
> Moving large files on the host itself doesn''t trigger a
> crash. However, piping data to an LVM partition (and in one case,
> piping from LVM to /dev/null) has triggered a crash, though the panic
> seems to occur simultaneously with an SSH login when it''s
> happened. Minus an SSH login, I''ve successfully and repeatably
moved
> data from LVM => LVM, LVM => null, yes => LVM, without a crash
(logged
> in via SSH the whole time, I should add...there seems to be something
> about logins specifically)
>
> The system has ended up in a bad state more than once, too, which
> complicates things. It''s possible that the crash triggered by
moving
> OFF of LVM to /dev/null was the result of a bad state triggered
> earlier, because I wasn''t able to reproduce it.
>
> No such issues occur if I''m not running Xen, which seems to rule
out
> HW failure. To be clear, the host in question is the hypervisor, and
> the panics occur regardless of whether VMs are running.
>
> Version info:
>
> Essentially, it''s a fully up-to-date Ubuntu 12.04 system.
>
> HW - HP ProLiant DL145 G2/K85NL (rackmount)
>
> Xen - 4.1.2-2ubuntu2.6
>
> LVM:
>   LVM version:     2.02.66(2) (2010-05-20)
>   Library version: 1.02.48 (2010-05-20)
>   Driver version:  4.22.0
>
> SSH:
>   OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
>
> Kernel - 3.2.0-39-generic
>
> I can provide more info on request.
>
> I''ve also attached a sample kernel panic. This one''s a
little old, but
> the crashes don''t always dump to syslog, and the symptoms
haven''t
> changed.
>
> Thanks!
>
> Peter
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users