Xen users - May 2013 - GPLPV domu high cpu from System and System Interrupts processes

Having this weird problem of high cpu coming off from "System" and
"System Interrupts" processes on a GPLPV domu.

The latency seems to be related to a system interrupts.

Did anyone else notice the same issue? Running standard web server apps here.

Discard that. It appears that lvm snapshot size (the one I''m using for
backups) got exceeded with i/o errors resulting in high DCP latency
from ata.sys driver. Though lvm backups are quite risky but still it
questions the fact that a simple i/o related issue on an lvm based
domu can render it unbootable.

On Sun, May 26, 2013 at 2:29 PM, Micky <mickylmartin@gmail.com>
wrote:
> Having this weird problem of high cpu coming off from "System"
and
> "System Interrupts" processes on a GPLPV domu.
>
> The latency seems to be related to a system interrupts.
>
> Did anyone else notice the same issue? Running standard web server apps
here.

> 
> Discard that. It appears that lvm snapshot size (the one I''m using
for
> backups) got exceeded with i/o errors resulting in high DCP latency
> from ata.sys driver. Though lvm backups are quite risky but still it
> questions the fact that a simple i/o related issue on an lvm based
> domu can render it unbootable.
> 
If the snapshot becomes full then it becomes disabled. Are you running your domu
on the snapshot itself or are you just saying that there is a snapshot and it
got full?

James

The domu itself is running on a separate lvm. I was just using
snapshots for backups. Indeed, the snapshot got over flowed with i/o
errors. Sadly the last backup with lvm seemed to have few gigs more
than the original size of the lvm so i am gonna have to shit to other
means to restore it. LOL!

 Pretty weird as it happened when the snapshot got filled in.

Dmeventd should have auto expanded the snapshot according to lvm.conf
but it didn''t. Testing.

On Sun, May 26, 2013 at 4:50 PM, James Harper
<james.harper@bendigoit.com.au> wrote:
>>
>> Discard that. It appears that lvm snapshot size (the one I''m
using for
>> backups) got exceeded with i/o errors resulting in high DCP latency
>> from ata.sys driver. Though lvm backups are quite risky but still it
>> questions the fact that a simple i/o related issue on an lvm based
>> domu can render it unbootable.
>>
>
> If the snapshot becomes full then it becomes disabled. Are you running your
domu on the snapshot itself or are you just saying that there is a snapshot and
it got full?
>
> James

I believe I have figured it out.

The default value of snapshot_autoextend_threshold in
/etc/lvm/lvm.conf is 100, which just simply disables the automatic
snapshot expansion.
No where in the documentation mentions that.

After modifying both snapshot* values in lvm.conf, you have to
manually kill dmeventd daemon to reread the lvm.conf file, as it gets
restarted when a snapshot is created.

One more important defacto standard, dmevent doesn''t instantly expands
the snapshots but rather checks for thresholds every 10 seconds or so.
So, if the domu has high I/O activity, you are better of by setting
the extend value by 50% or similar.

*It is REALLY IMPORTANT to configure to AUTOMATIC SNAPSHOT EXPANSION
because if a snapshot overflows, YOUR BACKUP WILL BE CORRUPT*

On Sun, May 26, 2013 at 5:19 PM, Micky <mickylmartin@gmail.com>
wrote:
> The domu itself is running on a separate lvm. I was just using
> snapshots for backups. Indeed, the snapshot got over flowed with i/o
> errors. Sadly the last backup with lvm seemed to have few gigs more
> than the original size of the lvm so i am gonna have to shit to other
> means to restore it. LOL!
>
>  Pretty weird as it happened when the snapshot got filled in.
>
> Dmeventd should have auto expanded the snapshot according to lvm.conf
> but it didn''t. Testing.
>
> On Sun, May 26, 2013 at 4:50 PM, James Harper
> <james.harper@bendigoit.com.au> wrote:
>>>
>>> Discard that. It appears that lvm snapshot size (the one
I''m using for
>>> backups) got exceeded with i/o errors resulting in high DCP latency
>>> from ata.sys driver. Though lvm backups are quite risky but still
it
>>> questions the fact that a simple i/o related issue on an lvm based
>>> domu can render it unbootable.
>>>
>>
>> If the snapshot becomes full then it becomes disabled. Are you running
your domu on the snapshot itself or are you just saying that there is a snapshot
and it got full?
>>
>> James