Xen users - May 2005 - Xen reboots on dom-U disk stress...

Hi all

I tried to run bonnie++ disk stresser in dom-U, who''s disk is backed 
with non-local (on nfs) loop-back file.

The machine rebooted pretty quickly.

So, how do I tell what''s barfing? Is it Xen? Is it dom-0 (nfs or
loop-back)?

I looked in dom-0''s /var/log/messages and didn''t see any
obvious record
of a dom-0 whoopsie (but is that the right place to look etc...).

Any help greatfully received.

Incidently, I ran bonnie++ successfully in dom-U with disk backed by 
local (dom-0 ext3) loop-back file. This pleasantly surprised me, as 
loop-back files don''t have a good rep for stability under load.

Regards
Roland


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

How much memory does dom0 have?  I believe NFS + loopback files is very prone 
to out-of-memory problems so it seems most likely your dom0 died.

If you hook up a serial line, you might be able to get some useful output 
before it dies.

Had you considered using something like GNBD instead of NFS + loopback?  It 
should perform better and be less prone to OOM problems.

Cheers,
Mark

On Wednesday 11 May 2005 08:36, Roland Paterson-Jones
wrote:
> Hi all
>
> I tried to run bonnie++ disk stresser in dom-U, who''s disk is
backed
> with non-local (on nfs) loop-back file.
>
> The machine rebooted pretty quickly.
>
> So, how do I tell what''s barfing? Is it Xen? Is it dom-0 (nfs or
> loop-back)?
>
> I looked in dom-0''s /var/log/messages and didn''t see any
obvious record
> of a dom-0 whoopsie (but is that the right place to look etc...).
>
> Any help greatfully received.
>
> Incidently, I ran bonnie++ successfully in dom-U with disk backed by
> local (dom-0 ext3) loop-back file. This pleasantly surprised me, as
> loop-back files don''t have a good rep for stability under load.
>
> Regards
> Roland
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Yes, spot on, it was running out of page space in dom-0 after which 
Linux seemed to go on a process killing spree before turning the gun on 
itself and rebooting. Not very pretty.

And yes, NFS + loopback looks particularly ugly. I got it to finally run 
with 1G in dom-0!

We''re trying GNDB, iSCSI et al... (any preferences?)

Thanks
Roland

Mark Williamson wrote:
>How much memory does dom0 have?  I believe NFS + loopback files is very
prone
>to out-of-memory problems so it seems most likely your dom0 died.
>
>If you hook up a serial line, you might be able to get some useful output 
>before it dies.
>
>Had you considered using something like GNBD instead of NFS + loopback?  It 
>should perform better and be less prone to OOM problems.
>
>Cheers,
>Mark
>
>On Wednesday 11 May 2005 08:36, Roland Paterson-Jones wrote:
>  
>
>>Hi all
>>
>>I tried to run bonnie++ disk stresser in dom-U, who''s disk is
backed
>>with non-local (on nfs) loop-back file.
>>
>>The machine rebooted pretty quickly.
>>
>>So, how do I tell what''s barfing? Is it Xen? Is it dom-0 (nfs
or
>>loop-back)?
>>
>>I looked in dom-0''s /var/log/messages and didn''t see
any obvious record
>>of a dom-0 whoopsie (but is that the right place to look etc...).
>>
>>Any help greatfully received.
>>
>>Incidently, I ran bonnie++ successfully in dom-U with disk backed by
>>local (dom-0 ext3) loop-back file. This pleasantly surprised me, as
>>loop-back files don''t have a good rep for stability under load.
>>
>>Regards
>>Roland
>>
>>
>>_______________________________________________
>>Xen-users mailing list
>>Xen-users@lists.xensource.com
>>http://lists.xensource.com/xen-users
>>    
>>
>
>_______________________________________________
>Xen-users mailing list
>Xen-users@lists.xensource.com
>http://lists.xensource.com/xen-users
>
>
>  
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Yes, spot on, it was running out of page space in dom-0 after which
> Linux seemed to go on a process killing spree before turning the gun on
> itself and rebooting. Not very pretty.
Yuk.  Computer guts ;-)
> And yes, NFS + loopback looks particularly ugly. I got it to finally run
> with 1G in dom-0!
Yowee!  I''ve looked in portions of both the NFS and the loopback code
but I
admit I don''t really know why it''s such a memory hog :-/
> We''re trying GNDB, iSCSI et al... (any preferences?)
From what I''ve heard, GNBD works pretty well and has the best
performance of
the NBDs.  iSCSI is more sophisticated but may well be overkill for your 
needs.

With GNBD I imagine it''s a simple matter to get the daemon to serve out
the
disk file, so it should be pretty easy to convert your setup.  You could then 
either have the domain use GNBD root (probably booting with an initrd), or 
just re-export the device from dom0 as you have been doing.

Cheers,
Mark
> Thanks
> Roland
>
> Mark Williamson wrote:
> >How much memory does dom0 have?  I believe NFS + loopback files is very
> > prone to out-of-memory problems so it seems most likely your dom0
died.
> >
> >If you hook up a serial line, you might be able to get some useful
output
> >before it dies.
> >
> >Had you considered using something like GNBD instead of NFS + loopback?
> > It should perform better and be less prone to OOM problems.
> >
> >Cheers,
> >Mark
> >
> >On Wednesday 11 May 2005 08:36, Roland Paterson-Jones wrote:
> >>Hi all
> >>
> >>I tried to run bonnie++ disk stresser in dom-U, who''s disk
is backed
> >>with non-local (on nfs) loop-back file.
> >>
> >>The machine rebooted pretty quickly.
> >>
> >>So, how do I tell what''s barfing? Is it Xen? Is it dom-0
(nfs or
> >>loop-back)?
> >>
> >>I looked in dom-0''s /var/log/messages and didn''t
see any obvious record
> >>of a dom-0 whoopsie (but is that the right place to look etc...).
> >>
> >>Any help greatfully received.
> >>
> >>Incidently, I ran bonnie++ successfully in dom-U with disk backed
by
> >>local (dom-0 ext3) loop-back file. This pleasantly surprised me, as
> >>loop-back files don''t have a good rep for stability under
load.
> >>
> >>Regards
> >>Roland
> >>
> >>
> >>_______________________________________________
> >>Xen-users mailing list
> >>Xen-users@lists.xensource.com
> >>http://lists.xensource.com/xen-users
> >
> >_______________________________________________
> >Xen-users mailing list
> >Xen-users@lists.xensource.com
> >http://lists.xensource.com/xen-users
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mark Williamson wrote:
>>We''re trying GNDB, iSCSI et al... (any preferences?)
>>    
>>
>
>>From what I''ve heard, GNBD works pretty well and has the best
performance of
>the NBDs.  iSCSI is more sophisticated but may well be overkill for your 
>needs.
>
>With GNBD I imagine it''s a simple matter to get the daemon to serve
out the
>disk file, so it should be pretty easy to convert your setup.  You could
then
>either have the domain use GNBD root (probably booting with an initrd), or 
>just re-export the device from dom0 as you have been doing.
>  
>
The ''nice'' thing about loop-back files is that they are
sparse, and are
easy to dynamically create and destroy. The NBD''s, on the other hand, 
need a dedicated device the far end (which tends to be a fairly static 
configuration). I guess the server-side device could be a loop-back 
device, but then you''re pushing loop-back onto the server.

Mmmm ...
Roland


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> The ''nice'' thing about loop-back files is that they are
sparse, and are
> easy to dynamically create and destroy. The NBD''s, on the other
hand,
> need a dedicated device the far end (which tends to be a fairly static
> configuration). I guess the server-side device could be a loop-back
> device, but then you''re pushing loop-back onto the server.
ENBD appears to be able to export a file at the server end.  I can''t
find any
information as to whether GNBD does: the docs don''t provide an example
but it
still might work...

Even if the NBD server you choose doesn''t support exporting files
directly,
it''ll still be an improvement over NFS.

Cheers,
Mark

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users