Xen users - Jul 2005 - Serious performance problems - is Xen not ready for production use?

Hi,

I hope my current problems with Xen 2.0.5 are related to my own 
stupidity - otherwise I''m getting the impression that Xen is not ready 
for productional use at the current stage.

I tried to migrate a dedicated web/mailserver to a Xen machine 
yesterday. The original server is a Celeron 2Ghz with 256MB, the new 
machine running Xen is an Athlon 2400 with 512M. So I created a domU 
with 256M memory, made a copy of the old system, adjusted fstab, 
networking and started it up. OS is a debian woody.

Everything seems to work - all services are there and working so the 
system itself is working ok. But: when accessing resources like squirrel 
webmail or accessing the IMAP service the machine is very slow (compared 
to old setup). The load average climbs easily to values above 6, 12 was
the highest value that I got when playing with a single client. Old 
system was never used in a way that a lot CPU power was needed, load avg 
stays there normally below 0.05-0.1 -- on the Xen setup the load never 
dropped below 2.5 (!!).

If anyone has an idea what causes this and how to fix it I would be 
really happy because currently I see no way migrating the server to the 
Xen-based system.

Together with the still unsolved question about ballooning the memory 
this caused my impression that I shouldn''t use Xen in production 
environment (see also my message from 06/28/2005 03:47 ''Can''t
increase
memory size during runtime'' or ''problem to balloon memory
beyond initial
allocation'' from Xuehai Zhang (07/07/2005 06:30). This is really sad 
because I really like Xen and WOULD like to use it...

No, /lib/tls is not there ;)



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I hope my current problems with Xen 2.0.5 are related to my own
> stupidity - otherwise I''m getting the impression that Xen is not
ready
> for productional use at the current stage.
It''s a project that''s still evolving rapidly - particularly in
terms of the
installation and management toolkit - however lots of people are using it in 
production settings already.  The community sites at xensource.com run in Xen 
domains, so we do eat our own dogfood :-)

Performance is one of Xen''s key focus points.  Rest assured that the
problems
you''re seeing are not normal - hopefully we can get you up and running.
> Everything seems to work - all services are there and working so the
> system itself is working ok. But: when accessing resources like squirrel
> webmail or accessing the IMAP service the machine is very slow (compared
> to old setup). The load average climbs easily to values above 6, 12 was
> the highest value that I got when playing with a single client. Old
> system was never used in a way that a lot CPU power was needed, load avg
> stays there normally below 0.05-0.1 -- on the Xen setup the load never
> dropped below 2.5 (!!).
First check: Can you verify that IDE DMA is working on your system?  If not, 
you''ll probably need to build a new kernel with the appropriate
drivers.

If that doesn''t help:
* what sort of workload are you imposing in your tests?
* do you know where the CPU time is going in the domain?
* does the output of top and vmstat look weird?
* anything strange in the dmesg output (try comparing to the dmesg from a 
native boot)?
> Together with the still unsolved question about ballooning the memory
> this caused my impression that I shouldn''t use Xen in production
> environment (see also my message from 06/28/2005 03:47
''Can''t increase
> memory size during runtime'' or ''problem to balloon memory
beyond initial
> allocation'' from Xuehai Zhang (07/07/2005 06:30). This is really
sad
> because I really like Xen and WOULD like to use it...
The memory ballooning needs to be fixed.  I can''t remember what release
the
reported breakage was in - it''s probably an easy fix either way.  The
break
probably went unnoticed because people usually don''t balloon domains
above
their initial allocation.

HTH,
Mark

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Mark,

thanks for the reply!
> First check: Can you verify that IDE DMA is working on your system?  If
not,
> you''ll probably need to build a new kernel with the appropriate
drivers.
Yes, DMA is fine!
> If that doesn''t help:
I''ll come back to this and will some more test this afternoon. I will 
post here...
> The memory ballooning needs to be fixed.  I can''t remember what
release the
> reported breakage was in - it''s probably an easy fix either way. 
The break
> probably went unnoticed because people usually don''t balloon
domains above
> their initial allocation.
That''s great news! Not that it''s broken but that it''s
a known problem ;)
I can live easily with the current and wait for a fix in the future...

Regards,
   Patrick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

<snip>
> When I had a closer look to the mem line:
> Mem:    257488K total,   255368K used,     2120K free,      128K buffers
>
> ...it looked strange to me. I checked and found my config file for this
> domain looking like this:
>
> ...
> memory = 128
> extra = "mem=268435456"
> ...
Well spotted!
> This was left over from my tries to get the "xm balloon" working.
I
> removed the extra line from the config file - and voila, it works... Is
> this config above wrong at all or is the strange behaviour I found
> related to the bugs with balloon feature..?
The config line looks OK at first glance but...  I wonder what units the mem 
parameter expects: if it expects KB then you''ve just caused it to
allocate a
frame table for an absolutely *huge* physical memory, which would account for 
all the real memory being allocated.

I can''t guess any better without going through the code - it should
hopefully
become apparent when someone fixes the ballooning problem (does anyone want 
to volunteer for that, btw?  It''d be useful if someone could post a
patch.).

Cheers,
Mark

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Patrick,
Nice to know ballooning domains above their initial allocation is a known
problem to fix. I also
forwarded my initial post here to xen devel list and Keir replied my email. You
may find the
discussion at
http://lists.xensource.com/archives/html/xen-devel/2005-07/msg00352.html.
Xuehai

Patrick Hess wrote:
>> The memory ballooning needs to be fixed.  I can''t remember
what
>> release the reported breakage was in - it''s probably an easy
fix
>> either way.  The break probably went unnoticed because people usually 
>> don''t balloon domains above their initial allocation.
> 
> 
> That''s great news! Not that it''s broken but that
it''s a known problem ;)
> I can live easily with the current and wait for a fix in the future...
> 
> Regards,
>   Patrick
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello again,

I just had some time to check some more things...
> If that doesn''t help:
> * what sort of workload are you imposing in your tests?
I just used thunderbird to access my IMAP folders and used a single 
browser to access squirrel webmail (also using same IMAP).
> * do you know where the CPU time is going in the domain?
No idea :) An "xm list" on the dom0 didn''t showed much
seconds for the
"slow" machine as I expected when I saw this load avg.
> * does the output of top and vmstat look weird?
Yes :) See below...
> * anything strange in the dmesg output (try comparing to the dmesg from a 
> native boot)?
No, that seems to be fine.
vmstat does not work:

root@pbone:~# vmstat
Unknown HZ value! (39) Assume 100.
    procs                      memory    swap          io     system 
      cpu
  r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us 
  sy  id
Segmentation fault

...and top looks like this after a minute running all services:

  19:25:10 up 1 day,  6:10,  1 user,  load average: 14.41, 5.82, 2.75
137 processes: 135 sleeping, 2 running, 0 zombie, 0 stopped
CPU states:   0.8% user,  99.2% system,   0.0% nice,   0.0% idle
Mem:    257488K total,   255368K used,     2120K free,      128K buffers
Swap:        0K total,        0K used,        0K free,     2440K cached
   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  2567 list      18   0  3716 1004    72 D     0.3  0.3   0:00 python
  2582 root      18   0  1824  400   144 R     0.3  0.1   0:00 top
  2567 list      18   0  3664 1284   392 D     0.4  0.4   0:00 python
     8 root      15   0     0    0     0 DW    0.2  0.0   0:00 kswapd0
  2576 root      18   0 87092 5520   324 D     0.2  2.1   0:00 apache
  2577 root      18   0 87092 5520   324 D     0.2  2.1   0:00 apache
  2578 root      18   0 87092 5520   324 D     0.2  2.1   0:00 apache
  2580 root      18   0 87092 5520   324 D     0.2  2.1   0:00 apache
  2583 root      17   0  1760  656   524 D     0.2  0.2   0:00 cron


When I had a closer look to the mem line:
Mem:    257488K total,   255368K used,     2120K free,      128K buffers

...it looked strange to me. I checked and found my config file for this 
domain looking like this:

...
memory = 128
extra = "mem=268435456"
...

This was left over from my tries to get the "xm balloon" working. I 
removed the extra line from the config file - and voila, it works... Is 
this config above wrong at all or is the strange behaviour I found 
related to the bugs with balloon feature..?

Regards,
   Patrick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, 14 Jul 2005, Mark Williamson wrote:
> <snip>
>
> > When I had a closer look to the mem line:
> > Mem:    257488K total,   255368K used,     2120K free,      128K
buffers
> >
> > ...it looked strange to me. I checked and found my config file for
this
> > domain looking like this:
> >
> > ...
> > memory = 128
> > extra = "mem=268435456"
> > ...
>
> Well spotted!
>
> > This was left over from my tries to get the "xm balloon"
working. I
> > removed the extra line from the config file - and voila, it works...
Is
> > this config above wrong at all or is the strange behaviour I found
> > related to the bugs with balloon feature..?
>
> The config line looks OK at first glance but...  I wonder what units the
mem
> parameter expects: if it expects KB then you''ve just caused it to
allocate a
> frame table for an absolutely *huge* physical memory, which would account
for
> all the real memory being allocated.
My experiment shows changing "mem=268435456" to "mem=256m"
doesn''t
help with the ballooning above initial allocation problem.
(Linux BootPrompt-Howto
(http://en.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.3)
suggests both "k" and "m" suffixes can be used for
"mem=" argument).

BTW, i still think "mem=268435456" is a valid configuration to give
memory
in bytes. Both "dmesg" and "free" outputs form the domain
booted with this
configuration show the domain is booted with 256MB memory.

> I can''t guess any better without going through the code - it
should hopefully
> become apparent when someone fixes the ballooning problem (does anyone want
> to volunteer for that, btw?  It''d be useful if someone could post
a patch.).
>
> Cheers,
> Mark
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> My experiment shows changing "mem=268435456" to
"mem=256m" doesn''t
> help with the ballooning above initial allocation problem.
> (Linux BootPrompt-Howto
> (http://en.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.3)
> suggests both "k" and "m" suffixes can be used for
"mem=" argument).
Yep, that''s the syntax I would expect for a kernel arg: I
don''t think this is
responsible for the ballooning problem though - that error must be somewhere 
else in the code.
> BTW, i still think "mem=268435456" is a valid configuration to
give memory
> in bytes. Both "dmesg" and "free" outputs form the
domain booted with this
> configuration show the domain is booted with 256MB memory.
I meant the issue Patrick saw: he had the right amount of memory but none of 
it was free.  I was suspicious that it might all be taken up by kernel data 
structures.

Did you see lots of memory allocated out of that 256M?

Cheers,
Mark
> > I can''t guess any better without going through the code - it
should
> > hopefully become apparent when someone fixes the ballooning problem
(does
> > anyone want to volunteer for that, btw?  It''d be useful if
someone could
> > post a patch.).
> >
> > Cheers,
> > Mark
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 15 Jul 2005, Mark Williamson wrote:
> > My experiment shows changing "mem=268435456" to
"mem=256m" doesn''t
> > help with the ballooning above initial allocation problem.
> > (Linux BootPrompt-Howto
> > (http://en.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.3)
> > suggests both "k" and "m" suffixes can be used for
"mem=" argument).
>
> Yep, that''s the syntax I would expect for a kernel arg: I
don''t think this is
> responsible for the ballooning problem though - that error must be
somewhere
> else in the code.
>
> > BTW, i still think "mem=268435456" is a valid configuration
to give memory
> > in bytes. Both "dmesg" and "free" outputs form the
domain booted with this
> > configuration show the domain is booted with 256MB memory.
>
> I meant the issue Patrick saw: he had the right amount of memory but none
of
> it was free.  I was suspicious that it might all be taken up by kernel data
> structures.
>
> Did you see lots of memory allocated out of that 256M?
The following is the memory allocation result (commands run from inside of
the domain).

$ dmesg | grep mem
Kernel command line:
ip=128.135.164.83:1.2.3.4:128.135.164.65:255.255.255.192::eth0:off
root=/dev/sda1 ro mem=256m maxmem=256m
vmalloc area: d0800000-fbff9000, maxmem 34000000
Memory: 125376k/262144k available (1630k kernel code, 5324k reserved, 440k
data, 112k init, 0k highmem)
xen_mem: Initialising balloon driver.
Freeing unused kernel memory: 112k freed

$ free
             total       used       free     shared    buffers     cached
Mem:        256932     147452     109480          0       3072       7920
-/+ buffers/cache:     136460     120472
Swap:       204792          0     204792



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk>
wrote:
> <snip>
> 
> > When I had a closer look to the mem line:
> > Mem:    257488K total,   255368K used,     2120K free,      128K
buffers
> >
> > ...it looked strange to me. I checked and found my config file for
this
> > domain looking like this:
> >
> > ...
> > memory = 128
> > extra = "mem=268435456"
> > ...
> 
> Well spotted!
> 
> > This was left over from my tries to get the "xm balloon"
working. I
> > removed the extra line from the config file - and voila, it works...
Is
> > this config above wrong at all or is the strange behaviour I found
> > related to the bugs with balloon feature..?
> 
> The config line looks OK at first glance but...  I wonder what units the
mem
> parameter expects: if it expects KB then you''ve just caused it to
allocate a
> frame table for an absolutely *huge* physical memory, which would account
for
> all the real memory being allocated.
> 
in domain config file, you can give memory parameters this way:
- <number>: memory unit is Byte
- <number>{k.K}: memory unit is Kilobyte
- <number>{m.M}: memory unit is Megabyte
- <number>{g.G}: memory unit is Gigabyte

> I can''t guess any better without going through the code - it
should hopefully
> become apparent when someone fixes the ballooning problem (does anyone want
> to volunteer for that, btw?  It''d be useful if someone could post
a patch.).
i will take a look, if nobody does it before me ;-)

cheers,
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/15/05, aq <aquynh@gmail.com> wrote:
> On 7/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > <snip>
> >
> > > When I had a closer look to the mem line:
> > > Mem:    257488K total,   255368K used,     2120K free,      128K
buffers
> > >
> > > ...it looked strange to me. I checked and found my config file
for this
> > > domain looking like this:
> > >
> > > ...
> > > memory = 128
> > > extra = "mem=268435456"
> > > ...
> >
> > Well spotted!
> >
> > > This was left over from my tries to get the "xm
balloon" working. I
> > > removed the extra line from the config file - and voila, it
works... Is
> > > this config above wrong at all or is the strange behaviour I
found
> > > related to the bugs with balloon feature..?
> >
> > The config line looks OK at first glance but...  I wonder what units
the mem
> > parameter expects: if it expects KB then you''ve just caused
it to allocate a
> > frame table for an absolutely *huge* physical memory, which would
account for
> > all the real memory being allocated.
> >
> 
> in domain config file, you can give memory parameters this way:
> - <number>: memory unit is Byte
> - <number>{k.K}: memory unit is Kilobyte
> - <number>{m.M}: memory unit is Megabyte
> - <number>{g.G}: memory unit is Gigabyte
> 
oops, i took another look, and looks like the default memory unit is
Kilobyte, if you dont specify. if so, it is impossible to give the
maxmem memory in Byte unit, and  the configuration above is probably
*invalid* because that amount of memory is way too much.


regards,
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/15/05, aq <aquynh@gmail.com> wrote:
> On 7/15/05, aq <aquynh@gmail.com> wrote:
> > On 7/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk>
wrote:
> > > <snip>
> > >
> > > > When I had a closer look to the mem line:
> > > > Mem:    257488K total,   255368K used,     2120K free,     
128K buffers
> > > >
> > > > ...it looked strange to me. I checked and found my config
file for this
> > > > domain looking like this:
> > > >
> > > > ...
> > > > memory = 128
> > > > extra = "mem=268435456"
> > > > ...
> > >
> > > Well spotted!
> > >
> > > > This was left over from my tries to get the "xm
balloon" working. I
> > > > removed the extra line from the config file - and voila, it
works... Is
> > > > this config above wrong at all or is the strange behaviour I
found
> > > > related to the bugs with balloon feature..?
> > >
> > > The config line looks OK at first glance but...  I wonder what
units the mem
> > > parameter expects: if it expects KB then you''ve just
caused it to allocate a
> > > frame table for an absolutely *huge* physical memory, which would
account for
> > > all the real memory being allocated.
> > >
> >
> > in domain config file, you can give memory parameters this way:
> > - <number>: memory unit is Byte
> > - <number>{k.K}: memory unit is Kilobyte
> > - <number>{m.M}: memory unit is Megabyte
> > - <number>{g.G}: memory unit is Gigabyte
> >
> 
> oops, i took another look, and looks like the default memory unit is
> Kilobyte, if you dont specify. if so, it is impossible to give the
> maxmem memory in Byte unit, and  the configuration above is probably
> *invalid* because that amount of memory is way too much.
hmm.. another oops :~p

the unit for programming interface is KB, but for *domain*
configuration, if you dont specify then the memory unit is MB. so you
should adjust your domain config file, and give it another shoot.


sorry for so much confuses ;-)
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

aq wrote:
> the unit for programming interface is KB, but for *domain*
> configuration, if you dont specify then the memory unit is MB. so you
> should adjust your domain config file, and give it another shoot.
In the config file I cannot use any k|K|m|M|... - only numeric values in 
MB seem to be allowed here:

nd:/etc/xen# cat dev2.xcfg
kernel ="/boot/xen-linux-2.6.10nwxu"
memory = 64M
[...more stuff...]
nd:/etc/xen# xm create -c /etc/xen/dev2.xcfg
Using config file "/etc/xen/dev2.xcfg".
Traceback (most recent call last):
[...trace...]
   File "/etc/xen/dev2.xcfg", line 2
     memory = 64M
                ^
SyntaxError: invalid syntax


Regards,
   Patrick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/15/05, Patrick Hess <ph@nwnet.de> wrote:
> aq wrote:
> 
> > the unit for programming interface is KB, but for *domain*
> > configuration, if you dont specify then the memory unit is MB. so you
> > should adjust your domain config file, and give it another shoot.
> 
> In the config file I cannot use any k|K|m|M|... - only numeric values in
> MB seem to be allowed here:
> 
> nd:/etc/xen# cat dev2.xcfg
> kernel ="/boot/xen-linux-2.6.10nwxu"
> memory = 64M
no, dont do that. the domain config file is actually a python script,
and it will be executed by python interpreter. so "memory = 64M" is
not syntax-valid, and you will get an error like you saw.

there are some confuses: in domain config file, all memory unit is MB,
and you dont need to specify the memory unit. but if you config the
domain using xm interface (for example "xm maxmem DOM MEM" then you
have few choices when specifying MEM (ie you can use K/M/G suffix on
memory parameter)

so "memory = 64" is fine.

regards,
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users