Hello, I have gotten my new Xen server working almost perfectly, yes... almost. The server in question is based on Debian Lenny and Xen has been installed from the Debian repos. It works perfectly fine with about 8 domains (it''s getting a little cramped on 2GB of RAM but works fine). The last problem is that when a domain starts to generate some I/O, there is massive iowait. The load that is put on the domain in question is Bit Torrent at approx. 40 Mbits split amongst 6 torrents. Nothing really big it seems. I installed Munin to have a few graphs and be able to see a pattern or something. Here is the graph during the torrenting : http://img237.imageshack.us/img237/8493/commun0rgycpuday.png When the torrents stop progessively, the iowait stays the same until all torrents are finished then it goes right back up. I would expect it to change gradually but it doesn''t... You can see this with in the following graph : http://img207.imageshack.us/img207/9681/commun0rgycpuday2.png When I execute the command *iostat -x 1* during the high load in question (on the dom0) I get the following : %user %nice %system %iowait %steal %idle 0,00 0,00 0,00 99,01 0,99 0,00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 35,64 0,00 183,17 0,00 7968,32 0,00 43,50 12,02 48,99 5,43 99,41 Obviously during the process, everything becomes quite slow... The domain in question is the only domain really doing something, the others are just idling. Do you have any idea what is wrong with my setup ? Thank you in advance for your help, Antoine _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 28, 2009 at 2:39 PM, Antoine Benkemoun <antoine.benkemoun@gmail.com> wrote:> Hello, > > I have gotten my new Xen server working almost perfectly, yes... almost. The > server in question is based on Debian Lenny and Xen has been installed from > the Debian repos. It works perfectly fine with about 8 domains (it''s getting > a little cramped on 2GB of RAM but works fine).Seriously? My RHEL 5.3 domU, functioning as firewall only, uses about 200MB of RAM, and it won''t boot if I give it less than 250MB RAM (roughly). I''m wondering how you can cramp all those domUs.> > The last problem is that when a domain starts to generate some I/O, there is > massive iowait.Could it be that all domUs are swapping due to high memory use? what does /proc/meminfo from domU and dom0 looks like? What does "xm top" output on dom0 looks like? Another possibility is that I/O on your system sucks (which might happen for HVM domU). What kind of domUs are you using, PV or HVM? Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello,
Thank you for your answer.
The dom0 is a pretty minimal install of Debian with nothing else then Xen,
SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it
averagely uses 100MB of RAM. It barely ever uses more then 1MB of swap idle.
RHEL must be heavier, which wouldn''t be too surprising.
I have reproduced the problem by generating IOs with 3 torrents and 1 file
copy.
XM top shows :
commun ------ 2959 3.5 262144 12.7 262144
12.7 1 1 1828352 1567395 3 16267 3411443 1848929 32
Domain-0 -----r 2617 2.7 262240 12.7 no limit
n/a 1 0 0 0 0 0 0 0 32
Load average for "commun", the domain being loaded, is above 5. Memory
is
only used at about 50% and only 440k of swap is used. Along goes the usual
symptoms, really slow web page loads and really slow reactivity of about
everything in the domain. Reactivity in the other domains is rather normal.
Meminfo for dm0 :
0rgy:/home/antoine# cat /proc/meminfo
MemTotal: 262340 kB
MemFree: 14284 kB
Buffers: 52060 kB
Cached: 89976 kB
SwapCached: 12 kB
Active: 142984 kB
Inactive: 43772 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 262340 kB
LowFree: 14284 kB
SwapTotal: 522104 kB
SwapFree: 521976 kB
Dirty: 208 kB
Writeback: 0 kB
AnonPages: 44708 kB
Mapped: 7472 kB
Slab: 45184 kB
SReclaimable: 36780 kB
SUnreclaim: 8404 kB
PageTables: 0 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 653272 kB
Committed_AS: 203720 kB
VmallocTotal: 589816 kB
VmallocUsed: 3860 kB
VmallocChunk: 585852 kB
Meminfo for the loaded domain :
antoine@commun:~$ cat /proc/meminfo
MemTotal: 262340 kB
MemFree: 4356 kB
Buffers: 1548 kB
Cached: 126500 kB
SwapCached: 264 kB
Active: 114924 kB
Inactive: 104188 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 262340 kB
LowFree: 4356 kB
SwapTotal: 262136 kB
SwapFree: 261696 kB
Dirty: 11064 kB
Writeback: 16 kB
AnonPages: 91016 kB
Mapped: 21360 kB
Slab: 14236 kB
SReclaimable: 4876 kB
SUnreclaim: 9360 kB
PageTables: 0 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 393304 kB
Committed_AS: 567828 kB
VmallocTotal: 589816 kB
VmallocUsed: 1864 kB
VmallocChunk: 587660 kB
Everything is PV. Does anybody have an idea as to what could be going wrong
?
Thank you in advance,
Antoine
On Tue, Apr 28, 2009 at 10:54 AM, Fajar A. Nugraha <fajar@fajar.net>
wrote:
> On Tue, Apr 28, 2009 at 2:39 PM, Antoine Benkemoun
> <antoine.benkemoun@gmail.com> wrote:
> > Hello,
> >
> > I have gotten my new Xen server working almost perfectly, yes...
almost.
> The
> > server in question is based on Debian Lenny and Xen has been installed
> from
> > the Debian repos. It works perfectly fine with about 8 domains
(it''s
> getting
> > a little cramped on 2GB of RAM but works fine).
>
> Seriously?
> My RHEL 5.3 domU, functioning as firewall only, uses about 200MB of
> RAM, and it won''t boot if I give it less than 250MB RAM (roughly).
I''m
> wondering how you can cramp all those domUs.
>
> >
> > The last problem is that when a domain starts to generate some I/O,
there
> is
> > massive iowait.
>
> Could it be that all domUs are swapping due to high memory use?
> what does /proc/meminfo from domU and dom0 looks like? What does "xm
> top" output on dom0 looks like?
>
> Another possibility is that I/O on your system sucks (which might
> happen for HVM domU). What kind of domUs are you using, PV or HVM?
>
> Regards,
>
> Fajar
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Tue, Apr 28, 2009 at 6:51 PM, Antoine Benkemoun <antoine.benkemoun@gmail.com> wrote:> The dom0 is a pretty minimal install of Debian with nothing else then Xen, > SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it > averagely uses 100MB of RAM.Hmm, interesting. I should test Debian or Ubuntu for firewall domUs sometimes.> I have reproduced the problem by generating IOs with 3 torrents and 1 file > copy. > > XM top shows : > > commun ------ 2959 3.5 262144 12.7 262144 > 12.7 1 1 1828352 1567395 3 16267 3411443 1848929 32 > Domain-0 -----r 2617 2.7 262240 12.7 no limit > n/a 1 0 0 0 0 0 0 0 32Since you ommit the headers, the output is quite confusing :P What are vbd stats for the domU, does it have lots of reads/writes?> > Load average for "commun", the domain being loaded, is above 5. Memory is > only used at about 50% and only 440k of swap is used. Along goes the usual > symptoms, really slow web page loads and really slow reactivity of about > everything in the domain. Reactivity in the other domains is rather normal.So it''s not swap problem. However looking at your previous iostat output %util is over 99 (which means your disk is busy), r/s over 180, and rsec/s almost 8000 (meaning about 4 MB/s). Based on this numbers I''m GUESSING you only have one lowend disk (not raid), and it has reached its IOPS caps. In other words, get more disks :P A suggestion, it would probably be easier to manage if you use LVM and put dom0 and domU each on its own LV. That way iostat can show per-LV staticstic so you can easily determine who''s using most disk I/O. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thank you for your answer. Sorry for the headers ! 4MB/s read speed is just plain ridiculous for a hard drive or is it just me ? Smartctl gives me the following output : === START OF INFORMATION SECTION ==Device Model: Hitachi HDP725025GLA380 Serial Number: GEK234RS1DT3RA Firmware Version: GM2OA5CA User Capacity: 250 059 350 016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Tue Apr 28 14:57:48 2009 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled It gives me the following google result : http://www.newegg.com/Product/Product.aspx?Item=N82E16822145211 . Smells like a decent hard drive to me ... I''ll just go and whip my server provider... These domains are actually stored on LVM so that''s already in place but iostat doesn''t seem to differenciate... Thank you for your help, Antoine On Tue, Apr 28, 2009 at 2:43 PM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Tue, Apr 28, 2009 at 6:51 PM, Antoine Benkemoun > <antoine.benkemoun@gmail.com> wrote: > > The dom0 is a pretty minimal install of Debian with nothing else then > Xen, > > SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it > > averagely uses 100MB of RAM. > > Hmm, interesting. I should test Debian or Ubuntu for firewall domUs > sometimes. > > > I have reproduced the problem by generating IOs with 3 torrents and 1 > file > > copy. > > > > XM top shows : > > > > commun ------ 2959 3.5 262144 12.7 262144 > > 12.7 1 1 1828352 1567395 3 16267 3411443 1848929 32 > > Domain-0 -----r 2617 2.7 262240 12.7 no limit > > n/a 1 0 0 0 0 0 0 0 32 > > Since you ommit the headers, the output is quite confusing :P > What are vbd stats for the domU, does it have lots of reads/writes? > > > > > > Load average for "commun", the domain being loaded, is above 5. Memory is > > only used at about 50% and only 440k of swap is used. Along goes the > usual > > symptoms, really slow web page loads and really slow reactivity of about > > everything in the domain. Reactivity in the other domains is rather > normal. > > So it''s not swap problem. However looking at your previous iostat > output %util is over 99 (which means your disk is busy), r/s over 180, > and rsec/s almost 8000 (meaning about 4 MB/s). Based on this numbers > I''m GUESSING you only have one lowend disk (not raid), and it has > reached its IOPS caps. In other words, get more disks :P > > A suggestion, it would probably be easier to manage if you use LVM and > put dom0 and domU each on its own LV. That way iostat can show per-LV > staticstic so you can easily determine who''s using most disk I/O. > > Regards, > > Fajar >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 28, 2009 at 7:59 PM, Antoine Benkemoun <antoine.benkemoun@gmail.com> wrote:> Thank you for your answer. Sorry for the headers ! > > 4MB/s read speed is just plain ridiculous for a hard drive or is it just me > ?I don''t think its the transfer rate. Its the IOPS thats causing your problems. This link (from Googling "one disk iops") http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240923773781+28353475&threadId=1122630 indicates max IOPS for one disk is lower then your current numbers (180). So I guess you''re IOPS bound. Adding more disk with the right setup would increase the number of IOPS you can handle. Since the IOPS are mostly read, you MIGHT be able to reduce it by giving domU more RAM, enough to hold the torrent it''s running. So instead of having 8 domUs with 256 MB each, try using only 1 or 2 domUs with 1 GB memory.> These domains are actually stored on LVM so that''s already in place but > iostat doesn''t seem to differenciate...Weird. I swear in shows up on RHEL. Could be they have vendor-specific patch. What does /proc/partitions and /proc/diskstats show, does it have "dm" entries? Anyway, from the output of "xm top" you should also be able to determine which domU uses most I/O (look in VBD_RD column). In general bittorrent IS very I/O intensive. If you want to limit the amount of I/O a domU can use (so that whatever they do they never made the entire system crawl down), you might be interested in dm-ioband. Haven''t used it myself, but it looks good. http://people.valinux.co.jp/~ryov/dm-ioband/ Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Also, if you''re using file: vbd access, and you''re low on ram, I imagine your buffer cache on the dom0 is nearly nonexistent, which would kill any writes. You might want to get more RAM while you''re at it. On Tue, Apr 28, 2009 at 6:20 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Tue, Apr 28, 2009 at 7:59 PM, Antoine Benkemoun > <antoine.benkemoun@gmail.com> wrote: >> Thank you for your answer. Sorry for the headers ! >> >> 4MB/s read speed is just plain ridiculous for a hard drive or is it just me >> ? > > I don''t think its the transfer rate. Its the IOPS thats causing your problems. > This link (from Googling "one disk iops") > http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240923773781+28353475&threadId=1122630 > indicates max IOPS for one disk is lower then your current numbers > (180). So I guess you''re IOPS bound. Adding more disk with the right > setup would increase the number of IOPS you can handle. > > Since the IOPS are mostly read, you MIGHT be able to reduce it by > giving domU more RAM, enough to hold the torrent it''s running. So > instead of having 8 domUs with 256 MB each, try using only 1 or 2 > domUs with 1 GB memory. > >> These domains are actually stored on LVM so that''s already in place but >> iostat doesn''t seem to differenciate... > > Weird. I swear in shows up on RHEL. Could be they have vendor-specific > patch. What does /proc/partitions and /proc/diskstats show, does it > have "dm" entries? > > Anyway, from the output of "xm top" you should also be able to > determine which domU uses most I/O (look in VBD_RD column). In general > bittorrent IS very I/O intensive. If you want to limit the amount of > I/O a domU can use (so that whatever they do they never made the > entire system crawl down), you might be interested in dm-ioband. > Haven''t used it myself, but it looks good. > http://people.valinux.co.jp/~ryov/dm-ioband/ > > Regards, > > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Chris Chen <muffaleta@gmail.com> "I want the kind of six pack you can''t drink." -- Micah _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thank you all for your answers. I''m taking good note of the fact that this is most likely tied to the fact to BT is IO intensive and that I need more RAM. That will teach me to leave some room for dom0 :-) On Wed, Apr 29, 2009 at 8:15 PM, Christopher Chen <muffaleta@gmail.com>wrote:> Also, if you''re using file: vbd access, and you''re low on ram, I > imagine your buffer cache on the dom0 is nearly nonexistent, which > would kill any writes. > > You might want to get more RAM while you''re at it. > > On Tue, Apr 28, 2009 at 6:20 AM, Fajar A. Nugraha <fajar@fajar.net> wrote: > > On Tue, Apr 28, 2009 at 7:59 PM, Antoine Benkemoun > > <antoine.benkemoun@gmail.com> wrote: > >> Thank you for your answer. Sorry for the headers ! > >> > >> 4MB/s read speed is just plain ridiculous for a hard drive or is it just > me > >> ? > > > > I don''t think its the transfer rate. Its the IOPS thats causing your > problems. > > This link (from Googling "one disk iops") > > > http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240923773781+28353475&threadId=1122630 > > indicates max IOPS for one disk is lower then your current numbers > > (180). So I guess you''re IOPS bound. Adding more disk with the right > > setup would increase the number of IOPS you can handle. > > > > Since the IOPS are mostly read, you MIGHT be able to reduce it by > > giving domU more RAM, enough to hold the torrent it''s running. So > > instead of having 8 domUs with 256 MB each, try using only 1 or 2 > > domUs with 1 GB memory. > > > >> These domains are actually stored on LVM so that''s already in place but > >> iostat doesn''t seem to differenciate... > > > > Weird. I swear in shows up on RHEL. Could be they have vendor-specific > > patch. What does /proc/partitions and /proc/diskstats show, does it > > have "dm" entries? > > > > Anyway, from the output of "xm top" you should also be able to > > determine which domU uses most I/O (look in VBD_RD column). In general > > bittorrent IS very I/O intensive. If you want to limit the amount of > > I/O a domU can use (so that whatever they do they never made the > > entire system crawl down), you might be interested in dm-ioband. > > Haven''t used it myself, but it looks good. > > http://people.valinux.co.jp/~ryov/dm-ioband/<http://people.valinux.co.jp/%7Eryov/dm-ioband/> > > > > Regards, > > > > Fajar > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > > > -- > Chris Chen <muffaleta@gmail.com> > "I want the kind of six pack you can''t drink." > -- Micah >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users