Hello, I have gotten my new Xen server working almost perfectly, yes... almost. The server in question is based on Debian Lenny and Xen has been installed from the Debian repos. It works perfectly fine with about 8 domains (it''s getting a little cramped on 2GB of RAM but works fine). The last problem is that when a domain starts to generate some I/O, there is massive iowait. The load that is put on the domain in question is Bit Torrent at approx. 40 Mbits split amongst 6 torrents. Nothing really big it seems. I installed Munin to have a few graphs and be able to see a pattern or something. Here is the graph during the torrenting : http://img237.imageshack.us/img237/8493/commun0rgycpuday.png When the torrents stop progessively, the iowait stays the same until all torrents are finished then it goes right back up. I would expect it to change gradually but it doesn''t... You can see this with in the following graph : http://img207.imageshack.us/img207/9681/commun0rgycpuday2.png When I execute the command *iostat -x 1* during the high load in question (on the dom0) I get the following : %user %nice %system %iowait %steal %idle 0,00 0,00 0,00 99,01 0,99 0,00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 35,64 0,00 183,17 0,00 7968,32 0,00 43,50 12,02 48,99 5,43 99,41 Obviously during the process, everything becomes quite slow... The domain in question is the only domain really doing something, the others are just idling. Do you have any idea what is wrong with my setup ? Thank you in advance for your help, Antoine _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 28, 2009 at 2:39 PM, Antoine Benkemoun <antoine.benkemoun@gmail.com> wrote:> Hello, > > I have gotten my new Xen server working almost perfectly, yes... almost. The > server in question is based on Debian Lenny and Xen has been installed from > the Debian repos. It works perfectly fine with about 8 domains (it''s getting > a little cramped on 2GB of RAM but works fine).Seriously? My RHEL 5.3 domU, functioning as firewall only, uses about 200MB of RAM, and it won''t boot if I give it less than 250MB RAM (roughly). I''m wondering how you can cramp all those domUs.> > The last problem is that when a domain starts to generate some I/O, there is > massive iowait.Could it be that all domUs are swapping due to high memory use? what does /proc/meminfo from domU and dom0 looks like? What does "xm top" output on dom0 looks like? Another possibility is that I/O on your system sucks (which might happen for HVM domU). What kind of domUs are you using, PV or HVM? Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, Thank you for your answer. The dom0 is a pretty minimal install of Debian with nothing else then Xen, SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it averagely uses 100MB of RAM. It barely ever uses more then 1MB of swap idle. RHEL must be heavier, which wouldn''t be too surprising. I have reproduced the problem by generating IOs with 3 torrents and 1 file copy. XM top shows : commun ------ 2959 3.5 262144 12.7 262144 12.7 1 1 1828352 1567395 3 16267 3411443 1848929 32 Domain-0 -----r 2617 2.7 262240 12.7 no limit n/a 1 0 0 0 0 0 0 0 32 Load average for "commun", the domain being loaded, is above 5. Memory is only used at about 50% and only 440k of swap is used. Along goes the usual symptoms, really slow web page loads and really slow reactivity of about everything in the domain. Reactivity in the other domains is rather normal. Meminfo for dm0 : 0rgy:/home/antoine# cat /proc/meminfo MemTotal: 262340 kB MemFree: 14284 kB Buffers: 52060 kB Cached: 89976 kB SwapCached: 12 kB Active: 142984 kB Inactive: 43772 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 262340 kB LowFree: 14284 kB SwapTotal: 522104 kB SwapFree: 521976 kB Dirty: 208 kB Writeback: 0 kB AnonPages: 44708 kB Mapped: 7472 kB Slab: 45184 kB SReclaimable: 36780 kB SUnreclaim: 8404 kB PageTables: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 653272 kB Committed_AS: 203720 kB VmallocTotal: 589816 kB VmallocUsed: 3860 kB VmallocChunk: 585852 kB Meminfo for the loaded domain : antoine@commun:~$ cat /proc/meminfo MemTotal: 262340 kB MemFree: 4356 kB Buffers: 1548 kB Cached: 126500 kB SwapCached: 264 kB Active: 114924 kB Inactive: 104188 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 262340 kB LowFree: 4356 kB SwapTotal: 262136 kB SwapFree: 261696 kB Dirty: 11064 kB Writeback: 16 kB AnonPages: 91016 kB Mapped: 21360 kB Slab: 14236 kB SReclaimable: 4876 kB SUnreclaim: 9360 kB PageTables: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 393304 kB Committed_AS: 567828 kB VmallocTotal: 589816 kB VmallocUsed: 1864 kB VmallocChunk: 587660 kB Everything is PV. Does anybody have an idea as to what could be going wrong ? Thank you in advance, Antoine On Tue, Apr 28, 2009 at 10:54 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Tue, Apr 28, 2009 at 2:39 PM, Antoine Benkemoun > <antoine.benkemoun@gmail.com> wrote: > > Hello, > > > > I have gotten my new Xen server working almost perfectly, yes... almost. > The > > server in question is based on Debian Lenny and Xen has been installed > from > > the Debian repos. It works perfectly fine with about 8 domains (it''s > getting > > a little cramped on 2GB of RAM but works fine). > > Seriously? > My RHEL 5.3 domU, functioning as firewall only, uses about 200MB of > RAM, and it won''t boot if I give it less than 250MB RAM (roughly). I''m > wondering how you can cramp all those domUs. > > > > > The last problem is that when a domain starts to generate some I/O, there > is > > massive iowait. > > Could it be that all domUs are swapping due to high memory use? > what does /proc/meminfo from domU and dom0 looks like? What does "xm > top" output on dom0 looks like? > > Another possibility is that I/O on your system sucks (which might > happen for HVM domU). What kind of domUs are you using, PV or HVM? > > Regards, > > Fajar >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 28, 2009 at 6:51 PM, Antoine Benkemoun <antoine.benkemoun@gmail.com> wrote:> The dom0 is a pretty minimal install of Debian with nothing else then Xen, > SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it > averagely uses 100MB of RAM.Hmm, interesting. I should test Debian or Ubuntu for firewall domUs sometimes.> I have reproduced the problem by generating IOs with 3 torrents and 1 file > copy. > > XM top shows : > > commun ------ 2959 3.5 262144 12.7 262144 > 12.7 1 1 1828352 1567395 3 16267 3411443 1848929 32 > Domain-0 -----r 2617 2.7 262240 12.7 no limit > n/a 1 0 0 0 0 0 0 0 32Since you ommit the headers, the output is quite confusing :P What are vbd stats for the domU, does it have lots of reads/writes?> > Load average for "commun", the domain being loaded, is above 5. Memory is > only used at about 50% and only 440k of swap is used. Along goes the usual > symptoms, really slow web page loads and really slow reactivity of about > everything in the domain. Reactivity in the other domains is rather normal.So it''s not swap problem. However looking at your previous iostat output %util is over 99 (which means your disk is busy), r/s over 180, and rsec/s almost 8000 (meaning about 4 MB/s). Based on this numbers I''m GUESSING you only have one lowend disk (not raid), and it has reached its IOPS caps. In other words, get more disks :P A suggestion, it would probably be easier to manage if you use LVM and put dom0 and domU each on its own LV. That way iostat can show per-LV staticstic so you can easily determine who''s using most disk I/O. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thank you for your answer. Sorry for the headers ! 4MB/s read speed is just plain ridiculous for a hard drive or is it just me ? Smartctl gives me the following output : === START OF INFORMATION SECTION ==Device Model: Hitachi HDP725025GLA380 Serial Number: GEK234RS1DT3RA Firmware Version: GM2OA5CA User Capacity: 250 059 350 016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Tue Apr 28 14:57:48 2009 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled It gives me the following google result : http://www.newegg.com/Product/Product.aspx?Item=N82E16822145211 . Smells like a decent hard drive to me ... I''ll just go and whip my server provider... These domains are actually stored on LVM so that''s already in place but iostat doesn''t seem to differenciate... Thank you for your help, Antoine On Tue, Apr 28, 2009 at 2:43 PM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Tue, Apr 28, 2009 at 6:51 PM, Antoine Benkemoun > <antoine.benkemoun@gmail.com> wrote: > > The dom0 is a pretty minimal install of Debian with nothing else then > Xen, > > SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it > > averagely uses 100MB of RAM. > > Hmm, interesting. I should test Debian or Ubuntu for firewall domUs > sometimes. > > > I have reproduced the problem by generating IOs with 3 torrents and 1 > file > > copy. > > > > XM top shows : > > > > commun ------ 2959 3.5 262144 12.7 262144 > > 12.7 1 1 1828352 1567395 3 16267 3411443 1848929 32 > > Domain-0 -----r 2617 2.7 262240 12.7 no limit > > n/a 1 0 0 0 0 0 0 0 32 > > Since you ommit the headers, the output is quite confusing :P > What are vbd stats for the domU, does it have lots of reads/writes? > > > > > > Load average for "commun", the domain being loaded, is above 5. Memory is > > only used at about 50% and only 440k of swap is used. Along goes the > usual > > symptoms, really slow web page loads and really slow reactivity of about > > everything in the domain. Reactivity in the other domains is rather > normal. > > So it''s not swap problem. However looking at your previous iostat > output %util is over 99 (which means your disk is busy), r/s over 180, > and rsec/s almost 8000 (meaning about 4 MB/s). Based on this numbers > I''m GUESSING you only have one lowend disk (not raid), and it has > reached its IOPS caps. In other words, get more disks :P > > A suggestion, it would probably be easier to manage if you use LVM and > put dom0 and domU each on its own LV. That way iostat can show per-LV > staticstic so you can easily determine who''s using most disk I/O. > > Regards, > > Fajar >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 28, 2009 at 7:59 PM, Antoine Benkemoun <antoine.benkemoun@gmail.com> wrote:> Thank you for your answer. Sorry for the headers ! > > 4MB/s read speed is just plain ridiculous for a hard drive or is it just me > ?I don''t think its the transfer rate. Its the IOPS thats causing your problems. This link (from Googling "one disk iops") http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240923773781+28353475&threadId=1122630 indicates max IOPS for one disk is lower then your current numbers (180). So I guess you''re IOPS bound. Adding more disk with the right setup would increase the number of IOPS you can handle. Since the IOPS are mostly read, you MIGHT be able to reduce it by giving domU more RAM, enough to hold the torrent it''s running. So instead of having 8 domUs with 256 MB each, try using only 1 or 2 domUs with 1 GB memory.> These domains are actually stored on LVM so that''s already in place but > iostat doesn''t seem to differenciate...Weird. I swear in shows up on RHEL. Could be they have vendor-specific patch. What does /proc/partitions and /proc/diskstats show, does it have "dm" entries? Anyway, from the output of "xm top" you should also be able to determine which domU uses most I/O (look in VBD_RD column). In general bittorrent IS very I/O intensive. If you want to limit the amount of I/O a domU can use (so that whatever they do they never made the entire system crawl down), you might be interested in dm-ioband. Haven''t used it myself, but it looks good. http://people.valinux.co.jp/~ryov/dm-ioband/ Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Also, if you''re using file: vbd access, and you''re low on ram, I imagine your buffer cache on the dom0 is nearly nonexistent, which would kill any writes. You might want to get more RAM while you''re at it. On Tue, Apr 28, 2009 at 6:20 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Tue, Apr 28, 2009 at 7:59 PM, Antoine Benkemoun > <antoine.benkemoun@gmail.com> wrote: >> Thank you for your answer. Sorry for the headers ! >> >> 4MB/s read speed is just plain ridiculous for a hard drive or is it just me >> ? > > I don''t think its the transfer rate. Its the IOPS thats causing your problems. > This link (from Googling "one disk iops") > http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240923773781+28353475&threadId=1122630 > indicates max IOPS for one disk is lower then your current numbers > (180). So I guess you''re IOPS bound. Adding more disk with the right > setup would increase the number of IOPS you can handle. > > Since the IOPS are mostly read, you MIGHT be able to reduce it by > giving domU more RAM, enough to hold the torrent it''s running. So > instead of having 8 domUs with 256 MB each, try using only 1 or 2 > domUs with 1 GB memory. > >> These domains are actually stored on LVM so that''s already in place but >> iostat doesn''t seem to differenciate... > > Weird. I swear in shows up on RHEL. Could be they have vendor-specific > patch. What does /proc/partitions and /proc/diskstats show, does it > have "dm" entries? > > Anyway, from the output of "xm top" you should also be able to > determine which domU uses most I/O (look in VBD_RD column). In general > bittorrent IS very I/O intensive. If you want to limit the amount of > I/O a domU can use (so that whatever they do they never made the > entire system crawl down), you might be interested in dm-ioband. > Haven''t used it myself, but it looks good. > http://people.valinux.co.jp/~ryov/dm-ioband/ > > Regards, > > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Chris Chen <muffaleta@gmail.com> "I want the kind of six pack you can''t drink." -- Micah _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thank you all for your answers. I''m taking good note of the fact that this is most likely tied to the fact to BT is IO intensive and that I need more RAM. That will teach me to leave some room for dom0 :-) On Wed, Apr 29, 2009 at 8:15 PM, Christopher Chen <muffaleta@gmail.com>wrote:> Also, if you''re using file: vbd access, and you''re low on ram, I > imagine your buffer cache on the dom0 is nearly nonexistent, which > would kill any writes. > > You might want to get more RAM while you''re at it. > > On Tue, Apr 28, 2009 at 6:20 AM, Fajar A. Nugraha <fajar@fajar.net> wrote: > > On Tue, Apr 28, 2009 at 7:59 PM, Antoine Benkemoun > > <antoine.benkemoun@gmail.com> wrote: > >> Thank you for your answer. Sorry for the headers ! > >> > >> 4MB/s read speed is just plain ridiculous for a hard drive or is it just > me > >> ? > > > > I don''t think its the transfer rate. Its the IOPS thats causing your > problems. > > This link (from Googling "one disk iops") > > > http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240923773781+28353475&threadId=1122630 > > indicates max IOPS for one disk is lower then your current numbers > > (180). So I guess you''re IOPS bound. Adding more disk with the right > > setup would increase the number of IOPS you can handle. > > > > Since the IOPS are mostly read, you MIGHT be able to reduce it by > > giving domU more RAM, enough to hold the torrent it''s running. So > > instead of having 8 domUs with 256 MB each, try using only 1 or 2 > > domUs with 1 GB memory. > > > >> These domains are actually stored on LVM so that''s already in place but > >> iostat doesn''t seem to differenciate... > > > > Weird. I swear in shows up on RHEL. Could be they have vendor-specific > > patch. What does /proc/partitions and /proc/diskstats show, does it > > have "dm" entries? > > > > Anyway, from the output of "xm top" you should also be able to > > determine which domU uses most I/O (look in VBD_RD column). In general > > bittorrent IS very I/O intensive. If you want to limit the amount of > > I/O a domU can use (so that whatever they do they never made the > > entire system crawl down), you might be interested in dm-ioband. > > Haven''t used it myself, but it looks good. > > http://people.valinux.co.jp/~ryov/dm-ioband/<http://people.valinux.co.jp/%7Eryov/dm-ioband/> > > > > Regards, > > > > Fajar > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > > > -- > Chris Chen <muffaleta@gmail.com> > "I want the kind of six pack you can''t drink." > -- Micah >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users