Hello, I need help with a production system that I would like to design and roll out a my job. I was thinking about buy two server with 16 each and 8 processor and have both server connect to two dell 2850 server with raid 6 and 3x 300 ultra 320 hard drives. Both dell 2850¹s that are being used a nas server will be connected to the two new server through 1gigabit Ethernet. I wonder if this a bad Idea? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Hello, > > I need help with a production system that I would like to design and > roll > out a my job. I was thinking about buy two server with 16 each and 8 > processor and have both server connect to two dell 2850 server with > raid 6 > and 3x 300 ultra 320 hard drives. Both dell 2850¹s that are being > used a > nas server will be connected to the two new server through 1gigabit > Ethernet. I wonder if this a bad Idea?Hi Christopher, the setup you talk about is more or less the same I''m using. I''ve been using it in a production environment for about a year now without a single problem. Here''s my setup: 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on the server) 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN via iscsi (2TB aprox each). The SAN servers are paired and replicated using drbd. As said no problems running xen 3.1 on Ubuntu and Debian DomU, and running Debian and CentOS as domU. All network connections are 1 Gb. I have about 30~40 domU. Hope it helps, Al. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Christopher, no, I don''t limit to 4 the domU''s on my Xen machines, I have plenty of servers to spare. I have some servers running 10-12 domU''s while others only run 1-2. Some none, I have them just in case... or for when new domU''s needed. I have not run out of processing power but I have of RAM. If I had to recommend something that''d be: GET PLENTY OF RAM. As for drbd, yes, I''m running an active/passive setup. You could say a machine is always misused, it''s true, but I have an ''on-line'' backup just in case. You could always use an active/active setup with a cluster aware file system. I tried it but I really didn''t need it. Hope it helps, Al. En/na christopher andrews ha escrit:> Hey Aleix, > > I notice here that you have 12 dell server running xen for your domU''s, so > does this mean your limit to 4 vm per dom0? This going out the fact that you > have 30 to 40 domU''s running in total from your first e-mail. Also I see > that you using drbd for you iscsi san on the dell 2950''s, I''m also thing > about using this setup but I''m wonder are they 2950 running passive active > mode because I believe that drdb can only have one device accessing the > virtual disk interface at a time. > > Thank You, > Chris > > > On 8/23/09 8:08 AM, "Aleix Dorca" <adorca@uda.ad> wrote: > > >>> Hello, >>> >>> I need help with a production system that I would like to design and >>> roll >>> out a my job. I was thinking about buy two server with 16 each and 8 >>> processor and have both server connect to two dell 2850 server with >>> raid 6 >>> and 3x 300 ultra 320 hard drives. Both dell 2850¹s that are being >>> used a >>> nas server will be connected to the two new server through 1gigabit >>> Ethernet. I wonder if this a bad Idea? >>> >> Hi Christopher, >> >> the setup you talk about is more or less the same I''m using. I''ve been >> using it in a production environment for about a year now without a >> single problem. Here''s my setup: >> >> 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on the >> server) >> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN via >> iscsi (2TB aprox each). >> >> The SAN servers are paired and replicated using drbd. >> >> As said no problems running xen 3.1 on Ubuntu and Debian DomU, and >> running Debian and CentOS as domU. >> >> All network connections are 1 Gb. I have about 30~40 domU. >> >> Hope it helps, >> >> Al. >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Aleix, For the servers running over ten domU, have you seen any issues with unexpected latencies or domU sar showing %steal above 1? Are any of these domU running web services or web applications? Sent from my iPhone On Aug 23, 2009, at 3:14 PM, Aleix Dorca <adorca@uda.ad> wrote:> Hi Christopher, > > no, I don''t limit to 4 the domU''s on my Xen machines, I have plenty > of servers to spare. I have some servers running 10-12 domU''s while > others only run 1-2. Some none, I have them just in case... or for > when new domU''s needed. I have not run out of processing power but I > have of RAM. If I had to recommend something that''d be: GET PLENTY > OF RAM. > > As for drbd, yes, I''m running an active/passive setup. You could say > a machine is always misused, it''s true, but I have an ''on-line'' > backup just in case. You could always use an active/active setup > with a cluster aware file system. I tried it but I really didn''t > need it. > > Hope it helps, > > Al. > > En/na christopher andrews ha escrit: >> Hey Aleix, >> I notice here that you have 12 dell server running xen for your >> domU''s, so >> does this mean your limit to 4 vm per dom0? This going out the fact >> that you >> have 30 to 40 domU''s running in total from your first e-mail. Also >> I see >> that you using drbd for you iscsi san on the dell 2950''s, I''m also >> thing >> about using this setup but I''m wonder are they 2950 running passive >> active >> mode because I believe that drdb can only have one device accessing >> the >> virtual disk interface at a time. >> >> Thank You, >> Chris >> >> >> On 8/23/09 8:08 AM, "Aleix Dorca" <adorca@uda.ad> wrote: >> >> >>>> Hello, >>>> >>>> I need help with a production system that I would like to design >>>> and >>>> roll >>>> out a my job. I was thinking about buy two server with 16 each >>>> and 8 >>>> processor and have both server connect to two dell 2850 server with >>>> raid 6 >>>> and 3x 300 ultra 320 hard drives. Both dell 2850¹s that are being >>>> used a >>>> nas server will be connected to the two new server through 1gigabit >>>> Ethernet. I wonder if this a bad Idea? >>>> >>> Hi Christopher, >>> >>> the setup you talk about is more or less the same I''m using. I''ve >>> been >>> using it in a production environment for about a year now without a >>> single problem. Here''s my setup: >>> >>> 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on the >>> server) >>> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN via >>> iscsi (2TB aprox each). >>> >>> The SAN servers are paired and replicated using drbd. >>> >>> As said no problems running xen 3.1 on Ubuntu and Debian DomU, and >>> running Debian and CentOS as domU. >>> >>> All network connections are 1 Gb. I have about 30~40 domU. >>> >>> Hope it helps, >>> >>> Al. >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >> >> >> > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Peter, sorry to tell you I''ve never run into performance problems that lead me to run sar and check steal percentages. Also, I''d have to say that my VMs don''t work to hard most of the time... maybe this helps avoid latencies. The VMs run everything from Active Directory, Exchange, MSSQL (on Windows 2003 HVM), mysql, apache, tomcat, squid... you name it. Aleix. El 23/08/2009, a las 21:34, Peter Booth escribió:> Aleix, > > For the servers running over ten domU, have you seen any issues with > unexpected latencies or domU sar showing %steal above 1? > Are any of these domU running web services or web applications? > > Sent from my iPhone > > On Aug 23, 2009, at 3:14 PM, Aleix Dorca <adorca@uda.ad> wrote: > >> Hi Christopher, >> >> no, I don''t limit to 4 the domU''s on my Xen machines, I have plenty >> of servers to spare. I have some servers running 10-12 domU''s while >> others only run 1-2. Some none, I have them just in case... or for >> when new domU''s needed. I have not run out of processing power but >> I have of RAM. If I had to recommend something that''d be: GET >> PLENTY OF RAM. >> >> As for drbd, yes, I''m running an active/passive setup. You could >> say a machine is always misused, it''s true, but I have an ''on-line'' >> backup just in case. You could always use an active/active setup >> with a cluster aware file system. I tried it but I really didn''t >> need it. >> >> Hope it helps, >> >> Al. >> >> En/na christopher andrews ha escrit: >>> Hey Aleix, >>> I notice here that you have 12 dell server running xen for your >>> domU''s, so >>> does this mean your limit to 4 vm per dom0? This going out the >>> fact that you >>> have 30 to 40 domU''s running in total from your first e-mail. Also >>> I see >>> that you using drbd for you iscsi san on the dell 2950''s, I''m also >>> thing >>> about using this setup but I''m wonder are they 2950 running >>> passive active >>> mode because I believe that drdb can only have one device >>> accessing the >>> virtual disk interface at a time. >>> >>> Thank You, >>> Chris >>> >>> >>> On 8/23/09 8:08 AM, "Aleix Dorca" <adorca@uda.ad> wrote: >>> >>> >>>>> Hello, >>>>> >>>>> I need help with a production system that I would like to design >>>>> and >>>>> roll >>>>> out a my job. I was thinking about buy two server with 16 each >>>>> and 8 >>>>> processor and have both server connect to two dell 2850 server >>>>> with >>>>> raid 6 >>>>> and 3x 300 ultra 320 hard drives. Both dell 2850¹s that are being >>>>> used a >>>>> nas server will be connected to the two new server through >>>>> 1gigabit >>>>> Ethernet. I wonder if this a bad Idea? >>>>> >>>> Hi Christopher, >>>> >>>> the setup you talk about is more or less the same I''m using. I''ve >>>> been >>>> using it in a production environment for about a year now without a >>>> single problem. Here''s my setup: >>>> >>>> 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on the >>>> server) >>>> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN >>>> via >>>> iscsi (2TB aprox each). >>>> >>>> The SAN servers are paired and replicated using drbd. >>>> >>>> As said no problems running xen 3.1 on Ubuntu and Debian DomU, and >>>> running Debian and CentOS as domU. >>>> >>>> All network connections are 1 Gb. I have about 30~40 domU. >>>> >>>> Hope it helps, >>>> >>>> Al. >>>> _______________________________________________ >>>> Xen-users mailing list >>>> Xen-users@lists.xensource.com >>>> http://lists.xensource.com/xen-users >>>> >>> >>> >>> >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher, I don''t know if there''s a best way to do it. :-) The way I did it is: I set up three partitions on 2950 servers. 1 for the OS, 1 for swap, and 1 for drbd (in my case /dev/sda3). This on both servers replicating. Once drbd configured I use LVM on top of it, creating "logical volumes" as needed. I can snapshot any LV any time and all actions are replicated on the other drbd machine. There''s plenty of tutorials dealing with this particular setup. Others, depending on their necessities prefer to have drbd on top of LVM... it also works... Al. El 24/08/2009, a las 6:52, christopher andrews escribió:> Hey Aleix, > > What would be the best way to setup drdb with lvm and still be able > to take > lvm snapshoot for backups. > > > On 8/23/09 9:38 PM, "Aleix Dorca" <adorca@uda.ad> wrote: > >> Hi Peter, >> >> sorry to tell you I''ve never run into performance problems that lead >> me to run sar and check steal percentages. Also, I''d have to say that >> my VMs don''t work to hard most of the time... maybe this helps avoid >> latencies. >> >> The VMs run everything from Active Directory, Exchange, MSSQL (on >> Windows 2003 HVM), mysql, apache, tomcat, squid... you name it. >> >> Aleix. >> >> El 23/08/2009, a las 21:34, Peter Booth escribió: >> >>> Aleix, >>> >>> For the servers running over ten domU, have you seen any issues with >>> unexpected latencies or domU sar showing %steal above 1? >>> Are any of these domU running web services or web applications? >>> >>> Sent from my iPhone >>> >>> On Aug 23, 2009, at 3:14 PM, Aleix Dorca <adorca@uda.ad> wrote: >>> >>>> Hi Christopher, >>>> >>>> no, I don''t limit to 4 the domU''s on my Xen machines, I have plenty >>>> of servers to spare. I have some servers running 10-12 domU''s while >>>> others only run 1-2. Some none, I have them just in case... or for >>>> when new domU''s needed. I have not run out of processing power but >>>> I have of RAM. If I had to recommend something that''d be: GET >>>> PLENTY OF RAM. >>>> >>>> As for drbd, yes, I''m running an active/passive setup. You could >>>> say a machine is always misused, it''s true, but I have an ''on-line'' >>>> backup just in case. You could always use an active/active setup >>>> with a cluster aware file system. I tried it but I really didn''t >>>> need it. >>>> >>>> Hope it helps, >>>> >>>> Al. >>>> >>>> En/na christopher andrews ha escrit: >>>>> Hey Aleix, >>>>> I notice here that you have 12 dell server running xen for your >>>>> domU''s, so >>>>> does this mean your limit to 4 vm per dom0? This going out the >>>>> fact that you >>>>> have 30 to 40 domU''s running in total from your first e-mail. Also >>>>> I see >>>>> that you using drbd for you iscsi san on the dell 2950''s, I''m also >>>>> thing >>>>> about using this setup but I''m wonder are they 2950 running >>>>> passive active >>>>> mode because I believe that drdb can only have one device >>>>> accessing the >>>>> virtual disk interface at a time. >>>>> >>>>> Thank You, >>>>> Chris >>>>> >>>>> >>>>> On 8/23/09 8:08 AM, "Aleix Dorca" <adorca@uda.ad> wrote: >>>>> >>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I need help with a production system that I would like to design >>>>>>> and >>>>>>> roll >>>>>>> out a my job. I was thinking about buy two server with 16 each >>>>>>> and 8 >>>>>>> processor and have both server connect to two dell 2850 server >>>>>>> with >>>>>>> raid 6 >>>>>>> and 3x 300 ultra 320 hard drives. Both dell 2850’s that are >>>>>>> being >>>>>>> used a >>>>>>> nas server will be connected to the two new server through >>>>>>> 1gigabit >>>>>>> Ethernet. I wonder if this a bad Idea? >>>>>>> >>>>>> Hi Christopher, >>>>>> >>>>>> the setup you talk about is more or less the same I''m using. I''ve >>>>>> been >>>>>> using it in a production environment for about a year now >>>>>> without a >>>>>> single problem. Here''s my setup: >>>>>> >>>>>> 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on >>>>>> the >>>>>> server) >>>>>> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN >>>>>> via >>>>>> iscsi (2TB aprox each). >>>>>> >>>>>> The SAN servers are paired and replicated using drbd. >>>>>> >>>>>> As said no problems running xen 3.1 on Ubuntu and Debian DomU, >>>>>> and >>>>>> running Debian and CentOS as domU. >>>>>> >>>>>> All network connections are 1 Gb. I have about 30~40 domU. >>>>>> >>>>>> Hope it helps, >>>>>> >>>>>> Al. >>>>>> _______________________________________________ >>>>>> Xen-users mailing list >>>>>> Xen-users@lists.xensource.com >>>>>> http://lists.xensource.com/xen-users >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-users mailing list >>>> Xen-users@lists.xensource.com >>>> http://lists.xensource.com/xen-users >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Aug-24 06:50 UTC
Re: [Xen-users] Re: I need advice about xen project
On Sun, Aug 23, 2009 at 10:08 PM, Aleix Dorca<adorca@uda.ad> wrote:> Here''s my setup: > > 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on the server) > 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN via iscsi > (2TB aprox each). > > The SAN servers are paired and replicated using drbd.Any reason why you''re not using 1TB drives, or server with lots of internal disks (like HP DL 185 with 14 drives) for SAN? Do you use some kind of fencing as well? -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Martti Kuparinen
2009-Aug-24 07:07 UTC
Re: [Xen-users] Re: I need advice about xen project
Fajar A. Nugraha wrote:>> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN via iscsi > > Any reason why you''re not using 1TB drives, or server with lots ofI assume this is about SAS 15000 rpm vs SATA 7200 rpm... Martti _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Aug-24 07:14 UTC
Re: [Xen-users] Re: I need advice about xen project
On Mon, Aug 24, 2009 at 2:07 PM, Martti Kuparinen<martti.kuparinen@iki.fi> wrote:> Fajar A. Nugraha wrote: >>> >>> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN via >>> iscsi >> >> Any reason why you''re not using 1TB drives, or server with lots of > > I assume this is about SAS 15000 rpm vs SATA 7200 rpm...There''s the 3.5" 1TB 10k rpm SAS. At least HP has them, although I haven''t seen them in IBM servers. Won''t those drives give you the most balance in terms of price, space, and performance? -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
The reason is that we had no more money :-) Also, we needed only 4TB tops... Al. El 24/08/2009, a las 8:50, Fajar A. Nugraha escribió:> On Sun, Aug 23, 2009 at 10:08 PM, Aleix Dorca<adorca@uda.ad> wrote: >> Here''s my setup: >> >> 12 Dell 1950 Servers with 8 Cores and 4 to 16 Gb (depending on the >> server) >> 4 Dell 2950 Servers with 5 450Gb disks on RAID 5 acting as a SAN >> via iscsi >> (2TB aprox each). >> >> The SAN servers are paired and replicated using drbd. > > Any reason why you''re not using 1TB drives, or server with lots of > internal disks (like HP DL 185 with 14 drives) for SAN? > Do you use some kind of fencing as well? > > -- > Fajar_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Booth <peter_booth@me.com> writes:> For the servers running over ten domU, have you seen any issues with > unexpected latencies or domU sar showing %steal above 1? > Are any of these domU running web services or web applications?Speaking as a service provider who runs a whole lot of domains one one server, (well, I use 32GiB/8 core servers, you can see what sizes I provide, you can do the math.) well, you hit lots of problems if you don''t dedicate a core to the dom0 (this means setting dom0_vcpus_pin=true in menu.lst, dom0-cpus 1 in /etc/xen/xend-config.sxp, and then setting cpus=1-7 in all the DomU configs) you see some significant problems with I/O latency if you don''t do this. Dropped packets, domains freezing, etc... if the Dom0 hasn''t enough CPU time, nobody is happy. As for CPU contention, there''s always some joker running mprime. You will see significant CPU contention if you give your guests many VCPUs. I give each guest 1 VCPU, and I don''t have a problem. Usually cpu stays below 50% so steal has got to be pretty low. The credit scheduler is actually pretty good. Disk I/O sucks, but that''s ''cause I''m putting everyone on a mirror of 7500rpm SATA disks. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don''t assume you are stupid. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users