Hello Everyone, I recently had a question that got no responses about GFS+DRBD clusters for Xen VM storage, but after some consideration (and a lot of Googling) I have a couple of new questions. Basically what we have here are two servers that will each have a RAID-5 array filled up with 5 x 320GB SATA drives, I want to have these as useable file systems on both servers (as they will both be used for Xen VM storage) but they will be replicating in the background for disaster recovery purposes over a GbE link. First of all, I need to know if this is good practice because I can see a looming clusterf**k if both machines are running VMs from the same shared storage location. Second, I ran a test on two identical servers with DRBD and GFS in a Primary/Primary cluster setup and the performance numbers were appalling compared to local ext3 storage, for example: 5 Concurrent Sessions in iozone gave me the following: Average Throughput for Writers per process: EXT3: 41395.96 KB/s DRBD+GFS (2 nodes): 10884.23 KB/s Average Throughput for Re-Writers per process: EXT3: 91709.05 KB/s DRBD+GFS (2 nodes): 15347.23 KB/s Average Throughput for Readers per process: EXT3: 210302.31 KB/s DRBD+GFS (2 nodes): 5383.27 KB/s <-------- a bit ridiculous And more of the same where basically it can range from being 4x to however many times slower reading was. I can only assume that this would be a garbage setup for Xen VM storage and was wondering if anyone could point me to a solution that may be more promising. We currently are running out of space on our NetApp (that does snapshots for backups) for VMs not to mention the I/O available for multiple VMs on a single NetApp directory is already dangerously low. Anyone have thoughts as to what might solve my problems? I am thinking a few things: - Experiment with DRBD again with another Filesystem (XFS?) and have it re-exported as NFS to both machines (so they can both bring up VMs from the "pool") - Export one of the machines as iSCSI and software raid it on a primary (not really what I want but might work) - Write a custom script that will backup the VM storage directories to a 3rd server (don''t really have the budget for a redundant backup server) using something like rsync And finally, what kind of redundant server to server storage do most people use here? Thanks a lot for reading my novel of a question :) Best, Tait Clarridge _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
El Martes 27 Enero 2009, Tait Clarridge escribió:> Hello Everyone,Hi, I''m no expert but I''m in the same path as you, so let''s try to help each other... and get help from others as we go :)> I recently had a question that got no responses about GFS+DRBD clusters for > Xen VM storage, but after some consideration (and a lot of Googling) I have > a couple of new questions. > > Basically what we have here are two servers that will each have a RAID-5 > array filled up with 5 x 320GB SATA drives, I want to have these as useable > file systems on both servers (as they will both be used for Xen VM storage) > but they will be replicating in the background for disaster recovery > purposes over a GbE link.OK,> First of all, I need to know if this is good practice because I can see a > looming clusterf**k if both machines are running VMs from the same shared > storage location.Well, it shouldn''t happen if you''re using GFS or another cluster aware filesystem.> Second, I ran a test on two identical servers with DRBD and GFS in a > Primary/Primary cluster setup and the performance numbers were appalling > compared to local ext3 storage, for example:Yes, cluster filesystem have lower performance than non-cluster filesystems, due to the former performing lokcs on files/dirs. Add DRBD replication on top of that and performance will be lower.> 5 Concurrent Sessions in iozone gave me the following: > > Average Throughput for Writers per process: > EXT3: 41395.96 KB/s > DRBD+GFS (2 nodes): 10884.23 KB/s > > Average Throughput for Re-Writers per process: > EXT3: 91709.05 KB/s > DRBD+GFS (2 nodes): 15347.23 KB/s > > Average Throughput for Readers per process: > EXT3: 210302.31 KB/s > DRBD+GFS (2 nodes): 5383.27 KB/s <-------- a bit ridiculousRidiculous indeed> And more of the same where basically it can range from being 4x to however > many times slower reading was. I can only assume that this would be a > garbage setup for Xen VM storage and was wondering if anyone could point me > to a solution that may be more promising. We currently are running out of > space on our NetApp (that does snapshots for backups) for VMs not to > mention the I/O available for multiple VMs on a single NetApp directory is > already dangerously low. > > Anyone have thoughts as to what might solve my problems?Have you tried any GFS optimizations? e.g. use noatime and nodiratime, disable gfs quotas, etc. The first two should improve reading performance.> I am thinking a few things: > > - Experiment with DRBD again with another Filesystem (XFS?) and have it > re-exported as NFS to both machines (so they can both bring up VMs from the > "pool")I guess NFS could work, unless you have too many machines using it (Linux''s NFS sucks)> - Export one of the machines as iSCSI and software raid it on a primary (not > really what I want but might work)This one sound interesting.> - Write a custom script that will backup the VM storage directories to a 3rd > server (don''t really have the budget for a redundant backup server) using > something like rsync > > And finally, what kind of redundant server to server storage do most people > use here?From what I''been reading on the list, most people uses some form of DRBD + AoE or iSCSI. Check the thread with subject "disk backend performance" from November 27, 2008. There started a very nice discussion involving Thomas Halinka and Stefan de Konink about AoE vs. iSCSI (thank you both!). Also, the thread with subject "lenny amd64 and xen" will be of your interest, on November 27 Thomas started a description of his self-build SAN which is very insightful.> Thanks a lot for reading my novel of a question :) > > Best, > > Tait ClarridgeBest regards, -- Ricardo J. Barberis Senior SysAdmin - I+D Dattatec.com :: Soluciones de Web Hosting Su Hosting hecho Simple..! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Second, I ran a test on two identical servers with DRBD and GFS in a > Primary/Primary cluster setup and the performance numbers were > appalling compared to local ext3 storage, for example:Well, first off, ask yourself why you''re using a cluster filesystem in the first place. Do you have active-active writers on the same filesystem at the same time? If the answer is no, then get rid of GFS -- you don''t need it. Second, what''s this about DRDB *and* SAN? Are the disks shared (SAN) or DRDB (local RAID)? John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello John, To answer your questions:>Well, first off, ask yourself why you''re using a cluster filesystem in >the first place. Do you have active-active writers on the same >filesystem at the same time? If the answer is no, then get rid of GFS >-- you don''t need it.The reason for this was so that I could have two Xen hosts using their local storage but to also have it replicating between servers for backup purposes. Maybe I was going about this the wrong way, but I wanted to have the ability to use the large storage pool (created by the cluster) as a platform for storing VMs as well as have it back everything up on two separate locations of disk.>Second, what''s this about DRDB *and* SAN? Are the disks shared (SAN) >or DRDB (local RAID)?To clarify, the disks themselves will be a RAID-5 local to each machine (1 array per machine, 2 in total) with DRBD running between to sort of RAID-1 them over the network. Does that help? I want to take the local RAID from both machines and turn it into a SAN. I was wondering whether using DRBD and a cluster filesystem would work better than using XFS+DRBD and exporting NFS. Thanks for the quick response! Best, Tait _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Guys, I did start a thread on this before this one. I''ve been asking about using NAS/SAN, local attached vs FC and Ethernet. I''ve also been asking about clustering, redundancy and have been given a lot of good information, especially from Fajar who sounds like a guru. See Optimizing I/O, Distributed vs Cluster and I can''t recall the other thread now. Mike On Tue, 27 Jan 2009 15:58:37 -0200, Ricardo J. Barberis wrote:> El Martes 27 Enero 2009, Tait Clarridge escribió: > >> Hello Everyone, >> > Hi, I''m no expert but I''m in the same path as you, so let''s try to help each > other... and get help from others as we go :) > >> I recently had a question that got no responses about GFS+DRBD clusters >> for >> Xen VM storage, but after some consideration (and a lot of Googling) I >> have >> a couple of new questions. >> >> Basically what we have here are two servers that will each have a RAID-5 >> array filled up with 5 x 320GB SATA drives, I want to have these as >> useable >> file systems on both servers (as they will both be used for Xen VM >> storage) >> but they will be replicating in the background for disaster recovery >> purposes over a GbE link. >> > OK, > >> First of all, I need to know if this is good practice because I can see a >> looming clusterf**k if both machines are running VMs from the same shared >> storage location. >> > Well, it shouldn''t happen if you''re using GFS or another cluster aware > filesystem. > >> Second, I ran a test on two identical servers with DRBD and GFS in a >> Primary/Primary cluster setup and the performance numbers were appalling >> compared to local ext3 storage, for example: >> > Yes, cluster filesystem have lower performance than non-cluster filesystems, > due to the former performing lokcs on files/dirs. > Add DRBD replication on top of that and performance will be lower. > >> 5 Concurrent Sessions in iozone gave me the following: >> >> Average Throughput for Writers per process: >> EXT3: 41395.96 KB/s >> DRBD+GFS (2 nodes): 10884.23 KB/s >> >> Average Throughput for Re-Writers per process: >> EXT3: 91709.05 KB/s >> DRBD+GFS (2 nodes): 15347.23 KB/s >> >> Average Throughput for Readers per process: >> EXT3: 210302.31 KB/s >> DRBD+GFS (2 nodes): 5383.27 KB/s <-------- a bit ridiculous >> > Ridiculous indeed > >> And more of the same where basically it can range from being 4x to however >> many times slower reading was. I can only assume that this would be a >> garbage setup for Xen VM storage and was wondering if anyone could point >> me >> to a solution that may be more promising. We currently are running out of >> space on our NetApp (that does snapshots for backups) for VMs not to >> mention the I/O available for multiple VMs on a single NetApp directory is >> already dangerously low. >> >> Anyone have thoughts as to what might solve my problems? >> > Have you tried any GFS optimizations? e.g. use noatime and nodiratime, > disable > gfs quotas, etc. The first two should improve reading performance. > >> I am thinking a few things: >> >> - Experiment with DRBD again with another Filesystem (XFS?) and have it >> re-exported as NFS to both machines (so they can both bring up VMs from >> the >> "pool") >> > I guess NFS could work, unless you have too many machines using it (Linux''s > NFS sucks) > >> - Export one of the machines as iSCSI and software raid it on a primary >> (not >> really what I want but might work) >> > This one sound interesting. > >> - Write a custom script that will backup the VM storage directories to a >> 3rd >> server (don''t really have the budget for a redundant backup server) using >> something like rsync >> >> And finally, what kind of redundant server to server storage do most >> people >> use here? >> > From what I''been reading on the list, most people uses some form of DRBD + > AoE > or iSCSI. > > Check the thread with subject "disk backend performance" from November 27, > 2008. There started a very nice discussion involving Thomas Halinka and > Stefan de Konink about AoE vs. iSCSI (thank you both!). > > Also, the thread with subject "lenny amd64 and xen" will be of your > interest, > on November 27 Thomas started a description of his self-build SAN which is > very insightful. > >> Thanks a lot for reading my novel of a question :) >> >> Best, >> >> Tait Clarridge >> > Best regards,_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Mike, I did see that and didn''t want to hijack your thread. However, I didn''t want to add some non-cluster questions into a thread that I thought was predominantly about clustering. Sorry about that, I can try and append my questions into your line of questioning... but sometimes it helps when questions asked and answered provide slightly different responses.. like when trolling Google for responses and a mix from different pages is what solves the problem. Cheers, Tait -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of lists@grounded.net Sent: Tuesday, January 27, 2009 1:31 PM To: xen-users Subject: Re: [Xen-users] Xen SAN Questions Guys, I did start a thread on this before this one. I''ve been asking about using NAS/SAN, local attached vs FC and Ethernet. I''ve also been asking about clustering, redundancy and have been given a lot of good information, especially from Fajar who sounds like a guru. See Optimizing I/O, Distributed vs Cluster and I can''t recall the other thread now. Mike On Tue, 27 Jan 2009 15:58:37 -0200, Ricardo J. Barberis wrote:> El Martes 27 Enero 2009, Tait Clarridge escribió: > >> Hello Everyone, >> > Hi, I''m no expert but I''m in the same path as you, so let''s try to help each > other... and get help from others as we go :) > >> I recently had a question that got no responses about GFS+DRBD clusters >> for >> Xen VM storage, but after some consideration (and a lot of Googling) I >> have >> a couple of new questions. >> >> Basically what we have here are two servers that will each have a RAID-5 >> array filled up with 5 x 320GB SATA drives, I want to have these as >> useable >> file systems on both servers (as they will both be used for Xen VM >> storage) >> but they will be replicating in the background for disaster recovery >> purposes over a GbE link. >> > OK, > >> First of all, I need to know if this is good practice because I can see a >> looming clusterf**k if both machines are running VMs from the same shared >> storage location. >> > Well, it shouldn''t happen if you''re using GFS or another cluster aware > filesystem. > >> Second, I ran a test on two identical servers with DRBD and GFS in a >> Primary/Primary cluster setup and the performance numbers were appalling >> compared to local ext3 storage, for example: >> > Yes, cluster filesystem have lower performance than non-cluster filesystems, > due to the former performing lokcs on files/dirs. > Add DRBD replication on top of that and performance will be lower. > >> 5 Concurrent Sessions in iozone gave me the following: >> >> Average Throughput for Writers per process: >> EXT3: 41395.96 KB/s >> DRBD+GFS (2 nodes): 10884.23 KB/s >> >> Average Throughput for Re-Writers per process: >> EXT3: 91709.05 KB/s >> DRBD+GFS (2 nodes): 15347.23 KB/s >> >> Average Throughput for Readers per process: >> EXT3: 210302.31 KB/s >> DRBD+GFS (2 nodes): 5383.27 KB/s <-------- a bit ridiculous >> > Ridiculous indeed > >> And more of the same where basically it can range from being 4x to however >> many times slower reading was. I can only assume that this would be a >> garbage setup for Xen VM storage and was wondering if anyone could point >> me >> to a solution that may be more promising. We currently are running out of >> space on our NetApp (that does snapshots for backups) for VMs not to >> mention the I/O available for multiple VMs on a single NetApp directory is >> already dangerously low. >> >> Anyone have thoughts as to what might solve my problems? >> > Have you tried any GFS optimizations? e.g. use noatime and nodiratime, > disable > gfs quotas, etc. The first two should improve reading performance. > >> I am thinking a few things: >> >> - Experiment with DRBD again with another Filesystem (XFS?) and have it >> re-exported as NFS to both machines (so they can both bring up VMs from >> the >> "pool") >> > I guess NFS could work, unless you have too many machines using it (Linux''s > NFS sucks) > >> - Export one of the machines as iSCSI and software raid it on a primary >> (not >> really what I want but might work) >> > This one sound interesting. > >> - Write a custom script that will backup the VM storage directories to a >> 3rd >> server (don''t really have the budget for a redundant backup server) using >> something like rsync >> >> And finally, what kind of redundant server to server storage do most >> people >> use here? >> > From what I''been reading on the list, most people uses some form of DRBD + > AoE > or iSCSI. > > Check the thread with subject "disk backend performance" from November 27, > 2008. There started a very nice discussion involving Thomas Halinka and > Stefan de Konink about AoE vs. iSCSI (thank you both!). > > Also, the thread with subject "lenny amd64 and xen" will be of your > interest, > on November 27 Thomas started a description of his self-build SAN which is > very insightful. > >> Thanks a lot for reading my novel of a question :) >> >> Best, >> >> Tait Clarridge >> > Best regards,_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Ricardo, I will do some testing with XFS and report some numbers of performance using both native and DRBD. And to answer some of your responses... El Martes 27 Enero 2009, Tait Clarridge escribió:>> Hello Everyone,>Hi, I''m no expert but I''m in the same path as you, so let''s try to help each >other... and get help from others as we go :)Sounds good man, like I said above I can provide you with some of the figures for the tests I am doing. :)>> First of all, I need to know if this is good practice because I can see a >> looming clusterf**k if both machines are running VMs from the same shared >> storage location.>Well, it shouldn''t happen if you''re using GFS or another cluster aware >filesystem.True, but from what I have seen, the two primary DRBD nodes (that were implemented so I could write actively to both) suffered some brutal split-brains and terrible slowness and aren''t the performance I need.. so effectively I think "clusters" are out of the picture.>> Second, I ran a test on two identical servers with DRBD and GFS in a >> Primary/Primary cluster setup and the performance numbers were appalling >> compared to local ext3 storage, for example:>Yes, cluster filesystem have lower performance than non-cluster filesystems, >due to the former performing lokcs on files/dirs. >Add DRBD replication on top of that and performance will be lower.So maybe I should use DRBD in a non-clustered setup?>> And more of the same where basically it can range from being 4x to however >> many times slower reading was. I can only assume that this would be a >> garbage setup for Xen VM storage and was wondering if anyone could point me >> to a solution that may be more promising. We currently are running out of >> space on our NetApp (that does snapshots for backups) for VMs not to >> mention the I/O available for multiple VMs on a single NetApp directory is >> already dangerously low. >> >> Anyone have thoughts as to what might solve my problems?>Have you tried any GFS optimizations? e.g. use noatime and nodiratime, disable >gfs quotas, etc. The first two should improve reading performance.I used both noatime and nodiratime but I recently destroyed the GFS setup in preparation for testing DRBD+XFS+NFS exporting>> I am thinking a few things: >> >> - Experiment with DRBD again with another Filesystem (XFS?) and have it >> re-exported as NFS to both machines (so they can both bring up VMs from the >> "pool")>I guess NFS could work, unless you have too many machines using it (Linux''s >NFS sucks)There would only be two machines using it with a 3rd if there was some planned maintenance.>> - Export one of the machines as iSCSI and software raid it on a primary (not >> really what I want but might work)>This one sound interesting.I will let you know if the software RAID approach works better than DRBD, but I doubt it will as someone else posted in another thread that since DRBD is essentially optimized for networking stuff.>> - Write a custom script that will backup the VM storage directories to a 3rd >> server (don''t really have the budget for a redundant backup server) using >> something like rsync >> >> And finally, what kind of redundant server to server storage do most people >> use here?>From what I''been reading on the list, most people uses some form of DRBD + AoE >or iSCSI.>Check the thread with subject "disk backend performance" from November 27, >2008. There started a very nice discussion involving Thomas Halinka and >Stefan de Konink about AoE vs. iSCSI (thank you both!).>Also, the thread with subject "lenny amd64 and xen" will be of your interest, >on November 27 Thomas started a description of his self-build SAN which is >very insightful.Alright, I will check those out after I finish some more testing. Thanks a lot for the reply, if I find anything I will report it back here. All the best, Tait Clarridge _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>> Well, first off, ask yourself why you''re using a cluster filesystem in >> the first place. Do you have active-active writers on the same >> filesystem at the same time? If the answer is no, then get rid of GFS >> -- you don''t need it. > > The reason for this was so that I could have two Xen hosts using > their local storage but to also have it replicating between servers > for backup purposes. Maybe I was going about this the wrong way, but > I wanted to have the ability to use the large storage pool (created > by the cluster) as a platform for storing VMs as well as have it > back everything up on two separate locations of disk.I think DRDB will work here (never used it personally), but I think I''d create a DRDB device for each VM and replicate each one separately so you could have a primary on the remote side running a vm that can be started locally if the remote site blows up and vice-versa. You don''t need the added overhead of a cluster filesystem for this. DRDB will certainly add overhead for replicating writes, but this is highly tweakable (consider you''re replication write rate, for example, where maybe you''re writing to the disk at 100MB/s but only replicating at 10KB/s, thus saving a ton of i/o).> To clarify, the disks themselves will be a RAID-5 local to each > machine (1 array per machine, 2 in total) with DRBD running between > to sort of RAID-1 them over the network. Does that help? I want to > take the local RAID from both machines and turn it into a SAN.Cool, that''s fine, but don''t say "SAN" here, they''re different. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks again for the reply.>> The reason for this was so that I could have two Xen hosts using >> their local storage but to also have it replicating between servers >> for backup purposes. Maybe I was going about this the wrong way, but >> I wanted to have the ability to use the large storage pool (created >> by the cluster) as a platform for storing VMs as well as have it >> back everything up on two separate locations of disk.>I think DRDB will work here (never used it personally), but I think >I''d create a DRDB device for each VM and replicate each one separately >so you could have a primary on the remote side running a vm that can >be started locally if the remote site blows up and vice-versa. You >don''t need the added overhead of a cluster filesystem for this. DRDB >will certainly add overhead for replicating writes, but this is highly >tweakable (consider you''re replication write rate, for example, where >maybe you''re writing to the disk at 100MB/s but only replicating at >10KB/s, thus saving a ton of i/o).Alright, however we do already have the images set as raw files and converting some of the systems (which are running in production) to a DRBD devices might prove difficult. It''s not a question of whether we can do that, but it is a good idea (for eliminating the need for clustering). I will definitely have a look in the replication write rate, but wouldn''t a lower rate mean that some of the more I/O intensive VMs will lose data in the event of something going wrong? I am kind of a newb when it comes to replication :) And since everything is in image files, what do you think about exporting the DRBD''d storage as NFS for both machines to read and write? Risky? Inefficient?>> To clarify, the disks themselves will be a RAID-5 local to each >> machine (1 array per machine, 2 in total) with DRBD running between >> to sort of RAID-1 them over the network. Does that help? I want to >> take the local RAID from both machines and turn it into a SAN.>Cool, that''s fine, but don''t say "SAN" here, they''re different.My mistake, sometimes I confuse terms. Best, Tait _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 27 Jan 2009, Tait Clarridge wrote:> To clarify, the disks themselves will be a RAID-5 local to each machine > (1 array per machine, 2 in total) with DRBD running between to sort of > RAID-1 them over the network. Does that help? I want to take the local > RAID from both machines and turn it into a SAN.We get about 166 MB/s write with our DRBD setup, what numbers are you seeing? Our hardware is: Supermicro X7DWN+ Dual Xeon 5410 3ware 9650SE-16ML (in RAID 6 mode) Centos 5.2 Xen 3.3.0 DRBD 2.3.0><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
With the DRBD and GFS (in primary/primary mode) setup I was seeing an average of roughly 55MB/s as opposed to ~206MB/s for EXT3 and ~334 MB/s on XFS... (unless my numbers are horribly wrong). I am running tests on two identical Dell Poweredge 1850''s. Each with a Xeon Dual Core @ 2.8 GHz + 1GB ram (to benchmark) Centos 5.2 Xen 3.1 DRBD 8.2.6 Which program are you benchmarking I/O with? Tait -----Original Message----- From: Nathan Stratton [mailto:nathan@robotics.net] Sent: Tuesday, January 27, 2009 2:47 PM To: Tait Clarridge Cc: John Madden; xen-users@lists.xensource.com Subject: RE: [Xen-users] Xen SAN Questions On Tue, 27 Jan 2009, Tait Clarridge wrote:> To clarify, the disks themselves will be a RAID-5 local to each machine > (1 array per machine, 2 in total) with DRBD running between to sort of > RAID-1 them over the network. Does that help? I want to take the local > RAID from both machines and turn it into a SAN.We get about 166 MB/s write with our DRBD setup, what numbers are you seeing? Our hardware is: Supermicro X7DWN+ Dual Xeon 5410 3ware 9650SE-16ML (in RAID 6 mode) Centos 5.2 Xen 3.3.0 DRBD 2.3.0><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 27 Jan 2009, Tait Clarridge wrote:> With the DRBD and GFS (in primary/primary mode) setup I was seeing an > average of roughly 55MB/s as opposed to ~206MB/s for EXT3 and ~334 MB/s > on XFS... (unless my numbers are horribly wrong).Hmm, what is your DRBD setup? Try the following: no-disk-barrier; no-disk-flushes; no-md-flushes; no-disk-drain;> Which program are you benchmarking I/O with?dd if=/dev/zero of=/share/test bs=1G count=8><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
El Martes 27 Enero 2009, Tait Clarridge escribió:> >> Second, I ran a test on two identical servers with DRBD and GFS in a > >> Primary/Primary cluster setup and the performance numbers were appalling > >> compared to local ext3 storage, for example: > > > >Yes, cluster filesystem have lower performance than non-cluster > > filesystems, due to the former performing lokcs on files/dirs. > >Add DRBD replication on top of that and performance will be lower. > > So maybe I should use DRBD in a non-clustered setup?Someone else should answer this, as I stil had no experience with DRBD, but maybe using it in active-passive mode could improve things, at the expense of having unusable storage (?)> I used both noatime and nodiratime but I recently destroyed the GFS setup > in preparation for testing DRBD+XFS+NFS exportingAlright, I have a GFS partition with no quota for sharing VM config files between the xen hosts. I''ll try to make some benchmarks with and without quota to see the impact on performance.> >> - Experiment with DRBD again with another Filesystem (XFS?) and have it > >> re-exported as NFS to both machines (so they can both bring up VMs from > >> the "pool") > > > >I guess NFS could work, unless you have too many machines using it > > (Linux''s NFS sucks) > > There would only be two machines using it with a 3rd if there was some > planned maintenance.OK, then you shouldn''t have major issues.> >Check the thread with subject "disk backend performance" from November 27, > >2008. There started a very nice discussion involving Thomas Halinka and > >Stefan de Konink about AoE vs. iSCSI (thank you both!). > > > >Also, the thread with subject "lenny amd64 and xen" will be of your > > interest, on November 27 Thomas started a description of his self-build > > SAN which is very insightful. > > Alright, I will check those out after I finish some more testing. > > Thanks a lot for the reply, if I find anything I will report it back here. > > All the best, > Tait ClarridgeCeers, -- Ricardo J. Barberis Senior SysAdmin - I+D Dattatec.com :: Soluciones de Web Hosting Su Hosting hecho Simple..! ------------------------------------------ Nota de confidencialidad: Este mensaje y los archivos adjuntos al mismo son confidenciales, de uso exclusivo para el destinatario del mismo. La divulgación y/o uso del mismo sin autorización por parte de Dattatec.com queda prohibida. Dattatec.com no se hace responsable del mensaje por la falsificación y/o alteración del mismo. De no ser Ud. el destinatario del mismo y lo ha recibido por error, por favor notifique al remitente y elimínelo de su sistema. Confidentiality Note: This message and any attachments (the message) are confidential and intended solely for the addressees. Any unauthorised use or dissemination is prohibited by Dattatec.com. Dattatec.com shall not be liable for the message if altered or falsified. If you are not the intended addressee of this message, please cancel it immediately and inform the sender. Nota de Confidencialidade: Esta mensagem e seus eventuais anexos podem conter dados confidenciais ou privilegiados. Se você os recebeu por engano ou não é um dos destinatários aos quais ela foi endereçada, por favor destrua-a e a todos os seus eventuais anexos ou copias realizadas, imediatamente. É proibida a retenção, distribuição, divulgação ou utilização de quaisquer informações aqui contidas. Por favor, informe-nos sobre o recebimento indevido desta mensagem, retornando-a para o autor. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Well I figured out one thing, that my numbers were totally off for EXT3 and XFS without DRBD haha. I will do some more testing, I can''t believe I ruled out "dd" as a viable benchmark. *smacks forehead* Thanks for the help, I am testing with those DRBD config options now. Tait -----Original Message----- From: Nathan Stratton [mailto:nathan@robotics.net] Sent: Tuesday, January 27, 2009 3:08 PM To: Tait Clarridge Cc: John Madden; xen-users@lists.xensource.com Subject: RE: [Xen-users] Xen SAN Questions On Tue, 27 Jan 2009, Tait Clarridge wrote:> With the DRBD and GFS (in primary/primary mode) setup I was seeing an > average of roughly 55MB/s as opposed to ~206MB/s for EXT3 and ~334 MB/s > on XFS... (unless my numbers are horribly wrong).Hmm, what is your DRBD setup? Try the following: no-disk-barrier; no-disk-flushes; no-md-flushes; no-disk-drain;> Which program are you benchmarking I/O with?dd if=/dev/zero of=/share/test bs=1G count=8><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 28, 2009 at 9:09 PM, Tait Clarridge <Tait.Clarridge@lavalife.com> wrote:> Well I figured out one thing, that my numbers were totally off for EXT3 and XFS without DRBD haha. > > I will do some more testing, I can''t believe I ruled out "dd" as a viable benchmark. > > *smacks forehead* > > Thanks for the help, I am testing with those DRBD config options now.I''d be interested to hear about your results. To tell the truth, I was tempted to try a similar setup. I decided against it though, because : - using local disks provide higher I/O throughput. Using network-attach disks, however, is mostly limited by the network interconnect speed. For example, 1 Gbps network link could only give max (theoretical) throughput of 125 MBps while local disks can easily give 235 MBps (tested with dd) - active-active DRBD setup can produce split-brain So in the end I settled for scheduled zfs-based backup. That is : - when using opensolaris dom0, I can use zvol-backed storage and do backups from dom0 - when using Linux dom0, I use zfs-fuse on domU and perform backups there. Again, I''d be interested to hear about your results. If you can get something like 200 MBps then I''d probably try to implement a similar setup. Hint : You probably want to stay away from GFS as domU''s backend storage. Just use LVM-backed storage (with cLVM, of course) for MUCH faster performance. To measure its performance, a simple way is using dd on the block device. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Just to clarify, which GFS are you talking about? Two exist. On Wed, Jan 28, 2009 at 7:01 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Wed, Jan 28, 2009 at 9:09 PM, Tait Clarridge > <Tait.Clarridge@lavalife.com> wrote: >> Well I figured out one thing, that my numbers were totally off for EXT3 and XFS without DRBD haha. >> >> I will do some more testing, I can''t believe I ruled out "dd" as a viable benchmark. >> >> *smacks forehead* >> >> Thanks for the help, I am testing with those DRBD config options now. > > I''d be interested to hear about your results. > > To tell the truth, I was tempted to try a similar setup. I decided > against it though, because : > - using local disks provide higher I/O throughput. Using > network-attach disks, however, is mostly limited by the network > interconnect speed. For example, 1 Gbps network link could only give > max (theoretical) throughput of 125 MBps while local disks can easily > give 235 MBps (tested with dd) > - active-active DRBD setup can produce split-brain > > So in the end I settled for scheduled zfs-based backup. That is : > - when using opensolaris dom0, I can use zvol-backed storage and do > backups from dom0 > - when using Linux dom0, I use zfs-fuse on domU and perform backups there. > > Again, I''d be interested to hear about your results. If you can get > something like 200 MBps then I''d probably try to implement a similar > setup. > > Hint : You probably want to stay away from GFS as domU''s backend > storage. Just use LVM-backed storage (with cLVM, of course) for MUCH > faster performance. To measure its performance, a simple way is using > dd on the block device. > > Regards, > > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jan 29, 2009 at 11:34 AM, Matthew Sacks <ntwrkd@gmail.com> wrote:> Just to clarify, which GFS are you talking about? > Two exist. >Both. Unless you have some info/benchmark results which says cluster fs (OCFS2, GFS, or GFS2) can perform as well as ext3/block device as domU backend storage? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Well I ran the numbers and made the changes and they helped a little bit, but I am thinking that the non Primary/Primary setup is what really helped. The setup I think I am going to go with is 2 Hardware RAID5 arrays connected using DRBD and formatted as XFS. Once I actually get the arrays setup and connected I will try a DD (I am testing on two servers that are older and much slower than the ones we are going to put into production as our xen hosts) and let you know if the final average speed. Thanks to everyone for their help! Best, Tait Clarridge -----Original Message----- From: Fajar A. Nugraha [mailto:fajar@fajar.net] Sent: Wednesday, January 28, 2009 10:01 AM To: Tait Clarridge Cc: xen-users list Subject: Re: [Xen-users] Xen SAN Questions On Wed, Jan 28, 2009 at 9:09 PM, Tait Clarridge <Tait.Clarridge@lavalife.com> wrote:> Well I figured out one thing, that my numbers were totally off for EXT3 and XFS without DRBD haha. > > I will do some more testing, I can''t believe I ruled out "dd" as a viable benchmark. > > *smacks forehead* > > Thanks for the help, I am testing with those DRBD config options now.I''d be interested to hear about your results. To tell the truth, I was tempted to try a similar setup. I decided against it though, because : - using local disks provide higher I/O throughput. Using network-attach disks, however, is mostly limited by the network interconnect speed. For example, 1 Gbps network link could only give max (theoretical) throughput of 125 MBps while local disks can easily give 235 MBps (tested with dd) - active-active DRBD setup can produce split-brain So in the end I settled for scheduled zfs-based backup. That is : - when using opensolaris dom0, I can use zvol-backed storage and do backups from dom0 - when using Linux dom0, I use zfs-fuse on domU and perform backups there. Again, I''d be interested to hear about your results. If you can get something like 200 MBps then I''d probably try to implement a similar setup. Hint : You probably want to stay away from GFS as domU''s backend storage. Just use LVM-backed storage (with cLVM, of course) for MUCH faster performance. To measure its performance, a simple way is using dd on the block device. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Once I actually get the arrays setup and connected I will try a DD > (I am testing on two servers that are older and much slower than the > ones we are going to put into production as our xen hosts) and let > you know if the final average speed.Keep in mind that dd is fine for testing raw sequential i/o but that almost nothing in the real world works this way. (How often do you say to yourself "well, I need to go ahead and create an empty 16GB file?") John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
True, so would running IOZONE tests be better? Because I have done a lot of that on these setups as well. -----Original Message----- From: John Madden [mailto:jmadden@ivytech.edu] Sent: Thursday, January 29, 2009 9:45 AM To: Tait Clarridge Cc: Fajar A. Nugraha; xen-users list Subject: RE: [Xen-users] Xen SAN Questions> Once I actually get the arrays setup and connected I will try a DD > (I am testing on two servers that are older and much slower than the > ones we are going to put into production as our xen hosts) and let > you know if the final average speed.Keep in mind that dd is fine for testing raw sequential i/o but that almost nothing in the real world works this way. (How often do you say to yourself "well, I need to go ahead and create an empty 16GB file?") John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> True, so would running IOZONE tests be better? Because I have done a > lot of that on these setups as well.It''s better, at least and does seem to be a reasonable de facto test. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users