Xen users - May 2011 - Alternatives to a dual-primary DRBD setup

Hi there,

usually I am just lurking around here collecting ideas about issues that 
I could run into in the future but now it seems my time has come to ask 
for your help...

Until now I ran a Citrix Xenserver 5.6 on a dual-primary DRBD setup. But 
for any reason I am experiencing sync problems now, maybe due to the 
raised I/O load (setup without guest machines, now ten virtual servers).

I am tapping a bit in darkness at searching for the needle in the 
haystack causing the problems. Therefore I am thinking about possible 
alternatives that allow useful things like XenMotion but possibly at a 
lower performance usage level.

My current setup:
- two servers with dual 12-core Opterons and 32 GB RAM, each
- each server equipped with a 500 MB (hardware) RAID1 for the host and 
with a 14 TB (hardware) RAID5 for the guests that''s being replicated 
using DRBD
- each server with a 10 GbE NIC
- Citrix Xenserver 5.6, both running in one pool

What are your ideas? Is it worth thinking about other storage solutions?

CU,
Mészi.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 05/20/11 20:51, Daniel Meszaros wrote:
> Hi there,
>
> usually I am just lurking around here collecting ideas about issues 
> that I could run into in the future but now it seems my time has come 
> to ask for your help...
>
> Until now I ran a Citrix Xenserver 5.6 on a dual-primary DRBD setup. 
> But for any reason I am experiencing sync problems now, maybe due to 
> the raised I/O load (setup without guest machines, now ten virtual 
> servers).
>
> I am tapping a bit in darkness at searching for the needle in the 
> haystack causing the problems. Therefore I am thinking about possible 
> alternatives that allow useful things like XenMotion but possibly at a 
> lower performance usage level.
>
> My current setup:
> - two servers with dual 12-core Opterons and 32 GB RAM, each
> - each server equipped with a 500 MB (hardware) RAID1 for the host and 
> with a 14 TB (hardware) RAID5 for the guests that''s being
replicated
> using DRBD
> - each server with a 10 GbE NIC
> - Citrix Xenserver 5.6, both running in one pool
>
> What are your ideas? Is it worth thinking about other storage solutions?
>
> CU,
> Mészi.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
To me it sounds your setup should be fine. It does depends on the nature 
of your guests of course. 10 is not a lot though.

I would try to find the bottleneck:

- do an effective measurement of the DRBD link with netperf
- investigate none of your guests is swapping heavily consuming a lot of I/O
- investigate your disk performance. RAID5 is not ideal, more striping 
would be better.

I don''t think you''re very likely to have better performance
with other
setups, like iSCSI.

.B


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> Hi there,
> 
> usually I am just lurking around here collecting ideas about issues
that
> I could run into in the future but now it seems my time has come to
ask
> for your help...
> 
> Until now I ran a Citrix Xenserver 5.6 on a dual-primary DRBD setup.
But
> for any reason I am experiencing sync problems now, maybe due to the
> raised I/O load (setup without guest machines, now ten virtual
servers).
> 
> I am tapping a bit in darkness at searching for the needle in the
> haystack causing the problems. Therefore I am thinking about possible
> alternatives that allow useful things like XenMotion but possibly at a
> lower performance usage level.
> 
> My current setup:
> - two servers with dual 12-core Opterons and 32 GB RAM, each
> - each server equipped with a 500 MB (hardware) RAID1 for the host and
> with a 14 TB (hardware) RAID5 for the guests that''s being
replicated
> using DRBD
> - each server with a 10 GbE NIC
> - Citrix Xenserver 5.6, both running in one pool
> 
> What are your ideas? Is it worth thinking about other storage
solutions?
> 
I would consider splitting up the storage and VM functions onto separate
servers. So 2 servers running DRBD and exporting LUN''s via iSCSI (or
AoE
or FCoE or whatever works for you) and then 2 (or more) servers running
VMs. This obviously needs more hardware though.

Having said that, can you elaborate on the "sync issues" you are
having
with DRBD? Anything that breaks under high load is still IMHO a bug and
can probably also break under low load, just less often, Are you running
the latest version?

James

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi!

Am 05/21/2011 02:16 AM, schrieb James Harper:
> I would consider splitting up the storage and VM functions onto separate
> servers. So 2 servers running DRBD and exporting LUN''s via iSCSI
(or AoE
> or FCoE or whatever works for you) and then 2 (or more) servers running
> VMs. This obviously needs more hardware though.
Unfortunately this is not possible at the moment due to limited budget. 
Besides this I expect other performance issues when introducing another 
layer like iSCSI or AoE to the game.

What I could do is to take for instance two harddisks into an own RAID1 
being DRBD-replicated parallelly to the data but I guess this would make 
no difference to my current situation in the end.

I also thought about leaving the DRBD idea behind and to setup two 
independent servers, one holding the system backups and one being the 
productive machine if my systems couldn''t cope with the DRBD thing. :-/

Another idea is to put the virtual systems onto the harddrive that the 
Xenserver host resides on (and not replicating them on-the-fly) and just 
leaving data on the DRBD-synced RAID5s.

The last both ideas would make XenMotion impossible which was the reason 
why I decided to run the Citrix solution. In that case I could possibly 
also run some newer (= better?) Xen version, i.e. under Debian Squeeze. :-/

Or some other idea that hasn''t yet been considered...

> Having said that, can you elaborate on the "sync issues" you are
having
> with DRBD? Anything that breaks under high load is still IMHO a bug and
> can probably also break under low load, just less often, Are you running
> the latest version?
No, it isn''t the latest DRBD version (= 8.3.8.1). Also my Citrix 
Xenserver isn''t latest. It is still 5.6 without any FP or SP.

When I set up the virtualization system I saw everything running fine 
but the host machines weren''t filled with virtual guests yet, 
unfortunately. It must be the (raised) amount of machines having reached 
an I/O performance consumption level that somewhen started concurring 
with the DRBD sync.

At least no system or data got harmed yet ... *knock knock* ... but
it''s
not redundant at the moment. :-(

Currently when running a benchmark from one dom0 to the other (with 
"netio") I get throughput rates of > 200 MByte/s. And this is the
speed
that I used to have with DRBD, too. But as soon as I start syncing from 
the "younger" primary to the other DRBD machine my virtual
guests'' logs
show timeout error messages.

In my investigations I read that a 10 GbE NIC can cause high CPU loads 
when not being configured properly. Is there possibly a way to reserve 
more of the CPU time and/or RAM for the Xen host/ dom0? I have plenty of 
CPU cores in the hosts and RAM is also far from being exceeded, still.

CU,
Daniel.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello once again,

the more I think about my DRBD issue and the more I research in the net 
the more I tend to explain the issue with limited CPU time for dom0. It 
will be better resolving _this_ instead of possibly reaching the same 
stage later on again but using another replication technique, wouldn''t
it?

Reasons why I think it is an I/O and/or CPU time issue:

1. It worked properly when I still did not have 8 virtual guest systems 
installed.
2. As soon as I start a DRBD resync my virtual guests bring kernel error 
messages like "INFO: task exim4:2336 blocked for more than 120 seconds.
".
3. When starting both Xenserver machines and syncing before starting the 
virtual guests a startup that''s usually done in <5 minutes takes up
to
60 minutes.

I checked the XenWiki accordingly and found two promising entries that 
I''d like to follow, if it''s possible to apply them under a
Citrix
Xenserver 5.6 system:

http://wiki.xensource.com/xenwiki/XenCommonProblems#head-413e1d74442772fd5a0a94f0655a009744096627

1. How can I limit the number of vcpus my dom0 has?
2. Can I dedicate a cpu core (or cores) only for dom0?

Especially the 2nd one appears to meet what I expect. So I would be 
going to check if I can configure that. How do _you_ think about it?

CU,
Daniel.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 23, 2011 at 2:46 PM, Daniel Brockmann <meszi@musiqus.org>
wrote:
> Hello once again,
>
> the more I think about my DRBD issue and the more I research in the net the
> more I tend to explain the issue with limited CPU time for dom0.
First thing first.
By "sync problems" in your previous post, did you mean both nodes
experience split brain for the drbd resource?

When setup properly, you should NOT experience it, regardless of how
much CPU resource dom0 has. You should only experience SLOW disk I/O.
split brain usually occur if you don''t setup fencing properly.
> It will be
> better resolving _this_ instead of possibly reaching the same stage later
on
> again but using another replication technique, wouldn''t it?
>
> Reasons why I think it is an I/O and/or CPU time issue:
>
> 1. It worked properly when I still did not have 8 virtual guest systems
> installed.
> 2. As soon as I start a DRBD resync my virtual guests bring kernel error
> messages like "INFO: task exim4:2336 blocked for more than 120
seconds. ".
> 3. When starting both Xenserver machines and syncing before starting the
> virtual guests a startup that''s usually done in <5 minutes
takes up to 60
> minutes.
... which is exactly the SLOW I/O I mentioned above.
>
> I checked the XenWiki accordingly and found two promising entries that
I''d
> like to follow, if it''s possible to apply them under a Citrix
Xenserver 5.6
> system:
>
>
http://wiki.xensource.com/xenwiki/XenCommonProblems#head-413e1d74442772fd5a0a94f0655a009744096627
>
> 1. How can I limit the number of vcpus my dom0 has?
> 2. Can I dedicate a cpu core (or cores) only for dom0?
>
> Especially the 2nd one appears to meet what I expect. So I would be going
to
> check if I can configure that. How do _you_ think about it?
This thread might be able to help you:
http://xen.1045712.n5.nabble.com/XenServer-adding-additional-CPUs-to-the-control-domain-td3325379.html


Personally, I suggest you step back and evaluate several things:
- do you REALLY need active-active setup?
Active-active drbd mandates protocol C (sync replication), which can
GREATLY slow down your throughput. If you can afford a small downtime
better stick with async replication.

- do you KNOW how much IOPS you need?
Disk IOPS is especially important since it''s usually the bottleneck in
virtualized environment. For example, a big time virtualization
provider that I know of use 30 IOPS per VM for sizing purposes (the
assumption is that not all VMs will be IO-intensive, so they use a low
number like 30 for simplifaction purposes). Then they multiply it by
the number of VM, and use sizing tool from storage-appliance-vendor to
calculate the number and type of disk required. Of course if you know
that your VM will be IO-intensive (e.g busy mail server), the
asumption above will not be valid for you, and you need to adjust it
to something higher.

- do you HAVE the necessary resource to support IOPS and replication
requirement?
For example, let''s say you use 30 IOPS per vm number above, and you
have 20 VM per host. So you need 30*20 = 600 IOPS. Lets assume one
7200 rpm disk can support 100 IOPS, so you need a MINIMUM of 6 disk
(if you use raid0) or 12 disk (if you use raid10). Then assume
active-active DRBD will make performance drop by 75%, so you''ll need
12 * 4 = 48 disk in raid10. Do you have that?

All things considered, it might be that your best option would be
something like:
- get a separate server with lots of disks, setup raid10, install a
storage appliance OS on top (e.g.
http://www.napp-it.org/index_en.html) then export it to your xenserver
either as nfs or scsi. While nfs/scsci induce some overhead, it should
be lower compared to using drbd, OR
- drop active-active requirement, OR
- beef-up your xenserver (e.g. use fast storage like SSD), upgrade
XenServer/XCP version to get dom0 to use multiple CPU core on dom0,
upgrade DRBD version to the latest, and setup proper fencing.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

A good rule of thumb is to use jumbo frames, that will get you less cpu
usage, although if you saw you don''t have resource issues nowadays,
DRBD
shouldn''t break using regular MTU.

Regards,
Ciro
El may 21, 2011 8:10 a.m., "Daniel Meszaros" <meszi@musiqus.org>
escribió:


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi!

Sorry for not answering you. I just found your email as a false-positive 
of my SPAM filter. :-(

Am 05/20/2011 09:28 PM, schrieb Bart Coninckx:
> To me it sounds your setup should be fine. It does depends on the nature
> of your guests of course. 10 is not a lot though.
>
> I would try to find the bottleneck:
>
> - do an effective measurement of the DRBD link with netperf
> - investigate none of your guests is swapping heavily consuming a lot of
> I/O
> - investigate your disk performance. RAID5 is not ideal, more striping
> would be better.
>
> I don''t think you''re very likely to have better
performance with other
> setups, like iSCSI.
At the moment I am moving away from Xen as it appears to limit the 
resources I expect to be needed a bit too much, especially Xenserver 
that just reserves up to four VCPUs for dom0.

In the meantime I did a test install of Xenserver 5.6 SP2 and compiled 
and installed the latest version of DRBD on the asynchronous partner of 
the DRBD pair. And it froze already when trying to create the Xenserver 
storage on the DRBD device - in that moment it wasn''t yet syncing 
towards the other machine at all.

Now I have Debian Squeeze with KVM running and it runs fine but still 
Standalone-DRBD as I am still migrating my machines over to KVM (mostly 
installing from scratch and migrating the data) ... I am gonna switch 
the other machine to the same setup ... keeping my fingers crossed when 
it''s gonna start the DRBD sync again.

Anyway: This is already a lot more than what I got with my Xenserver 5.6 
SP2 attempt, even if I would not have a primary-primary DRBD setup in 
the end.

Another change in the new installation is that I switched over to RAID6 
instead of RAID5 before but I guess this should not make a huge 
difference in means of performance.

I tend to explain the results with a far better support for my 
components due to a newer kernel and less limitations (compared with 
Citrix variant of CentOS in Xenserver).

CU,
Mészi.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi!

I am sorry for not answering. My SPAM filter wasn''t your friend and I 
missed to check it in time... :-(

Am 05/23/2011 10:28 AM, schrieb Fajar A. Nugraha:
> On Mon, May 23, 2011 at 2:46 PM, Daniel Brockmann wrote:
>>
>> the more I think about my DRBD issue and the more I research in the net
the
>> more I tend to explain the issue with limited CPU time for dom0.
>
> First thing first.
> By "sync problems" in your previous post, did you mean both nodes
> experience split brain for the drbd resource?
One machine -luckily that one that didn''t have any important machines 
running- lost sync so the other machine was the "newer" DRBD partner. 
But when starting the sync again all virtual guests of the "newer"
DRBD
partner weren''t available anymore until I stopped the sync forcefully. 
Besides this the sync speed was far below that what I had when I set up 
the DRBD pair on both Xenserver machines.

> When setup properly, you should NOT experience it, regardless of how
> much CPU resource dom0 has. You should only experience SLOW disk I/O.
> split brain usually occur if you don''t setup fencing properly.
The funny thing: It worked fine in the beginning. I was playing around 
with Xenmotion with a productive mail server machine without any 
problems. It continuously synced with rates between 250 and 350 MByte/s 
which is more than okay for our needs.

>> It will be
>> better resolving _this_ instead of possibly reaching the same stage
later on
>> again but using another replication technique, wouldn''t it?
>>
>> Reasons why I think it is an I/O and/or CPU time issue:
>>
>> 1. It worked properly when I still did not have 8 virtual guest systems
>> installed.
>> 2. As soon as I start a DRBD resync my virtual guests bring kernel
error
>> messages like "INFO: task exim4:2336 blocked for more than 120
seconds. ".
>> 3. When starting both Xenserver machines and syncing before starting
the
>> virtual guests a startup that''s usually done in<5 minutes
takes up to 60
>> minutes.
>
> ... which is exactly the SLOW I/O I mentioned above.
What I cannot understand is why this was occuring weeks after the setup 
was tested successfully. :-/

Okay, one thing changed: A few machines were added to the virtual pool. 
But even when these machines have been halted I could not sync the DRBD 
pair anymore.

>> I checked the XenWiki accordingly and found two promising entries that
I''d
>> like to follow, if it''s possible to apply them under a Citrix
Xenserver 5.6
>> system:
>>
>>
http://wiki.xensource.com/xenwiki/XenCommonProblems#head-413e1d74442772fd5a0a94f0655a009744096627
>>
>> 1. How can I limit the number of vcpus my dom0 has?
>> 2. Can I dedicate a cpu core (or cores) only for dom0?
>>
>> Especially the 2nd one appears to meet what I expect. So I would be
going to
>> check if I can configure that. How do _you_ think about it?
>
> This thread might be able to help you:
>
http://xen.1045712.n5.nabble.com/XenServer-adding-additional-CPUs-to-the-control-domain-td3325379.html
I tried it differently then by installing the DRBD "loser" with 
Xenserver 5.6 SP2 that reserves 4 VCPUs to dom0. I additionally cut 4 GB 
RAM away for dom0. And I compiled DRBD from the latest sources this time 
(on the old systems I took precompiled binaries). But this time I 
couldn''t even create a Xenserver storage on the DRBD device.

> Personally, I suggest you step back and evaluate several things:
> - do you REALLY need active-active setup?
> Active-active drbd mandates protocol C (sync replication), which can
> GREATLY slow down your throughput. If you can afford a small downtime
> better stick with async replication.
I could afford it in the worst case. And I will keep your suggestion in 
mind for the case my alternative tests appears to fail again.

Actually I run Debian Squeeze on the former DRBD "loser" with a 
still-not-syncing DRBD device. However, now I try KVM ... and it appears 
to run at least as fine as my Xenserver setup before ... let''s see 
what''s gonna happen in the moment when I kill the Xenserver machine in 
order to be installed with Squeeze, too ... and start syncing both.

(In this moment I wished to have more than just two hosts.)

> - do you KNOW how much IOPS you need?
> [...]
Hmm, It''s not gonna be a lot more than the machines I have. Most 
performance-consuming would be a virtualized Samba machine, I guess. 
I''ll check that next week when I am at the machines again. Thanks for 
the suggestion.

> All things considered, it might be that your best option would be
> something like:
> - get a separate server with lots of disks, setup raid10, install a
> storage appliance OS on top (e.g.
> http://www.napp-it.org/index_en.html) then export it to your xenserver
> either as nfs or scsi. While nfs/scsci induce some overhead, it should
> be lower compared to using drbd, OR
Hmm, my experience with NFS is that it''s a lot slower than DRBD - 
however, ignoring the syncer rates I had lately. ;-)

iSCSI would be an option if I had the budget to have that separate 
storage server. But I unfortunately don''t have it. :-(

> - drop active-active requirement, OR
Dual-primary worked for a while. I want to check at first if I can get 
that running again the way it was (but using a different platform). But 
if it was the only way I could live with a "usual" DRBD setup, too.
:-)

> - beef-up your xenserver (e.g. use fast storage like SSD), upgrade
> XenServer/XCP version to get dom0 to use multiple CPU core on dom0,
> upgrade DRBD version to the latest, and setup proper fencing.
I am not that convenient anymore if Xenserver and my MagnyCours CPUs 
work well together at all. That''s why I tried Squeeze and KVM now ... 
shipping around the limitations that Citrix included in their product.

Thank you very much for your suggestions. Regardless if it''s KVM or Xen
... it will help me a lot.

CU,
Mészi.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I tend to explain the results with a far better support for my
> components due to a newer kernel and less limitations (compared with
> Citrix variant of CentOS in Xenserver).
FWIW, I''d never recommend Xenserver to anyone who wants Xen in the same
way I''d never recommend a scooter to someone who wants to ride a
motorcycle.

John



-- 
John Madden
Sr UNIX Systems Engineer / Office of Technology
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users