Hello all,
we have a new setup with xen on centos5.3
I run drbd from lvm volumes to mirror data between the two servers.
both servers are 1U nec rack mounts with 8GB RAM, 2x mirrored 1TB
seagate satas.
The one is a dual core xeon, and the other a quad-core xeon.
I have a gigabit crossover link between the two with an MTU of 9000 on
each end.
I currently have 6 drbds mirroring across that link.
The highest speed I can get through that link with drbd is 11 MB/sec
(megabytes)
But if I copy a 1 gig file over that link I get 110 MB/sec.
Why is DRBD so slow?
I am not using drbd encryption because of the back to back link.
Here is a part of my drbd config:
# cat /etc/drbd.conf
global {
usage-count yes;
}
common {
protocol C;
syncer { rate 80M; }
net {
allow-two-primaries;
}
}
resource xenotrs {
device /dev/drbd6;
disk /dev/vg0/xenotrs;
meta-disk internal;
on baldur.somedomain.local {
address 10.99.99.1:7793;
}
on thor.somedomain.local {
address 10.99.99.2:7793;
}
}
Kind regards,
Coert
On Wed, 2009-07-22 at 11:16 +0200, Coert Waagmeester wrote:> Hello all, > > we have a new setup with xen on centos5.3 > > I run drbd from lvm volumes to mirror data between the two servers. > > both servers are 1U nec rack mounts with 8GB RAM, 2x mirrored 1TB > seagate satas. > > The one is a dual core xeon, and the other a quad-core xeon. > > I have a gigabit crossover link between the two with an MTU of 9000 on > each end. > > I currently have 6 drbds mirroring across that link. > > The highest speed I can get through that link with drbd is 11 MB/sec > (megabytes) > > But if I copy a 1 gig file over that link I get 110 MB/sec. > > Why is DRBD so slow? > > I am not using drbd encryption because of the back to back link. > Here is a part of my drbd config: > > # cat /etc/drbd.conf > global { > usage-count yes; > } > common { > protocol C; > syncer { rate 80M; } > net { > allow-two-primaries; > } > } > resource xenotrs { > device /dev/drbd6; > disk /dev/vg0/xenotrs; > meta-disk internal; > > on baldur.somedomain.local { > address 10.99.99.1:7793; > } > on thor.somedomain.local { > address 10.99.99.2:7793; > } > } > > > Kind regards, > Coert > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centosI am reading up on this on the internet as well, but all the tcp settings and disk settings make me slightly nervous...
On Jul 22, 2009, at 5:16 AM, Coert Waagmeester <lgroups at waagmeester.co.za > wrote:> Hello all, > > we have a new setup with xen on centos5.3 > > I run drbd from lvm volumes to mirror data between the two servers. > > both servers are 1U nec rack mounts with 8GB RAM, 2x mirrored 1TB > seagate satas. > > The one is a dual core xeon, and the other a quad-core xeon. > > I have a gigabit crossover link between the two with an MTU of 9000 on > each end. > > I currently have 6 drbds mirroring across that link. > > The highest speed I can get through that link with drbd is 11 MB/sec > (megabytes) > > But if I copy a 1 gig file over that link I get 110 MB/sec. > > Why is DRBD so slow? > > I am not using drbd encryption because of the back to back link. > Here is a part of my drbd config: > > # cat /etc/drbd.conf > global { > usage-count yes; > } > common { > protocol C; > syncer { rate 80M; } > net { > allow-two-primaries; > } > } > resource xenotrs { > device /dev/drbd6; > disk /dev/vg0/xenotrs; > meta-disk internal; > > on baldur.somedomain.local { > address 10.99.99.1:7793; > } > on thor.somedomain.local { > address 10.99.99.2:7793; > } > }Use iperf to measure the bandwidth/latency on those nics between the two hosts. If iperf comes back clean use dd with the oflag=direct to test the performance of the drives on both sides (create a test LV). Roughly you can multiply with the max number of outstanding I/Os your application does to get a real number, use 4 if you don't know what that is. DRBD protocol C is completely synchronous and won't return a write until it has been committed to disk on both sides. Having disk controllers with nvram cache can make all the difference in the world for this setup. -Ross
On Wed, 2009-07-22 at 11:16 +0200, Coert Waagmeester wrote:> The highest speed I can get through that link with drbd is 11 MB/sec > (megabytes)Not good...> But if I copy a 1 gig file over that link I get 110 MB/sec.That tells me that the network connection is fine. The issue is at a higher layer...> Why is DRBD so slow?Let's see...> common { > protocol C; > syncer { rate 80M; } > net { > allow-two-primaries; > } > }You want allow-two-primaries? That implies that you're using something like ocfs2, but that's probably immaterial to the discussion... Here's a question - do you have another syncer statement in the resource definition that's set to a lower number? That would definitely throttle the sync rate... -I