Hi, I want to put it all together. So far, I have a running machine with multiple domUs, and I am very excited. But now I want to go a step ahead: I have two identical machines with a big amount of RAM and disk space. My goal is to hve all the domUs almost redundant - and want to ask you for your advice. The plan is to have every VM in its own logical volume and have DRBD replicate this volume to the other machine. Assumed Linux with Linux-HA runs in a VM, here comes my question: If the complete Linux VM runs in its own logical volume, Linux-HA cannot work, since the configuration must not be the same on both sides. One solution would be to have a minor file based backend that holds /etc/, /proc/, /var/, /boot/ (and maybe more?). Or maybe there is a more sophisticated solution? How would you start? Thanks, Rainer _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rainer Sokoll <rainer@sokoll.com> wrote:> > I want to put it all together. > So far, I have a running machine with multiple domUs, and I am very > excited. But now I want to go a step ahead: > I have two identical machines with a big amount of RAM and disk space. > My goal is to hve all the domUs almost redundant - and want to ask you > for your advice. > The plan is to have every VM in its own logical volume and have DRBD > replicate this volume to the other machine. Assumed Linux with > Linux-HA runs in a VM, here comes my question: > If the complete Linux VM runs in its own logical volume, Linux-HA > cannot work, since the configuration must not be the same on both > sides. One solution would be to have a minor file based backend that > holds /etc/, /proc/, /var/, /boot/ (and maybe more?). Or maybe there > is a more sophisticated solution? > How would you start?I''ve developing quite the same solution :) Machine A will have two separate disks - one for ''local'' domUs, second for machine B domUs replicas). Machine B will also have two disk drives - one for local domUs and second as A domUs secondary device. Right now I''m probably in the same place you are (I''m also very excited :]) but I hope all this is possible due to the fact, that block device names are exported to domU as virtual names. So if Dom1a uses /dev/vg0/dom1vda as ''xvda'' on machine A, there is no problem that the same Dom1a would use /dev/vg1/dom1vda as ''xvda'' on machine B. That''s just a difference of /etc/xen/dom1a.cfg on dom0 @A and dom0 @ B, which probaby should be sligtly different anyway (percheps less vcpsu and less memeory). Some script that notices crased Dom1a @ boxA could not only re-create Dom1A @ boxB, but also execute some other commads like: xm vcpu-set Dom1b ... and xm mem-set Dom1b ... to temporarly lower regular recource usage of Dom1b @ boxB. The only issue I worry about right now is a disk performance, i.e. how much do I drop of my current ~100MB/s decent speed (ext3 on LVM2 on S/W Raid10,f2 on 2xSAS SFF 10K) putting somewhere into all that one more layer - DRDB. -- Tomek _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, maybe this link is helpful for you (German how-to of a Xen, DRDB, Heartbeat configuration) : http://www.pro-linux.de/work/virtual-ha/virtual-ha5.html Regards, Andreas 2008/4/22, Rainer Sokoll <rainer@sokoll.com>:> > Hi, > > I want to put it all together. > So far, I have a running machine with multiple domUs, and I am very > excited. But now I want to go a step ahead: > I have two identical machines with a big amount of RAM and disk space. > My goal is to hve all the domUs almost redundant - and want to ask you > for your advice. > The plan is to have every VM in its own logical volume and have DRBD > replicate this volume to the other machine. Assumed Linux with Linux-HA > runs in a VM, here comes my question: > If the complete Linux VM runs in its own logical volume, Linux-HA cannot > work, since the configuration must not be the same on both sides. One > solution would be to have a minor file based backend that holds /etc/, > /proc/, /var/, /boot/ (and maybe more?). Or maybe there is a more > sophisticated solution? > How would you start? > > Thanks, > Rainer > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tomasz Nowak wrote:> Rainer Sokoll <rainer@sokoll.com> wrote: >> >> I want to put it all together. >> So far, I have a running machine with multiple domUs, and I am very >> excited. But now I want to go a step ahead: >> I have two identical machines with a big amount of RAM and disk space. >> My goal is to hve all the domUs almost redundant - and want to ask you >> for your advice. >> The plan is to have every VM in its own logical volume and have DRBD >> replicate this volume to the other machine. Assumed Linux with >> Linux-HA runs in a VM, here comes my question: >> If the complete Linux VM runs in its own logical volume, Linux-HA >> cannot work, since the configuration must not be the same on both >> sides. One solution would be to have a minor file based backend that >> holds /etc/, /proc/, /var/, /boot/ (and maybe more?). Or maybe there >> is a more sophisticated solution? >> How would you start?Overall, what you''re really trying to create is something like a live CD or NFS boot environment, both of which are fairly well documented and worked out for you. Why not look into these as a starting point?> I''ve developing quite the same solution :) > Machine A will have two separate disks - one for ''local'' domUs, second > for machine B domUs replicas). Machine B will also have two disk > drives - one for local domUs and second as A domUs secondary device. > > Right now I''m probably in the same place you are (I''m also very > excited :]) but I hope all this is possible due to the fact, > that block device names are exported to domU as virtual names. > > So if Dom1a uses /dev/vg0/dom1vda as ''xvda'' on machine A, > there is no problem that the same Dom1a would use > /dev/vg1/dom1vda as ''xvda'' on machine B.Hold it. The contents of /dev/ are not usually part of a local file system these days. They''re usually ''udev'' provided, at least on Linux, and get auto-regenerated and mounted as a special type of file system at boot time. That said, distinguishing between different file systems to mount is often vastly simplified by using NFS mounting over a local network file system, rather than playing around in the Xen configurations. That makes it easier to reconfigure, dynamically, as well. And if you run autofs, you can symlink from /net/[server]/[exportdir] to your local designated target, such as /var/log/httpd or /usr/local or such. Backup needs to be done with caution for various reasons, but it''s workable.> That''s just > a difference of /etc/xen/dom1a.cfg on dom0 @A and dom0 @ B, > which probaby should be sligtly different anyway (percheps > less vcpsu and less memeory). Some script that notices > crased Dom1a @ boxA could not only re-create Dom1A @ boxB, > but also execute some other commads like: > xm vcpu-set Dom1b ... and xm mem-set Dom1b ... > to temporarly lower regular recource usage of Dom1b @ boxB. > > The only issue I worry about right now is a disk performance, > i.e. how much do I drop of my current ~100MB/s decent speed > (ext3 on LVM2 on S/W Raid10,f2 on 2xSAS SFF 10K) > putting somewhere into all that one more layer - DRDB. >Good question. I''ve not played with DRBD, and would also welcome information on it. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rainer Sokoll <rainer@sokoll.com> writes:> Hi, > > I want to put it all together. > So far, I have a running machine with multiple domUs, and I am very > excited. But now I want to go a step ahead: > I have two identical machines with a big amount of RAM and disk space. > My goal is to hve all the domUs almost redundant - and want to ask you > for your advice. > The plan is to have every VM in its own logical volume and have DRBD > replicate this volume to the other machine. Assumed Linux with Linux-HA > runs in a VM, here comes my question: > If the complete Linux VM runs in its own logical volume, Linux-HA cannot > work, since the configuration must not be the same on both sides. One > solution would be to have a minor file based backend that holds /etc/, > /proc/, /var/, /boot/ (and maybe more?). Or maybe there is a more > sophisticated solution? > How would you start?I don''t really understand what exactly do you want to make redundant? We have similar setup to yours, both cluster nodes simply have DRBD devices in LVs for ease of administration and inside we have regular disk images. we sync the domU config files and the rest of the junk between nodes using csync2. before you undertake this adventure, be aware that xen is currently not mature enough for such a setup, we had machines (both domUs and dom0s) crashing regularly when having drbd in primary/primary and drbd in primary/secondary simply does not work without heavy patching, which is simply not worth the trouble. right now we settled with quasi-migration, we save the domU, rsync the memory file over to the other machine, change drbd setup to primary and restore the machine and that more or less works ok with cca 10s downtime. if you only want your domUs to come up on the other node if a node goes down, this might not be an issue though. We wanted to be able to migrate domUs between the nodes to balance the load. -- To sto si frustriran, zavidan tko zna na cemu i sto ne vidis dalje od svoje guzice je tuzno. Da onda barem imas toliko samokontrole da sutis umjesto da pravis budalu od sebe... izgleda da si prestar da se promjenis na bolje. - Davor Pasaric, hr.comp.mac _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nico Kadel-Garcia <nkadel@gmail.com> wrote:> > > I''ve developing quite the same solution :) > > Machine A will have two separate disks - one for ''local'' domUs, > > second for machine B domUs replicas). Machine B will also have two > > disk drives - one for local domUs and second as A domUs secondary > > device. Right now I''m probably in the same place you are (I''m also > > very > > excited :]) but I hope all this is possible due to the fact, > > that block device names are exported to domU as virtual names. > > > > So if Dom1a uses /dev/vg0/dom1vda as ''xvda'' on machine A, > > there is no problem that the same Dom1a would use > > /dev/vg1/dom1vda as ''xvda'' on machine B. > Hold it. The contents of /dev/ are not usually part of a local file > system these days. They''re usually ''udev'' provided, at least on Linux, > and get auto-regenerated and mounted as a special type of file system > at boot time.You mean /dev/ of dom0''s? I don''t want to sync dom0s, but domUs only. Probably somehow: drbd0 => xvda, where drdb0 is a sync of /dev/vg0/dom1vda at one machine and /dev/vg1/dom1va at the second. -- Tomek Dwie tajemnice gwarantujace sukces: 1. Nikomu nie mów wszystkiego. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nico Kadel-Garcia <nkadel@gmail.com> writes: [...]>> The only issue I worry about right now is a disk performance, >> i.e. how much do I drop of my current ~100MB/s decent speed >> (ext3 on LVM2 on S/W Raid10,f2 on 2xSAS SFF 10K) >> putting somewhere into all that one more layer - DRDB. >> > Good question. I''ve not played with DRBD, and would also welcome > information on it.here are some tests i ran when testing my cluster all 32bit dom0/domU, centos5.1 xen 3.0 drbd 8.0.11 disks are sata Vendor: ATA Model: Hitachi HUA72105 Rev: GK6O drbd is synced via gigabit crossover cable on a pair of BCM5721 hopefully this formats well ;) bonie++ results name file_size putc putc_cpu put_block put_block_cpu rewrite rewrite_cpu getc getc_cpu get_block get_block_cpu seeks xxx 1G 32835 84 36672 3 15330 0 45565 37 81288 0 214.4 xxx 1G 33447 85 36397 3 17187 0 43458 35 80600 0 226 xxx 1G 32904 84 35061 2 16423 0 44402 34 79767 0 222.4 xxx 1G 35623 55 36679 2 16757 0 43347 31 80110 0 210.3 xxx 1G 36225 57 36461 3 14908 0 39560 54 78780 0 214.9 xxx 1G 33914 53 33150 2 15838 0 43829 34 81146 0 212.4 xxx 1G 34580 73 35840 2 12504 0 32321 67 81201 0 216 xxx 1G 35957 58 35184 2 14923 0 44902 35 81854 0 222.7 xxx 1G 36743 58 36733 3 14998 0 46162 35 80467 0 223.2 xxx 1G 35407 55 33088 2 15780 0 38761 30 81010 0 225.1 avg 34763.5 66.2 35526.5 2.4 15464.8 0 42230.7 39.2 80622.3 0 218.74 noatime,nodiratime,data=writeback xxx 1G 37214 59 36412 2 22132 0 43897 40 78985 0 250.9 xxx 1G 35971 57 39619 3 21283 0 44129 38 79904 0 254.8 xxx 1G 35299 55 38084 2 21073 0 43567 36 78576 0 250.2 xxx 1G 35728 56 39035 2 23278 0 43812 36 79060 0 252.2 xxx 1G 36807 58 40318 3 21518 0 43926 40 78694 0 251.1 xxx 1G 36934 59 38663 3 23104 0 43691 36 67848 0 235 xxx 1G 34884 55 39836 3 22323 0 42778 34 78294 0 248.8 xxx 1G 36630 58 40599 2 20535 0 43880 34 79439 0 255 xxx 1G 35963 57 40535 3 20716 0 43539 34 78697 0 253.7 xxx 1G 36111 57 36946 2 23118 0 44323 34 79656 0 247.1 avg 36154.1 57.1 39004.7 2.5 21908 0 43754.2 36.2 77915.3 0 249.88 to base 1390.6 -9.1 3478.2 0.1 6443.2 0 1523.5 -3 -2707 0 31.14 3.85% -15.94% 8.92% 4.00% 29.41% 3.48% -8.29% -3.47% 12.46% protocol=B,noatime,nodiratime,data=writeback xxx 1G 35641 56 38721 2 22342 0 43493 39 79768 0 247.6 xxx 1G 35846 56 37661 2 22608 0 43794 38 79423 0 252.4 xxx 1G 36957 59 38804 2 23436 0 44014 37 80370 0 248 xxx 1G 37655 59 36757 2 23312 0 43821 36 79384 0 251.2 xxx 1G 36126 57 37796 2 22240 0 43322 35 79189 0 243.3 xxx 1G 37010 59 37124 3 22273 0 43703 36 78480 0 246.1 xxx 1G 36342 57 37965 2 22615 0 43825 35 80078 0 256.1 xxx 1G 37205 59 38391 2 21258 0 43524 33 80328 0 251.1 xxx 1G 36284 57 37569 2 22638 0 43666 34 80152 0 243.5 xxx 1G 37751 60 37565 3 21748 0 43702 35 78545 0 248.3 avg 36681.7 57.9 37835.3 2.2 22447 0 43686.4 35.8 79571.7 0 248.76 to noatime 527.6 0.8 -1169.4 -0.3 539 0 -67.8 -0.4 1656.4 0 -1.12 1.44% 1.38% -3.09% -13.64% 2.40% -0.16% -1.12% 2.08% -0.45% to base 1918.2 0.173186453 -0.120081513 -0.176363636 -0.270090493 0 -0.036371485 0.071699744 0.055559301 0 -0.129122149 5.23% 0.30% 0.00% -8.02% 0.00% 0.00% 0.20% 0.00% -0.05% protocol=A,noatime,nodiratime,data=writeback xxx 1G 36481 58 39849 3 22559 0 45600 35 78342 0 260.4 xxx 1G 38373 61 41659 3 25262 0 46306 39 79128 0 254.4 xxx 1G 38403 61 40145 3 22839 0 41231 32 78267 0 260.3 xxx 1G 38048 61 41926 3 21506 0 44942 36 79753 0 255 xxx 1G 37734 60 39419 3 22692 0 44637 37 80066 0 261.3 xxx 1G 38553 62 38847 2 24376 0 45162 38 80295 0 257.7 xxx 1G 38605 91 38709 2 23814 0 37731 63 80322 0 256.9 xxx 1G 37455 59 39987 3 21526 0 45214 40 79139 0 251.2 xxx 1G 38451 61 40734 3 24739 0 45160 37 78289 0 250.9 xxx 1G 37001 91 38912 3 22429 0 40696 61 66910 0 257.5 avg 37910.4 66.5 40018.7 2.8 23174.2 0 43667.9 41.8 78051.1 0 256.56 to protocol=b 1228.7 8.6 2183.4 0.6 727.2 0 -18.5 6 -1520.6 0 7.8 3.24% 12.93% 5.46% 21.43% 3.14% #DIV/0! -0.04% 14.35% -1.95% #DIV/0! 3.04% to protocol=C, noatime 1756.3 9.4 1014 0.3 1266.2 -86.3 5.6 135.8 6.68 4.63% 14.14% 2.53% 10.71% 5.46% -0.20% 13.40% 0.17% 2.60% to base 3146.90 0.30 4492.20 0.40 7709.40 1437.20 2.60 -2571.20 37.82 8.30% 0.45% 11.23% 14.29% 33.27% 3.29% 6.22% -3.29% 14.74% protocol=A,al_extents=3833 xxx 1G 36926 58 38374 2 25660 0 43575 34 80513 0 245.8 xxx 1G 40684 65 42660 3 27165 0 43255 35 81155 0 247.2 xxx 1G 42181 67 44296 3 27211 0 39922 38 78175 0 245.4 xxx 1G 41712 67 42931 3 26529 0 43312 37 81270 0 246.7 xxx 1G 40893 65 44333 3 26047 0 42691 35 80717 0 240.8 xxx 1G 41157 65 44047 3 28368 0 43275 35 81241 0 244.1 xxx 1G 40660 65 43618 3 24242 0 43647 39 82371 0 222.8 xxx 1G 39230 62 43198 3 25762 0 42809 35 81119 0 209 xxx 1G 39602 63 41687 3 23352 0 43099 37 82070 0 222.3 xxx 1G 39928 63 42956 3 24787 0 42539 35 64510 0 209.6 avg 40297.3 64 42810 2.9 25912.3 0 42812.4 36 79314.1 0 233.37 to extents=287 2386.9 -2.5 2791.3 0.1 2738.1 -855.5 -5.8 1263 0 -23.19 5.92% -3.91% 6.52% 3.45% 10.57% -2.00% -16.11% 1.59% #DIV/0! -9.94% to base 5533.8 -2.2 7283.5 0.5 10447.5 581.7 -3.2 -1308.2 0 14.63 13.73% -3.44% 17.01% 17.24% 40.32% 1.36% -8.89% -1.65% #DIV/0! 6.27% protocol=A,bmbv,extents=3833 xxx 1G 38774 62 41954 3 26871 0 44548 38 80508 0 252.4 xxx 1G 37119 59 41707 3 26761 0 44931 38 78643 0 246.5 xxx 1G 37099 59 41287 3 24634 0 44618 40 79872 0 251.5 xxx 1G 38029 60 41831 3 25955 0 44247 37 80158 0 253.6 xxx 1G 38063 60 41692 3 26362 0 39694 37 77554 0 247.1 xxx 1G 38167 61 41958 3 25310 0 44360 36 80196 0 250.4 xxx 1G 37958 60 42119 3 25936 0 44882 36 80044 0 249.5 xxx 1G 38432 61 41406 3 26224 0 44737 35 80305 0 248.8 xxx 1G 38521 62 40864 3 25105 0 44246 36 81041 0 229 xxx 1G 37439 60 41781 3 24687 0 43397 36 79838 0 230.7 avg 37960.1 60.4 41659.9 3 25784.5 0 43966 36.9 79815.9 0 245.95 to base 3196.6 -5.8 6133.4 0.6 10319.7 1735.3 -2.3 -806.4 0 27.21 8.42% -9.60% 14.72% 20.00% 40.02% 3.95% -6.23% -1.01% 11.06% to protocol=A, no bmbv -2337.2 -3.6 -1150.1 0.1 -127.8 0 1153.6 0.9 501.8 0 12.58 -6.16% -5.96% -2.76% 3.33% -0.50% 2.62% 2.44% 0.63% 5.11% unlimited rate xxx 1G 38672 61 40177 3 25345 0 44294 38 79754 0 250.5 xxx 1G 39369 63 41914 2 25069 0 43973 38 79088 0 252.2 xxx 1G 40102 64 41508 3 26866 0 40644 35 80110 0 253.3 xxx 1G 39183 62 42734 3 25661 0 44282 36 80201 0 254.7 xxx 1G 38521 61 41672 3 25149 0 44723 36 80452 0 239.2 xxx 1G 37976 60 43040 3 24480 0 44304 37 80280 0 232.8 xxx 1G 39870 63 41722 3 24268 0 44183 38 79207 0 233.5 xxx 1G 38650 61 42041 3 25777 0 44533 36 66930 1 236.3 xxx 1G 39515 63 42254 3 26524 0 44643 37 80044 0 251.9 xxx 1G 40085 64 40896 3 25889 0 45097 36 79824 0 248.8 39194.3 62.2 41795.8 2.9 25502.8 0 44067.6 36.7 78589 0.1 245.32 to base 4430.8 -4 6269.3 0.5 10038 0 1836.9 -2.5 -2033.3 0.1 26.58 11.30% -6.43% 15.00% 17.24% 39.36% 4.17% -6.81% -2.59% 100.00% 10.83% to prev 1234.2 1.8 135.9 -0.1 -281.7 0 101.6 -0.2 -1226.9 0.1 -0.63 3.15% 2.89% 0.33% -3.45% -1.10% 0.23% -0.54% -1.56% 100.00% -0.26% to protocol=A,al_extents=3832 -1103 -1.8 -1014.2 0 -409.5 0 1255.2 0.7 -725.1 0.1 11.95 -2.81% -2.89% -2.43% 0.00% -1.61% #DIV/0! 2.85% 1.91% -0.92% 100.00% 4.87% -- To sto si frustriran, zavidan tko zna na cemu i sto ne vidis dalje od svoje guzice je tuzno. Da onda barem imas toliko samokontrole da sutis umjesto da pravis budalu od sebe... izgleda da si prestar da se promjenis na bolje. - Davor Pasaric, hr.comp.mac _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I am going to reply to this thread but I''m going to start from
something
new as it doesn''t seem to be covered in this thread so far.
I have been testing drbd under Xen and found some very disturbing things.
I''d like to implement this in a production system but this scares the
hell out of me...
I have two Dom0 servers connected with a crossover cable between two
gigabit e1000 NICs. No switch involved.
One DomU on each server with a 20G drbd device shared between them.
The drbd config contains:
syncer {
rate 10M;
group 1;
al-extents 257;
}
net {
on-disconnect reconnect;
}
so the net section is working at defaults. At first I had thought that
the problems I was seeing was due to timeout values etc and tried
various parameters in the net section but nothing made any difference.
When, on the current secondary node, I execute
drbdadm invalidate all
I get frequent errors such as:
drbd0: PingAck did not arrive in time.
drbd0: drbd0_asender [1572]: cstate SyncSource --> NetworkFailure
drbd0: asender terminated
drbd0: drbd_send_block() failed
drbd0: drbd0_receiver [1562]: cstate NetworkFailure --> BrokenPipe
drbd0: short read expecting header on sock: r=-512
drbd0: worker terminated
drbd0: ASSERT( mdev->ee_in_use == 0 ) in
/usr/src/modules/drbd/drbd/drbd_receiver.c:1880
drbd0: drbd0_receiver [1562]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.
I observe the xm top in both Dom0''s and I note a HUGE amount of dropped
RX packets being reported on both DomU''s vif interfaces. The dropping
of
RX packets is continuous throughout the drbd resync and grows extremely
large.
The ifconfig output within the DomU''s do not show any dropped packets.
I have used iperf to test the performance of the crossover link and it
is fine when there is no drbd syncing going on.
I have tried various things such as setting sysctl.conf options:
net.core.rmem_default=65536
net.core.wmem_default=65536
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
but so far the only thing that prevents the "PingAck did not arrive in
time" errors is to take the sync rate down to 1M.
My Xen version info is:
Xen version 3.0.3-1 (Debian 3.0.3-0-4)
Please advise...
Thanks!
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Please do not hijack threads, please put all your drbd questions to the drbd-user list. If properly setup and maintained drbd should work properly. -Ross> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Steve Wray > Sent: Tuesday, April 22, 2008 4:19 PM > To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Re: Xen, LVM, DRBD, Linux-HA > > I am going to reply to this thread but I''m going to start > from something > new as it doesn''t seem to be covered in this thread so far. > > I have been testing drbd under Xen and found some very > disturbing things. > > I''d like to implement this in a production system but this scares the > hell out of me... > > I have two Dom0 servers connected with a crossover cable between two > gigabit e1000 NICs. No switch involved. > > One DomU on each server with a 20G drbd device shared between them. > > > The drbd config contains: > > syncer { > rate 10M; > group 1; > al-extents 257; > } > > net { > on-disconnect reconnect; > } > > > so the net section is working at defaults. At first I had > thought that > the problems I was seeing was due to timeout values etc and tried > various parameters in the net section but nothing made any difference. > > When, on the current secondary node, I execute > > drbdadm invalidate all > > I get frequent errors such as: > > drbd0: PingAck did not arrive in time. > drbd0: drbd0_asender [1572]: cstate SyncSource --> NetworkFailure > drbd0: asender terminated > drbd0: drbd_send_block() failed > drbd0: drbd0_receiver [1562]: cstate NetworkFailure --> BrokenPipe > drbd0: short read expecting header on sock: r=-512 > drbd0: worker terminated > drbd0: ASSERT( mdev->ee_in_use == 0 ) in > /usr/src/modules/drbd/drbd/drbd_receiver.c:1880 > drbd0: drbd0_receiver [1562]: cstate BrokenPipe --> Unconnected > drbd0: Connection lost. > > > I observe the xm top in both Dom0''s and I note a HUGE amount > of dropped > RX packets being reported on both DomU''s vif interfaces. The > dropping of > RX packets is continuous throughout the drbd resync and grows > extremely > large. > > The ifconfig output within the DomU''s do not show any dropped packets. > > I have used iperf to test the performance of the crossover > link and it > is fine when there is no drbd syncing going on. > > I have tried various things such as setting sysctl.conf options: > > net.core.rmem_default=65536 > net.core.wmem_default=65536 > net.core.rmem_max=16777216 > net.core.wmem_max=16777216 > > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_wmem = 4096 65536 16777216 > > but so far the only thing that prevents the "PingAck did not > arrive in > time" errors is to take the sync rate down to 1M. > > > > My Xen version info is: > > Xen version 3.0.3-1 (Debian 3.0.3-0-4) > > > Please advise... > > Thanks! > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ross S. W. Walker wrote:> Please do not hijack threads, please put all your drbd questions > to the drbd-user list.This is a xen issue. I''m pretty sure its not a problem with drbd itself but with drbd under xen and probably with a clash between xen networking and disk IO. For people who are trying to use drbd under xen this is an important issue.> If properly setup and maintained drbd should work properly.I''ve been checking and many of our xen servers are showing similar packet loss under disk IO load. However, so far, only drbd shows such catastrophic problems. Please advise as to how to configure such that disk IO does not kill the network, as this *seems* to be what is happening here.> -Ross > > >> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com >> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Steve Wray >> Sent: Tuesday, April 22, 2008 4:19 PM >> To: xen-users@lists.xensource.com >> Subject: Re: [Xen-users] Re: Xen, LVM, DRBD, Linux-HA >> >> I am going to reply to this thread but I''m going to start >> from something >> new as it doesn''t seem to be covered in this thread so far. >> >> I have been testing drbd under Xen and found some very >> disturbing things. >> >> I''d like to implement this in a production system but this scares the >> hell out of me... >> >> I have two Dom0 servers connected with a crossover cable between two >> gigabit e1000 NICs. No switch involved. >> >> One DomU on each server with a 20G drbd device shared between them. >> >> >> The drbd config contains: >> >> syncer { >> rate 10M; >> group 1; >> al-extents 257; >> } >> >> net { >> on-disconnect reconnect; >> } >> >> >> so the net section is working at defaults. At first I had >> thought that >> the problems I was seeing was due to timeout values etc and tried >> various parameters in the net section but nothing made any difference. >> >> When, on the current secondary node, I execute >> >> drbdadm invalidate all >> >> I get frequent errors such as: >> >> drbd0: PingAck did not arrive in time. >> drbd0: drbd0_asender [1572]: cstate SyncSource --> NetworkFailure >> drbd0: asender terminated >> drbd0: drbd_send_block() failed >> drbd0: drbd0_receiver [1562]: cstate NetworkFailure --> BrokenPipe >> drbd0: short read expecting header on sock: r=-512 >> drbd0: worker terminated >> drbd0: ASSERT( mdev->ee_in_use == 0 ) in >> /usr/src/modules/drbd/drbd/drbd_receiver.c:1880 >> drbd0: drbd0_receiver [1562]: cstate BrokenPipe --> Unconnected >> drbd0: Connection lost. >> >> >> I observe the xm top in both Dom0''s and I note a HUGE amount >> of dropped >> RX packets being reported on both DomU''s vif interfaces. The >> dropping of >> RX packets is continuous throughout the drbd resync and grows >> extremely >> large. >> >> The ifconfig output within the DomU''s do not show any dropped packets. >> >> I have used iperf to test the performance of the crossover >> link and it >> is fine when there is no drbd syncing going on. >> >> I have tried various things such as setting sysctl.conf options: >> >> net.core.rmem_default=65536 >> net.core.wmem_default=65536 >> net.core.rmem_max=16777216 >> net.core.wmem_max=16777216 >> >> net.ipv4.tcp_rmem = 4096 87380 16777216 >> net.ipv4.tcp_wmem = 4096 65536 16777216 >> >> but so far the only thing that prevents the "PingAck did not >> arrive in >> time" errors is to take the sync rate down to 1M. >> >> >> >> My Xen version info is: >> >> Xen version 3.0.3-1 (Debian 3.0.3-0-4) >> >> >> Please advise... >> >> Thanks! >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > ______________________________________________________________________ > This e-mail, and any attachments thereto, is intended only for use by > the addressee(s) named herein and may contain legally privileged > and/or confidential information. If you are not the intended recipient > of this e-mail, you are hereby notified that any dissemination, > distribution or copying of this e-mail, and any attachments thereto, > is strictly prohibited. If you have received this e-mail in error, > please immediately notify the sender and permanently delete the > original and any copy or printout thereof. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steve Wray wrote:> Ross S. W. Walker wrote: > > Please do not hijack threads, please put all your drbd questions > > to the drbd-user list. > > This is a xen issue.Are you positive that the same symptoms wouldn''t happen if you were running a highly loaded postgres, or oracle system on top of these drbd devices?> I''m pretty sure its not a problem with drbd itself but with drbd under > xen and probably with a clash between xen networking and disk IO.Or maybe you are reaching the capacity of your hardware?> For people who are trying to use drbd under xen this is an > important issue.It is probably an important issue for drbd users who are running other applications too, so it wouldn''t hurt to post it to the drbd-users list as well.> > If properly setup and maintained drbd should work properly. > > I''ve been checking and many of our xen servers are showing similar > packet loss under disk IO load. However, so far, only drbd shows such > catastrophic problems. > > Please advise as to how to configure such that disk IO does not kill the > network, as this *seems* to be what is happening here.Take a ''vmstat'' sampling during these periods of packet loss. If the # of interrupts are too high then it''s time to think about, 1) interrupt throttling, 2) jumbo frames or 3) better server. The ''too high'' mark will depend on the class cpu, bus, memory you are running. There are ratings on the manufacturers web sites. -Ross ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, I get the impression that you want to run Linux-HA inside of the VM. Why not put it on the Dom0 in your two physical nodes and have them control all your VMs? Have you considered that or is it not fulfilling your needs? I have a setup like that with two physical nodes. For each VM I create the needed LVMs, same size on both nodes. I create a DRBD resource for each LVM to have the data mirrored between the two nodes. Have each VM configured to use its DRBD resources. Have heartbeat install on both nodes that controls fail over of all VMs. Live migration works for all para-VMs but not for HVMs. With HVMs I use heartbeat to shutdown it down before starting it on the other node. Cheers, Daniel On Tue, Apr 22, 2008 at 12:28 AM, Rainer Sokoll <rainer@sokoll.com> wrote:> Hi, > > I want to put it all together. > So far, I have a running machine with multiple domUs, and I am very > excited. But now I want to go a step ahead: > I have two identical machines with a big amount of RAM and disk space. > My goal is to hve all the domUs almost redundant - and want to ask you > for your advice. > The plan is to have every VM in its own logical volume and have DRBD > replicate this volume to the other machine. Assumed Linux with Linux-HA > runs in a VM, here comes my question: > If the complete Linux VM runs in its own logical volume, Linux-HA cannot > work, since the configuration must not be the same on both sides. One > solution would be to have a minor file based backend that holds /etc/, > /proc/, /var/, /boot/ (and maybe more?). Or maybe there is a more > sophisticated solution? > How would you start? > > Thanks, > Rainer > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I am using the latest stable version of Debian, 4.0. I compiled Xen 3.2.0 from source as well as DRBD 8.2.5. At first I also thought I had performances issue as Samba was acting very weird. Copying a new file to my samba share went without problem(speed of 7-8MB/s) over a 100Mbps link, but when I tried to copy the file back just a few seconds later I had terrible performance. But I am sure this is an issue only related to samba. With FTP in different directions and from different hosts I have perfect transmissions. From one VM to a standalone Linux box I reached speeds of 11MB/s, both transmit and receive. Running hdparm On Wed, Apr 23, 2008 at 12:59 AM, Tom Brown <tbrown@baremetal.com> wrote:> On Tue, 22 Apr 2008, Daniel Asplund wrote: > > Hi, > > > > I get the impression that you want to run Linux-HA inside of the VM. Why > > not > > put it on the Dom0 in your two physical nodes and have them control all > > your > > VMs? Have you considered that or is it not fulfilling your needs? > > I have a setup like that with two physical nodes. For each VM I create > > the > > needed LVMs, same size on both nodes. I create a DRBD resource for each > > LVM > > > > can I ask what you''re using for a dom0 distribution? I''ve done this > (without the HA stuff), but recently tried to do it with the default centos > 5.1 kernels and drbd modules, and disk I/O performance went through the > floor :( it was fine with a homebuilt 0.7 drbd module, but with the centos > 0.8 and 0.8.2 modules it sucked. > > Unfortunately, the 0.7 module was implicated in at least one system crash > and there were several with short uptimes, so I quit using that... and with > the .8 and .82 modules having horrible performance I was forced to quit > using drbd at least temporarily. > > -Tom > > > to have the data mirrored between the two nodes. Have each VM configured > > to > > use its DRBD resources. Have heartbeat install on both nodes that > > controls > > fail over of all VMs. > > > > Live migration works for all para-VMs but not for HVMs. With HVMs I use > > heartbeat to shutdown it down before starting it on the other node. > > > > Cheers, Daniel > > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sorry, managed to send the mail by mistake... ***** I am using the latest stable version of Debian, 4.0. I compiled Xen 3.2.0 from source as well as DRBD 8.2.5. At first I also thought I had performances issue as Samba was acting very weird. Copying a new file to my samba share went without problem(speed of 7-8MB/s) over a 100Mbps link, but when I tried to copy the file back just a few seconds later I had terrible performance. But I am sure this is an issue only related to samba. With FTP in different directions and from different hosts I have perfect transmissions. From one VM to a standalone Linux box I reached speeds of 11MB/s, both transmit and receive. Running hdparm -t on the device gives me same performance from within the VM as running it from the physical node. I have readings of 89-91MB/sec. Will continue testing, especially getting samba to work properly, but so far I am very happy with this setup. //Daniel> > > > > > > > Hi, > > > > > > I get the impression that you want to run Linux-HA inside of the VM. > > > Why not > > > put it on the Dom0 in your two physical nodes and have them control > > > all your > > > VMs? Have you considered that or is it not fulfilling your needs? > > > I have a setup like that with two physical nodes. For each VM I create > > > the > > > needed LVMs, same size on both nodes. I create a DRBD resource for > > > each LVM > > > > > > > can I ask what you''re using for a dom0 distribution? I''ve done this > > (without the HA stuff), but recently tried to do it with the default centos > > 5.1 kernels and drbd modules, and disk I/O performance went through the > > floor :( it was fine with a homebuilt 0.7 drbd module, but with the centos > > 0.8 and 0.8.2 modules it sucked. > > > > Unfortunately, the 0.7 module was implicated in at least one system > > crash and there were several with short uptimes, so I quit using that... and > > with the .8 and .82 modules having horrible performance I was forced to quit > > using drbd at least temporarily. > > > > -Tom > > > > > > to have the data mirrored between the two nodes. Have each VM > > > configured to > > > use its DRBD resources. Have heartbeat install on both nodes that > > > controls > > > fail over of all VMs. > > > > > > Live migration works for all para-VMs but not for HVMs. With HVMs I > > > use > > > heartbeat to shutdown it down before starting it on the other node. > > > > > > Cheers, Daniel > > > > > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 22, 2008 at 11:24:43PM +0200, Daniel Asplund wrote:> I get the impression that you want to run Linux-HA inside of the VM.Yes, that''s correct.> Why not put it on the Dom0 in your two physical nodes and have them > control all your VMs?Because I won''t get automatic failover if only a VM gets down. Rainer _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users