Hi There, In creating a "backup" dom0 server using drbd and using lvm partitions for the domU data, whats the best way to implement the drbd part? Should i have drbd mirroring the lvm partitions them selves or the data within them? Are there pros and cons of each? Thanks Simon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jun 20, 2006 at 02:49:48PM +1200, Simon wrote:> In creating a "backup" dom0 server using drbd and using lvm partitions > for the domU data, whats the best way to implement the drbd part? > > Should i have drbd mirroring the lvm partitions them selves or the > data within them?Depends. Primarily, on whether you''ve going to be sharing the entire PV between both machines, and never sharing some LVs with any other host. If you have a separate VG (on it''s own PV) for your domUs, and are just mirroring your domUs for availability, then mirroring the PVs is fine. Otherwise (if you''ve got the dom0 disk on the same VG) then you''ll have to mirror the LVs individually, and incur the overhead of lots-o-drbd. - Matt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Matthew, On 6/20/06, Matthew Palmer <mpalmer@hezmatt.org> wrote:> On Tue, Jun 20, 2006 at 02:49:48PM +1200, Simon wrote: > > In creating a "backup" dom0 server using drbd and using lvm partitions > > for the domU data, whats the best way to implement the drbd part? > > > > Should i have drbd mirroring the lvm partitions them selves or the > > data within them? > > Depends. Primarily, on whether you''ve going to be sharing the entire PV > between both machines, and never sharing some LVs with any other host. If > you have a separate VG (on it''s own PV) for your domUs, and are just > mirroring your domUs for availability, then mirroring the PVs is fine. > Otherwise (if you''ve got the dom0 disk on the same VG) then you''ll have to > mirror the LVs individually, and incur the overhead of lots-o-drbd.Im only concerned with two individual servers here: dom0(1) will hold all the domUs dom0(2) will only be used in a DR case when dom0(1) is down So reading your above comment.... LVM on DRBD? Simon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
xen-users@escapekey.org
2006-Jun-21 17:54 UTC
Re: [Xen-users] Re: LVM on DRBD -OR- DRBD on LVM?
Simon,
I think Matthew was making a recommendation based upon how you have your
domU''s setup ... i.e. are your domU''s living on LVM
LV''s or disk files
in one big LV, multiple PV''s (i.e. 1 for the boot/root of the dom0 and
one (or more) for the domU''s), etc.
We need to know how you have your primary server configured ... I''m not
sure you can reliably mirror your entire root PV since the kernel needs
to be up and have the DRBD driver in place to mount the device the
kernel will be booting on (chicken and egg) ... it may be possible, but
I''ve not thought about it long enough to decide.
What we''ve done is set up a pair of physically identical servers with a
GigE crossover cable for the DRBD devices to sync across (and a 100M Eth
for the public interface), and each guest has a set of LV''s dedicated
to
it. Each LV is created identically on both servers and are bound under
a DRBD device. The guest mounts the DRBD device, so all writes to the
domU''s "disk" gets replicated via DRBD to both servers.
Now, that said, that''s not our entire story, as we have some more
mirroring stuff under the LV''s, but you get the point for where we have
DRBD in place.
Each guest generally has 3 LV''s associated with it, and only 2 of those
replicated via DRBD:
xen0:GuestA-root -> drbd -> xen1:GuestA-root (2gb)
xen0:GuestA-swap <not replicated> xen1:GuestA-swap (1gb)
xen0:GuestA-data -> drbd -> xen1:GuestA-data (anywhere
from 5gb to 100gb)
I felt that replicating swap was just a waste of resources as we can''t
do hot migration anyway, so upon reboot, anything in swap is gone anyhow.
Currently, we are running with 14 DRBD devices, but none of the guests
are in production yet, so I don''t have good load and performance data
yet ... I''m still hunting for good performance monitoring tools (got
one
suggestion for ganglia, and I also found bandwidthd as well as the usual
friends like cacti+snmpd), and I''ve been watching the privage GigE as
well as /proc/drbd.
I''ll let the list know how performance seems to be after it''s
all up and
running! :)
Oh, and currently, we are running all guests on the primary server, but
we plan to distribute the guests across both servers after all is running.
Hope this helps,
Sam
Simon wrote:> Hi Matthew,
>
> On 6/20/06, Matthew Palmer <mpalmer@hezmatt.org> wrote:
>> On Tue, Jun 20, 2006 at 02:49:48PM +1200, Simon wrote:
>> > In creating a "backup" dom0 server using drbd and using
lvm partitions
>> > for the domU data, whats the best way to implement the drbd part?
>> >
>> > Should i have drbd mirroring the lvm partitions them selves or the
>> > data within them?
>>
>> Depends. Primarily, on whether you''ve going to be sharing the
entire PV
>> between both machines, and never sharing some LVs with any other
>> host. If
>> you have a separate VG (on it''s own PV) for your domUs, and
are just
>> mirroring your domUs for availability, then mirroring the PVs is fine.
>> Otherwise (if you''ve got the dom0 disk on the same VG) then
you''ll
>> have to
>> mirror the LVs individually, and incur the overhead of lots-o-drbd.
>
> Im only concerned with two individual servers here:
>
> dom0(1) will hold all the domUs
> dom0(2) will only be used in a DR case when dom0(1) is down
>
> So reading your above comment.... LVM on DRBD?
>
> Simon
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Hi Guys, Thanks for the replies... see inline below: On 6/22/06, xen-users@escapekey.org <xen-users@escapekey.org> wrote:> Simon, > > I think Matthew was making a recommendation based upon how you have your > domU''s setup ... i.e. are your domU''s living on LVM LV''s or disk files > in one big LV, multiple PV''s (i.e. 1 for the boot/root of the dom0 and > one (or more) for the domU''s), etc.Each domU has its own LVs in this case (hda/swap). So im mirroring the entire pv using drbd0.> We need to know how you have your primary server configured ... I''m not > sure you can reliably mirror your entire root PV since the kernel needs > to be up and have the DRBD driver in place to mount the device the > kernel will be booting on (chicken and egg) ... it may be possible, but > I''ve not thought about it long enough to decide.So this means i have to have a script to start lvm, then start each domU... I *seem* to have this working i think, here is how its currently configured (note, no XEN yet): drbd starts before lvm and then i have a script to make server1 the primary (is there a way to do this in the configuration?). On server2 i have removed lvm from boot init. Once at the command line i can mount one of the lvs on server1, read/write to it etc. Turn the power off of server one to fake a hw problem. Then i have a script that i can run on server2 to make it primary and start lvm, mount the drive see the changes from server1. And then boot server1 etc... What i have found is that if i try to start lvm on the secondary i get lots of errors: /server2:~# /etc/init.d/lvm start Setting up LVM Volume Groups... Reading all physical volumes. This may take a while... /dev/drbd0: read failed after 0 of 1024 at 0: Input/output error /dev/drbd0: read failed after 0 of 2048 at 0: Input/output error No volume groups found No volume groups found /dev/drbd0: read failed after 0 of 2048 at 0: Input/output error No volume groups found Jun 21 13:37:41 server2 kernel: drbd0: 65 messages suppressed in /usr/src/modules/drbd/drbd/drbd_req.c:197. Jun 21 13:37:41 server2 kernel: drbd0: Not in Primary state, no IO requests allowed So im gussing im correct it leaving lvm off on the secondary? But all changes are mirrored and resynced etc.> What we''ve done is set up a pair of physically identical servers with a > GigE crossover cable for the DRBD devices to sync across (and a 100M Eth > for the public interface), and each guest has a set of LV''s dedicated to > it. Each LV is created identically on both servers and are bound under > a DRBD device. The guest mounts the DRBD device, so all writes to the > domU''s "disk" gets replicated via DRBD to both servers. > > Now, that said, that''s not our entire story, as we have some more > mirroring stuff under the LV''s, but you get the point for where we have > DRBD in place. > > Each guest generally has 3 LV''s associated with it, and only 2 of those > replicated via DRBD: > > xen0:GuestA-root -> drbd -> xen1:GuestA-root (2gb) > xen0:GuestA-swap <not replicated> xen1:GuestA-swap (1gb) > xen0:GuestA-data -> drbd -> xen1:GuestA-data (anywhere > from 5gb to 100gb) > > I felt that replicating swap was just a waste of resources as we can''t > do hot migration anyway, so upon reboot, anything in swap is gone anyhow.Noted - had the same thoughts.> Currently, we are running with 14 DRBD devices, but none of the guests > are in production yet, so I don''t have good load and performance data > yet ... I''m still hunting for good performance monitoring tools (got one > suggestion for ganglia, and I also found bandwidthd as well as the usual > friends like cacti+snmpd), and I''ve been watching the privage GigE as > well as /proc/drbd. > > I''ll let the list know how performance seems to be after it''s all up and > running! :) > > Oh, and currently, we are running all guests on the primary server, but > we plan to distribute the guests across both servers after all is running.Hmm - good point. I''ll think about that one. For simplistys sake, i just really want one dom0 where all our domUs run. This will be running web(LAMP)/mail/dns and secondary mail/dns is on a seperate server in a different location. Regards, Simon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
xen-users@escapekey.org
2006-Jun-21 22:12 UTC
Re: [Xen-users] Re: LVM on DRBD -OR- DRBD on LVM?
<stuff deleted>> Each domU has its own LVs in this case (hda/swap). So im mirroring the > entire pv using drbd0.So, just to be sure, you have a PV just for the domU LV''s then? Like a whole partition or separate hard drive? (like, in our case, we created a PV out of /dev/sda3 ... sorta)> > So this means i have to have a script to start lvm, then start each > domU...LVM generally starts up with the OS startup scripts, but if _all_ of your domU data is on a single PV (and _none_ of your dom0 data is on that PV), then you could do this ... just move the position of the LVM startup script in your /etc/rc?.d to after drbd''s startup.> > I *seem* to have this working i think, here is how its currently > configured (note, no XEN yet): > > drbd starts before lvm and then i have a script to make server1 the > primary (is there a way to do this in the configuration?).Not that I know of, but I''ve not looked for that, really.> On server2 > i have removed lvm from boot init. Once at the command line i can > mount one of the lvs on server1, read/write to it etc. Turn the power > off of server one to fake a hw problem. Then i have a script that i > can run on server2 to make it primary and start lvm, mount the drive > see the changes from server1. And then boot server1 etc...Good.> What i have found is that if i try to start lvm on the secondary i get > lots of errors:You mean start LVM on server2 while server1 is still the primary? No can do! DRBD is active/passive (in it''s current incarnation), and besides, unless you are running something like RedHat''s Cluster LVM (clvm), the plain old LVM doesn''t understand multi access (active/active) VGs yet, I don''t think.>> >> I felt that replicating swap was just a waste of resources as we can''t >> do hot migration anyway, so upon reboot, anything in swap is gone >> anyhow. > > Noted - had the same thoughts.The downside to replicating the whole PV, you are stuck inherently replicating swap activity, too. That''s one reason why we are replicating at the LV level (that and I want to be able to run the guests *not* only on one primary server).> > For simplistys sake, i just really want one dom0 where all our domUs > run. This will be running web(LAMP)/mail/dns and secondary mail/dns is > on a seperate server in a different location.Are you building lots of LAMP servers or just one or ? I''m trying to see where Xen will fit into this :) I also saw your posts on the DRBD list, and it sounds like you are making good progress! --Sam _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wednesday 21 June 2006 5:12 pm, xen-users@escapekey.org wrote:> besides, unless you are running something like RedHat''s Cluster LVM > (clvm), the plain old LVM doesn''t understand multi access > (active/active) VGs yet, I don''t think.not exactly true; you can certainly use plain LVM on several nodes using the same PV, just as long as: a) no two nodes use the same LV at the same time and, b) can''t modify the LVM metadata while other nodes use it. IOW: 1) use pvcreate, vgcreate, lvcreate, lvextend, etc on _one_ node. 2) do a vgscan on all nodes; all see the LVs, use them as desired; just don''t fight over a single LV 3) to do ANY modification to the VG, deactivate all nodes except one (shut down or "vgchange -a n") 4) use lvcreate, lvresize, etc. on the only node left 5) do "vgscan; vgchange -a y" on the other nodes (or turn them on) the point is that the only really ''shared'' data is the LVM metadata; so use it ''read only''. the clvm package (it''s on Debian and Ubuntu, besides RH and Centos) uses the GFS cluster locking to arbitrate access to the LVM metadata. with it, you can use LVM commands on any node, and all would see the changes immediately and without conflicts (except pvmove, i think). each LV contents goes as usual, that is, don''t mount at more than one node (unless you put a cluster filesystem on it). -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users