Hello everybody,
I am in the process of setting up a really cool xen serverfarm. Backend
storage will be an LVMed AoE-device on top of DRBD.
The goal is to have the backend storage completely redundant.
Picture:
|RAID| |RAID|
|DRBD1| <----> |DRBD2|
\ /
|VMAC|
| AoE |
|global LVM VG|
/ | \
|Dom0a| |Dom0b| |Dom0c|
| | |
| | |
|LVMLV1| |LVMLV2| |LVMLV3|
|DomU1| |DomU2| |DomU3|
Each Dom0 has the global LVM VG visible but only uses the relevant LVs
for its DomUs.
There are two open questions for me:
1. Is is possible to failover AoE from drbd1 to drbd2? Did anyone try
this yet? Normal drbd-setups use heartbeat with a virtual IP for
failover, but in this case I need a virtual Mac, since AoE is layer2.
Does an AoE-Client recognize mac-changes automatically (I don''t think
so), or do I have to "failover" a custom virtual mac with "ip set
hwaddr" or such?
In the archives I didn''t find anything close to it.
2. Do I need the Redhat-Cluster-framework for this to work? If I make
changes to the shared LVM VG (e.g. create a new LV for a new DomU), I
have to execute vgscan on all Dom0s.
This would not be necessary if I would use the cluster framework. But I
don''t need GFS or such, so I don''t want to add additional
complexity.
But perhaps I have overseen some metadata problems here? Anyone? :)
Regards,
Bjoern
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Thu, Oct 12, 2006 at 12:09:56PM +0200, Bjoern Metzdorf wrote:> Hello everybody, > > I am in the process of setting up a really cool xen serverfarm. Backend > storage will be an LVMed AoE-device on top of DRBD. > > The goal is to have the backend storage completely redundant. > > Picture: > > |RAID| |RAID| > |DRBD1| <----> |DRBD2| > \ / > |VMAC| > | AoE | > |global LVM VG| > / | \ > |Dom0a| |Dom0b| |Dom0c| > | | | > | | | > |LVMLV1| |LVMLV2| |LVMLV3| > |DomU1| |DomU2| |DomU3| > > Each Dom0 has the global LVM VG visible but only uses the relevant LVs > for its DomUs. > > There are two open questions for me: > > 1. Is is possible to failover AoE from drbd1 to drbd2? Did anyone try > this yet? Normal drbd-setups use heartbeat with a virtual IP for > failover, but in this case I need a virtual Mac, since AoE is layer2. > > Does an AoE-Client recognize mac-changes automatically (I don''t think > so), or do I have to "failover" a custom virtual mac with "ip set > hwaddr" or such? > > In the archives I didn''t find anything close to it. > > 2. Do I need the Redhat-Cluster-framework for this to work? If I make > changes to the shared LVM VG (e.g. create a new LV for a new DomU), I > have to execute vgscan on all Dom0s. > This would not be necessary if I would use the cluster framework. But I > don''t need GFS or such, so I don''t want to add additional complexity. > But perhaps I have overseen some metadata problems here? Anyone? :) >You need CLVM ("Cluster LVM") from Redhat-Cluster-framework. No need for GFS. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Pasi,>> 2. Do I need the Redhat-Cluster-framework for this to work? If I make >> changes to the shared LVM VG (e.g. create a new LV for a new DomU), I >> have to execute vgscan on all Dom0s. >> This would not be necessary if I would use the cluster framework. But I >> don''t need GFS or such, so I don''t want to add additional complexity. >> But perhaps I have overseen some metadata problems here? Anyone? :) >> > > You need CLVM ("Cluster LVM") from Redhat-Cluster-framework. No need for > GFS.Ok, what exactly is the clvm daemon for? Propagating changes to the VGs to all nodes accessing the VGs? Is it possible to do this manually with vgscan or such? Regards, Bjoern _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Bjoern Metzdorf wrote:> Ok, what exactly is the clvm daemon for? Propagating changes to the > VGs to all nodes accessing the VGs? Is it possible to do this manually > with vgscan or such?This might help: http://tldp.org/HOWTO/LVM-HOWTO/sharinglvm1.html Also try http://www.cs.utexas.edu/~abhinay/research_papers/clvm.pdf <http://www.cs.utexas.edu/%7Eabhinay/research_papers/clvm.pdf> which seems to be related to this: http://sourceware.org/cluster/ Roland _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Roland,>> Ok, what exactly is the clvm daemon for? Propagating changes to the >> VGs to all nodes accessing the VGs? Is it possible to do this manually >> with vgscan or such? > > This might help: > > http://tldp.org/HOWTO/LVM-HOWTO/sharinglvm1.html>> http://www.cs.utexas.edu/~abhinay/research_papers/clvm.pdfOkay, I see I have to use clvm. Luckily Debian etch provides it along with the neccessary redhat-cluster-modules. Thanks! Regards, Bjoern _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thursday 12 October 2006 7:25 am, Bjoern Metzdorf wrote:> > You need CLVM ("Cluster LVM") from Redhat-Cluster-framework. No need for > > GFS. > > Ok, what exactly is the clvm daemon for? Propagating changes to the VGs > to all nodes accessing the VGs? Is it possible to do this manually with > vgscan or such?clvm isn''t a daemon, it''s just a couple of libraries and a few changes to lvm.conf: locking_library = "liblvm2clusterlock.so" locking_type = 2 library_dir = "/lib/lvm2" in the global{} block. this libraries link with the cluster management tools of GFS (but doesn''t need the whole package). you can''t just "do vgscan on all nodes", because between the time you do any change on the VG and the moment you do vgscan on the other nodes there MUST NOT be any access to the VG on any node. the LVM docs say that it''s safe to do: vgchange <vgname> -a n on all but one node <do wanted changes on the only active node> vgchange <vgname> -a y on the other nodes but that would mean stopping all work on all nodes. the CLVM libraries simply use the cluster tools to propagate a lock to suspend all nodes for a few miliseconds, apply the change and then release the lock. that way you can issue LVM administration commands from any node and it would be applied atomically on all nodes. once it''s installed it just works, well worth the trouble of setting up the cluster -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users