hi, has anyone experience with the use of iscsi (as iscsi initiator) in Xen (which Xen version?). Does it work in dom0 and domU? Does it work (good)? Or are there any problems (besides the normal linux problems with iscsi)? How is the performance in domU? Background is that i am planing virtual servers within xen and they should get their disks from the network so that i can perform live migration. Has anyone a setup like this working? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Using CentOS 4.2 (which came with a cleanly packaged iscsi-initiator-tools) I''ve gotten it to work and I find the performance preferrable to NFS Root. I still haven''t gotten the hang of booting using iSCSI Root over the network, but it should just be a couple more adjustments to the initrd. Currently, I have dom0 serve as the iSCSI initiator and then have the domUs use the block devices. Some limitations I''ve run into: - I can''t manage to get the iSCSI initiator to see changes to the exported LUNs (e.g. block device has been grown, or see more LUNs) - For multiple LUNs from different iSCSI Targets, establish a fixed order/names for the block devices Other than that, you shouldn''t have too much problems with it. :-) - gino ledesma On 1/28/06, Markus Hochholdinger <Markus@hochholdinger.net> wrote:> hi, > > has anyone experience with the use of iscsi (as iscsi initiator) in Xen (which > Xen version?). Does it work in dom0 and domU? > > Does it work (good)? Or are there any problems (besides the normal linux > problems with iscsi)? How is the performance in domU? > > Background is that i am planing virtual servers within xen and they should get > their disks from the network so that i can perform live migration. > Has anyone a setup like this working? > > > -- > greetings > > eMHa > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Saturday 28 January 2006 11:24 am, Markus Hochholdinger wrote:> has anyone experience with the use of iscsi (as iscsi initiator) in Xen > (which Xen version?). Does it work in dom0 and domU?i haven''t tried it, but i guess performance wouldn''t be as good. iSCSI is a complex protocol, and t needs a lot of help from the NIC to be competitive with other forms of SAN. that''s why you see iSCSI cards from Adaptec and Intel. those cards look like regular SCSI hardware (i think the Adaptec cards might even use the same driver for SCSI and iSCSI) under Xen, it''s somewhat more complex to get TCP offloading, and i don''t think you could use iSCSI cards from domU. the CPU utilization would be way higher. what i plan to do (once i get a few test boxes) is to use coraid''s AoE protocol (the driver is in the main Linux tree). it''s designed to be software driven, without any special card; and it''s CPU utilization seems to be around ATA levels. with any kind of SAN, i think the best strategy would be to use CLVM in dom0 and export the logical volumes to the domUs; this way you only have to setup the SAN drivers in dom0, the domUs would treat those as local disks. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Samstag, 28. Januar 2006 18:42 schrieb Gino LV. Ledesma:> Using CentOS 4.2 (which came with a cleanly packaged > iscsi-initiator-tools) I''ve gotten it to work and I find the > performance preferrable to NFS Root. I still haven''t gotten the hangnfs is not that fast. With iscsi i hope i can saturate the 1GBit/s LAN and get most of it to the block device.> of booting using iSCSI Root over the network, but it should just be a > couple more adjustments to the initrd.OK, that''s just the same as for non virtual servers which get their rootfs on iscsi i think.> Currently, I have dom0 serve as the iSCSI initiator and then have theSo it works in dom0. That''s good so i am at least able to connect to the storage anyway.> domUs use the block devices. Some limitations I''ve run into: > - I can''t manage to get the iSCSI initiator to see changes to the > exported LUNs (e.g. block device has been grown, or see more LUNs)Is it possible to disconnect a iscsi device and then reconnect so you can see the new size? I mean without reboot.> - For multiple LUNs from different iSCSI Targets, establish a fixed > order/names for the block devicesOK.> Other than that, you shouldn''t have too much problems with it. :-)Fine :-) -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Samstag, 28. Januar 2006 18:48 schrieb Javier Guerra:> On Saturday 28 January 2006 11:24 am, Markus Hochholdinger wrote: > > has anyone experience with the use of iscsi (as iscsi initiator) in Xen > > (which Xen version?). Does it work in dom0 and domU? > i haven''t tried it, but i guess performance wouldn''t be as good. iSCSI is > a complex protocol, and t needs a lot of help from the NIC to be > competitive with other forms of SAN. that''s why you see iSCSI cards from > Adaptec and Intel. those cards look like regular SCSI hardware (i think > the Adaptec cards might even use the same driver for SCSI and iSCSI)so the cpu has to work a lot for iscsi. But should performance be better in dom0 than in domU? I''m planning to use iscsi in domU, if possible.> under Xen, it''s somewhat more complex to get TCP offloading, and i don''t > think you could use iSCSI cards from domU. the CPU utilization would be > way higher.I don''t want to use iscsi cards because they are very expensive. How is network performance in domU? Is it possible to get almost the throughput of the rela nic?> what i plan to do (once i get a few test boxes) is to use coraid''s AoE > protocol (the driver is in the main Linux tree). it''s designed to be > software driven, without any special card; and it''s CPU utilization seems > to be around ATA levels.I also read about this. My thoughts were that iscsi exists longer and is a standard. AoE is no standard and can only be used by linux by now (i think). You also can make iscsi targets and initiators without any special hardware.> with any kind of SAN, i think the best strategy would be to use CLVM in > dom0 and export the logical volumes to the domUs; this way you only have to > setup the SAN drivers in dom0, the domUs would treat those as local disks.Is this safe for migration? How can you get the block device, which is exported to the domU, to other Xen hosts when migrating an domU? If all could take place in domU (iscsi initiator and rootfs on iscsi) you are completly independent from the dom0 block device configuration. I would like to prefer this if possible and if performance is not to bad. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>> Using CentOS 4.2 (which came with a cleanly packaged >> iscsi-initiator-tools) I''ve gotten it to work and I find the >> performance preferrable to NFS Root.How did you get past the "FATAL: module iscsi_sfnet not found." error when starting up the iscsi initiator on CentOS 4.2? That''s stopping me from getting the iscsi initiator to work on CentOS 4.2. Evidently the iscsi_sfnet module doesn''t exist on my CentOS 4.2 system after I installed the iscsi-initiator-utils RPM from the CentOS 4.2 CD media. Thanks for your help. Datto ---------------- Using CentOS 4.2 (which came with a cleanly packaged iscsi-initiator-tools) I''ve gotten it to work and I find the performance preferrable to NFS Root. I still haven''t gotten the hang of booting using iSCSI Root over the network, but it should just be a couple more adjustments to the initrd. Currently, I have dom0 serve as the iSCSI initiator and then have the domUs use the block devices. Some limitations I''ve run into: - I can''t manage to get the iSCSI initiator to see changes to the exported LUNs (e.g. block device has been grown, or see more LUNs) - For multiple LUNs from different iSCSI Targets, establish a fixed order/names for the block devices Other than that, you shouldn''t have too much problems with it. :-) - gino ledesma On 1/28/06, Markus Hochholdinger <Markus@hochholdinger.net> wrote:> hi, > > has anyone experience with the use of iscsi (as iscsi initiator) in Xen (which > Xen version?). Does it work in dom0 and domU? > > Does it work (good)? Or are there any problems (besides the normal linux > problems with iscsi)? How is the performance in domU? > > Background is that i am planing virtual servers within xen and they should get > their disks from the network so that i can perform live migration. > Has anyone a setup like this working? > > > -- > greetings > > eMHa > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hi, Am Samstag, 28. Januar 2006 19:24 schrieb Datto:> >> Using CentOS 4.2 (which came with a cleanly packaged > >> iscsi-initiator-tools) I''ve gotten it to work and I find the > >> performance preferrable to NFS Root. > How did you get past the "FATAL: module iscsi_sfnet not found." error when > starting up the iscsi initiator on CentOS 4.2? That''s stopping me from > getting the iscsi initiator to work on CentOS 4.2. Evidently the > iscsi_sfnet module doesn''t exist on my CentOS 4.2 system after I installed > the iscsi-initiator-utils RPM from the CentOS 4.2 CD media.is there not also an iscsi_init module (http://iscsi-init.sourceforge.net/) to get the iscsi devices at boot time? In your case you could perhaps check if the module iscsi_sfnet is there and/or if your initrd has this module included? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > > what i plan to do (once i get a few test boxes) is to use coraid''s AoE > > protocol (the driver is in the main Linux tree). it''s designed to be > > software driven, without any special card; and it''s CPU utilization seems > > to be around ATA levels. > > I also read about this. My thoughts were that iscsi exists longer and is a > standard. AoE is no standard and can only be used by linux by now (i think). > You also can make iscsi targets and initiators without any special hardware.There is a Linux user-space AoE "server" (vblade), but in my experience read performance sucks (5 MByte/s), write performance was fine (40+ MB/s). I had a look at the CorRAID site and they seemed to show the slow numbers, so I am still looking elsewhere for a good network attached "block device"... [I suspect the read performance issue was just an issue with not being able to handle multiple reads simultaneously, but I''m not strong enough with kernel space drivers to validate that...]> > with any kind of SAN, i think the best strategy would be to use CLVM > > in dom0 and export the logical volumes to the domUs; this way you only > > have to setup the SAN drivers in dom0, the domUs would treat those as > > local disks. > > Is this safe for migration? How can you get the block device, which isdoesn''t sound like it. I started to type the same question.> exported to the domU, to other Xen hosts when migrating an domU? If all > could take place in domU (iscsi initiator and rootfs on iscsi) you are > completly independent from the dom0 block device configuration. I would > like to prefer this if possible and if performance is not to bad.AFAICS, performance should be better if the domU attaches directly to the network storage instead of having to relay all reads/writes through dom0 -Tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra wrote:>(snip) iSCSI is a >complex protocol, and t needs a lot of help from the NIC to be competitive >with other forms of SAN. that''s why you see iSCSI cards from Adaptec and >Intel. those cards look like regular SCSI hardware (i think the Adaptec >cards might even use the same driver for SCSI and iSCSI) > > >There are dissenting opinions on this. Excerpt from http://en.wikipedia.org/wiki/ISCSI : "In fact, in modern high-performance servers, a plain NIC with efficient network driver code can outperform a TOE card because fewer interrupts <http://en.wikipedia.org/wiki/Interrupt> and DMA <http://en.wikipedia.org/wiki/Direct_memory_access> memory transfers are required." Regards, Dan. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Saturday 28 January 2006 1:18 pm, Markus Hochholdinger wrote:> so the cpu has to work a lot for iscsi. But should performance be better in > dom0 than in domU? I''m planning to use iscsi in domU, if possible.if you do heavy network in domU, either it has to pass through the dom0 kernel, or you have to dedicate a NIC to each domU. so, i think it could be slower (or more CPU demanding) to do the initiator in domU than dom0.> > what i plan to do (once i get a few test boxes) is to use coraid''s AoE > > protocol (the driver is in the main Linux tree). it''s designed to be > > software driven, without any special card; and it''s CPU utilization seems > > to be around ATA levels. > > I also read about this. My thoughts were that iscsi exists longer and is a > standard. AoE is no standard and can only be used by linux by now (i > think). You also can make iscsi targets and initiators without any special > hardware.all very true; but software-only iSCSI is a _real_ cpu hog. i''ve read about 40-50% CPU usage for 1Gb on 3GHz P4 (a couple of years ago, might have improved with better FSBs or PCI busses) most 1Gb NICs do some offloading, but only iSCSI ones manage the SCSI part. and with the paravirtualized network drivers, i think you can''t use either TCP or IP processing in the card; at best the card will do only the Ethernet frame checksum (i think even 100BaseT NICs do that). i wouldn''t be surprised if iSCSI on domU uses twice as much CPU than on dom0 (where it would use the full offloading capabilities of the NIC)> > with any kind of SAN, i think the best strategy would be to use CLVM in > > dom0 and export the logical volumes to the domUs; this way you only have > > to setup the SAN drivers in dom0, the domUs would treat those as local > > disks. > > Is this safe for migration? How can you get the block device, which is > exported to the domU, to other Xen hosts when migrating an domU? If allwith CLVM you can get several boxes looking at the same SAN drives and seeing the same LVs. of course you can''t mount a filesystem on two boxes (unless it''s a cluster filesystem, like GFS); but its perfectly safe to use a LV for a VM on box A, then migrate to box B, where it can find the same LV. at no point in time the LV is used from two different machines (real or virtual) at the same time. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hi, Am Samstag, 28. Januar 2006 20:20 schrieb tbrown@baremetal.com: [..]> AFAICS, performance should be better if the domU attaches directly to the > network storage instead of having to relay all reads/writes through dom0seems to be that there is no definitely answer until tested. In dom0 only dom0 has to make the network stuff and is closer to the nic. But perhaps dom0 gets the (big) load when traffic occurs. And there is one layer more to the black device (iscsi <-> dom0 <-> domU). In domU the network traffic has to go through dom0. But all other things are handled in domU. So domU is closer to the block device. Well, two servers are ordered. When they arrive i will test performance of each constellation. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Markus, Markus Hochholdinger wrote:> Background is that i am planing virtual servers within xen and they should get > their disks from the network so that i can perform live migration. > Has anyone a setup like this working? >Sure. I''ve got a similar setup with gnbd. gnbd works like a charm. I''ve heard of people combining gnbd with drbd to get a true HA setup. I''ve only used mine for test and development. Per. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Jan 28, 2006 at 12:48:54PM -0500, Javier Guerra wrote:> On Saturday 28 January 2006 11:24 am, Markus Hochholdinger wrote: > > has anyone experience with the use of iscsi (as iscsi initiator) in Xen > > (which Xen version?). Does it work in dom0 and domU? > > i haven''t tried it, but i guess performance wouldn''t be as good. iSCSI is a > complex protocol, and t needs a lot of help from the NIC to be competitive > with other forms of SAN. that''s why you see iSCSI cards from Adaptec and > Intel. those cards look like regular SCSI hardware (i think the Adaptec > cards might even use the same driver for SCSI and iSCSI) > > under Xen, it''s somewhat more complex to get TCP offloading, and i don''t think > you could use iSCSI cards from domU. the CPU utilization would be way > higher. > > what i plan to do (once i get a few test boxes) is to use coraid''s AoE > protocol (the driver is in the main Linux tree). it''s designed to be > software driven, without any special card; and it''s CPU utilization seems to > be around ATA levels. > > with any kind of SAN, i think the best strategy would be to use CLVM in dom0 > and export the logical volumes to the domUs; this way you only have to setup > the SAN drivers in dom0, the domUs would treat those as local disks. >Hi! So you have used SAN+CLVM with Xen ? Is CLVM considered stable? Have you had problems with it? And also it would be nice to know what version of the distribution, kernel, clvm and xen you have been using.. I''m going to try CLVM in couple of weeks, and it would be nice to know other people''s thoughts about it. Thanks! -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 12:38 schrieb Per Andreas Buer:> Markus Hochholdinger wrote: > > Background is that i am planing virtual servers within xen and they > > should get their disks from the network so that i can perform live > > migration. Has anyone a setup like this working? > Sure. I''ve got a similar setup with gnbd. gnbd works like a charm. I''ve > heard of people combining gnbd with drbd to get a true HA setup. I''ve > only used mine for test and development.well, my idea of HA is as follows: - Two storage servers on individual SANs connected to the Xen hosts. Each storage server provides block devices per iscsi. - On domU two iscsi block devices are combined to a raid1. On this raid1 we will have the rootfs. Advantages: - storage servers can easily upgraded. Because of raid1 you can savely disconnect on storage server and upgrade hard disk space. After resync the raid1 you can do the same with the other storage server. - If you use a kind of lvm on the storage servers you can easily expand the exportet iscsi block devices (the raid1 and the filesystem has also to be expanded). - You can make live migration without configuring the destination Xen host specially (e.g. provide block devices in dom0 to export to domU) because all is done in domU. - If one domU dies or the Xen host you can easily start the domUs on other Xen hosts. Disadvantages: - When one storage server dies ALL domU have to rebuild their raid1 when storage this storage server comes back. High traffic on the SANs. - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) Not sure: - Performance? Can we get full network performance in domU? Ideal is we can use full bandwith of the SANs (e.g. 1GBit/s). And if the SANs can handle this (i will make raid0 with three SATA disks in each storage server). - How is the CPU load on dom0 and domU when using iscsi in domU? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Jan 29, 2006 at 02:00:11PM +0100, Markus Hochholdinger wrote:> Hi, > > Am Sonntag, 29. Januar 2006 12:38 schrieb Per Andreas Buer: > > Markus Hochholdinger wrote: > > > Background is that i am planing virtual servers within xen and they > > > should get their disks from the network so that i can perform live > > > migration. Has anyone a setup like this working? > > Sure. I''ve got a similar setup with gnbd. gnbd works like a charm. I''ve > > heard of people combining gnbd with drbd to get a true HA setup. I''ve > > only used mine for test and development. > > well, my idea of HA is as follows: > - Two storage servers on individual SANs connected to the Xen hosts. Each > storage server provides block devices per iscsi. > - On domU two iscsi block devices are combined to a raid1. On this raid1 we > will have the rootfs. > > Advantages: > - storage servers can easily upgraded. Because of raid1 you can savely > disconnect on storage server and upgrade hard disk space. After resync the > raid1 you can do the same with the other storage server. > - If you use a kind of lvm on the storage servers you can easily expand the > exportet iscsi block devices (the raid1 and the filesystem has also to be > expanded). > - You can make live migration without configuring the destination Xen host > specially (e.g. provide block devices in dom0 to export to domU) because all > is done in domU. > - If one domU dies or the Xen host you can easily start the domUs on other > Xen hosts. > > Disadvantages: > - When one storage server dies ALL domU have to rebuild their raid1 when > storage this storage server comes back. High traffic on the SANs. > - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) > > Not sure: > - Performance? Can we get full network performance in domU? Ideal is we can > use full bandwith of the SANs (e.g. 1GBit/s). And if the SANs can handle this > (i will make raid0 with three SATA disks in each storage server). > - How is the CPU load on dom0 and domU when using iscsi in domU? > >Wouldn''t it be better to use iSCSI initiator in dom0, and map luns from two different storage servers (and SANs).. raid1 them, and create LVM (or CLVM) on top of raid.. and export LV to each domU. Then you could more easily upgrade the storage servers and only rebuild the raid in dom0.. domU''s wouldn''t see any problems / failures. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 15:02 schrieb Pasi Kärkkäinen: [..]> Wouldn''t it be better to use iSCSI initiator in dom0, and map luns from two > different storage servers (and SANs).. raid1 them, and create LVM (or CLVM) > on top of raid.. and export LV to each domU.define better!? It is easier to set up. But if you want to use migration it becomes complicatet. If you use more than one Xen host you have to setup on each Xen host the block devices (iscsi, raid, lvm). And i''m not sure what happens if you have two active raid1 on the same iscsi devices on different Xen hosts!?> Then you could more easily upgrade the storage servers and only rebuild the > raid in dom0.. domU''s wouldn''t see any problems / failures.Well, rebuild will take the same amount of network traffic. I''m not sure if it is really better? My idea is to have the domU not depend on the block device configuration of dom0. So its easier to migrate domUs. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Jan 29, 2006 at 03:11:54PM +0100, Markus Hochholdinger wrote:> Hi, > > Am Sonntag, 29. Januar 2006 15:02 schrieb Pasi Kärkkäinen: > [..] > > Wouldn''t it be better to use iSCSI initiator in dom0, and map luns from two > > different storage servers (and SANs).. raid1 them, and create LVM (or CLVM) > > on top of raid.. and export LV to each domU. > > define better!? It is easier to set up. But if you want to use migration it > becomes complicatet. If you use more than one Xen host you have to setup on > each Xen host the block devices (iscsi, raid, lvm). And i''m not sure what > happens if you have two active raid1 on the same iscsi devices on different > Xen hosts!? >CLVM (Cluster LVM) is supposed to handle this.. it allows to create volumes to shared storage (FC, iSCSI, etc). And Xen itself handles the migration.. access to block device on host1 is stopped when the access is started on host2 (after the migration).> > > Then you could more easily upgrade the storage servers and only rebuild the > > raid in dom0.. domU''s wouldn''t see any problems / failures. > > Well, rebuild will take the same amount of network traffic. I''m not sure if it > is really better? > My idea is to have the domU not depend on the block device configuration of > dom0. So its easier to migrate domUs. >Personally I''d like to keep all the raid/iscsi stuff out of domU''s.. to keep domU''s simple and easy to manage :) -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sunday 29 January 2006 9:11 am, Markus Hochholdinger wrote:> define better!? It is easier to set up. But if you want to use migration it > becomes complicatet. If you use more than one Xen host you have to setup on > each Xen host the block devices (iscsi, raid, lvm). And i''m not sure what > happens if you have two active raid1 on the same iscsi devices on different > Xen hosts!?as long as you don''t mount the same LV on two hosts (real or virtual) you''re safe.> > Then you could more easily upgrade the storage servers and only rebuild > > the raid in dom0.. domU''s wouldn''t see any problems / failures. > > Well, rebuild will take the same amount of network traffic. I''m not sure if > it is really better? > My idea is to have the domU not depend on the block device configuration of > dom0. So its easier to migrate domUs.just have the dom0s access the same SAN drives. LVM (or CLVM) will find the same VGs and setup exactly the same LVs on each dom0. the domUs would be far simpler each. (no iSCSI config, the devices would look like local disks to them) -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 15:24 schrieb Pasi Kärkkäinen: [..]> > define better!? It is easier to set up. But if you want to use migration > > it becomes complicatet. If you use more than one Xen host you have to > > setup on each Xen host the block devices (iscsi, raid, lvm). And i''m not > > sure what happens if you have two active raid1 on the same iscsi devices > > on different Xen hosts!? > CLVM (Cluster LVM) is supposed to handle this.. it allows to create volumes > to shared storage (FC, iSCSI, etc).ah. It sounds suspect to me but reading the clvm docs it should really handle this. This is really an option.> And Xen itself handles the migration.. access to block device on host1 is > stopped when the access is started on host2 (after the migration).It seems a lot of people do this this way. And there were never any problems in this setup?> > > Then you could more easily upgrade the storage servers and only rebuild > > > the raid in dom0.. domU''s wouldn''t see any problems / failures. > > Well, rebuild will take the same amount of network traffic. I''m not sure > > if it is really better? > > My idea is to have the domU not depend on the block device configuration > > of dom0. So its easier to migrate domUs. > Personally I''d like to keep all the raid/iscsi stuff out of domU''s.. to > keep domU''s simple and easy to manage :)OK, everyone can have his own opinion :-) Perhaps when i get deeper in the setup i will agree to you :-) For migrating existing servers it could really be better to get the iscsi and raid stuff out of the domUs. So the change to the servers will be minimal. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger wrote:> > well, my idea of HA is as follows: > - Two storage servers on individual SANs connected to the Xen hosts. Each > storage server provides block devices per iscsi. >I guess gnbd can be a drop-in replacement for iSCSI. I would think performance is better as gnbd is written for the Linux kernel - the SCSI protocol is written for hardware. I _know_ gnbd is easier to set up. You just point the client to the server and the client populates /dev/gnbd/ with the named entries (the devices are given logical names - no SCSI buses, devices or LUNS). If we compare your iSCSI-based setup to a setup with Heartbeat/DRBD/GNBD-setup there might be some interesting points. You can choose for yourself if you want the DomUs to act as GNBD clients or if you want to access the GNBD servers directly from your DomU - or a combination (through Dom0 for rootfs/swap - and via GNBD for data volumes).> - On domU two iscsi block devices are combined to a raid1. On this raid1 we > will have the rootfs. > > Advantages: > - storage servers can easily upgraded. Because of raid1 you can savely > disconnect on storage server and upgrade hard disk space. After resync the > raid1 you can do the same with the other storage server. >The same with Heartbeat/DRBD/GNBD. You just fail one of the storage servers and upgrade it. After it is back up DRBD does an _incremental_ sync witch usually just takes a few seconds. With such a setup you can use a _dedicated_ link for DRBD.> - If you use a kind of lvm on the storage servers you can easily expand the > exportet iscsi block devices (the raid1 and the filesystem has also to be > expanded). >The same goes for Hearbeat/DRBD/GNBD I would guess.> - You can make live migration without configuring the destination Xen host > specially (e.g. provide block devices in dom0 to export to domU) because all > is done in domU. >GNBD clients are more or less stateless.> - If one domU dies or the Xen host you can easily start the domUs on other > Xen hosts. > > Disadvantages: > - When one storage server dies ALL domU have to rebuild their raid1 when > storage this storage server comes back. High traffic on the SANs.You will also have to rebuild a volume if a XenU dies while writing to disk.> - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) >iSCSI for rootfs sounds lke a lot of pain.> Not sure: > - Performance? Can we get full network performance in domU? Ideal is we can > use full bandwith of the SANs (e.g. 1GBit/s). And if the SANs can handle this > (i will make raid0 with three SATA disks in each storage server). >Remember that every write has to be written twice. So your write capacity might suffer a bit. Per. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Jan 29, 2006 at 03:39:20PM +0100, Markus Hochholdinger wrote:> > Am Sonntag, 29. Januar 2006 15:24 schrieb Pasi Kärkkäinen: > [..] > > > define better!? It is easier to set up. But if you want to use migration > > > it becomes complicatet. If you use more than one Xen host you have to > > > setup on each Xen host the block devices (iscsi, raid, lvm). And i''m not > > > sure what happens if you have two active raid1 on the same iscsi devices > > > on different Xen hosts!? > > CLVM (Cluster LVM) is supposed to handle this.. it allows to create volumes > > to shared storage (FC, iSCSI, etc). > > ah. It sounds suspect to me but reading the clvm docs it should really handle > this. This is really an option. > > > > And Xen itself handles the migration.. access to block device on host1 is > > stopped when the access is started on host2 (after the migration). > > It seems a lot of people do this this way. And there were never any problems > in this setup? > > > > > > Then you could more easily upgrade the storage servers and only rebuild > > > > the raid in dom0.. domU''s wouldn''t see any problems / failures. > > > Well, rebuild will take the same amount of network traffic. I''m not sure > > > if it is really better? > > > My idea is to have the domU not depend on the block device configuration > > > of dom0. So its easier to migrate domUs. > > Personally I''d like to keep all the raid/iscsi stuff out of domU''s.. to > > keep domU''s simple and easy to manage :) > > OK, everyone can have his own opinion :-) Perhaps when i get deeper in the > setup i will agree to you :-) > For migrating existing servers it could really be better to get the iscsi and > raid stuff out of the domUs. So the change to the servers will be minimal. >In fact, even better would be to do the RAID in the storage arrays / servers (=iSCSI targets), and provide multiple paths to the targets.. using separate network (switch) for both paths. In the dom0 you would only set up multipath device to the shared RAID array (over iSCSI), and then create CLVM on top of that. No need to do the RAID in dom0 :) -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger wrote:>Hi, > >Am Sonntag, 29. Januar 2006 12:38 schrieb Per Andreas Buer: > > >>Markus Hochholdinger wrote: >> >> >>>Background is that i am planing virtual servers within xen and they >>>should get their disks from the network so that i can perform live >>>migration. Has anyone a setup like this working? >>> >>> >>Sure. I''ve got a similar setup with gnbd. gnbd works like a charm. I''ve >>heard of people combining gnbd with drbd to get a true HA setup. I''ve >>only used mine for test and development. >> >> > >well, my idea of HA is as follows: > - Two storage servers on individual SANs connected to the Xen hosts. Each >storage server provides block devices per iscsi. > - On domU two iscsi block devices are combined to a raid1. On this raid1 we >will have the rootfs. >We are doing doing this. Well sort of. We are having the dom0 attach to the iscsi devices and then pass them up as hda/hdb. The domU''s deal with the raiding of the devices. My reason for doing this is that the domU''s don''t need any access to the network that contains the iscsi devices and has all the iscsi information hidden from them so that they don''t know anthing about the arrays. All my dom0''s have access to the all the iscsi devices so that migration is possible.> >Advantages: > - storage servers can easily upgraded. Because of raid1 you can savely >disconnect on storage server and upgrade hard disk space. After resync the >raid1 you can do the same with the other storage server. > - If you use a kind of lvm on the storage servers you can easily expand the >exportet iscsi block devices (the raid1 and the filesystem has also to be >expanded). > - You can make live migration without configuring the destination Xen host >specially (e.g. provide block devices in dom0 to export to domU) because all >is done in domU. > - If one domU dies or the Xen host you can easily start the domUs on other >Xen hosts. > >I have done the resync thing to upgrade the storage server software. It is a pain but it is do-able. The other advantages hold true even if the dom0''s attach to the iscsi devices.>Disadvantages: > - When one storage server dies ALL domU have to rebuild their raid1 when >storage this storage server comes back. High traffic on the SANs. > - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) >Rebuilding the arrays can suck back a lot of network bandwidth. It is much better to rebuild the raid arrays consecutivly. rebuilding them concurrently really slows things down and causes the drives in the iscsi targets to really go mad seeking. building a new domU is quite easy. I build the raid array in the dom0 and then mount it locally and extract a minimal install onto the array. I then unmount the array and stop the raid. I can then boot the domU. then ..... bob''s your uncle.> >Not sure: > - Performance? Can we get full network performance in domU? Ideal is we can >use full bandwith of the SANs (e.g. 1GBit/s). And if the SANs can handle this >(i will make raid0 with three SATA disks in each storage server). > - How is the CPU load on dom0 and domU when using iscsi in domU? > > > >It would be interesting to know if running the iscsi devices in the domU is faster or slower than running the device in the dom0 and exporting the block device to the domU. To get all this working for me I had to hack together a way to insure that all the iscsi devices had consistant names in each dom0 Important safty tips. 1) do not have 2 domU''s trying to run out of the same raid array. 2) do not have 2 domU''s try to run the same filesystem. 3) raid 5 is a little too delicate to run with iscsi devices. -- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 22:04 schrieb Pasi Kärkkäinen: [..]> In fact, even better would be to do the RAID in the storage arrays / > servers (=iSCSI targets), and provide multiple paths to the targets.. using > separate network (switch) for both paths. > In the dom0 you would only set up multipath device to the shared RAID array > (over iSCSI), and then create CLVM on top of that. No need to do the RAID > in dom0 :)and if the backplane of the storage server dies? Or just the hard disk controller in the storage? I''d like to have no signle point of failure :-) -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 22:58 schrieb Alvin Starr:> Markus Hochholdinger wrote:[..]> >well, my idea of HA is as follows: > > - Two storage servers on individual SANs connected to the Xen hosts. Each > >storage server provides block devices per iscsi. > > - On domU two iscsi block devices are combined to a raid1. On this raid1 > > we will have the rootfs. > We are doing doing this. Well sort of. > We are having the dom0 attach to the iscsi devices and then pass them up > as hda/hdb. The domU''s deal with the raiding of the devices.And the domU makes the raid1?> My reason for doing this is that the domU''s don''t need any access to the > network that contains the iscsi devices and has all the iscsi > information hidden from them so that they don''t know anthing about the > arrays.Yes, thats a good point.> All my dom0''s have access to the all the iscsi devices so that > migration is possible.But you have to take care that no two dom0s write to the same iscsi target!> I have done the resync thing to upgrade the storage server software. It > is a pain but it is do-able.Sounds not that good i hoped.> The other advantages hold true even if the dom0''s attach to the iscsi > devices.Yes.> >Disadvantages: > > - When one storage server dies ALL domU have to rebuild their raid1 when > >storage this storage server comes back. High traffic on the SANs. > > - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) > Rebuilding the arrays can suck back a lot of network bandwidth. It is > much better to rebuild the raid arrays consecutivly. rebuilding them > concurrently really slows things down and causes the drives in the iscsi > targets to really go mad seeking.OK.> building a new domU is quite easy. > I build the raid array in the dom0 and then mount it locally and extract > a minimal install onto the array. I then unmount the array and stop the > raid. I can then boot the domU. > then ..... bob''s your uncle.It is easy to handle for the domUs. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sunday 29 January 2006 6:53 pm, Markus Hochholdinger wrote:> > All my dom0''s have access to the all the iscsi devices so that > > migration is possible. > > But you have to take care that no two dom0s write to the same iscsi target!that''s where you have it wrong. you have to take care that no two dom0s write to the same VOLUME. (that is, partitions, LUNs or LVs). two different LVs on the same disk can be written by different dom0s. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hi, i''ve tested the load of the machine when i do network i/o. And the results say: Do not make the iscsi stuff in domU! I''ve testet with the live cd on my notebook against a server (nc -l -p 1234) in the network. dom0: dd if=/dev/zero bs=10M count=100 | nc server 1234 domU: dd if=/dev/zero bs=10M count=100 | nc server 1234 OK, on the live cd the network is not bridged but routed. But this shouldn''t change that much. So i checked the cpu usage with xm top when doing the dd. I did this all on the console with no running X. When i make dd in dom0 i''ve a cpu load about 15% in dom0 and no cpu load in domU. When i make dd in domU i''ve a cpu load about 20% in domU and a cpu load about 10% in dom0!!! So if i will do my iscsi in domU i will load my cpu twice. Perhaps i should thinking about iscsi in dom0... -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hi, Am Montag, 30. Januar 2006 00:59 schrieb Javier Guerra:> On Sunday 29 January 2006 6:53 pm, Markus Hochholdinger wrote: > > > All my dom0''s have access to the all the iscsi devices so that > > > migration is possible. > > But you have to take care that no two dom0s write to the same iscsi > > target! > that''s where you have it wrong. you have to take care that no two dom0s > write to the same VOLUME. (that is, partitions, LUNs or LVs). two > different LVs on the same disk can be written by different dom0s.you are right. I should have said: "But you have to take care that no two dom0s write to the same partition on the same iscsi target." If i have the partition for my rootfs on multiple dom0s available it could accidently be mounted. Or the same domU could be startet on different dom0s. That is what i have fear of. But to be able to make live migration the partitions have to be available. But as my little network test shows it should be better for performance to do it this way. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sunday 29 January 2006 7:10 pm, Markus Hochholdinger wrote:> If i have the partition for my rootfs on multiple dom0s available it could > accidently be mounted. Or the same domU could be startet on different > dom0s. That is what i have fear of. But to be able to make live migration > the partitions have to be available.yeah, having several volumes on a "don''t touch!" mode seems risky. i guess that as long as they''re never mounted manually, just referenced in the domU definitions it should be safe enough. also, if the domUs have access to the SAN environment, wouldn''t that multiply several times the risk? I mean, if only the dom0 see the SAN, there are as many points of confusion as real boxes; but if the domUs see the SAN, any one of them could mess with anyone other.> But as my little network test shows it should be better for performance to > do it this way.yeah, very interesting that the dom0 CPU load is bigger as a router (20%) than as initiator (15%). it''s also a neat coincidence that the total load adds to twice (20+10, opposed to 15). does anybody have a working bridged network to make a similar test? i''m guessing the overhead would be a bit lower, measurable but not too significant, maybe 15+10 instead of 20+10. any bets? -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger wrote:>Hi, > >Am Sonntag, 29. Januar 2006 22:58 schrieb Alvin Starr: > > >>Markus Hochholdinger wrote: >> >> >[..] > > >>>well, my idea of HA is as follows: >>>- Two storage servers on individual SANs connected to the Xen hosts. Each >>>storage server provides block devices per iscsi. >>>- On domU two iscsi block devices are combined to a raid1. On this raid1 >>>we will have the rootfs. >>> >>> >>We are doing doing this. Well sort of. >>We are having the dom0 attach to the iscsi devices and then pass them up >>as hda/hdb. The domU''s deal with the raiding of the devices. >> >> > >And the domU makes the raid1? > > >Yes.> > >>My reason for doing this is that the domU''s don''t need any access to the >>network that contains the iscsi devices and has all the iscsi >>information hidden from them so that they don''t know anthing about the >>arrays. >> >> > >Yes, thats a good point. > > > > >> All my dom0''s have access to the all the iscsi devices so that >>migration is possible. >> >> > >But you have to take care that no two dom0s write to the same iscsi target! > > >I have burned myself there more than a few times. Currently I use the vmid as a unique handle. I have a simple mapping between vmid and iscsi device and I use the vmid to create the MAC address of the domU. My quick hack idea is to ping and see if a domU has taken the MAC address and if so then I will not start the new domain. It would be really nice if Xen had some global idea of what domains are running that would make most of these problems go away.> > >>I have done the resync thing to upgrade the storage server software. It >>is a pain but it is do-able. >> >> > >Sounds not that good i hoped. > > >The process is workable and I was able to keep 8 domains running on 3 systems for my switch over and back.> > >>The other advantages hold true even if the dom0''s attach to the iscsi >>devices. >> >> > >Yes. > > > > >>>Disadvantages: >>>- When one storage server dies ALL domU have to rebuild their raid1 when >>>storage this storage server comes back. High traffic on the SANs. >>>- Not easy to setup a new domU in this environment (lvm, iscsi, raid1) >>> >>> >>Rebuilding the arrays can suck back a lot of network bandwidth. It is >>much better to rebuild the raid arrays consecutivly. rebuilding them >>concurrently really slows things down and causes the drives in the iscsi >>targets to really go mad seeking. >> >> > >OK. > > > > >>building a new domU is quite easy. >>I build the raid array in the dom0 and then mount it locally and extract >>a minimal install onto the array. I then unmount the array and stop the >>raid. I can then boot the domU. >>then ..... bob''s your uncle. >> >> > >It is easy to handle for the domUs. > > >-- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Jan 30, 2006 at 12:47:50AM +0100, Markus Hochholdinger wrote:> Hi, > > Am Sonntag, 29. Januar 2006 22:04 schrieb Pasi Kärkkäinen: > [..] > > In fact, even better would be to do the RAID in the storage arrays / > > servers (=iSCSI targets), and provide multiple paths to the targets.. using > > separate network (switch) for both paths. > > In the dom0 you would only set up multipath device to the shared RAID array > > (over iSCSI), and then create CLVM on top of that. No need to do the RAID > > in dom0 :) > > and if the backplane of the storage server dies? Or just the hard disk > controller in the storage? I''d like to have no signle point of failure :-) >Well, make them double. Use two target boxes, use DRBD or similar to sync the data between the boxes, and hearbeat to do failover (high-availability) between the boxes (iSCSI targets). http://www.pcpro.co.uk/realworld/82284/san-on-the-cheap.html Tutorial there. No need to have single point of failure :) And now, you only need to have multipath setup in dom0, all the RAID stuff is in the storage server. And all the volumes (in CLVM) are visible to all the xen hosts.. so you can easily migrate domU''s from host to another, and the domU''s know nothing about the underlying LVM/iSCSI/SAN/RAID. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> And Xen itself handles the migration.. access to block device on host1 is > stopped when the access is started on host2 (after the migration).Are you saying that the change from dom0 handling the block device to the second dom0 handling the block device is completely transparent? E.g. I have a dom0 on one machine, A, importing a SAN block device and exporting it as /dev/sda to a domU, and an identical machine, B, with the dom0 importing the same block device. If I migrate the domU from A to B, will the migration be seamless in the same way it would be if the domU was using iscsi to access the SAN? If this is the case then why would anyone want the overhead of running iscsi in the domU? BTW, this is an interesting discussion for me as I now have my dom0 setup to export evms drives as iscsi root to my domUs, before I go any further down this avenue, I am interested in investigating the alternative setups that allow easy migration. Andy -- pteron.net services _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Andy Repton wrote:>>And Xen itself handles the migration.. access to block device on host1 is >>stopped when the access is started on host2 (after the migration). >> >> > >Are you saying that the change from dom0 handling the block device to >the second dom0 handling the block device is completely transparent? > >E.g. I have a dom0 on one machine, A, importing a SAN block device and >exporting it as /dev/sda to a domU, and an identical machine, B, with >the dom0 importing the same block device. If I migrate the domU from >A to B, will the migration be seamless in the same way it would be if >the domU was using iscsi to access the SAN? >Yes this seems to work correctly.> >If this is the case then why would anyone want the overhead of running >iscsi in the domU? >I am not sure why but I am sure that someone has a good reason. It just did not seem to make sense for what I wanted to do.>BTW, this is an interesting discussion for me as I now have my dom0 >setup to export evms drives as iscsi root to my domUs, before I go any >further down this avenue, I am interested in investigating the >alternative setups that allow easy migration. > >-- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Greetings, Thanks to Stefan and Mike among others! Now when I do a xm top I have something!!!! domU is naughty though. And when I launch Yast >> Software and Xen It loops into asking for a network resource and when I put http://localhost/sources (where I have the suse 10 sources) its is sort of happy than closes immidiately. Any ideas? Cheers, Jimmy _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> domU is naughty though. And when I launch Yast >> Software and Xen It > loops into > asking for a network resource and when I put http://localhost/sources > (where I have the suse 10 sources) its is sort of happy than closes > immidiately.What''s naughty about the domU? ;-) The second problem sounds like it could be a Yast bug... Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Per Andreas Buer wrote:> Markus Hochholdinger wrote: > >> >> well, my idea of HA is as follows: >> - Two storage servers on individual SANs connected to the Xen hosts. >> Each storage server provides block devices per iscsi. >> > > I guess gnbd can be a drop-in replacement for iSCSI. I would think > performance is better as gnbd is written for the Linux kernel - the > SCSI protocol is written for hardware. I _know_ gnbd is easier to set > up. You just point the client to the server and the client populates > /dev/gnbd/ with the named entries (the devices are given logical names > - no SCSI buses, devices or LUNS).If I remember correctly gnbd is not quite the same as iscsi. When I looked into using gnbd I figured I could not create a target disk device that would present 10-20 unique devices to the xen clients. I am using lvm to break apart a set of disks and then presenting each volume as a separate iscsi target. I did not think the same thing could be done with GNBD but then I started this about a year ago so the rules may have changed in the interfening time.> > If we compare your iSCSI-based setup to a setup with > Heartbeat/DRBD/GNBD-setup there might be some interesting points. You > can choose for yourself if you want the DomUs to act as GNBD clients > or if you want to access the GNBD servers directly from your DomU - or > a combination (through Dom0 for rootfs/swap - and via GNBD for data > volumes). > >> - On domU two iscsi block devices are combined to a raid1. On this >> raid1 we will have the rootfs. >> >> Advantages: >> - storage servers can easily upgraded. Because of raid1 you can >> savely disconnect on storage server and upgrade hard disk space. >> After resync the raid1 you can do the same with the other storage >> server. >> > > The same with Heartbeat/DRBD/GNBD. You just fail one of the storage > servers and upgrade it. After it is back up DRBD does an _incremental_ > sync witch usually just takes a few seconds. With such a setup you can > use a _dedicated_ link for DRBD.That is a nice feature.>> - If you use a kind of lvm on the storage servers you can easily >> expand the exportet iscsi block devices (the raid1 and the filesystem >> has also to be expanded). >> > > The same goes for Hearbeat/DRBD/GNBD I would guess. > >> - You can make live migration without configuring the destination >> Xen host specially (e.g. provide block devices in dom0 to export to >> domU) because all is done in domU. >> > > GNBD clients are more or less stateless. > >> - If one domU dies or the Xen host you can easily start the domUs on >> other Xen hosts. >> >> Disadvantages: >> - When one storage server dies ALL domU have to rebuild their raid1 >> when storage this storage server comes back. High traffic on the SANs. > > > You will also have to rebuild a volume if a XenU dies while writing to > disk. > >> - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) >> > > > iSCSI for rootfs sounds lke a lot of pain. > >> Not sure: >> - Performance? Can we get full network performance in domU? Ideal is >> we can use full bandwith of the SANs (e.g. 1GBit/s). And if the SANs >> can handle this (i will make raid0 with three SATA disks in each >> storage server). >> > > Remember that every write has to be written twice. So your write > capacity might suffer a bit. >Has anybody built a system using gnbd that supports several dom0 systms and migrating domU''s? -- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
As a side note, have you guys taken a look @ the ATA-over-Ethernet stuff? It''s much lighter than iSCSI. I have a little test bench setup with four 20GB USB drives in a RAID5 being exported (I know, I know but it''s just for play or is it (forplay!) couldn''t resist). I have to start using gfs (or another filesystem which allows data sharing) in order to properly allow sharing of files between various servers without corruption. I haven''t tried any of this stuff with xen yet. Badiane __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Badiane Ka wrote:>As a side note, have you guys taken a look @ the >ATA-over-Ethernet stuff? It''s much lighter than >iSCSI. I have a little test bench setup with four >20GB USB drives in a RAID5 being exported (I know, I >know but it''s just for play or is it (forplay!) >couldn''t resist). I have to start using gfs (or >another filesystem which allows data sharing) in order >to properly allow sharing of files between various >servers without corruption. > >I haven''t tried any of this stuff with xen yet. > >Badiane > >It looks like AOE only makes whole drives available so that to make a system of xen servers you would need to use somthing like GFS and then distribute that around and slice that up for the individual domU''s It would be interesting to see a comparison between the 2 setups in terms of performance. I did in the past play with HyperSCSI and it is a package that is somewhere between iSCSI and AOE. It is too bad that the project looks to be dead. -- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alvin Starr wrote:> Has anybody built a system using gnbd that supports several dom0 systms > and migrating domU''s?Yupp. We got system with a single (no DRBD - its only a test system) gnbd-server and four attached dom0 systems. Migration works like a charm. Per. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Per Andreas Buer a écrit : Alvin Starr wrote: Has anybody built a system using gnbd that supports several dom0 systms and migrating domU''s? Yupp. We got system with a single (no DRBD - its only a test system) gnbd-server and four attached dom0 systems. Migration works like a charm. Per. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users I have a two node cluster base on 3.0-testing working with aoe. Migration works, but i was not able to get a success with iscsi initiators ... Lee. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
The whole drives would be available to dom0 to be partitioned like any other drives are and the "targeting" would be handled by vblade I think. The reason you would use GFS or any other clustering filesystem is, as far as understand it, to allow sharing of files via some sort of mitigated locking mechanism, since trying that with single system OS'' could lead to corruption of the data without a node knowing if a file was locked or not . The requirements in terms of resources need by iSCSI, as far as I can tell from the literature, is greater than AoE. One thing that I noticed but haven''t really confirmed is that it "was" subject to the 48bit LBA limit which would somewhat suck when considering that one cant get a 300GB drive very cheaply. Ubuntu comes with gfs on install so I''ve been thinking that I wouldn''t mind working with it. Here is an interesting article I just came across (I''m still researching more hands on experiences): http://eric_rollins.home.mindspring.com/genezzo/cluster.html http://opengfs.sourceforge.net/ http://opendlm.sourceforge.net/ http://hilli.dk/howtos/ata-over-ethernet-aoe-in-gentoo-linux/ http://lpk.com.price.ru/~lelik/AoE/ http://freshmeat.net/projects/vblade/ http://www.linuxjournal.com/node/8149/print http://www.coraid.com/documents/AoEDescription.pdf http://acd.ucar.edu/~fredrick/linux/etherdrive/ http://evblade.kwaak.net/ Badiane __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Mark,
I forgot to mention that somewhere on my CV there is School Certificate from
Cambridge University in 1972 :-)
Anyway here is what I get :
suse76:/ # xm create -c /etc/xen/inhouse
Unexpected error: exceptions.ImportError
Please report to xen-devel@lists.xensource.com
Traceback (most recent call last):
File "/usr/sbin/xm", line 10, in ?
main.main(sys.argv)
File "/usr/lib/python2.4/xen/xm/main.py", line 897, in main
rc = cmd(args)
File "<string>", line 1, in <lambda>
File "/usr/lib/python2.4/xen/xm/main.py", line 532, in xm_subcommand
cmd = __import__(command, globals(), locals(), ''xen.xm'')
File "/usr/lib/python2.4/xen/xm/create.py", line 30, in ?
import xen.lowlevel.xc
ImportError: /usr/lib/python2.4/xen/lowlevel/xc.so: undefined symbol:
xc_domain_get_vcpu_info
~
The /etc/xen/inhouse file
# -*- mode: python; -*-
# configuration name:
name = "inhouse"
# usable ram:
memory = 2048
# kernel und initrd:
kernel = "/boot/vmlinuz-xen"
ramdisk = "/boot/initrd-xen"
# boot device:
root = "/dev/sda2"
# boot to run level:
extra = "3"
# network interface:
vif = [ ''mac=00:60:B0:3C:39:70, bridge=xenbr0'' ]
dhcp = "dhcp"
hostname = xen76
How dirty is this config?
Cheers,
Jimmy
PS Did I also mention about the Solutions Linux 2006 to morrow?
-----Original Message-----
From: Mark Williamson [mailto:mark.williamson@cl.cam.ac.uk]
Sent: Monday, January 30, 2006 2:21 PM
To: xen-users@lists.xensource.com
Cc: Jimmy Pierre
Subject: Re: [Xen-users] Xen 3 and SuSe 10
> domU is naughty though. And when I launch Yast >> Software and Xen It
> loops into
> asking for a network resource and when I put http://localhost/sources
> (where I have the suse 10 sources) its is sort of happy than closes
> immidiately.
What''s naughty about the domU? ;-)
The second problem sounds like it could be a Yast bug...
Cheers,
Mark
--
Dave: Just a question. What use is a unicyle with no seat? And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
> Has anybody built a system using gnbd that supports several dom0 systms > and migrating domU''s?Hi, I had several scenarios for testing purposes running domU migrations between dom0s. GNBD and iSCSI (iSCSI Enterprise Target and open-iscsi as initiator) both worked fine for me and my stress tests :) Michael -- ---------------------------------------------------------------------------------------- Michael Mey Thinking Objects Software GmbH | mailto: michael.mey@to.com Lilienthalstrasse 2/1 | phone: +49 711 88770-147 70825 Stuttgart-Korntal, Germany | fax: +49 711 88770-449 ---------------------------------------------------------------------------------------- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Samstag, 28. Januar 2006 21:46 schrieb Javier Guerra:> On Saturday 28 January 2006 1:18 pm, Markus Hochholdinger wrote: > > so the cpu has to work a lot for iscsi. But should performance be better > > in dom0 than in domU? I''m planning to use iscsi in domU, if possible. > if you do heavy network in domU, either it has to pass through the dom0 > kernel, or you have to dedicate a NIC to each domU. so, i think it could be > slower (or more CPU demanding) to do the initiator in domU than dom0.yes, that''s also what my tests show.> > > what i plan to do (once i get a few test boxes) is to use coraid''s AoE > > > protocol (the driver is in the main Linux tree). it''s designed to be > > > software driven, without any special card; and it''s CPU utilization > > > seems to be around ATA levels. > > I also read about this. My thoughts were that iscsi exists longer and is > > a standard. AoE is no standard and can only be used by linux by now (i > > think). You also can make iscsi targets and initiators without any > > special hardware. > all very true; but software-only iSCSI is a _real_ cpu hog. i''ve read > about 40-50% CPU usage for 1Gb on 3GHz P4 (a couple of years ago, might > have improved with better FSBs or PCI busses)Hm, i''ve not found anything about cpu load with software iscsi. Maybe i''ll test this.> most 1Gb NICs do some offloading, but only iSCSI ones manage the SCSI part. > and with the paravirtualized network drivers, i think you can''t use either > TCP or IP processing in the card; at best the card will do only the > Ethernet frame checksum (i think even 100BaseT NICs do that). i wouldn''t > be surprised if iSCSI on domU uses twice as much CPU than on dom0 (where it > would use the full offloading capabilities of the NIC)Yes, ack.> > > with any kind of SAN, i think the best strategy would be to use CLVM in > > > dom0 and export the logical volumes to the domUs; this way you only > > > have to setup the SAN drivers in dom0, the domUs would treat those as > > > local disks. > > Is this safe for migration? How can you get the block device, which is > > exported to the domU, to other Xen hosts when migrating an domU? If all > with CLVM you can get several boxes looking at the same SAN drives and > seeing the same LVs. of course you can''t mount a filesystem on two boxes > (unless it''s a cluster filesystem, like GFS); but its perfectly safe to use > a LV for a VM on box A, then migrate to box B, where it can find the same > LV. at no point in time the LV is used from two different machines (real > or virtual) at the same time.Yes. This was a misunderstanding for my part. I thought this is not possible, but as i see the responds here it is possible to make it so. Just the block devices exportet in dom0 to domU have to be equal on each Xen host (each dom0). So i will think over my ideas. -- greetingd eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 13:06 schrieb Pasi Kärkkäinen: [..]> So you have used SAN+CLVM with Xen ? > Is CLVM considered stable? Have you had problems with it? > And also it would be nice to know what version of the distribution, kernel, > clvm and xen you have been using.. > I''m going to try CLVM in couple of weeks, and it would be nice to know > other people''s thoughts about it.as far as i read the docs clvm is a part on top of lvm2. But i don''t really see what''s the benefit of clvm? It must be that clvm is able to work on top of shared storage (like iscsi, gnbd, ..) from different hosts? Means that multiple clvm on different hosts use the same block devices and clvm manages that nothing bad happens (write from two lokations to the same block device (partition) are managed). But if anyone knows for real, let us know? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 12:38 schrieben Sie:> Markus Hochholdinger wrote: > > Background is that i am planing virtual servers within xen and they > > should get their disks from the network so that i can perform live > > migration. Has anyone a setup like this working? > Sure. I''ve got a similar setup with gnbd. gnbd works like a charm. I''ve > heard of people combining gnbd with drbd to get a true HA setup. I''ve > only used mine for test and development.hey, you''re completely right :-) gnbd seams to be very easy and is exactly what i need! As of the docs (http://sourceware.org/cluster/gnbd/gnbd_usage.txt): server: 1. Start the gnbd server daemon # gnbd_serv 2. Export the block devices # gnbd_export -c -e <unique_gnbd_device_name> -d <local_partition_name> client: 1. Mount sysfs, if not already running # mount -t sysfs sysfs /sys 2. Load the gnbd module # modprobe gnbd 3. Import the gnbd devices # gnbd_import -i <gnbd_server_machine> This imports all of the exported gnbd devices from a server. The gnbd devices will appear as /dev/gnbd/<unique_gnbd_device_name>. From this point on, continue the setup as if these were regular shared storage. So the cool thing is, if you export block devices they are all automatically visible to the client with defined names. With iscsi you have to configure each block device on the server AND on the client (client can be multiple dom0s)! With gnbd you only have to configure them on the server. That makes management very easy! So next point to me is where to make my raid1? This multipath thing looks complicated to me. You need a block device sync and heartbeat. So my ideas now are: - two storage servers - on each storage server ~ lvm2 over all hard disks with striping (fast) ~ make logical volumes ~ export logical volumes with gnbd - on each dom0 import gnbd devices from both storage servers - configure domU to use these gnbd devices from dom0 - in domU make raid1 advantages: + easy to set up + domU can use black devices as normal disks + (live) migration possible disadvantages: + domU has to take care of redundancy (raid1) things to keep in mind resp. to test: + how does disconnect of a gnbd device will be handled? + do you need the fence daemon? + does resizing of gnbd devices work transparently? hint: My opinion is, if you make the raid1 thing in dom0 you can''t make live migration!? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, 30. Januar 2006 10:13 schrieb Andy Repton:> > And Xen itself handles the migration.. access to block device on host1 is > > stopped when the access is started on host2 (after the migration). > Are you saying that the change from dom0 handling the block device to > the second dom0 handling the block device is completely transparent?well, as the answers on this list shows it seems to be the case. I also wasn''t aware of this.> E.g. I have a dom0 on one machine, A, importing a SAN block device and > exporting it as /dev/sda to a domU, and an identical machine, B, with > the dom0 importing the same block device. If I migrate the domU from > A to B, will the migration be seamless in the same way it would be if > the domU was using iscsi to access the SAN?It seems so.> If this is the case then why would anyone want the overhead of running > iscsi in the domU?Well, i am now at the point to query the iscsi thing. It seems that iscsi has no advantages over other network block devices. Rather it is more tricky to set up.> BTW, this is an interesting discussion for me as I now have my dom0 > setup to export evms drives as iscsi root to my domUs, before I go any > further down this avenue, I am interested in investigating the > alternative setups that allow easy migration.Uh, i think we could make a new thread: "Possible block devices for domU". I now have a lot of new input. And my first opinion is fading. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jan 31, 2006 at 01:45:15PM +0100, Markus Hochholdinger wrote:> Hi, > > Am Sonntag, 29. Januar 2006 13:06 schrieb Pasi Kärkkäinen: > [..] > > So you have used SAN+CLVM with Xen ? > > Is CLVM considered stable? Have you had problems with it? > > And also it would be nice to know what version of the distribution, kernel, > > clvm and xen you have been using.. > > I''m going to try CLVM in couple of weeks, and it would be nice to know > > other people''s thoughts about it. > > as far as i read the docs clvm is a part on top of lvm2. But i don''t really > see what''s the benefit of clvm? It must be that clvm is able to work on top > of shared storage (like iscsi, gnbd, ..) from different hosts? Means that > multiple clvm on different hosts use the same block devices and clvm manages > that nothing bad happens (write from two lokations to the same block device > (partition) are managed). > > But if anyone knows for real, let us know? >CLVM makes it possible to have /dev/vg-shared/domU1-lv volume available in both xen hosts.. so you can migrate domU''s easily between xen hosts. Basicly you set up only one block device (over iSCSI or gnbd) for each xen host (dom0), and then use CLVM with that device.. really easy & simple to share all the logical volumes between xen hosts. If you don''t use CLVM, you need to set up separate iSCSI/gnbd device in xen host/dom0 for yeach domU.. much harder to manage & setup. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 15:25 schrieb Javier Guerra:> > Well, rebuild will take the same amount of network traffic. I''m not sure > > if it is really better? > > My idea is to have the domU not depend on the block device configuration > > of dom0. So its easier to migrate domUs. > just have the dom0s access the same SAN drives. LVM (or CLVM) will find the > same VGs and setup exactly the same LVs on each dom0. > the domUs would be far simpler each. (no iSCSI config, the devices would > look like local disks to them)yeah, that''s a big point. I thought i make the overhead of iscsi in domU to have the domU independent of the dom0. But when using gnbd you have all the advantages like with iscsi but it is easier to set up. I also thought that using iscsi is better for running other OS because it''s a standard. But with block device configuration in dom0 you lose nothing. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Sonntag, 29. Januar 2006 16:59 schrieb Per Andreas Buer:> Markus Hochholdinger wrote: > > well, my idea of HA is as follows: > > - Two storage servers on individual SANs connected to the Xen hosts. > > Each storage server provides block devices per iscsi. > I guess gnbd can be a drop-in replacement for iSCSI. I would think > performance is better as gnbd is written for the Linux kernel - the > SCSI protocol is written for hardware. I _know_ gnbd is easier to set > up. You just point the client to the server and the client populates > /dev/gnbd/ with the named entries (the devices are given logical names - > no SCSI buses, devices or LUNS).fully acknowledged. I was wrong with my thinking. I read the gnbd docs and it Is really what i want!> If we compare your iSCSI-based setup to a setup with > Heartbeat/DRBD/GNBD-setup there might be some interesting points. You > can choose for yourself if you want the DomUs to act as GNBD clients or > if you want to access the GNBD servers directly from your DomU - or a > combination (through Dom0 for rootfs/swap - and via GNBD for data volumes).Yes. The set up with heartbeat and drbd sounds (still) complicated to me. But using gnbd only for block devices over network seems to be the easiest way. And as you are saying, you can do a lot of things with gnbd. You are not restricted to use heartbeat, drbd or multipath!> > - On domU two iscsi block devices are combined to a raid1. On this raid1 > > we will have the rootfs. > > Advantages: > > - storage servers can easily upgraded. Because of raid1 you can savely > > disconnect on storage server and upgrade hard disk space. After resync > > the raid1 you can do the same with the other storage server. > The same with Heartbeat/DRBD/GNBD. You just fail one of the storage > servers and upgrade it. After it is back up DRBD does an _incremental_ > sync witch usually just takes a few seconds. With such a setup you can > use a _dedicated_ link for DRBD.Or you make the rad1 stuff in domU. So the setup of the storage servers would be easier. And setting up raid1 in domU is also very easy.> > - If you use a kind of lvm on the storage servers you can easily expand > > the exportet iscsi block devices (the raid1 and the filesystem has also > > to be expanded). > The same goes for Hearbeat/DRBD/GNBD I would guess.And also without heartbeat and drbd.> > - You can make live migration without configuring the destination Xen > > host specially (e.g. provide block devices in dom0 to export to domU) > > because all is done in domU. > GNBD clients are more or less stateless.Ack.> > - If one domU dies or the Xen host you can easily start the domUs on > > other Xen hosts. > > Disadvantages: > > - When one storage server dies ALL domU have to rebuild their raid1 when > > storage this storage server comes back. High traffic on the SANs. > You will also have to rebuild a volume if a XenU dies while writing to > disk.Yes, that''s a drawback. But somewhere you have to rebuild a raid1 if something fails. I am able to rebuild the raid1 of important servers first if i make the raid1 in domU.> > - Not easy to setup a new domU in this environment (lvm, iscsi, raid1) > iSCSI for rootfs sounds lke a lot of pain.Yeah. That#s right. You have to make all iscsi modules and iscsi programs available in initrd. Also the configuration is poss. in the initrd.> > Not sure: > > - Performance? Can we get full network performance in domU? Ideal is we > > can use full bandwith of the SANs (e.g. 1GBit/s). And if the SANs can > > handle this (i will make raid0 with three SATA disks in each storage > > server). > Remember that every write has to be written twice. So your write > capacity might suffer a bit.Yes, that is true. Making network load in domU makes the cpu load twice as making the traffic in dom0. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, 30. Januar 2006 15:18 schrieb Alvin Starr:> Per Andreas Buer wrote: > > I guess gnbd can be a drop-in replacement for iSCSI. I would think > > performance is better as gnbd is written for the Linux kernel - the > > SCSI protocol is written for hardware. I _know_ gnbd is easier to set > > up. You just point the client to the server and the client populates > > /dev/gnbd/ with the named entries (the devices are given logical names > > - no SCSI buses, devices or LUNS). > If I remember correctly gnbd is not quite the same as iscsi. When I > looked into using gnbd I figured I could not create a target disk device > that would present 10-20 unique devices to the xen clients. > I am using lvm to break apart a set of disks and then presenting each > volume as a separate iscsi target.as far as i read the docs of gnbd the only limitation is a unique name for each gnbd device. You probably could include the hostname in the gnbd devices? [..] -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, 30. Januar 2006 18:06 schrieb Per Andreas Buer:> Alvin Starr wrote: > > Has anybody built a system using gnbd that supports several dom0 systms > > and migrating domU''s? > Yupp. We got system with a single (no DRBD - its only a test system) > gnbd-server and four attached dom0 systems. Migration works like a charm.that''s great to hear this. Gives me hope to implement this project right :-) -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Dienstag, 31. Januar 2006 12:25 schrieb Michael Mey:> > Has anybody built a system using gnbd that supports several dom0 systms > > and migrating domU''s? > I had several scenarios for testing purposes running domU migrations > between dom0s. GNBD and iSCSI (iSCSI Enterprise Target and open-iscsi as > initiator) both worked fine for me and my stress tests :)which one have you decided to use? And why? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, 30. Januar 2006 01:20 schrieb Javier Guerra:> On Sunday 29 January 2006 7:10 pm, Markus Hochholdinger wrote: > > If i have the partition for my rootfs on multiple dom0s available it > > could accidently be mounted. Or the same domU could be startet on > > different dom0s. That is what i have fear of. But to be able to make live > > migration the partitions have to be available. > yeah, having several volumes on a "don''t touch!" mode seems risky. i guess > that as long as they''re never mounted manually, just referenced in the domU > definitions it should be safe enough.well, and you shouldn''t start the same domU on two different dom0s! Is xend aware of this? If not, you can mess up your rootfs when running the same domU on the same block device on different Xen hosts.> also, if the domUs have access to the SAN environment, wouldn''t that > multiply several times the risk? I mean, if only the dom0 see the SAN, > there are as many points of confusion as real boxes; but if the domUs see > the SAN, any one of them could mess with anyone other.That''s right. In the point of security it should be better to have the SAN only available in dom0.> > But as my little network test shows it should be better for performance > > to do it this way. > yeah, very interesting that the dom0 CPU load is bigger as a router (20%) > than as initiator (15%). it''s also a neat coincidence that the total load > adds to twice (20+10, opposed to 15). > does anybody have a working bridged network to make a similar test? i''m > guessing the overhead would be a bit lower, measurable but not too > significant, maybe 15+10 instead of 20+10. any bets?My figures are approximately. I run the tests a few minutes and looked at xen top. When i have the new hardware i will test it more precise. Well, ok, i''ve to test a lot... -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger wrote:> Hi, > > Am Sonntag, 29. Januar 2006 13:06 schrieb Pasi Kärkkäinen: > [..] > >> So you have used SAN+CLVM with Xen ? >> Is CLVM considered stable? Have you had problems with it? >> And also it would be nice to know what version of the distribution, kernel, >> clvm and xen you have been using.. >> I''m going to try CLVM in couple of weeks, and it would be nice to know >> other people''s thoughts about it. >> > > as far as i read the docs clvm is a part on top of lvm2. But i don''t really > see what''s the benefit of clvm? It must be that clvm is able to work on top > of shared storage (like iscsi, gnbd, ..) from different hosts? Means that > multiple clvm on different hosts use the same block devices and clvm manages > that nothing bad happens (write from two lokations to the same block device > (partition) are managed). > > But if anyone knows for real, let us know? >During the last few days. I''ve started to experiment with GNBD and CLVM. CLVM is usefull for the situation, where you''ve exported a VG to more than one host and want to modify it (create a LV). Without CLVM, the other host wont know the change until it is told to resync (via vgchange). Without CLVM you''ve to umount all LV on the other hosts and than make a chance. After that, the other have to update their LVM (vgchange) and remount their LV. For changes in bigger running system. This would be a overkill. Greets Arnd ___________________________________________________________ Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, 30. Januar 2006 03:57 schrieb Alvin Starr:> Markus Hochholdinger wrote: > >Am Sonntag, 29. Januar 2006 22:58 schrieb Alvin Starr: > >>Markus Hochholdinger wrote: > >>We are doing doing this. Well sort of. > >>We are having the dom0 attach to the iscsi devices and then pass them up > >>as hda/hdb. The domU''s deal with the raiding of the devices. > >And the domU makes the raid1? > Yes.so i will not be the only one who will do the raid1 in domU. [..]> >But you have to take care that no two dom0s write to the same iscsi > > target! > I have burned myself there more than a few times.Sounds very bad.> Currently I use the vmid as a unique handle. I have a simple mapping > between vmid and iscsi device and I use the vmid to create the MAC > address of the domU. > My quick hack idea is to ping and see if a domU has taken the MAC > address and if so then I will not start the new domain. > It would be really nice if Xen had some global idea of what domains are > running that would make most of these problems go away.Yes! Thats a big point. Perhaps someone would try to make a automatic fail over if one Xen host dies and starts a domU on another Xen host. But if domU, at what reason ever, is not dead you mess up your fs! And on a production system this will really be bad!> >>I have done the resync thing to upgrade the storage server software. It > >>is a pain but it is do-able. > >Sounds not that good i hoped. > The process is workable and I was able to keep 8 domains running on 3 > systems for my switch over and back.A little raw calculation: - 1 TByte (1000 GByte) disk space on each storage server - 1 GBit/s LAN (~ 100 MByte/s) => complete resync will last about 3 hours So i saw hardware raid controllers takeing longer for much lesser disk space! [..] -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tuesday 31 January 2006 14:41, Markus Hochholdinger wrote:> Hi, > > Am Dienstag, 31. Januar 2006 12:25 schrieb Michael Mey: > > > Has anybody built a system using gnbd that supports several dom0 systms > > > and migrating domU''s? > > > > I had several scenarios for testing purposes running domU migrations > > between dom0s. GNBD and iSCSI (iSCSI Enterprise Target and open-iscsi as > > initiator) both worked fine for me and my stress tests :) > > which one have you decided to use? And why?It was for getting to know which technology really works under high domU load for live-migration. The storage decision depends on what you want to do and what budget you have. The first choice disregarding the costs would be a san. fc works fine, but san-boxes with iscsi are cheaper as the required equipment is (you don''t need expensive fc-hba, fc-switch etc.). If you want to implement a solution on your own, go for gnbd because it''s simple, fast and reliable and can run on common hardware. You really wouldn''t want to mess up with nfs. ok, a vbd as nfsroot is stable for high i/o load in contrast to a file image on a nfs server, but there a security and performance issues. If you don''t want a third box for live-migration and load-balancing, the next version of drbd could be interesting (with multiple masters) or you want to customize xen for usage with the current stable drbd release. There''s also an interesting commercial product called peerfs which also works nicely for live-migration and isn''t that expensive. So, it''s up to your equipment, needs and budget what you want to implement :) Michael -- ---------------------------------------------------------------------------------------- Michael Mey Thinking Objects Software GmbH | mailto: michael.mey@to.com Lilienthalstrasse 2/1 | phone: +49 711 88770-147 70825 Stuttgart-Korntal, Germany | fax: +49 711 88770-449 ---------------------------------------------------------------------------------------- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Dienstag, 31. Januar 2006 14:18 schrieb Pasi Kärkkäinen:> On Tue, Jan 31, 2006 at 01:45:15PM +0100, Markus Hochholdinger wrote: > > Am Sonntag, 29. Januar 2006 13:06 schrieb Pasi Kärkkäinen: > > But if anyone knows for real, let us know? > CLVM makes it possible to have /dev/vg-shared/domU1-lv volume available in > both xen hosts.. so you can migrate domU''s easily between xen hosts.so clvm is a replacement for gnbd but not for network block device? i.g.: (storage server, lvm2, lv -> gnbd) -> (Xen host1, gnbd -> hda1 for domU) \> (Xen host2, gnbd -> hda1 for domU) (storage server, iscsi) -> (Xen host, iscsi -> clvm -> hda1 for domU) \> (Xen host, iscsi -> clvm -> hda1 for domU) So clvm takes the lvm part from the storage server to the Xen hosts? Are i am right?> Basicly you set up only one block device (over iSCSI or gnbd) for each > xen host (dom0), and then use CLVM with that device.. really easy & simple > to share all the logical volumes between xen hosts.And how complicated ist the management of clvm?> If you don''t use CLVM, you need to set up separate iSCSI/gnbd device in xen > host/dom0 for yeach domU.. much harder to manage & setup.Hm, what i see from gnbd it''s very easy! You don''t need to have a config on the gnbd client. Only on the gnbd server. But also have all the gnbd volumes available. With clvm you have to run some sort of cluster management on each client which distribute the config for the clvm (as i read in the docs). Sounds to me complicated. Or i am wrong? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger wrote:> A little raw calculation: > - 1 TByte (1000 GByte) disk space on each storage server > - 1 GBit/s LAN (~ 100 MByte/s) > => complete resync will last about 3 hoursJust a FYI. Newest versions of MD has "bitmap based intent logging", which should reduce rebuild times for big arrays drastically.> So i saw hardware raid controllers takeing longer for much lesser disk space!After having used both, I trust MD over hardware controllers any day. (Haven''t actually tested which performed better, since both were adequate for my needs at the time.) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Dienstag, 31. Januar 2006 14:52 schrieben Sie:> Markus Hochholdinger wrote:[..]> During the last few days. I''ve started to experiment with GNBD and CLVM. > CLVM is usefull for the situation, where you''ve exported a VG to more > than one host and want to modify it (create a LV). Without CLVM, the > other host wont know the change until it is told to resync (via vgchange).doesn''t gnbd recognize size changes? If you export logical volumes (lvm2) from the storage server(s) with gnbd you are able to resize the logical volumes!? Do the gnbd clients notice the changes?> Without CLVM you''ve to umount all LV on the other hosts and than make a > chance. After that, the other have to update their LVM (vgchange) and > remount their LV. For changes in bigger running system. This would be a > overkill.Without clvm you could use gnbd. Does the gnbd client notice when the gnbd server adds a new gnbd export? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tuesday 31 January 2006 9:08 am, Markus Hochholdinger wrote:> so clvm is a replacement for gnbd but not for network block device?no, CLVM is an option of LVM: under LVM, all the volumes metadata (that is, the mapping info) is stored in the block devices themselves. this way, if you have a host with some disks, you join them, and split in any way using LVM. later on, you shut down the machine, take off the disks and install them in another Linux box. on the new host, you just do "vgscan", and presto! all the LVs are there! not so surprising... but that means that the joining, splitting, and all is in the block devices. now... what happens if the block devices are on a SAN with several hosts seeing them? using LVM, you first turn on ONE host; configure ALL the LVs as you wish. then turn on all hosts, and everyone will do vgscan as part of the startup. from then on, all hosts will see the same LVs. but if you then create, delete,extend a new LV, or add, remove, move, etc. a PV... then the other hosts will have inconsistent mappings until they do vgscan again. that''s why in LVM docs it says that to modify a shared volume group you have to unmount and "vgchange -an" all from all hosts but one (effectively disconnecting them from the SAN), do all changes on ONE host, and then do "vgchange -ay" on the other hosts. IOW: with shared drives, you lose the ability to do online modifications to the volume group. that''s where CLVM enters. the GFS stack includes lots of kernel modules and daemons to manage the "Cluster" concept, where there are some shared resources and locking. there''s a kind of plugin for LVM that uses those systems to distribute and synchronize all the LVM managing. so, if you configure the ''cluster'' system, the CLVM managing is really simple: all the LVM commands work as expected, and do all the communications needed to keep the device mappings consistent in all hosts. you just change anything on a shared volume group, and all hosts are notified and transparently reread the device mappings. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger wrote:> Hi, > > Am Dienstag, 31. Januar 2006 14:52 schrieben Sie: > >> Markus Hochholdinger wrote: >> > [..] > >> During the last few days. I''ve started to experiment with GNBD and CLVM. >> CLVM is usefull for the situation, where you''ve exported a VG to more >> than one host and want to modify it (create a LV). Without CLVM, the >> other host wont know the change until it is told to resync (via vgchange). >> > > doesn''t gnbd recognize size changes? If you export logical volumes (lvm2) from > the storage server(s) with gnbd you are able to resize the logical volumes!? > Do the gnbd clients notice the changes? > >I haven''t given a clear answer. Sorry. In my setup. I exported the physical device. And on the imported end, there i configured a LVM. So i import on all DOM0 Hosts the physical device only. If i know create a LV, it exitsts automaticly on all other hosts. So the LV for the DomU will be exported via the local block device backend. Greets Arnd ___________________________________________________________ Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tuesday 31 January 2006 8:03 am, Markus Hochholdinger wrote:> So my ideas now are: > - two storage servers > - on each storage server > ~ lvm2 over all hard disks with striping (fast) > ~ make logical volumes > ~ export logical volumes with gnbd > - on each dom0 import gnbd devices from both storage servers > - configure domU to use these gnbd devices from dom0 > - in domU make raid1i think a better use of the resources would be: - two storage servers - on each storage server: ~ md raid to join all disks on one volume ~ export volume with gnbd ~ drbd to ''mirror'' both volumes, making it HA - two or more ''process boxes'' with Xen - on each process box: ~ CLVM on dom0 splits the HA volume on several LVs ~ domUs see simple volume devices advantages: + full data redundancy + domU can use block devices as normal disks + live migration possible and easy + the simplest possible domU settings + the replication managed by drbd doesn''t tax the xen boxes + the replication managed by drbd can use a different network + LV resizing is managed on any dom0, the volume servers doesn''t care. + the storage can be easily expanded online just adding more volume servers and joining them in the volume group. disadvangates: - CLVM needs some GFS modules and daemons on each dom0 if you use AoE instead of gnbd, you have the option later to use hardware drive cages (only coraid, unfortunately). but i don''t know if there''s a way to make them HA. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Dienstag, 31. Januar 2006 15:06 schrieb Michael Mey:> On Tuesday 31 January 2006 14:41, Markus Hochholdinger wrote: > > Am Dienstag, 31. Januar 2006 12:25 schrieb Michael Mey: > > > > Has anybody built a system using gnbd that supports several dom0 > > > > systms and migrating domU''s? > > > I had several scenarios for testing purposes running domU migrations > > > between dom0s. GNBD and iSCSI (iSCSI Enterprise Target and open-iscsi > > > as initiator) both worked fine for me and my stress tests :) > > which one have you decided to use? And why? > It was for getting to know which technology really works under high domU > load for live-migration. > The storage decision depends on what you want to do and what budget you > have.i want to consolidate servers in consideration of upgrades. So my points are: 1. inexpensively state of the art computers, but NOT highend (=expensive) means sata disks, 1GBit/s ethernt, .. 2. failover because of standard hardware there should be (almost) no single point of failuer. at least two storage servers and two Xen hosts 3. expandable easy to expand. when we use Xen with storage servers we can separately expand storage (hard disks) and cpu power (Xen hosts) So i now think that gnbd should be enough for this purpose.> The first choice disregarding the costs would be a san. fc works fine, but > san-boxes with iscsi are cheaper as the required equipment is (you don''t > need expensive fc-hba, fc-switch etc.).OK, a san with fc is faster (really that much?) but also really expensive!> If you want to implement a solution on your own, go for gnbd because it''s > simple, fast and reliable and can run on common hardware. You reallyYeah, thats the point: "common hardware". I don''t want to invest in uncommon hardware which will be unsupported tomorrow.> wouldn''t want to mess up with nfs. ok, a vbd as nfsroot is stable for high > i/o load in contrast to a file image on a nfs server, but there a security > and performance issues.Well, to make nfs failsafe is not that easy as setting up a raid1.> If you don''t want a third box for live-migration and load-balancing, the > next version of drbd could be interesting (with multiple masters) or you > want to customize xen for usage with the current stable drbd release.I don''t like the thing with master slave in drbd. So the next version "could" be interessting. But i also like to use approved solutions. I will set up a live environment with at least 4 boxes. For this I''d like to use proved techniques. And i also will set up a development and working environment with two storage servers and about four Xen hosts. Here i can experiment a little. So i can implement tested things in the live environment.> There''s also an interesting commercial product called peerfs which also > works nicely for live-migration and isn''t that expensive.Well, i don''t like commercial products ;-) I will be able to fix things my self if they don''t work as i''d expect.> So, it''s up to your equipment, needs and budget what you want to implement > :)The need is all and the budget is small ;-) All jokes aside, i will try to make a highend solution with cheap hardware. Just like RAID does but for servers. OK, the hardware will not that cheap. But thinking about scsi vs. sata you can save a lot of money. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tuesday 31 January 2006 10:00 am, Markus Hochholdinger wrote:> I don''t like the thing with master slave in drbd. So the next versionwhy? this scheme let''s you eliminate the "write-twice" scheme of raid1. ok, a load balancing is nicer, but raid1 seems the worst.> I will set up a live environment with at least 4 boxes. For this I''d like > to use proved techniques. And i also will set up a development and working > environment with two storage servers and about four Xen hosts. Here i can > experiment a little. So i can implement tested things in the live > environment.i''m getting green with envy... here in Peru is so hard to get any budget to experiment! all my tests so far are simulations with several QEMU instances.> The need is all and the budget is small ;-) > All jokes aside, i will try to make a highend solution with cheap hardware. > Just like RAID does but for servers. OK, the hardware will not that cheap. > But thinking about scsi vs. sata you can save a lot of money.don''t forget about the mechanical quality... i''ve been bitted (hard!) by not-so-good SATA backplanes that introduced lots of failures. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Dienstag, 31. Januar 2006 16:08 schrieb Javier Guerra:> On Tuesday 31 January 2006 10:00 am, Markus Hochholdinger wrote: > > I don''t like the thing with master slave in drbd. So the next version > why? this scheme let''s you eliminate the "write-twice" scheme of raid1. > ok, a load balancing is nicer, but raid1 seems the worst.correct. Writing twice is not that good. But my opinion is that this is easier to handle.> > I will set up a live environment with at least 4 boxes. For this I''d like > > to use proved techniques. And i also will set up a development and > > working environment with two storage servers and about four Xen hosts. > > Here i can experiment a little. So i can implement tested things in the > > live environment. > i''m getting green with envy... here in Peru is so hard to get any budget to > experiment! all my tests so far are simulations with several QEMU > instances.OK, this sounds a little big but the company i will do this for has at least three years spend no money for hardware (because of the bad economic situation). And now all the hardware is very old and also the economic situation is getting better. So i''ve to implement a solution that has a future.> > The need is all and the budget is small ;-) > > All jokes aside, i will try to make a highend solution with cheap > > hardware. Just like RAID does but for servers. OK, the hardware will not > > that cheap. But thinking about scsi vs. sata you can save a lot of money. > don''t forget about the mechanical quality... i''ve been bitted (hard!) by > not-so-good SATA backplanes that introduced lots of failures.Yeah, i know sata is not that good. But that''s the reason i will do it all full redundant. -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jan 31, 2006 at 03:08:35PM +0100, Molle Bestefich wrote:> After having used both, I trust MD over hardware controllers any day.The only thing that worries me about MD in 2.6.12 is that if you simulate a total disk failure (unplug the power cord in a running disk), MD just freezes the system for an hour (actually not the whole system, but any process which tries to access the filesystem on the array hangs until MD has done 100 retries, one every 30 seconds) as the disk appears "Busy", instead of just using the other disk. This _seems_ fixed in 2.6.14 (the code has changed, so I supposed it''s better now, but I haven''t actually tested this yet - maybe someone here has?). Otherwise typical disk failures get managed by MD just fine. Marcin -- Marcin Owsiany porridge@expro.pl _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Freitag, 3. Februar 2006 11:05 schrieb Marcin Owsiany:> On Tue, Jan 31, 2006 at 03:08:35PM +0100, Molle Bestefich wrote: > > After having used both, I trust MD over hardware controllers any day. > > The only thing that worries me about MD in 2.6.12 is that if you > simulate a total disk failure (unplug the power cord in a running disk), > MD just freezes the system for an hour (actually not the whole system, > but any process which tries to access the filesystem on the array hangs > until MD has done 100 retries, one every 30 seconds) as the disk appears > "Busy", instead of just using the other disk. > > This _seems_ fixed in 2.6.14 (the code has changed, so I supposed it''s > better now, but I haven''t actually tested this yet - maybe someone here > has?).as far as I know 2.6.12 doesn''t support hotplugging at all. It seems that this is the reason. If a device is not accessible anymore then the kernel will have a serious problem with that. I don''t know if 2.6.14 is really the kernel this problem is gone for the first time, but at least for 2.6.15 I know that for sure. more or less all servers (I saw in the past) with disk failures using mdadm on an < 2.6.15 kernel were not working normaly after a hdd died. All of them needed a reboot. all servers with >=2.6.15 doesn''t seem to have this problem anymore. Here an log from a server that "lost" a disk about 1 month ago (on 2.6.15): hdi: dma_intr: status=0x53 { DriveReady SeekComplete Index Error } hdi: dma_intr: error=0x86 { DriveStatusError BadCRC TrackZeroNotFound } ide: failed opcode was: unknown PDC202XX: Primary channel reset. ide4: reset: master: sector buffer error end_request: I/O error, dev hdi, sector 95980216 raid1: Disk failure on hdi, disabling device. Operation continuing on 1 devices raid1: hdi: rescheduling sector 95980216 RAID1 conf printout: --- wd:1 rd:2 disk 0, wo:0, o:1, dev:hde disk 1, wo:1, o:0, dev:hdi RAID1 conf printout: --- wd:1 rd:2 disk 0, wo:0, o:1, dev:hde raid1: hde: redirecting sector 95980216 to another mirror after that the system was running without any problems. --Ralph> Otherwise typical disk failures get managed by MD just fine. > > Marcin_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, 30. Januar 2006 07:55 schrieb Pasi Kärkkäinen:> On Mon, Jan 30, 2006 at 12:47:50AM +0100, Markus Hochholdinger wrote: > > Am Sonntag, 29. Januar 2006 22:04 schrieb Pasi Kärkkäinen: > > > In fact, even better would be to do the RAID in the storage arrays / > > > servers (=iSCSI targets), and provide multiple paths to the targets.. > > > using separate network (switch) for both paths. > > > In the dom0 you would only set up multipath device to the shared RAID > > > array (over iSCSI), and then create CLVM on top of that. No need to do > > > the RAID in dom0 :) > > and if the backplane of the storage server dies? Or just the hard disk > > controller in the storage? I''d like to have no signle point of failure > > :-) > Well, make them double. > Use two target boxes, use DRBD or similar to sync the data between the > boxes, and hearbeat to do failover (high-availability) between the boxes > (iSCSI targets).slowly i''m getting it :-) The Idea to make the raid stuff outside domU is very good and even better is to make the raid stuff on the storage servers themselves. So drbd is the right approach. But what i dislike is this heartbeat thing. Have you read the drbd docs? As long as i can see there could be some damage to the storage. If drdb splits (master disconnect from slave) it is possible that both gnbd servers get master and when reconnecting you have a big problem. Or I am wrong? OK, for us this scenario can hardly appear but it is possible (heartbeat fails, both drbd servers get master, gnbd client writes to one master, then the connection to this master fails and gnbd client writes to the other "master" => lose of date integrity). So my summery on this (improvements are welcome): * (storage1,drbd/gnbd/multipath) <=> (multipath/gnbd,dom0) <=> domU (storage2,drbd/gnbd/multipath) </ + traffic the real bytes over only one san. lesser cpu load for io on dom0. + redundancy over direct connection between san1 and san2. - need heartbeat, need cluster things, complicated (for me!?) - possible failure scenario * (storage1,gnbd) <=> (gnbd,dom0) <=> (domU,raid1) (storage1,gnbd) <=> (gnbd,dom0) </ + no heartbeat needed, failures are handled by the raid1 in domU + raid1 is well proved - twice the raffic, one time in san1 and one time in san2. - double the cpu load for io on dom0 - domU has to do raid1 (more cpu load?) So from my point of view i will take the disadvantages of the raid1 thing to get a easier setup. Also these techniques (nbd resp. gnbd, raid1) are well proven over many years. When the load gets to high on my dom0s I am also able to change to drdb.> http://www.pcpro.co.uk/realworld/82284/san-on-the-cheap.html > Tutorial there.Have to register in page three :-( But as long as i read they also use drbd. Are there any other solutions like drbd?> No need to have single point of failure :)I''m not sure if this is the case!? Only major failure scenarios are covered by solutions with drbd. As my example above shows there could be some kind of data corruption. And with Murphy''s Law i''ll get it. Or i''m wrong?> And now, you only need to have multipath setup in dom0, all the RAID stuff > is in the storage server. And all the volumes (in CLVM) are visible to all > the xen hosts.. so you can easily migrate domU''s from host to another, and > the domU''s know nothing about the underlying LVM/iSCSI/SAN/RAID.Really, i would like to use a thing like drbd but i don''t trust (understand?) it. The people from drbd also have no best answer (see http://svn.drbd.org/drbd/trunk/ROADMAP point 5). The idea is really good. Perhpas it would be best when the syncronisation is controlled by the host using the block devices (like raid1, but the data is send only one time). So the host is the only one accessing the block device. But this would assume that drbd and gnbd would go hand in hand. Please can anybody instruct me that i''m wrong with this drbd thing? -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 06 February 2006 6:49 pm, Markus Hochholdinger wrote:> So drbd is the right approach. But what i dislike is this heartbeat thing. > Have you read the drbd docs? As long as i can see there could be some > damage to the storage. If drdb splits (master disconnect from slave) it is > possible that both gnbd servers get master and when reconnecting you have a > big problem. Or I am wrong?use some kind of STOMOTH (Shoot The Other Machine On The Head). as soon as heartbeat fails, one of the hosts will kill the other. usual weapons are ethernet-managed power strips and manageable ethernet switches (to disable the other machine''s port). that way, you''re assured that if there won''t be two surviving masters. i don''t know if drbd (or heartbeat) includes support for that, but GFS (and therefore, CLVM) uses it extensively. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra wrote:> i think a better use of the resources would be: > > - two storage servers > - on each storage server: > ~ md raid to join all disks on one volume > ~ export volume with gnbd > ~ drbd to ''mirror'' both volumes, making it HAUsing gnbd over drbd sounds cool. Do you or does anyone else have a heartbeat resource script for gnbd (to be used with drbd)? Best Regards, Michael Paesold _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Mittwoch, 15. Februar 2006 17:32 schrieb Michael Paesold:> Javier Guerra wrote: > > i think a better use of the resources would be: > > > > - two storage servers > > - on each storage server: > > ~ md raid to join all disks on one volume > > ~ export volume with gnbd > > ~ drbd to ''mirror'' both volumes, making it HAI had a very well setup that looked like: Server1: Physical Disk / DRBD / LVM / GNBD | HA | Server2: Physical Disk / DRBD / LVM / GNBD connected via a gbit switch to 2 xen hosts. I used LVM on the "SAN"-Hosts to have single volumes that I can export to both xenhosts for every vm. Both xenhost could access the gnbd device at same (but ofcourse I just used a normal filesystem for the vm''s this wouldn''t be a good idea). But that makes live migration possible (and working well with such a setup). The DRBD devices are fault-tolerant because of heartbeat. It worked perfectly even if the active san hosts was switched off between a live migration of a vm from on xenhost to the other... On the next (most likely production) setup I will use multiple disk in each san server (to have better performance) and I think then the setup will look like: Physical Disk / RAID0 (Stripes) / DRBD / LVM / GNBD -Ralph> Using gnbd over drbd sounds cool. Do you or does anyone else have a > heartbeat resource script for gnbd (to be used with drbd)?no, only the normal drbd files but they are working very well (as long as gnbd itself is not the problem, and I never had problems with gnbd within my tests).> Best Regards, > Michael Paesold > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ralph Passgang wrote:> Physical Disk / RAID0 (Stripes) / DRBD / LVM / GNBDI was thinking about a similar setup. Although the other option mentioned above, i.e.: Physical / RAID ... / DRBD / GNBD / CLVM also has its merrits.>> Using gnbd over drbd sounds cool. Do you or does anyone else have a >> heartbeat resource script for gnbd (to be used with drbd)? > > no, only the normal drbd files but they are working very well (as long as > gnbd itself is not the problem, and I never had problems with gnbd within > my tests).I am just wondering what happens if you do /etc/init.d/heartbeat stop (or equivalent) on the current primary. The resource.d/drbddisk script will just do "drbdadm secondary $res", so the resource will suddenly become inaccessible to gnbd_serv. If you don''t stop it before, will it not crash or something? Ah well, there is still the LVM layer in your setup. So will LVM_stop(), i.e. vgchange -a -n $1 work at all while gnbd_serv is accessing the logical volumes? The clients will afterwards access the other host because of IP migration, ok, but what will happen at switching back to the first host without some cleanup? Well, I will setup a "cluster" of 4 servers for testing next week (two xen hosts, two "san" hosts). I will hopefully be able to answer these questions myself then. :-) Best Regards, Michael Paesold _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Donnerstag, 16. Februar 2006 11:13 schrieb Michael Paesold:> Ralph Passgang wrote: > > Physical Disk / RAID0 (Stripes) / DRBD / LVM / GNBD > > I was thinking about a similar setup. Although the other option mentioned > above, i.e.: > Physical / RAID ... / DRBD / GNBD / CLVM > also has its merrits. > > >> Using gnbd over drbd sounds cool. Do you or does anyone else have a > >> heartbeat resource script for gnbd (to be used with drbd)? > > > > no, only the normal drbd files but they are working very well (as long as > > gnbd itself is not the problem, and I never had problems with gnbd within > > my tests). > > I am just wondering what happens if you do /etc/init.d/heartbeat stop (or > equivalent) on the current primary. The resource.d/drbddisk script will > just do "drbdadm secondary $res", so the resource will suddenly become > inaccessible to gnbd_serv. If you don''t stop it before, will it not crash > or something? > > Ah well, there is still the LVM layer in your setup. So will LVM_stop(), > i.e. vgchange -a -n $1 work at all while gnbd_serv is accessing the logical > volumes?I have added two or three lines to the drbd ha scripts, but only to start lvm & gnbd if the host is primary and to stop lvm & gnbd if the drbd device goes to secondary state. nothing more... it was working very well. if I executed /etc/init.d/heartbeat stop on the primary host the secondary immidently took over the ip and started the gnbd export. for the xen systems that was completly transparent.> The clients will afterwards access the other host because of IP migration, > ok, but what will happen at switching back to the first host without some > cleanup?not needed, as long as the ha scripts start & stop lvm & gnbd correct. You can then just start ha on the primary host again without having two active nodes. ha takes care that only one host is really "primary". at least I never had any troubles with it.> Well, I will setup a "cluster" of 4 servers for testing next week (two xen > hosts, two "san" hosts). I will hopefully be able to answer these questions > myself then. :-)that sounds exactly like mine setup... I think you will be suprised how good this is really working. --Ralph> Best Regards, > Michael Paesold > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users