Hello, I''d like to ask if someone has experience with setting up Xen (I use 3.2 from Lenny) in a diskless scenario - so not just the DomUs on a central storage (therefore, I use open-iscsi), but also Xen itself and the Dom0? I tried that because it would make adding nodes and other changings quite easy, and save disks. I used PXE (with mboot.c32) and NFS-Root, and it worked quite well after building a nfsroot capable initrd and considering that the network device used for nfsroot is not usable for building a bridge (without some initramfs customizations, I read) because the link will be lost when the bridge comes up. But when I tried to live migrate DomUs between the nodes, I faced severe troubles. Migrations failed because, AFAIR, the block device frontend couldn''t be disconnected? After that, I couldn''t even get block device stats from libvirt... Has anyone done such a setup, preferrably even with iSCSI, but without dedicated hardware such as HBAs etc.? Or does someone know a link to a howto? I''ve searched the mailing list and Google quite some time, and surprisingly, most people care only about central storage for domUs... tia & kr, tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Oct 28, 2009 at 10:53 AM, Thomas Treutner <thomas@scripty.at> wrote:> Hello, > > I''d like to ask if someone has experience with setting up Xen (I use 3.2 from > Lenny) in a diskless scenario - so not just the DomUs on a central storage > (therefore, I use open-iscsi), but also Xen itself and the Dom0?not first-hand experience; but a couple advices: - be sure to initiate the iSCSI devices on Dom0. IOW, the DomU''s won''t know their storage isn''t local. That means the same LUNs have to be available on every Dom0, so they''ll be ready when the DomU gets migrated. - you can get rid of NFS totally, either put your Dom0 in iSCSI too, or just keep the initrd. there''s no requirement for initrd to yield to another root. in either case, you''ll find it much easier to manage. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thomas Treutner wrote:> Hello, > > I''d like to ask if someone has experience with setting up Xen (I use 3.2 from > Lenny) in a diskless scenario - so not just the DomUs on a central storage > (therefore, I use open-iscsi), but also Xen itself and the Dom0? > > I tried that because it would make adding nodes and other changings quite > easy, and save disks. I used PXE (with mboot.c32) and NFS-Root, and it worked > quite well after building a nfsroot capable initrd and considering that the > network device used for nfsroot is not usable for building a bridge (without > some initramfs customizations, I read) because the link will be lost when the > bridge comes up. > > But when I tried to live migrate DomUs between the nodes, I faced severe > troubles. Migrations failed because, AFAIR, the block device frontend > couldn''t be disconnected? After that, I couldn''t even get block device stats > from libvirt... > > Has anyone done such a setup, preferrably even with iSCSI, but without > dedicated hardware such as HBAs etc.? Or does someone know a link to a howto? > I''ve searched the mailing list and Google quite some time, and surprisingly, > most people care only about central storage for domUs... >This is almost exactly what I am trying to do, except voor the NFS Root part. I want all to be available n the iSCSI storage and the virtual environment, and that means my NFS server is also a virtual one. That must be up to be able to start Dom0, but I need Dom0 to start the DomU actins as the NFS server.... So I choose to do the latter: I''ve built a Xen3.2 live install on a USB disk, and use that as the local storage for Dom0. I do have HBAs, but for now, I cannot use the hardware iSCSI support, because the driver is not yet available for my current kernel. The storage is indeed iSCSI, with the open-iSCSI initiator (also in the initrd). For the DomUs, I use a nilfs partition on the iSCSI storage, so this can be shared by the different Dom0s. I''m still in the development phase of this setup. I am also considering to put the Dom0 root on iSCSI, just leaving the kernel and initrd on the USB disk. The final step is migrating to iSCSI boot, as soon as the drivers become available. Putting that on a PXE server should be easy to implement. As I was saying, I''m still developing this, and I did not yet reach the point of live migration of DomUs. You know, normal work also continues.... The next 2 weeks I''m working somewhere else, putting this on the low-priority list... Hope this helps, Marcel> > tia & kr, > tom > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users- -- - --------------------------------------------------------------- ing. Marcel van Dorp (CCDP, CCNP, CCSP) WiWo Support http://www.wiwo.nl Postbus 1098 tel. 071-523 77 91 2340 BB Oegstgeest gsm 0653-50 77 76 - --------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrobXwACgkQIA1+xqauveotowCdGYB8yGRnMWFq83r3G0momWw4 T4EAn2llIVU/o1dPJdy2oUFq3e0M+NDo =p+vL -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Oct 28, 2009 at 9:12 AM, Marcel van Dorp <M.vanDorp@wiwo.nl> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Thomas Treutner wrote: > > Hello, > > > > I''d like to ask if someone has experience with setting up Xen (I use 3.2 > from > > Lenny) in a diskless scenario - so not just the DomUs on a central > storage > > (therefore, I use open-iscsi), but also Xen itself and the Dom0? > > > > I tried that because it would make adding nodes and other changings quite > > easy, and save disks. I used PXE (with mboot.c32) and NFS-Root, and it > worked > > quite well after building a nfsroot capable initrd and considering that > the > > network device used for nfsroot is not usable for building a bridge > (without > > some initramfs customizations, I read) because the link will be lost when > the > > bridge comes up. > > > > But when I tried to live migrate DomUs between the nodes, I faced severe > > troubles. Migrations failed because, AFAIR, the block device frontend > > couldn''t be disconnected? After that, I couldn''t even get block device > stats > > from libvirt... > > > > Has anyone done such a setup, preferrably even with iSCSI, but without > > dedicated hardware such as HBAs etc.? Or does someone know a link to a > howto? > > I''ve searched the mailing list and Google quite some time, and > surprisingly, > > most people care only about central storage for domUs... > > > > This is almost exactly what I am trying to do, except voor the NFS Root > part. I want all to be available n the iSCSI storage and the virtual > environment, and that means my NFS server is also a virtual one. That > must be up to be able to start Dom0, but I need Dom0 to start the DomU > actins as the NFS server.... So I choose to do the latter: > > I''ve built a Xen3.2 live install on a USB disk, and use that as the > local storage for Dom0. I do have HBAs, but for now, I cannot use the > hardware iSCSI support, because the driver is not yet available for my > current kernel. > > The storage is indeed iSCSI, with the open-iSCSI initiator (also in the > initrd). For the DomUs, I use a nilfs partition on the iSCSI storage, so > this can be shared by the different Dom0s. > > I''m still in the development phase of this setup. I am also considering > to put the Dom0 root on iSCSI, just leaving the kernel and initrd on the > USB disk. The final step is migrating to iSCSI boot, as soon as the > drivers become available. Putting that on a PXE server should be easy to > implement. > > As I was saying, I''m still developing this, and I did not yet reach the > point of live migration of DomUs. You know, normal work also continues.... > > The next 2 weeks I''m working somewhere else, putting this on the > low-priority list... > > Hope this helps, > > Marcel > > > > > > tia & kr, > > tom >Are you using pv or hvm domUs? Could you post your grub and domU config files? I''ve been trying to get hvm domUs booting from NFS root filesystems with not much success. Thanks -Bruce _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wednesday 28 October 2009 17:07:39 Javier Guerra wrote:> On Wed, Oct 28, 2009 at 10:53 AM, Thomas Treutner <thomas@scripty.at> wrote: > > Hello, > > > > I''d like to ask if someone has experience with setting up Xen (I use 3.2 > > from Lenny) in a diskless scenario - so not just the DomUs on a central > > storage (therefore, I use open-iscsi), but also Xen itself and the Dom0? > > not first-hand experience; but a couple advices:thanks for your input!> - be sure to initiate the iSCSI devices on Dom0. IOW, the DomU''s > won''t know their storage isn''t local.Yep, I did it that way, using a block-iscsi script floating around in the Internet somewhere and then iscsi: instead of phy: etc.> That means the same LUNs have > to be available on every Dom0, so they''ll be ready when the DomU gets > migrated.Of course. I think I resolved my problems in the second try. Problem was I was changing my IP subnet after I discovered the nfsroot/bridging troubles to separate that clearly. Being new to open-iscsi, I wasn''t aware that in /etc/iscsi/nodes and sendtargets old information was lurking around ;-) So the iSCSI-initiator was confused about which path to take (my target host was still reachable at the old address too) and had hickups. After resolving that the initiator still tried at the old address first, which ran into a timeout, so live migration was very slow (~20s of waiting for vdb actions). Resolved that too and now works like a charm!> - you can get rid of NFS totally, either put your Dom0 in iSCSI too,I''ve searched for that, it seems to be quite a hack without HBAs to me, even for vanilla kernels.> or just keep the initrd. there''s no requirement for initrd to yield > to another root. in either case, you''ll find it much easier to > manage.Hm, that would be interesting. So the Dom0 would reside in a ramdisk? Wouldn''t one need quite some (additional) amounts of RAM for that? I have a node with just 2G which isn''t very much for Xen anyways, when subtracting 256-512M for Dom0. And I think I would need something like NFS anyways for the DomU.cfgs, if I don''t want to regenerate the initrds every time. But thanks anyways for your ideas! kr, tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Oct 28, 2009 at 1:45 PM, Thomas Treutner <thomas@scripty.at> wrote:> On Wednesday 28 October 2009 17:07:39 Javier Guerra wrote: >> - be sure to initiate the iSCSI devices on Dom0. IOW, the DomU''s >> won''t know their storage isn''t local. > > Yep, I did it that way, using a block-iscsi script floating around in the > Internet somewhere and then iscsi: instead of phy: etc.you can keep the phy: if you attach the iSCSI devices to all Dom0''s. that way, for Xen the LUNs are just block devices. i have no idea if that implies more (or less) overhead, though.>> - you can get rid of NFS totally, either put your Dom0 in iSCSI too, > > I''ve searched for that, it seems to be quite a hack without HBAs to me, even > for vanilla kernels.no need for special hardware; just put the iSCSI drivers in your initrd and do the LUN attaching before switching roots (and after setting up the bridge).>> or just keep the initrd. there''s no requirement for initrd to yield >> to another root. in either case, you''ll find it much easier to >> manage. > > Hm, that would be interesting. So the Dom0 would reside in a ramdisk? Wouldn''t > one need quite some (additional) amounts of RAM for that? I have a node witha small linux distro fits in less than 30MB of disk (even less if you use busybox instead of the full GNU tools) and if you use cramfs for the initrd, you can keep it compressed while mounted in RAM. the Xen daemons aren''t too big either. the biggest requirement is Python, IFRC.> just 2G which isn''t very much for Xen anyways, when subtracting 256-512M for > Dom0. And I think I would need something like NFS anyways for the DomU.cfgs, > if I don''t want to regenerate the initrds every time.you could fetch the configs from a central config server via http, or tftp... makes you less dependent on the continued availability of the network (significant only if iSCSI is on a separate network) -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users