Hi Everyone, I''m currently mulling over the idea of running our Dom0 on a USB flash disk. Currently, we have the Dom0 and DomU running of the same disk array. I''d like to know the list''s opinion on doing this. It seems like a very nice idea to split the single critical Dom0 and the tens of DomUs. I''d like to stop the DomUs from affecting the disk IOPS available for the Dom0 as I''m sure you''ll appreciate the critical nature of the it. Any opinions are very much appreciated Thanks
On Tue, Dec 13, 2011 at 3:35 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> I''m currently mulling over the idea of running our Dom0 on a USB flash disk. > Currently, we have the Dom0 and DomU running of the same disk array. I''d > like to know the list''s opinion on doing this. It seems like a very nice > idea to split the single critical Dom0 and the tens of DomUs.better yet, make all of Dom0 fit in the rdimage, and pull it via PXE -- Javier
You can certainly do it, but I''d suggest using something like PXEboot instead. On Tue, Dec 13, 2011 at 3:35 PM, Jonathan Tripathy <jonnyt@abpni.co.uk>wrote:> Hi Everyone, > > I''m currently mulling over the idea of running our Dom0 on a USB flash > disk. Currently, we have the Dom0 and DomU running of the same disk array. > I''d like to know the list''s opinion on doing this. It seems like a very > nice idea to split the single critical Dom0 and the tens of DomUs. > > I''d like to stop the DomUs from affecting the disk IOPS available for the > Dom0 as I''m sure you''ll appreciate the critical nature of the it. > > Any opinions are very much appreciated > > Thanks > > ______________________________**_________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/**xen-users<http://lists.xensource.com/xen-users> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 13/12/2011 20:41, Javier Guerra Giraldez wrote:> On Tue, Dec 13, 2011 at 3:35 PM, Jonathan Tripathy<jonnyt@abpni.co.uk> wrote: >> I''m currently mulling over the idea of running our Dom0 on a USB flash disk. >> Currently, we have the Dom0 and DomU running of the same disk array. I''d >> like to know the list''s opinion on doing this. It seems like a very nice >> idea to split the single critical Dom0 and the tens of DomUs. > better yet, make all of Dom0 fit in the rdimage, and pull it via PXEThanks for the suggestion. Can you please explain to me what an rdimage is? Also, I''m guessing that PXE is more reliably than USB sticks as PXE images can ultimately come from a metal spindle on another server? Thanks
On 13/12/2011 20:53, Jonathan Tripathy wrote:> > On 13/12/2011 20:41, Javier Guerra Giraldez wrote: >> On Tue, Dec 13, 2011 at 3:35 PM, Jonathan >> Tripathy<jonnyt@abpni.co.uk> wrote: >>> I''m currently mulling over the idea of running our Dom0 on a USB >>> flash disk. >>> Currently, we have the Dom0 and DomU running of the same disk array. >>> I''d >>> like to know the list''s opinion on doing this. It seems like a very >>> nice >>> idea to split the single critical Dom0 and the tens of DomUs. >> better yet, make all of Dom0 fit in the rdimage, and pull it via PXE > Thanks for the suggestion. Can you please explain to me what an > rdimage is? Also, I''m guessing that PXE is more reliably than USB > sticks as PXE images can ultimately come from a metal spindle on > another server? > > ThanksAh silly me! That''s the initrd of course. Does anyone have any tips on how to put an entire Dom0 in the initrd? Would an alternative to this be the nfs mount the root filesystem (Assuming that Xen and Dom0 kernel came from a PXE server)? Thanks
Jonathan Tripathy wrote:>I''m currently mulling over the idea of running our Dom0 on a USB >flash disk. Currently, we have the Dom0 and DomU running of the same >disk array. I''d like to know the list''s opinion on doing this. It >seems like a very nice idea to split the single critical Dom0 and >the tens of DomUs. > >I''d like to stop the DomUs from affecting the disk IOPS available >for the Dom0 as I''m sure you''ll appreciate the critical nature of >the it.I''m not sure what you''d gain. Once the system is running, Dom0 shouldn''t be doing much (if any) disk I/O - and if it is, then doing it to a USB memory stick is likely to be more of a bottleneck to the system as a whole than a few deeks on the hard disk(s). -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books.
On 13/12/2011 22:28, Simon Hobson wrote:> Jonathan Tripathy wrote: > >> I''m currently mulling over the idea of running our Dom0 on a USB >> flash disk. Currently, we have the Dom0 and DomU running of the same >> disk array. I''d like to know the list''s opinion on doing this. It >> seems like a very nice idea to split the single critical Dom0 and the >> tens of DomUs. >> >> I''d like to stop the DomUs from affecting the disk IOPS available for >> the Dom0 as I''m sure you''ll appreciate the critical nature of the it. > > I''m not sure what you''d gain. Once the system is running, Dom0 > shouldn''t be doing much (if any) disk I/O - and if it is, then doing > it to a USB memory stick is likely to be more of a bottleneck to the > system as a whole than a few deeks on the hard disk(s).Hi Simon, Sometimes during times when the DomUs are busy, xm commands take a long time to execute. I always presumed that this was due to disk IO. Could it be something else? Thanks
On 13/12/2011 22:55, xenusers wrote:> > > On Tue, Dec 13, 2011 at 4:39 PM, Jonathan Tripathy <jonnyt@abpni.co.uk > <mailto:jonnyt@abpni.co.uk>> wrote: > > > On 13/12/2011 22:28, Simon Hobson wrote: > > Jonathan Tripathy wrote: > > I''m currently mulling over the idea of running our Dom0 on > a USB flash disk. Currently, we have the Dom0 and DomU > running of the same disk array. I''d like to know the > list''s opinion on doing this. It seems like a very nice > idea to split the single critical Dom0 and the tens of DomUs. > > I''d like to stop the DomUs from affecting the disk IOPS > available for the Dom0 as I''m sure you''ll appreciate the > critical nature of the it. > > > I''m not sure what you''d gain. Once the system is running, Dom0 > shouldn''t be doing much (if any) disk I/O - and if it is, then > doing it to a USB memory stick is likely to be more of a > bottleneck to the system as a whole than a few deeks on the > hard disk(s). > > Hi Simon, > > Sometimes during times when the DomUs are busy, xm commands take a > long time to execute. I always presumed that this was due to disk > IO. Could it be something else? > > Thanks > > > Hi Jonathan, > > I don''t know what hardware you''re running on, but I can tell you that > I''m running Xen on some clustered PowerEdge R815s (48 cores, 64GB > RAM). During initial setup and testing, I hadn''t pinned Dom0 to just > specific cores, we were just letting it run as-is while we were > ironing out the pacemaker configuration to cluster them. The Xen > resource agent was occasionally timing out (meaning an ''xm list --long > <domu name>'' was taking more than 30 seconds to complete) and > restarting the DomUs. I was somewhat baffled, as these servers were > oviously very bored with a testing-only load of a few guests. I could > run the command in a loop, sleeping for a second each time, and sure > enough it was rather slow... very frequently taking 10 to 15 seconds > to complete when there were two DomUs running, occasionally 45 seconds > or more. Similarly, ''xm info'' would take ages to return every few > times it was run. > > Long story short (too late?), I added the "dom0_max_vcpus=2 > dom0_vcpus_pin" arguments to the grub commandline for Xen. Configured > all DomUs to stay off of those cores with ''cpus="2-47"'' in their > config files. No other changes made to the setup at all, but now the > commands return almost instantly, every single time. Giving dom0 only > two cores completely resolved the issue for us. > > Regards, > MarkHi Mark, Thanks for sharing the above info. Indeed, my issue could be related to CPU. I have found that the issue isn''t so bad on our newer servers (which have better CPUs in them). Unfortunately, we don''t have enough cores to be able to dedicate an entire core to the Dom0. I guess this isn''t the worst issue in the world for us, as the delay is acceptable on our newer servers. Now, on our older servers (which are due to be replaced fairly soon), the times can be as long as 30 - 60 seconds which is clearly too long! Cheers Jonathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, tl;dr: - It''s a good idea and works great once you got it right - Don''t waste time doing it with a normal distro: instead help make Xen work nicely on Alpine Linux which is designed for robustness / usb boot - check you''re not using sched-credit2, Then use sched-credit -d Domain-0 -w 5120 -c 130 on your dom0. You can still pin it to specific cpus. Note you can use xenpm to find out which cpus are real ones and which are just HT. 2011/12/13 Jonathan Tripathy <jonnyt@abpni.co.uk>:> Hi Everyone,> > I''m currently mulling over the idea of running our Dom0 on a USB flash disk. > Currently, we have the Dom0 and DomU running of the same disk array. I''d > like to know the list''s opinion on doing this. It seems like a very nice > idea to split the single critical Dom0 and the tens of DomUs. > > I''d like to stop the DomUs from affecting the disk IOPS available for the > Dom0 as I''m sure you''ll appreciate the critical nature of the it.Most people, surprisingly, don''t.> Any opinions are very much appreciatedwe''ve done this in a large ha production system. HA not in the standard "we run important online shop" but in the older "yes, sure make a cluster. but btw: it''s simply not allowed to fail, period" sense. The good: Different fault domains. dom0 IO not impaired by heavy IO domUs, like you described. This means dom0 is fully operable even if the host is running multiple domUs pushing many K IOPS / manu 100 MB/s all RAID disks are strictly for "payload". The bad: USB sticks fail more easily than Raid disks, hotplug is a nightmare and udev is a failure. The ugly: Linux block device persistence in 2.16.XX is a mere joke, old CentOS kernels will re-enumerate a usb stick that comes back from sleep after being idle too long, and while they don''t have command that allows to disable powersaving, they seem to have it enabled... You can of course setup persistent device names, BUT udev fails anyway since it''s all running via callout shell scripts, and udev fails even more since it does not really change the kernel device names, only the stuff in /dev/ is affected. Even if you have rules that name your "replugged" root device "/dev/myUSBwonder" it will not help at all since the kernel device has still changed to "the usb thing right next to where the old usb thing was". That means you lose the root device node and thats it. In case anybody missed it: Yes, I think doing a kernels job with shellscripts in user space is not the smartest thing on earth. Most CentOS code proved to be quite fragile, and not even a reboot works if you have a disk issue of any notable scale. Why would anybody test what happens if things go wrong, right? The upside: We wanted stuff to be really robust. Using /etc/sysconfig/readonlyroot + rwtab tweakimg you can actually load 90% of the system into RAM and still keep 10% on readonly USB (i.e. stuff in /etc/ that you want to be change-able. We now have a system that *will* work for months even if the boot media fails. This allows for the OS to feel "unmodified" in case you need to apply some post-install updates w/o a reboot. Now on a failure of root usb thing very little bad can happen, and any failure on the Raid is something we can almost laugh at. So, if you need to build a highly robust embedded solution on basis of a standard distro system, and bring along a few dozen hours, then go for it. I''m quite happy now that things ended up pretty solid. Re: PXE booting: Well, if you want all your hosts to be in a reboot loop after a massive power failure until the DHCP is back? Otherwise, try PXE, then rather boot from some solid media (unless that is not installed via PXE yet. then... usual) Now for something better: But, alternatively, if you *wanna* spend time on such a thing, it would be even cooler if you joined #alpine-linux on irc at some time and help polishing the Xen port. Because Alpine has all the embedded-ultra-robustness stuff builtin. It was designed by network engineers and is much closer to what you want. Florian -- the purpose of libvirt is to provide an abstraction layer hiding all xen features added since 2006 until they were finally understood and copied by the kvm devs.
Andreas Balg
2013-Jan-26 16:23 UTC
Boot XCP from USB Stick - Anybody already did it - How might this be done ?
Hello everybody, we''d like to install and boot the XCP (1.6) Hypervisor from a USB-Stick - Why? We''d like to migrate from a couple of ESXi-Servers -which are in fact IBM-Blades sold with the ESX integrated. Those are installed on 2GB USB-Sticks as well that are installed inside the blades. So far I''ve been able to find some industrial grade USB-Sticks of 4Gb/8Gb that are specified to operate at temperatures up to 85°C (inside a blade server) and are built using SLC-flash. Those are specified at MTBF of 5.000.000 hours of operation without the usual wear of flash cells - even at high I/O rates. So this should easily be very good for reliable dom0 operation. DomUs are planned to be kept on FC-SAN - But I absolutely do not want to boot from there Why not install it on some small HDD? - those purpose built IBM Blade-Servers have no internal connectors for sas or sata disks but are already installed at that customer site Anybody been able to compile usb-drivers for the installer or a kernel with built in drivers for USB or create a bootable image to be copied directly to a stick? Any ideas of how to get this started ? Anybody did it before? Cheers Andreas Balg Am 13.12.2011 21:35, schrieb Jonathan Tripathy:> Hi Everyone, > > I''m currently mulling over the idea of running our Dom0 on a USB flash > disk. Currently, we have the Dom0 and DomU running of the same disk > array. I''d like to know the list''s opinion on doing this. It seems > like a very nice idea to split the single critical Dom0 and the tens > of DomUs. > > I''d like to stop the DomUs from affecting the disk IOPS available for > the Dom0 as I''m sure you''ll appreciate the critical nature of the it. > > Any opinions are very much appreciated > > Thanks > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users
Alexandre Kouznetsov
2013-Jan-28 16:07 UTC
Re: Boot XCP from USB Stick - Anybody already did it - How might this be done ?
Hello. El 26/01/13 10:23, Andreas Balg escribió:> So far I''ve been able to find some industrial grade USB-Sticks of > 4Gb/8Gb that are specified to operate at temperatures up to 85°C (inside > a blade server) and are built using SLC-flash. Those are specified at > MTBF of 5.000.000 hours of operation without the usual wear of flash > cells - even at high I/O rates. So this should easily be very good for > reliable dom0 operation. DomUs are planned to be kept on FC-SAN - But I > absolutely do not want to boot from thereCould you share the brand/model? Kingston USB thumbs I have used started to behave weired after some months of operation, so I dropped that setup.> Anybody been able to compile usb-drivers for the installer or a kernel > with built in drivers for USB or create a bootable image to be copied > directly to a stick?Yes, it''s not a big deal. I have don it with XCP 1.1 and 1.5, surely the setup of 1.6 would be similar. Check this last year''s thread. There I provide the links to the reference I used: http://lists.xen.org/archives/html/xen-users/2012-04/msg00236.html -- Alexandre Kouznetsov