Hey all, I''m thinking about moving my 2.0.7 install over to 3.0.0 for eventual use in a production environment. Two questions: 1) Do people think 3.0.0 is ready for production - or is it just a testing/unstable playpen? 2) What do I have to change - just the xen-2.0.7.gz for do I have to recompile the dom0/domU kernels too? Cheers, Matthew Walster _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello! On Jan 17, 2006 at 1040 +0000, Matthew Walster appeared and said:> [...] > I''m thinking about moving my 2.0.7 install over to 3.0.0 for eventual use in a > production environment. > > Two questions: > > 1) Do people think 3.0.0 is ready for production - or is it just a > testing/unstable playpen? > [...]A co-worker runs Xen 2.0.x. I used his system for testing and installed a Xen 3.0.0 test system in the office and a Xen 3.0.0 "production test" system on a co-located server. Both system runs stable for over four weeks, no reboots, no strange behaviour. If you really want to upgrade, why not make a copy of your guest domains on a test machine running Xen 3.0.0 and see how they are doing? The labels "stable", "testing" and "unstable" are usually defined by a set of requirements which vary from setup to setup. I''d say Xen 3.0.0 is stable for me, the ones that experience the random reboots probably won''t agree. Best, Lynx. -- "From the delicate strands, between minds we weave our mesh: a blanket to warm the soul." --- Lady Deirdre Skye (SMAC) --- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Matthew Walster wrote:> Hey all, > > I''m thinking about moving my 2.0.7 install over to 3.0.0 for eventual use in a > production environment. > > Two questions: > > 1) Do people think 3.0.0 is ready for production - or is it just a > testing/unstable playpen? >We''ve seen some strange behavior with migration and one incident where we could not create new XenU''s. As long as you stay away from migration I''d guess you''re safe. Per. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Mittwoch, 18. Januar 2006 20:10 schrieb Per Andreas Buer:> Matthew Walster wrote: > > Hey all, > > > > I''m thinking about moving my 2.0.7 install over to 3.0.0 for eventual use > > in a production environment. > > > > Two questions: > > > > 1) Do people think 3.0.0 is ready for production - or is it just a > > testing/unstable playpen? > > We''ve seen some strange behavior with migration and one incident where > we could not create new XenU''s. As long as you stay away from migration > I''d guess you''re safe.migration is working very well for us, but you have to be carefully on some aspects before trying to migrate productions domU''s... 1) you have to use the same xen version on all xen hosts that are involved in migrating... so not 32bit xen on one machine and 32bit + pae or even 64bit xen on the other... 2) you should use the absoulty exact kernel, or at least doesn''t optimize it for another architecture or something like that The kernel have to be available at the same location at every xen host. 3) you have to use a san (or at least something that works like a san)... If you want a opensource solution which doesn''t cost money and want to use (maybe old) standard servers for providing blockdevices to the xen machines, then I can recommend gnbd (from redhat''s cluster tools) for exporting block devices to more then one xen host at the same time (but as long as you don''t use gfs as fs, you should not try to mount it more than once). 4) you cannot migrate a domU from for example a xen to a celeron without that the domU has to be restartet. But you can migrate the very same domU from a celeron to xeon. (After you migrated it from a celeron to a xeon, you will be able to migrate it back to celeron)... I am not quite sure why the effect from my number 4 exists. I guess because the celeron has not as many instruction sets (for example mmx, sse, nx, and so on) but the domU needs the same sets to keep running after migrating.> Per.--Ralph> > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> migration is working very well for us, but you have to be carefully on some > aspects before trying to migrate productions domU''s... > > 1) you have to use the same xen version on all xen hosts that are involved > in migrating... so not 32bit xen on one machine and 32bit + pae or even > 64bit xen on the other...Have you ever tried this? Sounds like it could be quite entertaining :-)> 2) you should use the absoulty exact kernel, or at least doesn''t optimize > it for another architecture or something like that The kernel have to be > available at the same location at every xen host.Having the kernel in the same place on every host is only important for a reboot though - the migration itself should be fine. The long term solution to this should probably be to boot a kernel off the domain filesystem.> 3) you have to use a san (or at least something that works like a san)... > If you want a opensource solution which doesn''t cost money and want to use > (maybe old) standard servers for providing blockdevices to the xen > machines, then I can recommend gnbd (from redhat''s cluster tools) for > exporting block devices to more then one xen host at the same time (but as > long as you don''t use gfs as fs, you should not try to mount it more than > once).Yep.> 4) you cannot migrate a domU from for example a xen to a celeron without > that the domU has to be restartet. But you can migrate the very same domU > from a celeron to xeon. (After you migrated it from a celeron to a xeon, > you will be able to migrate it back to celeron)... > > I am not quite sure why the effect from my number 4 exists. I guess because > the celeron has not as many instruction sets (for example mmx, sse, nx, and > so on) but the domU needs the same sets to keep running after migrating.Linux enables more optimisations if it detects it''s booting on a newer processor. If you boot it on something where it can use these optimisations and then move it to a machine where it can''t, things will break nastily. Of course if you disable this (never done it, so not sure how to) then it won''t be an issue, at the expense of losing some efficiency on the more modern machines. The features of the Celeron are a strict subset of the Xeon, so if Linux boots on the Celeron it will only ever use things that the Celeron has - hence the behaviour you''re seeing. Out of interest for Xeon->Celeron migration did you see an immediate crash? Probably would be an invalid opcode exception of some kind. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi List, from our side we faced some mayor features missing in 3-testing. As the hide option for physical devices in dom0 seem not to work at all and for personal named interfaced a patch is needed we decided to stay at xen2. Unfortunatly there was no reliable roadmap on these issues from the developpers by now. Hope they''ll keep us updated cheers Mathias -----Original Message----- Sent: Mittwoch 18.01.06 20:10:52 Subject: Re: [Xen-users] 2.0.7 -> 3.0.0 upgrade>Matthew Walster wrote: >> Hey all, >> >> I''m thinking about moving my 2.0.7 install over to 3.0.0 for eventual use in a >> production environment. >> >> Two questions: >> >> 1) Do people think 3.0.0 is ready for production - or is it just a >> testing/unstable playpen? >> > >We''ve seen some strange behavior with migration and one incident where >we could not create new XenU''s. As long as you stay away from migration >I''d guess you''re safe. > >Per. > >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Donnerstag, 19. Januar 2006 14:50 schrieb Mark Williamson:> > migration is working very well for us, but you have to be carefully on > > some aspects before trying to migrate productions domU''s... > > > > 1) you have to use the same xen version on all xen hosts that are > > involved in migrating... so not 32bit xen on one machine and 32bit + pae > > or even 64bit xen on the other... > > Have you ever tried this? Sounds like it could be quite entertaining :-)no, but I think it will not even be possible to migrate, because xen detects the missmatch, but that''s just a guess. I have not tried it yet :) If you try to boot a dom0 with a pae enabled kernel on a non pae hypervisor (for example), then xen notices that and will not boot, so maybe it''s the same for migrating.> > 2) you should use the absoulty exact kernel, or at least doesn''t optimize > > it for another architecture or something like that The kernel have to be > > available at the same location at every xen host. > > Having the kernel in the same place on every host is only important for a > reboot though - the migration itself should be fine. The long term > solution to this should probably be to boot a kernel off the domain > filesystem.My long term solution is to have the kernel images on a san too, so that for every xen host the very same kernels are available... but your right, as long as you don''t want to reboot you wouldn''t need the kernel on the second host, I guess... But the first time I tried the migration I tried it from the xeon -> celeron and because the domain reboots immidently I got a error: "kernel not available/file not found" or something like this.> > 3) you have to use a san (or at least something that works like a san)... > > If you want a opensource solution which doesn''t cost money and want to > > use (maybe old) standard servers for providing blockdevices to the xen > > machines, then I can recommend gnbd (from redhat''s cluster tools) for > > exporting block devices to more then one xen host at the same time (but > > as long as you don''t use gfs as fs, you should not try to mount it more > > than once). > > Yep.I had a testsetup like this: GNBD Server1: hdd <--> drbd <--> lvm <--> gnbd | | (sync via network) | GNBD Server2: hdd <--> drbd (<--> lvm <--> gnbd) together with heartbeat for a failover if the master gnbd hosts fails... I imported the gnbd devices in two xen hosts for testing live migration. I was really impressed, it worked almost out of the box... live migration in about 5-10 seconds with a downtime definitly <100ms. and that statefull... xen is great ;-P I can even turn off the gnbd master (and having heartbeat detecting that and making the failover in under 1 sec.) and having a live migration running in the very same moment. no problem at all... that even impressed my boss :-)> > 4) you cannot migrate a domU from for example a xen to a celeron without > > that the domU has to be restartet. But you can migrate the very same domU > > from a celeron to xeon. (After you migrated it from a celeron to a xeon, > > you will be able to migrate it back to celeron)... > > > > I am not quite sure why the effect from my number 4 exists. I guess > > because the celeron has not as many instruction sets (for example mmx, > > sse, nx, and so on) but the domU needs the same sets to keep running > > after migrating. > > Linux enables more optimisations if it detects it''s booting on a newer > processor. If you boot it on something where it can use these > optimisations and then move it to a machine where it can''t, things will > break nastily.yeah, the domU gets restarted immidently, or at least very very fast...> Of course if you disable this (never done it, so not sure > how to) then it won''t be an issue, at the expense of losing some efficiency > on the more modern machines.I really would like to know how... We will probablly not have the very exact cpu in all xen hosts in future, so at the moment I would need to start a domU on the "oldest" machine and migrating it to the final host where it should run, but that is not very elegant, but needed if you want to be sure that you can move the domU later on to every other available host.> The features of the Celeron are a strict subset of the Xeon, so if Linux > boots on the Celeron it will only ever use things that the Celeron has - > hence the behaviour you''re seeing.I already was quite sure that exactly this was the problem. But now I can be sure, thanks... Btw... it''s the very same for a celeron and a sempron... you can migrate from celeron to sempron, but not the otherway around (if the domain was initially started on the sempron)...> Out of interest for Xeon->Celeron migration did you see an immediate crash? > Probably would be an invalid opcode exception of some kind.no... on the target host it looks like the domain was just started... in the moment the migration is finished the domain gets restarted... it''s to fast to have a "xm console" running on the target host... xm dmesg doesn''t show anything interessting... but I have not looked in /var/log/xend.log... Next time I try I will check that and let you know.> Cheers, > Mark--Ralph _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > > 1) you have to use the same xen version on all xen hosts that are > > > involved in migrating... so not 32bit xen on one machine and 32bit + > > > pae or even 64bit xen on the other... > > > > Have you ever tried this? Sounds like it could be quite entertaining :-) > > no, but I think it will not even be possible to migrate, because xen > detects the missmatch, but that''s just a guess. I have not tried it yet :)The tools *might* disallow a migration, I''ve never checked. I hope they do :-D> If you try to boot a dom0 with a pae enabled kernel on a non pae hypervisor > (for example), then xen notices that and will not boot, so maybe it''s the > same for migrating.It''s a slightly different mechanism since dom0 is a special case.> My long term solution is to have the kernel images on a san too, so that > for every xen host the very same kernels are available... > > but your right, as long as you don''t want to reboot you wouldn''t need the > kernel on the second host, I guess... But the first time I tried the > migration I tried it from the xeon -> celeron and because the domain > reboots immidently I got a error: "kernel not available/file not found" or > something like this.Yep, it''d be because of the immediate reboot showing the lack of kernel file, without the crash / reboot, you''d have been fine.> I can even turn off the gnbd master (and having heartbeat detecting that > and making the failover in under 1 sec.) and having a live migration > running in the very same moment. no problem at all... that even impressed > my boss :-)Now *that''s* a cool demo :-)> > The features of the Celeron are a strict subset of the Xeon, so if Linux > > boots on the Celeron it will only ever use things that the Celeron has - > > hence the behaviour you''re seeing. > > I already was quite sure that exactly this was the problem. But now I can > be sure, thanks... > > Btw... it''s the very same for a celeron and a sempron... you can migrate > from celeron to sempron, but not the otherway around (if the domain was > initially started on the sempron)...Could be the use of AMD specific instructions. For instance, the AMD supports all the Celeron''s media instructions, but the Celeron doesn''t have 3DNow. The software RAID checksumming code uses these. I guess the memcpy implementation *might* use them if available (???). There are also various other optimisations the kernel uses, depending on what system it boots on.> > Out of interest for Xeon->Celeron migration did you see an immediate > > crash? Probably would be an invalid opcode exception of some kind. > > no... on the target host it looks like the domain was just started... in > the moment the migration is finished the domain gets restarted...Yep, that''s what I meant. You may find that selecting your processor to be something like "Pentium Pro" (or anything that implements a subset of what you have!) will be enough to make Linux behave right, but without having tried it I can''t be certain. Bear in mind that even if you start virtual machines on the CPU with least capabilities you''ll still find that once they''ve rebooted they can''t migrate back there. It''s best to try and keep your cluster as homogenous as possible. Cheers, Mark> it''s to fast to have a "xm console" running on the target host... xm dmesg > doesn''t show anything interessting... but I have not looked > in /var/log/xend.log... Next time I try I will check that and let you know. > > > Cheers, > > Mark > > --Ralph > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Donnerstag, 19. Januar 2006 16:42 schrieb Mark Williamson:> > > > 1) you have to use the same xen version on all xen hosts that are > > > > involved in migrating... so not 32bit xen on one machine and 32bit + > > > > pae or even 64bit xen on the other... > > > > > > Have you ever tried this? Sounds like it could be quite entertaining > > > :-) > > > > no, but I think it will not even be possible to migrate, because xen > > detects the missmatch, but that''s just a guess. I have not tried it yet > > :) > > The tools *might* disallow a migration, I''ve never checked. I hope they > do :-D > > > If you try to boot a dom0 with a pae enabled kernel on a non pae > > hypervisor (for example), then xen notices that and will not boot, so > > maybe it''s the same for migrating. > > It''s a slightly different mechanism since dom0 is a special case. > > > My long term solution is to have the kernel images on a san too, so that > > for every xen host the very same kernels are available... > > > > but your right, as long as you don''t want to reboot you wouldn''t need the > > kernel on the second host, I guess... But the first time I tried the > > migration I tried it from the xeon -> celeron and because the domain > > reboots immidently I got a error: "kernel not available/file not found" > > or something like this. > > Yep, it''d be because of the immediate reboot showing the lack of kernel > file, without the crash / reboot, you''d have been fine. > > > I can even turn off the gnbd master (and having heartbeat detecting that > > and making the failover in under 1 sec.) and having a live migration > > running in the very same moment. no problem at all... that even impressed > > my boss :-) > > Now *that''s* a cool demo :-) > > > > The features of the Celeron are a strict subset of the Xeon, so if > > > Linux boots on the Celeron it will only ever use things that the > > > Celeron has - hence the behaviour you''re seeing. > > > > I already was quite sure that exactly this was the problem. But now I can > > be sure, thanks... > > > > Btw... it''s the very same for a celeron and a sempron... you can migrate > > from celeron to sempron, but not the otherway around (if the domain was > > initially started on the sempron)... > > Could be the use of AMD specific instructions. For instance, the AMD > supports all the Celeron''s media instructions, but the Celeron doesn''t have > 3DNow. > > The software RAID checksumming code uses these. I guess the memcpy > implementation *might* use them if available (???). There are also various > other optimisations the kernel uses, depending on what system it boots on. > > > > Out of interest for Xeon->Celeron migration did you see an immediate > > > crash? Probably would be an invalid opcode exception of some kind. > > > > no... on the target host it looks like the domain was just started... in > > the moment the migration is finished the domain gets restarted... > > Yep, that''s what I meant. > > You may find that selecting your processor to be something like "Pentium > Pro" (or anything that implements a subset of what you have!) will be > enough to make Linux behave right, but without having tried it I can''t be > certain. > > Bear in mind that even if you start virtual machines on the CPU with least > capabilities you''ll still find that once they''ve rebooted they can''t > migrate back there. It''s best to try and keep your cluster as homogenous > as possible.and that will not be the only problem... I made a small console tool for our customers, so that they can start/halt, reboot, and console to their domainU by their own (working on xen2&3 at the moment). The customers simply start a ssh shell to the xen host (dom0) and after logging in a small menu gets shown where they can choose what they want to do. It''s important to keep this "mangament console" safe, because the ssh user who logs in has to get sudo right on "xm". if the console is not safe it is possible to stop other domains... :) But this is working very good for us at the moment (even if it''s more a hack than a solid solution), but in future this will get definitly a problem. If domUs will get migrated, then this will not work anymore. The customer doesn''t know on which host his domain should run. And if he login in to the wrong xen host and start the same domain (running on another host already) would seriously brake the fs on the san. I have to think about a solution for that (maybe that''s where xensource''s optimizer can help). But maybe setting up a spererate managment host, which knows where which domainU is executed right now, would solve this. But if the "managment console" is not running on the local dom0, then I think I would have to use the xend http stuff (which I ignored totally for now *g*). And what about a console? For a console I always have to use dom0, if I got that correctly... but neverless, thinking about such stuff makes fun, so it''s a pleasure to work with xen :) --Ralph> Cheers, > Mark > > > it''s to fast to have a "xm console" running on the target host... xm > > dmesg doesn''t show anything interessting... but I have not looked > > in /var/log/xend.log... Next time I try I will check that and let you > > know. > > > > > Cheers, > > > Mark > > > > --Ralph > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users