Xen users - Nov 2011 - Best way to get Xen into Ubuntu Natty/64bit

Hi,

I had tried to run Xen on one system using the packages from a Xen
PPA, that seemed to have anything I wanted.
It came with Xen 4.1.2, so I added it, but I need something more
"finished" and "working"

When testing I found so many issues that I''m now uninstalling all of
this again:
ii  libxen-dev                      4.1.2~final-2ubuntu7
     Public headers and libs for Xen
ii  libxenstore3.0                  4.1.2~final-2ubuntu7
     Xenstore communications library for Xen
ii  xen-docs-4.1                    4.1.2~final-2ubuntu7
     Documentation for Xen
ii  xen-hypervisor-4.1-amd64        4.1.2~final-2ubuntu7
     Xen Hypervisor on AMD64
ii  xen-utils-4.1                   4.1.2~final-2ubuntu7
     XEN administrative tools
ii  xen-utils-common                4.1.0~rc6-1ubuntu1
     XEN administrative tools - common files
ii  xenstore-utils                  4.1.2~final-2ubuntu7
     Xenstore utilities for Xen

It didn''t even manage to make Xen the default boot option in the grub2
config :)
Bringing up VMs was also more of a luck thing.

What is the best choice if I want to be actually able to use the
following things:

- dom0_mem *must* work. It''s totally idiotic if it doesnt work and i
sit there watching dom0 balloon on each new VM boot, plus the OOM
killer hit a few times during that. How a less than 1.4GB OS install
can run out of Ram whilst ballooning from 96GB to 16GB, I do not
know.
>From what I understand this makes all early 3.0 kernels a bad choice
and definitely matches what I was seeing when I tried to set dom0
mem....
- Xen NUMA needs to work since it is a 2socket Opteron
- blktap2
- blkback
- tmem
- Ceph (this means a kernel where I can have very current btrfs and
ceph, most probably not the stock kernel anyway), but this can also
run inside a domU so this is not that important

I guess I''ll be going with xen-testing right away and I can also
switch distros if anyone has a recommendation.
(Oracle VM 2.2, 3.0.1 and 3.0.2 unfortunately didn''t work on the
system, otherwise I''d be running that instead of trying to run
something.

What are you using for Xen installs that are supposed to *work* on a
larger scale box?
Debian and a selfmade kernel?
RHEL + GitCo?

Alpine 2.2testing was the least of mess of all I tried but
- they''re just adding Xen support, so it''s not done yet
- only recognized 8 of 24 cpus.



-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I settled on OpenSUSE since they seemed to be the most commited to having
Dom0 support.  RHEL 6 doesn''t run as a Dom0 (Xen Host), you have to
RHEL 5,
I think Fedora Core 16 is trying to get this up to date though.

I''m not really a Debian guy so I can''t comment there, but I
had no problems
with OpenSuSE, and I had Windows 7 Pro running on it fine, using LVM,
within a day, so I can definitely recommend that.  It''s got KDE 4 too,
which I''m giving a chance even though I''m generally a Gnome
guy.

The only hitch I hit I''ve had is my Dom0 can''t currently
connect to the
windows box using bridged mode unless I''m root, but I need to try the
routing scripts to see if that fixes it. The rest of the machines on the
network can RDP to the hosted Windows machine fine, but I haven''t
looked at
that in the last week.

Best of Luck

- Robert

On Sun, Nov 13, 2011 at 9:10 AM, Florian Heigl
<florian.heigl@gmail.com>wrote:
> Hi,
>
> I had tried to run Xen on one system using the packages from a Xen
> PPA, that seemed to have anything I wanted.
> It came with Xen 4.1.2, so I added it, but I need something more
> "finished" and "working"
>
> When testing I found so many issues that I''m now uninstalling all
of this
> again:
> ii  libxen-dev                      4.1.2~final-2ubuntu7
>     Public headers and libs for Xen
> ii  libxenstore3.0                  4.1.2~final-2ubuntu7
>     Xenstore communications library for Xen
> ii  xen-docs-4.1                    4.1.2~final-2ubuntu7
>     Documentation for Xen
> ii  xen-hypervisor-4.1-amd64        4.1.2~final-2ubuntu7
>     Xen Hypervisor on AMD64
> ii  xen-utils-4.1                   4.1.2~final-2ubuntu7
>     XEN administrative tools
> ii  xen-utils-common                4.1.0~rc6-1ubuntu1
>     XEN administrative tools - common files
> ii  xenstore-utils                  4.1.2~final-2ubuntu7
>     Xenstore utilities for Xen
>
> It didn''t even manage to make Xen the default boot option in the
grub2
> config :)
> Bringing up VMs was also more of a luck thing.
>
> What is the best choice if I want to be actually able to use the
> following things:
>
> - dom0_mem *must* work. It''s totally idiotic if it doesnt work and
i
> sit there watching dom0 balloon on each new VM boot, plus the OOM
> killer hit a few times during that. How a less than 1.4GB OS install
> can run out of Ram whilst ballooning from 96GB to 16GB, I do not know.
> >From what I understand this makes all early 3.0 kernels a bad choice
> and definitely matches what I was seeing when I tried to set dom0
> mem....
> - Xen NUMA needs to work since it is a 2socket Opteron
> - blktap2
> - blkback
> - tmem
> - Ceph (this means a kernel where I can have very current btrfs and
> ceph, most probably not the stock kernel anyway), but this can also
> run inside a domU so this is not that important
>
> I guess I''ll be going with xen-testing right away and I can also
> switch distros if anyone has a recommendation.
> (Oracle VM 2.2, 3.0.1 and 3.0.2 unfortunately didn''t work on the
> system, otherwise I''d be running that instead of trying to run
> something.
>
> What are you using for Xen installs that are supposed to *work* on a
> larger scale box?
> Debian and a selfmade kernel?
> RHEL + GitCo?
>
> Alpine 2.2testing was the least of mess of all I tried but
> - they''re just adding Xen support, so it''s not done yet
> - only recognized 8 of 24 cpus.
>
>
>
> --
> the purpose of libvirt is to provide an abstraction layer hiding all
> xen features added since 2006 until they were finally understood and
> copied by the kvm devs.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

2011/11/13 Robert Zaleski <rlzaleski@gmail.com>:
> I settled on OpenSUSE since they seemed to be the most commited to having
> Dom0 support.  RHEL 6 doesn''t run as a Dom0 (Xen Host), you have
to RHEL 5,
> I think Fedora Core 16 is trying to get this up to date though.
You have a point there. SUSE has been delivering very clean Xen
packages for years (and without their kernel patches who knows where
Xen would have ended up) and I have to wonder if it can make sense to
spend my time fixing broken distros / packages when I just wanted to
build a VM?
This will be my first private SuSE install since 1999 or so.
I hope they stopped stuff like beautifying the system config files
after all these years.

> Best of Luck
> - Robert
Thanks, will need a lot, the download just aborted :))

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

FILE UNDER: Why people consider using Xen complicated.


In the second install run now ;)
I boot off a 2GB industrial flash drive. First time the installer just
had to create a swap space on that.
The box has 96GB ram, but the installer loves a swap space, and since
it''s so much ram, lets use 1.99GB for swap and the rest for the OS
install.

This would be easier if the chipsets SATA controller with those large
SATA drives would be detected, but right now I''m down to 2GB...

That aside, the best looking and fastest installer I have seen in
ages. Makes Ubuntu and RHEL installers look from the stone ages.

And holy hell, they use *kexec* to skip the post-install reboot. This mighty.

If there was just some global button to disable all the automagic.
Especially when it''s so broken:

"trying to download release notes... failed"
"select cancel to skip all stuff that needs internet access"...
cancel!
"now downloading updates"

30 minutes later i''m right in dependency hell because they have a
circular dependency for python.
python depends on python(abi) which would be installed with python and
I feel somewhere between just dding my root device and running away.
The "solution" would apparently be to remove the "minimal install
patterns" and with gods grace finally installing CUPS plus a few dozen
more desktop pkgs. No system can live without CUPS ever. Or cyrus, or
samba. I wonder if I''m allowed to uninstall those later.

After this I was actually able to installl Xen.
Including SDL and XORG and GNOME LIBS after I selected "no, don''t
install the GUI bits"

SUSE might in some fashion be the best, well-rounded smarted Linux
distro around. It might even have working Xen. And it looks like it
does.
But it doesn''t support my disk controllers  and it makes my head hurt
so much. :/

This story ends with a running xenstored, unmounted xenstore
mountpoint and xl / xm not working.

OpenSuse + headless xen server? Not so much.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Am Dienstag, 15. November 2011, 15:00:57 schrieb Florian
Heigl:
> SUSE might in some fashion be the best, well-rounded smarted Linux
> distro around. It might even have working Xen. And it looks like it
> does.
> But it doesn''t support my disk controllers  and it makes my head
hurt
> so much. :/
...just another point to give a short recommendation to try a look at gentoo 
again and build your distro as you need it ;)

sorry for the noise...


cheers,

Niels.

-- 
---
Niels Dettenbach
Syndicat IT&Internet
http://www.syndicat.com/

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Nov 15, 2011 at 9:00 PM, Florian Heigl <florian.heigl@gmail.com>
wrote:
> FILE UNDER: Why people consider using Xen complicated.
> SUSE might in some fashion be the best, well-rounded smarted Linux
> distro around. It might even have working Xen. And it looks like it
> does.
> But it doesn''t support my disk controllers  and it makes my head
hurt
> so much. :/
>
> This story ends with a running xenstored, unmounted xenstore
> mountpoint and xl / xm not working.
>
> OpenSuse + headless xen server? Not so much.
... and that''s why I still recommend good-old RHEL5 :D

To be fair, opensuse might be considered bleeding-edge enough that
some broken stuff is understandable. If you want something tested,
RHEL5.7 or SLES11.1 might be a better choice. Those who like
vmware-like functionality might even prefer using XenServer/XCP.

My last experiment was RHEL6 + xen 4.1 from
http://xenbits.xen.org/people/mayoung/EL6.xen/ + dom0 kernel from
http://xenbits.xen.org/people/mayoung/testing/ . It kinda work, but an
exsting ubuntu domU (on ext4) keep on having "ext4 journal aborted,
error, mounting read only" (or something like that) after abnormal
shutdown, which shouldn''t happen (any journaling FS should
be good enough to recover from a power failure-like scneario), and
might indicate something wrong with dom0''s xen-blkback module. So I
replaced the dom0 kernel with vanilla 3.1.0 (based on modified F16''s
SRPM), and it''s working as expected now.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Nov 15, 2011 at 8:44 AM, Fajar A. Nugraha <list@fajar.net>
wrote:
> On Tue, Nov 15, 2011 at 9:00 PM, Florian Heigl
<florian.heigl@gmail.com> wrote:
>> FILE UNDER: Why people consider using Xen complicated.
>
>
>> SUSE might in some fashion be the best, well-rounded smarted Linux
>> distro around. It might even have working Xen. And it looks like it
>> does.
>> But it doesn''t support my disk controllers  and it makes my
head hurt
>> so much. :/
>>
>> This story ends with a running xenstored, unmounted xenstore
>> mountpoint and xl / xm not working.
>>
>> OpenSuse + headless xen server? Not so much.
>
> ... and that''s why I still recommend good-old RHEL5 :D
>
> To be fair, opensuse might be considered bleeding-edge enough that
> some broken stuff is understandable. If you want something tested,
> RHEL5.7 or SLES11.1 might be a better choice. Those who like
> vmware-like functionality might even prefer using XenServer/XCP.
>
> My last experiment was RHEL6 + xen 4.1 from
> http://xenbits.xen.org/people/mayoung/EL6.xen/ + dom0 kernel from
> http://xenbits.xen.org/people/mayoung/testing/ . It kinda work, but an
> exsting ubuntu domU (on ext4) keep on having "ext4 journal aborted,
> error, mounting read only" (or something like that) after abnormal
> shutdown, which shouldn''t happen (any journaling FS should
> be good enough to recover from a power failure-like scneario), and
> might indicate something wrong with dom0''s xen-blkback module. So
I
> replaced the dom0 kernel with vanilla 3.1.0 (based on modified
F16''s
> SRPM), and it''s working as expected now.
>
> --
> Fajar
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
I am in the current Beta for Suse Enterprise, I am sure others on the
list are as well, it is mainly a SP to the current version 11, but it
seems they have really mucked around with the Xen portion.  I have had
10s of failures trying to setup Xen with it.  Has anyone on the list
had any luck with it, or is is a similar situation to OpenSuSE?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

2011/11/15 Niels Dettenbach <nd@syndicat.com>:
> Am Dienstag, 15. November 2011, 15:00:57 schrieb Florian Heigl:
>> SUSE might in some fashion be the best, well-rounded smarted Linux
>> distro around. It might even have working Xen. And it looks like it
>> does.
>> But it doesn''t support my disk controllers  and it makes my
head hurt
>> so much. :/
>
> ...just another point to give a short recommendation to try a look at
gentoo
> again and build your distro as you need it ;)
perfectly valid. I know at least one guy who ran his production Xen
hosts on Gentoo for quite some time. I had tested it and remember it
was the only distro where you could get close to using sHype and
others back then.

Vlan interface configuration is the other thing I remember from Gentoo
and that was not so great. That and the lack of (out of the box)
network installs was the reasons why I put away with it in the end. I
can install my oracle vm boxes in something around 4 minutes using
cobbler + kickstart into a defined state. I didn''t see how that would
work using Gentoo.

CentOS 5 also has it''s downsides. Running Ceph on that would be plain
impossible, and of course you''d totally not be running a distro kernel
for Xen there?
And I pretty much guess CentOS/RHEL 5 will also not boot if the
CentOS-binary-compatible OVM 2.2.2 panics upon disk detection.

It definitely needs to be something less 2006''ish than CentOS, thats
why I had given Ubuntu a shot.

Re: the person who was beta-testing the next SUSE update:
Maybe we should have a wiki page that just lists the features Xen
broken in each distro...

Anyway.
If I summarize this thread (in my always positive way) then I''m down
to:
- Use manual build of Xen on Ubuntu or Debian
- Use CentOS 5 (can''t since it wont work)
- Use CentOS 6 + GitCo (maybe?)
- Use SUSE (can''t since hw not working and tendency to hurt self after
prolonged use)
- Fedora 16 (how about that!?)
- revisit Gentoo and spend 2-3 days getting all configured properly
before I can tell if it does what I want. Ah, and add another 5-6 days
to make Infiniband really work there
- Stuff FC16 kernel into CentOS6 (uh. that''s not how we run servers out
here?)
- Stop trying all of that and put my time into the Alpine Linux Xen
port. Meaning I''ll have to fix anything I encounter and
there''s no
googling for issues.

Footnote:
Using some build like the xen testing one for RHEL6 is a road to hell.
I''ve done that a few times for RHEL5/RHEL6 and know well enough these
builds get rarely bugfixed, but never updated.
If something stops working then you''re left on your own.
Probably most prominent would be the FreeBSD XenPV versions where
every 2 years one or another dev made one working install, tarred it
up and then went off to do something else because only the bugfixing
was left to do. And so you still see people trying to follow FreeBSD 7
instructions that refer to an image that is on a webserver that died 2
years ago. You get the idea.


Thanks for your inputs!

-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Nov 15, 2011 at 11:03 PM, Florian Heigl <florian.heigl@gmail.com>
wrote:
> And I pretty much guess CentOS/RHEL 5 will also not boot if the
> CentOS-binary-compatible OVM 2.2.2 panics upon disk detection.
I wouldn''t be so sure about that. Redhat adds drivers on every point
release. And OVM 3 uses kernel 2.6.32, so both might actually work.
>
> It definitely needs to be something less 2006''ish than CentOS,
RHEL 5.7 is less than 6 monts old :D
> Anyway.
> If I summarize this thread (in my always positive way) then I''m
down to:
> - Use manual build of Xen on Ubuntu or Debian
... or RHEL/Centos
> - Use CentOS 5 (can''t since it wont work)
it might if your kernel is new enough
> - Use CentOS 6 + GitCo (maybe?)
I think gitco is for RHEL/Centos5. Might work for 6 though (or at
least you can rebuild its SRPM).
> - Fedora 16 (how about that!?)
might work. haven''t tried. I needed a distro that lasts longer than 1
year.
> - Stuff FC16 kernel into CentOS6 (uh. that''s not how we run
servers out here?)
News flash: Oracle grabs kernel 2.6.32, put in on their RHEL5 clone,
and markets it as "unbreakable enterprise kernel" :)

Personally I''m much more comfortable using latest vanilla kernel
(packaged as RPM, of course) on top of RHEL6. At least for the life
time of the server (usually about 3 years) I know that I only have to
update xen and the kernel manually, and not having to worry about
backporting openssl or sendmail (or any one of the hundreds of
packages normally present on a linux distro) bug fixes myself.

Then again if the next Ubuntu LTS properly supports Xen, i might be
tempted to use that as base instead.
> - Stop trying all of that and put my time into the Alpine Linux Xen
> port. Meaning I''ll have to fix anything I encounter and
there''s no
> googling for issues.
... or stop trying to treat Xen as a software you install on top of
existing OS, and use an appliance: Xenserver, XCP, OVM. Real
time-saver for certain situations.

And if neither of them fits your needs, then perhaps xen is simply not
the right tool for the job. In the end of they it''s getting the job
done that really counts, isn''t it?
>
> Footnote:
> Using some build like the xen testing one for RHEL6 is a road to hell.
> I''ve done that a few times for RHEL5/RHEL6 and know well enough
these
> builds get rarely bugfixed, but never updated.
> If something stops working then you''re left on your own.
I have several systems running RHEL5 + stock kernel and xen, has been
running for several years, works just fine.
I have other system running  RHEL5 + custom version of kernel and xen
(all from package, either gitco or self-built) because I needed some
new features (e.g. ability to run opensolaris domU, and pygrub that
supports zfs). They''re also managable (as long as you remember to
build/update the custom packages). If the "testing" package
isn''t
updated fast enough (e.g. Gitco is usually behind by a few weeks or
month) then I just build it myself using the last available SRPM as
base.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Nov 14, 2011 at 12:10 AM, Florian Heigl <florian.heigl@gmail.com>
wrote:
> (Oracle VM 2.2, 3.0.1 and 3.0.2 unfortunately didn''t work on the
> system, otherwise I''d be running that instead of trying to run
> something.
sorry, I just read that part.

Ignore my previous comments about RHEL5 and OVM 3 then. And
XCP/Xenserver probably won''t work either :(

Have you tried Vmware hypervisor? I''m starting to think it might be
the only supported, hassle-free, just-work option for your particular
hardware.

Either that, or learn to live with latest vanilla/backported kernels.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

for the record:
1. suse recognizes the disks, that was my fault. I had put the blade
in a different slot for easier reach to the network cables, but didnt
bring the disks along
2. traced down the suse xen error. the xencommons startup failed,
deleting /var/lib/xenstored/* and /var/run/xenstored/* has made it
work (for now)
I''ll see if it breaks on every boot.

it comes with 4.0.2+some minor levels, so this is roundabout on the
same level as OVM3.

2011/11/15 Fajar A. Nugraha <list@fajar.net>:
> On Mon, Nov 14, 2011 at 12:10 AM, Florian Heigl
<florian.heigl@gmail.com> wrote:
>> (Oracle VM 2.2, 3.0.1 and 3.0.2 unfortunately didn''t work on
the
>> system, otherwise I''d be running that instead of trying to run
>> something.
>
> sorry, I just read that part.
np, it was a wall of text anyway :)
> Ignore my previous comments about RHEL5 and OVM 3 then. And
fyi: ovm2.2.2 crash on sata init, ovm3 crash on loading xen.gz. Not
much hope there right now.
> XCP/Xenserver probably won''t work either :(
guess so. will not try, strong UUID allergy from a few years working
with XenServer.
> Have you tried Vmware hypervisor? I''m starting to think it might
be
> the only supported, hassle-free, just-work option for your particular
> hardware.
Hehe, yes ESXi would definitely work. But I think it can''t hurt to
stab at a few more of these issues. I''d also be happy to do a few
benchmarks on it later on.
i.e. run 60GB PV domU on Xen vs. 60GB KVM VM. I suspect Xen actually
understanding NUMA would show some benefits. But of course that''s
after making it work.
> Either that, or learn to live with latest vanilla/backported kernels.
*grins*
just like in the old days on etch :)



-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users