I''m trying to start a large number of domains (hopefully around 70)
each
backed by a separate qcow disk. They are booting via pygrub. I''ve
included an example config file below. I can start the first 16 DomUs
just fine, but any after that fail to start. It looks like I''m hitting
a limit to the number of blktap devices the Dom0 can have, so pygrub is
failing to load the new disks. I''ve posted relevant sections of
/var/log/xen/xend.log for both a successful and failed domain creation.
I''m nearly certain that the problem is with mounting the image in Dom0
so that pygrub can find a kernel on it; if I modify the config file for
a failed domain to boot directly from Dom0''s kernel, and then retry the
domain creation, it works. Furthermore, if I do an "xm shutdown" on
one
of the successfully created pygrub domains, I can start one of the
failed ones.
Could it be that there is a limit of 16 blktap devices in Dom0 and
they''re not being cleaned up properly after pygrub uses them? It looks
like they are. In the log file, after pygrub returns it says
"Unmounting /dev/xvdp from /dev/xvdp".
Thanks,
Jim Cipar
LOG FOR FAILED DOMAIN CREATION
---------------------------------------------------------------
[2008-08-30 10:06:11 5237] INFO (XendDomainInfo:2066) Mounting
/home/jcipar/Disks/VM-17-0.qcow on /dev/xvdp.
[2008-08-30 10:06:11 5237] DEBUG (DevController:117) DevController:
writing {''backend-id'': ''0'',
''virtual-device'': ''51952'',
''device-type'':
''disk'', ''state'': ''1'',
''backend'': ''/loca
l/domain/0/backend/tap/0/51952''} to /local/domain/0/device/vbd/51952.
[2008-08-30 10:06:11 5237] DEBUG (DevController:119) DevController:
writing {''domain'': ''Domain-0'',
''frontend'':
''/local/domain/0/device/vbd/51952'', ''uuid'':
''714fe8ee-0376-7cad-ab
0a-a3a8085166b6'', ''dev'':
''/dev/xvdp'', ''state'': ''1'',
''params'':
''qcow:/home/jcipar/Disks/VM-17-0.qcow'',
''mode'': ''w'', ''online'':
''1'',
''frontend-id'': ''0'',
''type'': ''tap''} to
/local/domain/0/backend/tap/0/51952.
[2008-08-30 10:06:11 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:11 5237] DEBUG (DevController:594)
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:12 5237] DEBUG (DevController:594)
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:12 5237] DEBUG (DevController:608)
hotplugStatusCallback 1.
[2008-08-30 10:06:12 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:12 5237] DEBUG (DevController:594)
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:12 5237] DEBUG (DevController:608)
hotplugStatusCallback 1.
[2008-08-30 10:06:12 5237] ERROR (XendBootloader:43) Disk isn''t
accessible
[2008-08-30 10:06:12 5237] INFO (XendDomainInfo:2084) Unmounting
/dev/xvdp from /dev/xvdp.
[2008-08-30 10:06:12 5237] DEBUG (XendDomainInfo:590)
XendDomainInfo.destroyDevice: deviceClass = tap, device = /dev/xvdp
[2008-08-30 10:06:12 5237] ERROR (XendDomainInfo:440) VM start failed
LOG FOR SUCCESSFUL DOMAIN CREATION
-------------------------------------------------------------------------
[2008-08-30 10:06:06 5237] DEBUG (DevController:119) DevController:
writing {''domain'': ''Domain-0'',
''frontend'':
''/local/domain/0/device/vbd/51952'', ''uuid'':
''3fbc58db-daa3-b555-21
4e-14745062eb03'', ''dev'':
''/dev/xvdp'', ''state'': ''1'',
''params'':
''qcow:/home/jcipar/Disks/VM-16-0.qcow'',
''mode'': ''w'', ''online'':
''1'',
''frontend-id'': ''0'', ''
type'': ''tap''} to /local/domain/0/backend/tap/0/51952.
[2008-08-30 10:06:06 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:06 5237] DEBUG (DevController:594)
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:06 5237] DEBUG (DevController:594)
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:06 5237] DEBUG (DevController:608)
hotplugStatusCallback 1.
[2008-08-30 10:06:06 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:06 5237] DEBUG (DevController:594)
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:06 5237] DEBUG (DevController:608)
hotplugStatusCallback 1.
[2008-08-30 10:06:06 12692] DEBUG (XendBootloader:100) Launching
bootloader as [''/usr/bin/pygrub'',
''--args=root=/dev/xvda1 xencons=tty'',
''--output=/var/run/xend/boot/xenbl.15861
'', ''-q'', ''/dev/xvdp''].
[2008-08-30 10:06:07 5237] INFO (XendDomainInfo:2084) Unmounting
/dev/xvdp from /dev/xvdp.
[2008-08-30 10:06:07 5237] DEBUG (XendDomainInfo:590)
XendDomainInfo.destroyDevice: deviceClass = tap, device = /dev/xvdp
EXAMPLE CONFIG FILE
------------------------------------------
bootloader="/usr/bin/pygrub"
disk=[''tap:qcow:/home/jcipar/Disks/VM-1-0.qcow,xvda1,w'']
vif = [''ip=172.19.158.1'']
memory=100
root="/dev/xvda1"
extra=''xencons=tty''
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Todd Deshane
2008-Sep-01 16:24 UTC
Re: [Xen-users] BLKTAP limit in Dom0, can''t boot with pygrub
> Could it be that there is a limit of 16 blktap devices in Dom0 and they''re > not being cleaned up properly after pygrub uses them? It looks like they > are. In the log file, after pygrub returns it says "Unmounting /dev/xvdp > from /dev/xvdp". >There shouldn''t be a limit, but maybe some version of Xen/Linux had one. What version of Xen/LInux are you using? Are you stuck at that version of Xen/Linux or can you use a newer version? I''ve had more 16 guest running on Ubuntu 8.04 + Xen from packages. I''ll be running some scalability tests on Xen 3.3 + Linux 2.6.27 soon, so feel free to ping me for the results if they would be useful to you. Cheers, Todd -- Todd Deshane http://todddeshane.net check out our book: http://runningxen.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Sep-01 18:48 UTC
Re: [Xen-users] BLKTAP limit in Dom0, can''t boot with pygrub
On Mon, Sep 1, 2008 at 2:10 PM, James Cipar <jcipar@andrew.cmu.edu> wrote:> I''m using Ubuntu 8.04, > ubuntu-xen-server 0.0.1-2ubuntu8 > Host (and guest) kernel 2.6.24-17.31 > Hypervisor 3.2.0-0ubuntu10 > xen-utils 3.2.0-0ubuntu10 > > I''m getting a warning from aptitude that the recommended package libc6-xen > is unavailable. If the solution is to simply upgrade to the newest version, > I''m willing to do that, but I don''t want to just upgrade to "see if it > works". I''d rather understand what''s causing the problem. >I''d expect this set of packages to work. Are you running 32 or 64 bit? My tests were done on a very similar set of packages on 64bit. I pushed to 28 guests, simply due to running out of memory. Personally I haven''t had problems with the Ubuntu 2.6.24 kernel, but others have reported problems. Maybe a 32bit or hardware compatibility problem. Have you ruled out that it is a memory problem? -- Todd Deshane http://todddeshane.net check out our book: http://runningxen.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Sep-01 19:04 UTC
Re: [Xen-users] BLKTAP limit in Dom0, can''t boot with pygrub
On Mon, Sep 1, 2008 at 2:59 PM, James Cipar <jcipar@andrew.cmu.edu> wrote:> I''m using 64 bit. Its an 8GB host, with 100MB VMs. If I don''t use pygrub I > can start 64 guests, so I don''t think it''s a memory limit. The problem only > occurs when I use pygrub, and it happens at 16 guests, well below the memory > limit for that machine. >I didn''t notice that before, so it is a pygrub issue. Pygrub has been deprecated as of Xen 3.3 in favor of a more secure solution called PV-GRUB. Cheers, Todd _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Sep-01 19:37 UTC
Re: [Xen-users] BLKTAP limit in Dom0, can''t boot with pygrub
On Mon, Sep 1, 2008 at 3:31 PM, James Cipar <jcipar@andrew.cmu.edu> wrote:> Are there any good docs about pvgrub online? A quick search didn''t really > come up with anything. Can I just replace pygrub with pvgrub? > >I would suggest starting here: http://blog.xen.org/index.php/2008/08/28/xen-33-feature-pv-grub/ I don''t think it is easy to get all the pvgrub stuff without using 3.3. Cheers, Todd _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users