libvirt users - Nov 2012 - Failed to access console after launching LXC through libvirt

Hi,

I'm trying to manage LXC instances through OpenStack, which use libvirt as
a virtualization driver layer. After launching LXC instance, I simply could
not attach to the console.

virsh # list
 Id Name                 State
----------------------------------
14366 instance-00000078    running

virsh # console 14366
Connected to domain instance-00000078
Escape character is ^]


And it keeps stuck here.

Initially I thought it could be a cgroup device config issue, and tried to
"echo ?c 5:1 rwm'
>/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow?, to add
/dev/console access right to the container. But it does not work, console
stilll not working.

Before the cgroup tweaking, devices.list are as follows

[root at localhost libvirt]# cat
/cgroup/devices/libvirt/lxc/instance-0000007a/devices.list

c 1:3 rwm
c 1:5 rwm
c 1:7 rwm
c 1:8 rwm
c 1:9 rwm
c 5:0 rwm
c 5:2 rwm
c 136:* rwm


I'm not quite sure what kind of issue could possibly cause this console
access issue. Cause I'm now using two different guest OS templates on two
different host OSes. Following are some symptoms I observed.


* (g) for container guest OS
* (h) for host OS
* NOTE: Ubuntu guest os are generated by lxc-create -t ubuntu, which is an
Ubuntu 12.04 based lxc template.
             CentOS guest os are handcrafted by myself, taking reference of
http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6

   1. Ubuntu (g) launched by lxc-start on Ubuntu 12.04 host, console works
   2. Ubuntu (g) launched by OpenStack / libvirt on Ubuntu 12.04 host,
   console works
   3. Ubuntu (g) launched by OpenStack / libvirt on CentOS 6.3 host,
   console NOT works
   4. CentOS (g) launched by lxc-start on Ubuntu 12.04 host, console works
   5. CentOS (g) launched by OpenStack / libvirt on Ubuntu12.04 host,
   console NOT works
   6. CentOS (g) launched by OpenStack / libvirt on CentOS 6.3 host,
   console NOT works

So somehow I feel it could be something related to LXC template rootfs, but
Host OS do make a difference (2 vs. 3). lxc-start / libvirt also makes a
difference (4 vs. 5), but that might be caused by cgroup setting.

Could someone shed me some clues to further digging this issue? Thanks!

--
Qiu Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121109/e3ff3be3/attachment.htm>

On Fri, Nov 9, 2012 at 12:09 AM, unicell <unicell at gmail.com> wrote:
> Hi,
>
> I'm trying to manage LXC instances through OpenStack, which use libvirt
as
> a virtualization driver layer. After launching LXC instance, I simply could
> not attach to the console.
>
> virsh # list
>  Id Name                 State
> ----------------------------------
> 14366 instance-00000078    running
>
> virsh # console 14366
> Connected to domain instance-00000078
> Escape character is ^]
>
>
> And it keeps stuck here.
>
> Initially I thought it could be a cgroup device config issue, and tried to
> "echo ?c 5:1 rwm'
> >/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow?, to add
> /dev/console access right to the container. But it does not work, console
> stilll not working.
>
> Before the cgroup tweaking, devices.list are as follows
>
> [root at localhost libvirt]# cat
> /cgroup/devices/libvirt/lxc/instance-0000007a/devices.list
>
> c 1:3 rwm
> c 1:5 rwm
> c 1:7 rwm
> c 1:8 rwm
> c 1:9 rwm
> c 5:0 rwm
> c 5:2 rwm
> c 136:* rwm
>
>
> I'm not quite sure what kind of issue could possibly cause this console
> access issue. Cause I'm now using two different guest OS templates on
two
> different host OSes. Following are some symptoms I observed.
>
>
> * (g) for container guest OS
> * (h) for host OS
> * NOTE: Ubuntu guest os are generated by lxc-create -t ubuntu, which is an
> Ubuntu 12.04 based lxc template.
>              CentOS guest os are handcrafted by myself, taking reference
> of http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6
>
>    1. Ubuntu (g) launched by lxc-start on Ubuntu 12.04 host, console works
>    2. Ubuntu (g) launched by OpenStack / libvirt on Ubuntu 12.04 host,
>    console works
>    3. Ubuntu (g) launched by OpenStack / libvirt on CentOS 6.3 host,
>    console NOT works
>    4. CentOS (g) launched by lxc-start on Ubuntu 12.04 host, console works
>    5. CentOS (g) launched by OpenStack / libvirt on Ubuntu12.04 host,
>    console NOT works
>    6. CentOS (g) launched by OpenStack / libvirt on CentOS 6.3 host,
>    console NOT works
>
> So somehow I feel it could be something related to LXC template rootfs,
> but Host OS do make a difference (2 vs. 3). lxc-start / libvirt also makes
> a difference (4 vs. 5), but that might be caused by cgroup setting.
>
> Could someone shed me some clues to further digging this issue? Thanks!
>
> --
> Qiu Yu
>
For the "Ubuntu (g) launched by OpenStack / libvirt on CentOS 6.3
host", I
also tried following command and console still not working.

echo "a *.* rwm"
>/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow

More information attached.

[root at localhost ~]# ls -l
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty*
crw-rw-rw- 1 root tty 5, 0 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty
crw------- 1 root tty 4, 0 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty0
crw------- 1 root tty 4, 1 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty1
crw------- 1 root tty 4, 2 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty2
crw------- 1 root tty 4, 3 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty3
crw------- 1 root tty 4, 4 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty4
crw------- 1 root tty 4, 5 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty5
crw------- 1 root tty 4, 6 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty6
crw------- 1 root tty 4, 7 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty7
crw------- 1 root tty 4, 8 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty8
crw------- 1 root tty 4, 9 Oct 23 20:01
/home/stack/nova_state/instances/instance-00000078/rootfs/dev/tty9


[root at localhost ~]# cat
/cgroup/devices/libvirt/lxc/instance-00000078/devices.list
c 1:3 rwm
c 1:5 rwm
c 1:7 rwm
c 1:8 rwm
c 1:9 rwm
c 5:0 rwm
c 5:2 rwm
c 136:* rwm
c 5:1 rwm
c 5:* rwm
a *:* rwm


[root at localhost ~]# cat /etc/libvirt/lxc/instance-00000078.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
  virsh edit instance-00000078
or other application using the libvirt API.
-->

<domain type='lxc'>
  <name>instance-00000078</name>
  <uuid>55f225d1-e49c-4a51-9c17-0a5dde6d1917</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
    <cmdline>console=ttyS0</cmdline>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source
dir='/home/stack/nova_state/instances/instance-00000078/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='fa:16:3e:08:4f:b9'/>
      <source bridge='br100'/>
      <filterref
filter='nova-instance-instance-00000078-fa163e084fb9'>
        <parameter name='DHCPSERVER' value='10.48.253.1'/>
        <parameter name='IP' value='10.48.253.2'/>
        <parameter name='PROJMASK' value='255.255.255.0'/>
        <parameter name='PROJNET' value='10.48.253.0'/>
      </filterref>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>


--
Qiu Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121109/d729619c/attachment.htm>

Hey Qiu Yu,

I am a beginner to both lxc and openstack swift. Can you please clarify
some of my doubts related to openstack swift and lxc?


On 8 November 2012 21:39, unicell <unicell at gmail.com> wrote:
> Hi,
>
> I'm trying to manage LXC instances through OpenStack, which use libvirt
as
> a virtualization driver layer. After launching LXC instance, I simply could
> not attach to the console.
>
> virsh # list
>  Id Name                 State
> ----------------------------------
> 14366 instance-00000078    running
>
> virsh # console 14366
> Connected to domain instance-00000078
> Escape character is ^]
>
>
> And it keeps stuck here.
>
> Initially I thought it could be a cgroup device config issue, and tried to
> "echo ?c 5:1 rwm'
> >/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow?, to add
> /dev/console access right to the container. But it does not work, console
> stilll not working.
>
> Before the cgroup tweaking, devices.list are as follows
>
> [root at localhost libvirt]# cat
> /cgroup/devices/libvirt/lxc/instance-0000007a/devices.list
>
> c 1:3 rwm
> c 1:5 rwm
> c 1:7 rwm
> c 1:8 rwm
> c 1:9 rwm
> c 5:0 rwm
> c 5:2 rwm
> c 136:* rwm
>
>
> I'm not quite sure what kind of issue could possibly cause this console
> access issue. Cause I'm now using two different guest OS templates on
two
> different host OSes. Following are some symptoms I observed.
>
>
> * (g) for container guest OS
> * (h) for host OS
> * NOTE: Ubuntu guest os are generated by lxc-create -t ubuntu, which is an
> Ubuntu 12.04 based lxc template.
>              CentOS guest os are handcrafted by myself, taking reference
> of http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6
>
>    1. Ubuntu (g) launched by lxc-start on Ubuntu 12.04 host, console works
>    2. Ubuntu (g) launched by OpenStack / libvirt on Ubuntu 12.04 host,
>    console works
>    3. Ubuntu (g) launched by OpenStack / libvirt on CentOS 6.3 host,
>    console NOT works
>    4. CentOS (g) launched by lxc-start on Ubuntu 12.04 host, console works
>    5. CentOS (g) launched by OpenStack / libvirt on Ubuntu12.04 host,
>    console NOT works
>    6. CentOS (g) launched by OpenStack / libvirt on CentOS 6.3 host,
>    console NOT works
>
> So somehow I feel it could be something related to LXC template rootfs,
> but Host OS do make a difference (2 vs. 3). lxc-start / libvirt also makes
> a difference (4 vs. 5), but that might be caused by cgroup setting.
>
> Could someone shed me some clues to further digging this issue? Thanks!
>
> --
> Qiu Yu
>
>
>
------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_nov
> _______________________________________________
> Lxc-users mailing list
> Lxc-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lxc-users
>
>

-- 
Best Regards,

Sujay M
Final year B.Tech
Computer Engineering
NITK Surathkal

contact: +918971897571
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121108/4faa519d/attachment.htm>

Hey Qiu Yu,

I am a beginner to both lxc and openstack swift. Can you please clarify
some of my doubts related to openstack swift and lxc?


1. I want to set up a small cloud storage environment in my laptop. I mean
i need to host storage services for some clients and i want to virtualize
this environment. Is openstack swift enough for the above purpose or should
i install nova, glance and swift?


On 8 November 2012 22:10, Sujay M <sujay.m17 at gmail.com> wrote:
> Hey Qiu Yu,
>
> I am a beginner to both lxc and openstack swift. Can you please clarify
> some of my doubts related to openstack swift and lxc?
>
>
> On 8 November 2012 21:39, unicell <unicell at gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to manage LXC instances through OpenStack, which use
libvirt
>> as a virtualization driver layer. After launching LXC instance, I
simply
>> could not attach to the console.
>>
>> virsh # list
>>  Id Name                 State
>> ----------------------------------
>> 14366 instance-00000078    running
>>
>> virsh # console 14366
>> Connected to domain instance-00000078
>> Escape character is ^]
>>
>>
>> And it keeps stuck here.
>>
>> Initially I thought it could be a cgroup device config issue, and tried
>> to "echo ?c 5:1 rwm'
>> >/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow?, to
add
>> /dev/console access right to the container. But it does not work,
console
>> stilll not working.
>>
>> Before the cgroup tweaking, devices.list are as follows
>>
>> [root at localhost libvirt]# cat
>> /cgroup/devices/libvirt/lxc/instance-0000007a/devices.list
>>
>> c 1:3 rwm
>> c 1:5 rwm
>> c 1:7 rwm
>> c 1:8 rwm
>> c 1:9 rwm
>> c 5:0 rwm
>> c 5:2 rwm
>> c 136:* rwm
>>
>>
>> I'm not quite sure what kind of issue could possibly cause this
console
>> access issue. Cause I'm now using two different guest OS templates
on two
>> different host OSes. Following are some symptoms I observed.
>>
>>
>> * (g) for container guest OS
>> * (h) for host OS
>> * NOTE: Ubuntu guest os are generated by lxc-create -t ubuntu, which is
>> an Ubuntu 12.04 based lxc template.
>>              CentOS guest os are handcrafted by myself, taking
reference
>> of http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6
>>
>>    1. Ubuntu (g) launched by lxc-start on Ubuntu 12.04 host, console
>>    works
>>    2. Ubuntu (g) launched by OpenStack / libvirt on Ubuntu 12.04 host,
>>    console works
>>    3. Ubuntu (g) launched by OpenStack / libvirt on CentOS 6.3 host,
>>    console NOT works
>>    4. CentOS (g) launched by lxc-start on Ubuntu 12.04 host, console
>>    works
>>    5. CentOS (g) launched by OpenStack / libvirt on Ubuntu12.04 host,
>>    console NOT works
>>    6. CentOS (g) launched by OpenStack / libvirt on CentOS 6.3 host,
>>    console NOT works
>>
>> So somehow I feel it could be something related to LXC template rootfs,
>> but Host OS do make a difference (2 vs. 3). lxc-start / libvirt also
makes
>> a difference (4 vs. 5), but that might be caused by cgroup setting.
>>
>> Could someone shed me some clues to further digging this issue? Thanks!
>>
>> --
>> Qiu Yu
>>
>>
>>
------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_nov
>> _______________________________________________
>> Lxc-users mailing list
>> Lxc-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/lxc-users
>>
>>
>
>
> --
> Best Regards,
>
> Sujay M
> Final year B.Tech
> Computer Engineering
> NITK Surathkal
>
> contact: +918971897571
>
>

-- 
Best Regards,

Sujay M
Final year B.Tech
Computer Engineering
NITK Surathkal

contact: +918971897571
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121108/bf328bd0/attachment.htm>