Xen users - Apr 2011 - Trouble with manual bridging on Xen3/CentOS 5

Hi all,

  While trying to sort out/test some patches, I was told that it''s best
to create bridges manually in
/etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented out
(network-script network-bridge) and restarted the network and Xen.

  With the bridges manually in place, and no longer having
''pethX''
devices, I tried to provision a VM and it failed with:

===# virt-install --connect xen --name vm0002_pppoe --ram 2048 --arch
x86_64 --vcpus 1 --cpuset 1-7 --location http://10.255.0.1/f9/x86_64/img
--os-type linux --os-variant rhel5.4 --disk
path=/dev/drbd_x4_vg0/vm0002_1 --network bridge=xenbr0 --network
bridge=xenbr2 --vnc --paravirt --debug

Fri, 29 Apr 2011 17:32:12 DEBUG    Requesting libvirt URI xen
Fri, 29 Apr 2011 17:32:12 ERROR    unable to connect to
''localhost:8000'': Connection refused
Traceback (most recent call last):
  File "/usr/sbin/virt-install", line 892, in ?
    main()
  File "/usr/sbin/virt-install", line 628, in main
    conn = cli.getConnection(options.connect)
  File "/usr/lib/python2.4/site-packages/virtinst/cli.py", line 126,
in
getConnection
    conn = libvirt.open(connect)
  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 169, in
open
    if ret is None:raise libvirtError(''virConnectOpen()
failed'')
libvirtError: unable to connect to ''localhost:8000'':
Connection refused
===
  Thinking that I still needed to use ''(network-script
network-bridge)'',
so I put it back (actually, a modified version I will link below). When
I start xend after putting it back, I got such an incredible flood that
all network communications were lost on the network. Obviously, I''m
reluctant to randomly try things now.

  So, if I manually build the bridges, how am I supposed to configure
Xen to use them?

Thanks!

PS: My modified network-bridge:

===#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0
bridge=xenbr0
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2
bridge=xenbr2
===
-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, Apr 30, 2011 at 6:08 AM, Digimer <linux@alteeve.com>
wrote:
> Hi all,
>
>  While trying to sort out/test some patches, I was told that it''s
best
> to create bridges manually in
> /etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented out
> (network-script network-bridge) and restarted the network and Xen.
Yes, using system network configuration will be better.  Which
distribution is this?  CentOS/RHEL 5?
You commented out (network-script network-bridge)?  If you are using
system network configuration to configure your bridge, have you try
the following instead of commented out?

(network-script /bin/true)
>
>  With the bridges manually in place, and no longer having
''pethX''
> devices, I tried to provision a VM and it failed with:
>
> ===> # virt-install --connect xen --name vm0002_pppoe --ram 2048 --arch
> x86_64 --vcpus 1 --cpuset 1-7 --location http://10.255.0.1/f9/x86_64/img
> --os-type linux --os-variant rhel5.4 --disk
> path=/dev/drbd_x4_vg0/vm0002_1 --network bridge=xenbr0 --network
> bridge=xenbr2 --vnc --paravirt --debug
brctl show outout?

Is your manual ifcfg-xenbr0 a bridge to ifcfg-eth0 and ifcfg-xenbr2 a
bridge to ifcfg-eth2?
Where is your ifcfg-eth1 and ifcfg-xenbr1?  Normal network
configuration will have ifcfg-eth* in order... ...

>
> Fri, 29 Apr 2011 17:32:12 DEBUG    Requesting libvirt URI xen
> Fri, 29 Apr 2011 17:32:12 ERROR    unable to connect to
> ''localhost:8000'': Connection refused
> Traceback (most recent call last):
>  File "/usr/sbin/virt-install", line 892, in ?
>    main()
>  File "/usr/sbin/virt-install", line 628, in main
>    conn = cli.getConnection(options.connect)
>  File "/usr/lib/python2.4/site-packages/virtinst/cli.py", line
126, in
> getConnection
>    conn = libvirt.open(connect)
>  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 169,
in open
>    if ret is None:raise libvirtError(''virConnectOpen()
failed'')
> libvirtError: unable to connect to ''localhost:8000'':
Connection refused
> ===>
>  Thinking that I still needed to use ''(network-script
network-bridge)'',
> so I put it back (actually, a modified version I will link below). When
> I start xend after putting it back, I got such an incredible flood that
> all network communications were lost on the network. Obviously,
I''m
> reluctant to randomly try things now.
>
>  So, if I manually build the bridges, how am I supposed to configure
> Xen to use them?
>
> Thanks!
>
> PS: My modified network-bridge:
>
> ===> #!/bin/sh
> dir=$(dirname "$0")
> "$dir/network-bridge" "$@" vifnum=0 netdev=eth0
bridge=xenbr0
> "$dir/network-bridge" "$@" vifnum=2 netdev=eth2
bridge=xenbr2
> ===
It is hard to tell what went wrong without seeing the following outputs:

brctl show
ifconfig
ip link

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, Apr 30, 2011 at 6:26 AM, Teck Choon Giam
<giamteckchoon@gmail.com> wrote:
> On Sat, Apr 30, 2011 at 6:08 AM, Digimer <linux@alteeve.com> wrote:
>> Hi all,
>>
>>  While trying to sort out/test some patches, I was told that
it''s best
>> to create bridges manually in
>> /etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented out
>> (network-script network-bridge) and restarted the network and Xen.
>
> Yes, using system network configuration will be better.  Which
> distribution is this?  CentOS/RHEL 5?
Oops... just noticed the subject... so this is CentOS 5 :p

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, Apr 30, 2011 at 5:08 AM, Digimer <linux@alteeve.com>
wrote:
> Hi all,
>
>  While trying to sort out/test some patches, I was told that it''s
best
> to create bridges manually in
> /etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented out
> (network-script network-bridge) and restarted the network and Xen.
Did you restart the dom0?
> # virt-install --connect xen --name vm0002_pppoe --ram 2048 --arch
> x86_64 --vcpus 1 --cpuset 1-7 --location http://10.255.0.1/f9/x86_64/img
> --os-type linux --os-variant rhel5.4 --disk
> path=/dev/drbd_x4_vg0/vm0002_1 --network bridge=xenbr0 --network
> bridge=xenbr2 --vnc --paravirt --debug
Do you have xenbr0 and xenbr2 bridge?
>
> Fri, 29 Apr 2011 17:32:12 DEBUG    Requesting libvirt URI xen
> Fri, 29 Apr 2011 17:32:12 ERROR    unable to connect to
> ''localhost:8000'': Connection refused
The message should be clear. Is xend running? Is it listening on port
8000 (try "netstat -anp")?
>  Thinking that I still needed to use ''(network-script
network-bridge)'',
> so I put it back (actually, a modified version I will link below). When
> I start xend after putting it back, I got such an incredible flood that
> all network communications were lost on the network. Obviously,
I''m
> reluctant to randomly try things now.
Duh :P

That''s part of the reason why I asked "did you restart the
dom0".
"service network restart" does not necessarily do what you think it
should do (for example, it determines what interface to take up/down
based on config file, not what interface is actually up).
>
>  So, if I manually build the bridges, how am I supposed to configure
> Xen to use them?
I simply modify Xen config file (/etc/xen/*) to use the bridge I
created (on vif line).

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 04/29/2011 06:26 PM, Teck Choon Giam wrote:
> On Sat, Apr 30, 2011 at 6:08 AM, Digimer <linux@alteeve.com> wrote:
>> Hi all,
>>
>>  While trying to sort out/test some patches, I was told that
it''s best
>> to create bridges manually in
>> /etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented out
>> (network-script network-bridge) and restarted the network and Xen.
> 
> Yes, using system network configuration will be better.  Which
> distribution is this?  CentOS/RHEL 5?
> You commented out (network-script network-bridge)?  If you are using
> system network configuration to configure your bridge, have you try
> the following instead of commented out?
> 
> (network-script /bin/true)
That was the ticket. I was able to provision a new VM with two
interfaces, however, the vifs were at MTU 1500 so I still have work to
do. :)
>>  With the bridges manually in place, and no longer having
''pethX''
>> devices, I tried to provision a VM and it failed with:
>>
>> ===>> # virt-install --connect xen --name vm0002_pppoe --ram 2048
--arch
>> x86_64 --vcpus 1 --cpuset 1-7 --location
http://10.255.0.1/f9/x86_64/img
>> --os-type linux --os-variant rhel5.4 --disk
>> path=/dev/drbd_x4_vg0/vm0002_1 --network bridge=xenbr0 --network
>> bridge=xenbr2 --vnc --paravirt --debug
> 
> brctl show outout?
> 
> Is your manual ifcfg-xenbr0 a bridge to ifcfg-eth0 and ifcfg-xenbr2 a
> bridge to ifcfg-eth2?
> Where is your ifcfg-eth1 and ifcfg-xenbr1?  Normal network
> configuration will have ifcfg-eth* in order... ...
I use eth1 on the dom0''s as storage-only interfaces. As such,
they''ll
never be used by a domU, so I don''t bother bridging them.
> It is hard to tell what went wrong without seeing the following outputs:
> 
> brctl show
> ifconfig
> ip link
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon
If I run into more problems, I''ll be sure to include the output from
those. Thanks again, that /bin/true was the ticket. :)

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 1, 2011 at 12:50 AM, Digimer <linux@alteeve.com>
wrote:
> On 04/29/2011 06:26 PM, Teck Choon Giam wrote:
>> On Sat, Apr 30, 2011 at 6:08 AM, Digimer <linux@alteeve.com>
wrote:
>>> Hi all,
>>>
>>>  While trying to sort out/test some patches, I was told that
it''s best
>>> to create bridges manually in
>>> /etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented
out
>>> (network-script network-bridge) and restarted the network and Xen.
>>
>> Yes, using system network configuration will be better.  Which
>> distribution is this?  CentOS/RHEL 5?
>> You commented out (network-script network-bridge)?  If you are using
>> system network configuration to configure your bridge, have you try
>> the following instead of commented out?
>>
>> (network-script /bin/true)
>
> That was the ticket. I was able to provision a new VM with two
> interfaces, however, the vifs were at MTU 1500 so I still have work to
> do. :)
Glad to hear that ;)

Whereby for MTU... maybe you want to try manually set those?  For example:

To get the link of various ip link... ...
ip link show

Then if you want to set the mtu for vif1.0 for instance... do something like:

ip link set dev vif1.0 mtu 1412

Of course in your case, vif1.0 or whatever appended with .0 for vif#.0
will be your xenbr0/eth0... so you might want to do something like:

ip link set dev eth0 mtu 1412 << This can be configure in ifcfg-eth0
ip link set dev xenbr0 mtu 1412 << This also can be set in ifcfg-xenbr0
ip link set dev vif1.0 mtu 1412
ip link set dev vif2.0 mtu 1412

OR using ifconfig to set the related interface mtu.

so on... ... I don''t think this is difficult to find the vif related
script in /etc/xen/scripts/ to add in the appropriate link for your
desired mtu value... ... IMO.  I think this is vif-common.sh as the
below codes:

if [ "$type_if" = vif ]; then
    # Check presence of compulsory args.
    XENBUS_PATH="${XENBUS_PATH:?}"
    dev="${dev:?}"

    vifname=$(xenstore_read_default "$XENBUS_PATH/vifname"
"")
    if [ "$vifname" ]
    then
        if [ "$command" == "online" ] && ! ip link
show "$vifname" >&/dev/null
        then
            do_or_die ip link set "$dev" name "$vifname"
<< THIS IS
WHERE YOU CAN ADD BUT I HAVE NOT EXPLODE THIS NOR TEST THIS WITH
APPENDED mtu "$mtu"
        fi
        dev="$vifname"
    fi
elif [ "$type_if" = tap ]; then
    # Check presence of compulsory args.
    : ${INTERFACE:?}

    # Get xenbus_path from device name.
    # The name is built like that: "tap${domid}.${devid}".
    dev_=${dev#tap}
    domid=${dev_%.*}
    devid=${dev_#*.}

    XENBUS_PATH="/local/domain/0/backend/vif/$domid/$devid"
fi


Something like:

do_or_die ip link set "$dev" name "$vifname" mtu
"$vifmtu"

Or if doesn''t work just split to another line:

do_or_die ip link set "$dev" name "$vifname"
do_or_die ip link set dev "$vifname" mtu "$vifmtu"

Or use ifconfig... ...

Of course you will need to set $vifmtu somewhere before the above or
use a function/line to get its bridge/parent interface name
(xenbr0/eth0) mtu value ... ... this is purely talking rubbish from me
without testing... just thoughts/ideas :p

Example:

Get parent/bridge interface mtu value for xenbr0:

local vifmtu=`ip link show xenbr0|head -n 1|sed
''s@.*\Wmtu\W@@''|sed ''s@\W.*@@''`

Or even better way is we know vif#.N where N will be the bridge number:

local myxenbrN=`echo $vifname | cut -f2 -d ''.''`
local myxenbrname="xenbr${myxenbrN}"
local vifmtu=`ip link show ${myxenbrname}|head -n 1|sed
''s@.*\Wmtu\W@@''|sed ''s@\W.*@@''`
do_or_die ip link set "$dev" name "$vifname" mtu
"$vifmtu"

Again, the above are ideas/thoughts without testing at all and I might
be totally wrong :p

Hope this helps!

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 04/30/2011 02:41 AM, Fajar A. Nugraha wrote:
> On Sat, Apr 30, 2011 at 5:08 AM, Digimer <linux@alteeve.com> wrote:
>> Hi all,
>>
>>  While trying to sort out/test some patches, I was told that
it''s best
>> to create bridges manually in
>> /etc/sysconfig/network-scripts/ifcfg-xenbr*. I did this, commented out
>> (network-script network-bridge) and restarted the network and Xen.
> 
> Did you restart the dom0?
Yup. :)
>> # virt-install --connect xen --name vm0002_pppoe --ram 2048 --arch
>> x86_64 --vcpus 1 --cpuset 1-7 --location
http://10.255.0.1/f9/x86_64/img
>> --os-type linux --os-variant rhel5.4 --disk
>> path=/dev/drbd_x4_vg0/vm0002_1 --network bridge=xenbr0 --network
>> bridge=xenbr2 --vnc --paravirt --debug
> 
> Do you have xenbr0 and xenbr2 bridge?
Yup.
>> Fri, 29 Apr 2011 17:32:12 DEBUG    Requesting libvirt URI xen
>> Fri, 29 Apr 2011 17:32:12 ERROR    unable to connect to
>> ''localhost:8000'': Connection refused
> 
> The message should be clear. Is xend running? Is it listening on port
> 8000 (try "netstat -anp")?
> 
>>  Thinking that I still needed to use ''(network-script
network-bridge)'',
>> so I put it back (actually, a modified version I will link below). When
>> I start xend after putting it back, I got such an incredible flood that
>> all network communications were lost on the network. Obviously,
I''m
>> reluctant to randomly try things now.
> 
> Duh :P
> 
> That''s part of the reason why I asked "did you restart the
dom0".
> "service network restart" does not necessarily do what you think
it
> should do (for example, it determines what interface to take up/down
> based on config file, not what interface is actually up).
Ya... I''ve long gotten into the habit of rebooting my test nodes
whenever I played with Xen''s networking for just this reason.
>>  So, if I manually build the bridges, how am I supposed to configure
>> Xen to use them?
> 
> I simply modify Xen config file (/etc/xen/*) to use the bridge I
> created (on vif line).
I finally got this sorted out, and here is the new patch and comments on
what was needed to make it work:

https://bugzilla.redhat.com/show_bug.cgi?id=697021#c11

Thanks for the reply!

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 04/30/2011 03:25 PM, Teck Choon Giam wrote:
> On Sun, May 1, 2011 at 12:50 AM, Digimer <linux@alteeve.com> wrote:
>> On 04/29/2011 06:26 PM, Teck Choon Giam wrote:
>>> On Sat, Apr 30, 2011 at 6:08 AM, Digimer <linux@alteeve.com>
wrote:
>>>
>>> (network-script /bin/true)
>>
>> That was the ticket. I was able to provision a new VM with two
>> interfaces, however, the vifs were at MTU 1500 so I still have work to
>> do. :)
> 
> Glad to hear that ;)
> 
> Whereby for MTU... maybe you want to try manually set those?  For example:
> 
> To get the link of various ip link... ...
> ip link show
> 
> Then if you want to set the mtu for vif1.0 for instance... do something
like:
> 
> ip link set dev vif1.0 mtu 1412
> 
> Of course in your case, vif1.0 or whatever appended with .0 for vif#.0
> will be your xenbr0/eth0... so you might want to do something like:
> 
> ip link set dev eth0 mtu 1412 << This can be configure in ifcfg-eth0
> ip link set dev xenbr0 mtu 1412 << This also can be set in
ifcfg-xenbr0
> ip link set dev vif1.0 mtu 1412
> ip link set dev vif2.0 mtu 1412
> 
> OR using ifconfig to set the related interface mtu.
> 
> so on... ... I don''t think this is difficult to find the vif
related
> script in /etc/xen/scripts/ to add in the appropriate link for your
> desired mtu value... ... IMO.  I think this is vif-common.sh as the
> below codes:
> 
> if [ "$type_if" = vif ]; then
>     # Check presence of compulsory args.
>     XENBUS_PATH="${XENBUS_PATH:?}"
>     dev="${dev:?}"
> 
>     vifname=$(xenstore_read_default "$XENBUS_PATH/vifname"
"")
>     if [ "$vifname" ]
>     then
>         if [ "$command" == "online" ] && ! ip
link show "$vifname" >&/dev/null
>         then
>             do_or_die ip link set "$dev" name
"$vifname" << THIS IS
> WHERE YOU CAN ADD BUT I HAVE NOT EXPLODE THIS NOR TEST THIS WITH
> APPENDED mtu "$mtu"
>         fi
>         dev="$vifname"
>     fi
> elif [ "$type_if" = tap ]; then
>     # Check presence of compulsory args.
>     : ${INTERFACE:?}
> 
>     # Get xenbus_path from device name.
>     # The name is built like that: "tap${domid}.${devid}".
>     dev_=${dev#tap}
>     domid=${dev_%.*}
>     devid=${dev_#*.}
> 
>     XENBUS_PATH="/local/domain/0/backend/vif/$domid/$devid"
> fi
> 
> 
> Something like:
> 
> do_or_die ip link set "$dev" name "$vifname" mtu
"$vifmtu"
> 
> Or if doesn''t work just split to another line:
> 
> do_or_die ip link set "$dev" name "$vifname"
> do_or_die ip link set dev "$vifname" mtu "$vifmtu"
> 
> Or use ifconfig... ...
> 
> Of course you will need to set $vifmtu somewhere before the above or
> use a function/line to get its bridge/parent interface name
> (xenbr0/eth0) mtu value ... ... this is purely talking rubbish from me
> without testing... just thoughts/ideas :p
> 
> Example:
> 
> Get parent/bridge interface mtu value for xenbr0:
> 
> local vifmtu=`ip link show xenbr0|head -n 1|sed
''s@.*\Wmtu\W@@''|sed ''s@\W.*@@''`
> 
> Or even better way is we know vif#.N where N will be the bridge number:
> 
> local myxenbrN=`echo $vifname | cut -f2 -d ''.''`
> local myxenbrname="xenbr${myxenbrN}"
> local vifmtu=`ip link show ${myxenbrname}|head -n 1|sed
> ''s@.*\Wmtu\W@@''|sed ''s@\W.*@@''`
> do_or_die ip link set "$dev" name "$vifname" mtu
"$vifmtu"
> 
> Again, the above are ideas/thoughts without testing at all and I might
> be totally wrong :p
> 
> Hope this helps!
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon
Hi Giam,

  This was the approach I took, which resulted in this patch:

https://bugzilla.redhat.com/attachment.cgi?id=495998&action=diff

  The trick with doing it manually is that the bridge''s MTU is set to
the lowest MTU of all attached ifs. So if a new vif comes up at 1500 and
is then connected to a bridge that had been at 9000, all traffic >1500
bytes will die on the bridge. So, as you explored, I had to edit the
userland scripts to up the vif''s MTU before it connected to the bridge.

  I''d done this earlier, but it was bugged. While trying to solve that
new bug, Pasi suggested I take the bridge control out of Xen''s control
and that prompted the experimenting that got me in trouble. :)

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 1, 2011 at 5:32 AM, Digimer <linux@alteeve.com>
wrote:
> On 04/30/2011 03:25 PM, Teck Choon Giam wrote:
>> On Sun, May 1, 2011 at 12:50 AM, Digimer <linux@alteeve.com>
wrote:
>>> On 04/29/2011 06:26 PM, Teck Choon Giam wrote:
>>>> On Sat, Apr 30, 2011 at 6:08 AM, Digimer
<linux@alteeve.com> wrote:
>>>>
>>>> (network-script /bin/true)
>>>
>>> That was the ticket. I was able to provision a new VM with two
>>> interfaces, however, the vifs were at MTU 1500 so I still have work
to
>>> do. :)
>>
>> Glad to hear that ;)
>>
>> Whereby for MTU... maybe you want to try manually set those?  For
example:
>>
>> To get the link of various ip link... ...
>> ip link show
>>
>> Then if you want to set the mtu for vif1.0 for instance... do something
like:
>>
>> ip link set dev vif1.0 mtu 1412
>>
>> Of course in your case, vif1.0 or whatever appended with .0 for vif#.0
>> will be your xenbr0/eth0... so you might want to do something like:
>>
>> ip link set dev eth0 mtu 1412 << This can be configure in
ifcfg-eth0
>> ip link set dev xenbr0 mtu 1412 << This also can be set in
ifcfg-xenbr0
>> ip link set dev vif1.0 mtu 1412
>> ip link set dev vif2.0 mtu 1412
>>
>> OR using ifconfig to set the related interface mtu.
>>
>> so on... ... I don''t think this is difficult to find the vif
related
>> script in /etc/xen/scripts/ to add in the appropriate link for your
>> desired mtu value... ... IMO.  I think this is vif-common.sh as the
>> below codes:
>>
>> if [ "$type_if" = vif ]; then
>>     # Check presence of compulsory args.
>>     XENBUS_PATH="${XENBUS_PATH:?}"
>>     dev="${dev:?}"
>>
>>     vifname=$(xenstore_read_default "$XENBUS_PATH/vifname"
"")
>>     if [ "$vifname" ]
>>     then
>>         if [ "$command" == "online" ] && !
ip link show "$vifname" >&/dev/null
>>         then
>>             do_or_die ip link set "$dev" name
"$vifname" << THIS IS
>> WHERE YOU CAN ADD BUT I HAVE NOT EXPLODE THIS NOR TEST THIS WITH
>> APPENDED mtu "$mtu"
>>         fi
>>         dev="$vifname"
>>     fi
>> elif [ "$type_if" = tap ]; then
>>     # Check presence of compulsory args.
>>     : ${INTERFACE:?}
>>
>>     # Get xenbus_path from device name.
>>     # The name is built like that: "tap${domid}.${devid}".
>>     dev_=${dev#tap}
>>     domid=${dev_%.*}
>>     devid=${dev_#*.}
>>
>>     XENBUS_PATH="/local/domain/0/backend/vif/$domid/$devid"
>> fi
>>
>>
>> Something like:
>>
>> do_or_die ip link set "$dev" name "$vifname" mtu
"$vifmtu"
>>
>> Or if doesn''t work just split to another line:
>>
>> do_or_die ip link set "$dev" name "$vifname"
>> do_or_die ip link set dev "$vifname" mtu "$vifmtu"
>>
>> Or use ifconfig... ...
>>
>> Of course you will need to set $vifmtu somewhere before the above or
>> use a function/line to get its bridge/parent interface name
>> (xenbr0/eth0) mtu value ... ... this is purely talking rubbish from me
>> without testing... just thoughts/ideas :p
>>
>> Example:
>>
>> Get parent/bridge interface mtu value for xenbr0:
>>
>> local vifmtu=`ip link show xenbr0|head -n 1|sed
''s@.*\Wmtu\W@@''|sed ''s@\W.*@@''`
>>
>> Or even better way is we know vif#.N where N will be the bridge number:
>>
>> local myxenbrN=`echo $vifname | cut -f2 -d ''.''`
>> local myxenbrname="xenbr${myxenbrN}"
>> local vifmtu=`ip link show ${myxenbrname}|head -n 1|sed
>> ''s@.*\Wmtu\W@@''|sed ''s@\W.*@@''`
>> do_or_die ip link set "$dev" name "$vifname" mtu
"$vifmtu"
>>
>> Again, the above are ideas/thoughts without testing at all and I might
>> be totally wrong :p
>>
>> Hope this helps!
>>
>> Thanks.
>>
>> Kindest regards,
>> Giam Teck Choon
>
> Hi Giam,
>
>  This was the approach I took, which resulted in this patch:
>
> https://bugzilla.redhat.com/attachment.cgi?id=495998&action=diff
>
>  The trick with doing it manually is that the bridge''s MTU is set
to
> the lowest MTU of all attached ifs. So if a new vif comes up at 1500 and
> is then connected to a bridge that had been at 9000, all traffic >1500
> bytes will die on the bridge. So, as you explored, I had to edit the
> userland scripts to up the vif''s MTU before it connected to the
bridge.
>
>  I''d done this earlier, but it was bugged. While trying to solve
that
> new bug, Pasi suggested I take the bridge control out of Xen''s
control
> and that prompted the experimenting that got me in trouble. :)
>
Pasi is right about to use system network configuration so that xend
and system network are separated.  Can you explain how it was bugged?
Is it sometimes get set and sometimes not?  Have you try to give a
small sleep interval such as 3 seconds before setting the related mtu?

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 04/30/2011 08:47 PM, Teck Choon Giam wrote:
> Pasi is right about to use system network configuration so that xend
> and system network are separated.  Can you explain how it was bugged?
> Is it sometimes get set and sometimes not?  Have you try to give a
> small sleep interval such as 3 seconds before setting the related mtu?
> 
> Thanks.
Sure.

  I''ve got three interfaces in my server;
- eth0; Internet-facing, bridged as xenbr0
- eth1; Storage network, not bridged or used by any domU.
- eth2; Back-channel/internal network, bridged as xenbr2, used by some domU.

  I had a patch[1] in xen-network-common.sh''s setup_bridge_port()
function that would look at the $dev passed in, parse it and pull the
MTU from the corresponding bridge. The bug was that this $dev matched
the ethX device in the domU rather than the bridge, so when the second
interface (xenbr2 <-> domU''s eth1) would look for the MTU of
''xenbr1'',
which didn''t exist.

  In trying to fix that with Paolo''s help, I found that it stopped
updating the vif0.0 and vif2.0 used by dom0. In trying to find out why,
Pasi mentioned that I should move the bridge''s outside of Xen, which I
did now. With the bridges outside of xen, they were set to the MTU of
dom0''s corresponding ethX devices on boot. Then I could focus on
getting
the domU vif''s at the corresponding bridge''s MTU and now you
know the
rest of the story. :)

Digimer

1. https://bugzilla.redhat.com/attachment.cgi?id=492964&action=diff

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 1, 2011 at 10:04 AM, Digimer <linux@alteeve.com>
wrote:
> On 04/30/2011 08:47 PM, Teck Choon Giam wrote:
>> Pasi is right about to use system network configuration so that xend
>> and system network are separated.  Can you explain how it was bugged?
>> Is it sometimes get set and sometimes not?  Have you try to give a
>> small sleep interval such as 3 seconds before setting the related mtu?
>>
>> Thanks.
>
> Sure.
>
>  I''ve got three interfaces in my server;
> - eth0; Internet-facing, bridged as xenbr0
> - eth1; Storage network, not bridged or used by any domU.
> - eth2; Back-channel/internal network, bridged as xenbr2, used by some
domU.
>
>  I had a patch[1] in xen-network-common.sh''s setup_bridge_port()
> function that would look at the $dev passed in, parse it and pull the
> MTU from the corresponding bridge. The bug was that this $dev matched
> the ethX device in the domU rather than the bridge, so when the second
> interface (xenbr2 <-> domU''s eth1) would look for the MTU of
''xenbr1'',
> which didn''t exist.
Your domUs are all HVM or PV or both in mix with the same problem?

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 05/01/2011 12:53 AM, Teck Choon Giam wrote:
> On Sun, May 1, 2011 at 10:04 AM, Digimer <linux@alteeve.com> wrote:
>> On 04/30/2011 08:47 PM, Teck Choon Giam wrote:
>>> Pasi is right about to use system network configuration so that
xend
>>> and system network are separated.  Can you explain how it was
bugged?
>>> Is it sometimes get set and sometimes not?  Have you try to give a
>>> small sleep interval such as 3 seconds before setting the related
mtu?
>>>
>>> Thanks.
>>
>> Sure.
>>
>>  I''ve got three interfaces in my server;
>> - eth0; Internet-facing, bridged as xenbr0
>> - eth1; Storage network, not bridged or used by any domU.
>> - eth2; Back-channel/internal network, bridged as xenbr2, used by some
domU.
>>
>>  I had a patch[1] in xen-network-common.sh''s
setup_bridge_port()
>> function that would look at the $dev passed in, parse it and pull the
>> MTU from the corresponding bridge. The bug was that this $dev matched
>> the ethX device in the domU rather than the bridge, so when the second
>> interface (xenbr2 <-> domU''s eth1) would look for the
MTU of ''xenbr1'',
>> which didn''t exist.
> 
> Your domUs are all HVM or PV or both in mix with the same problem?
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon
They''re a mix, and the issue was across the board.

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 1, 2011 at 1:17 PM, Digimer <linux@alteeve.com>
wrote:
> On 05/01/2011 12:53 AM, Teck Choon Giam wrote:
>> On Sun, May 1, 2011 at 10:04 AM, Digimer <linux@alteeve.com>
wrote:
>>> On 04/30/2011 08:47 PM, Teck Choon Giam wrote:
>>>> Pasi is right about to use system network configuration so that
xend
>>>> and system network are separated.  Can you explain how it was
bugged?
>>>> Is it sometimes get set and sometimes not?  Have you try to
give a
>>>> small sleep interval such as 3 seconds before setting the
related mtu?
>>>>
>>>> Thanks.
>>>
>>> Sure.
>>>
>>>  I''ve got three interfaces in my server;
>>> - eth0; Internet-facing, bridged as xenbr0
>>> - eth1; Storage network, not bridged or used by any domU.
>>> - eth2; Back-channel/internal network, bridged as xenbr2, used by
some domU.
>>>
>>>  I had a patch[1] in xen-network-common.sh''s
setup_bridge_port()
>>> function that would look at the $dev passed in, parse it and pull
the
>>> MTU from the corresponding bridge. The bug was that this $dev
matched
>>> the ethX device in the domU rather than the bridge, so when the
second
>>> interface (xenbr2 <-> domU''s eth1) would look for
the MTU of ''xenbr1'',
>>> which didn''t exist.
>>
>> Your domUs are all HVM or PV or both in mix with the same problem?
>>
>> Thanks.
>>
>> Kindest regards,
>> Giam Teck Choon
>
> They''re a mix, and the issue was across the board.
>
Ok, mind to show me your brctl show output?  I am curious about what
is the vif?.? assignment to your xenbr2 especially.  It is either
vif?.1 or vif?.2.  Have you tried with vifname set in your domU
configs to see whether the same problem persist?

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 05/02/2011 12:04 PM, Teck Choon Giam wrote:
> Ok, mind to show me your brctl show output?  I am curious about what
> is the vif?.? assignment to your xenbr2 especially.  It is either
> vif?.1 or vif?.2.  Have you tried with vifname set in your domU
> configs to see whether the same problem persist?
> 
> Thanks.
To get that info, I''d need to reconfigure a test cluster with the
bugged
patch. If it''s curiosity only, I''d like to set it aside. If it
could
help with development though, let me know and I''ll do so tonight.

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, May 3, 2011 at 12:10 AM, Digimer <linux@alteeve.com>
wrote:
> On 05/02/2011 12:04 PM, Teck Choon Giam wrote:
>> Ok, mind to show me your brctl show output?  I am curious about what
>> is the vif?.? assignment to your xenbr2 especially.  It is either
>> vif?.1 or vif?.2.  Have you tried with vifname set in your domU
>> configs to see whether the same problem persist?
>>
>> Thanks.
>
> To get that info, I''d need to reconfigure a test cluster with the
bugged
> patch. If it''s curiosity only, I''d like to set it aside.
If it could
> help with development though, let me know and I''ll do so tonight.
>
Curious only though but that might helps to determine one or more of
your patches issue.  One of your patch at least is using vif?.N to get
its physical ethN if I am not mistaken:

https://bugzilla.redhat.com/attachment.cgi?id=492964&action=diff

pdev=`echo $dev | sed -e "s/^vif\(.*\)\.\(.*\)$/peth\2/"`

So if $dev is vif?.N here, the pdev will be pethN right?  However,
since you are using system network configuration to configure your
bridge, your physical ethN remain the same as it is and there isn''t
any pethN here I believe.  Correct me if I am wrong though :p

And your next line is:

mtu=`cat /sys/class/net/$pdev/mtu`

So if $pdev not set or empty or pethN not found, you will have issue here... ...

Now let said you are using xen provided to do the bridging instead of
using system network configuration... so pethN will be set by xen or
rename real ethN to pethN and setup bridge name as ethN if I remember
correctly... ... now if vif?.N where N is 1 for your xenbr2/peth2 you
will have issue as well :p

You might want to try setting vifname for your testing domU config to
see whether does it resolve your issue?

I haven''t look closely for your other patches though and what I stated
above hopefully make sense :p

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 05/02/2011 01:33 PM, Teck Choon Giam wrote:
> On Tue, May 3, 2011 at 12:10 AM, Digimer <linux@alteeve.com> wrote:
>> On 05/02/2011 12:04 PM, Teck Choon Giam wrote:
>>> Ok, mind to show me your brctl show output?  I am curious about
what
>>> is the vif?.? assignment to your xenbr2 especially.  It is either
>>> vif?.1 or vif?.2.  Have you tried with vifname set in your domU
>>> configs to see whether the same problem persist?
>>>
>>> Thanks.
>>
>> To get that info, I''d need to reconfigure a test cluster with
the bugged
>> patch. If it''s curiosity only, I''d like to set it
aside. If it could
>> help with development though, let me know and I''ll do so
tonight.
>>
> 
> Curious only though but that might helps to determine one or more of
> your patches issue.  One of your patch at least is using vif?.N to get
> its physical ethN if I am not mistaken:
> 
> https://bugzilla.redhat.com/attachment.cgi?id=492964&action=diff
That patch is now outdated, and was the one with the flaw.
> pdev=`echo $dev | sed -e "s/^vif\(.*\)\.\(.*\)$/peth\2/"`
> 
> So if $dev is vif?.N here, the pdev will be pethN right?  However,
> since you are using system network configuration to configure your
> bridge, your physical ethN remain the same as it is and there
isn''t
> any pethN here I believe.  Correct me if I am wrong though :p
peth=pethN, yes. At the time though, I was /not/ configuring the xenbrX
devices in the system, I was letting Xen do it. When Xen does it, that
is when the ''pethX'' devices were created. Now that I am using
system-built bridges, pethX devices no longer exist.
> And your next line is:
> 
> mtu=`cat /sys/class/net/$pdev/mtu`
> 
> So if $pdev not set or empty or pethN not found, you will have issue
here... ...
Exactly, however, $pdev was set to ''peth1'' when the vif was
about to be
connected to xenbr2. The results were the same; The script failed, but
the cause was different (the xenbr2 <-> peth1 mapping).
> Now let said you are using xen provided to do the bridging instead of
> using system network configuration... so pethN will be set by xen or
> rename real ethN to pethN and setup bridge name as ethN if I remember
> correctly... ... now if vif?.N where N is 1 for your xenbr2/peth2 you
> will have issue as well :p
Bingo.
> You might want to try setting vifname for your testing domU config to
> see whether does it resolve your issue?
> 
> I haven''t look closely for your other patches though and what I
stated
> above hopefully make sense :p
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon
I was able to make it work with a slight modification to a suggestion
that Paolo gave me on IRC:

https://bugzilla.redhat.com/attachment.cgi?id=496016&action=diff

The change was that, at this point in the xen-network-common[-bonded].sh
scripts, the ''vif'' special file didn''t exist, so I
directly grabbed the
MTU of the bridge instead. So far, this seems to be the winner. Testing
will tell if there are new/remaining issues.

-- 
Digimer
E-Mail: digimer@alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, May 3, 2011 at 1:51 AM, Digimer <linux@alteeve.com>
wrote:
> On 05/02/2011 01:33 PM, Teck Choon Giam wrote:
>> On Tue, May 3, 2011 at 12:10 AM, Digimer <linux@alteeve.com>
wrote:
>>> On 05/02/2011 12:04 PM, Teck Choon Giam wrote:
>>>> Ok, mind to show me your brctl show output?  I am curious about
what
>>>> is the vif?.? assignment to your xenbr2 especially.  It is
either
>>>> vif?.1 or vif?.2.  Have you tried with vifname set in your domU
>>>> configs to see whether the same problem persist?
>>>>
>>>> Thanks.
>>>
>>> To get that info, I''d need to reconfigure a test cluster
with the bugged
>>> patch. If it''s curiosity only, I''d like to set it
aside. If it could
>>> help with development though, let me know and I''ll do so
tonight.
>>>
>>
>> Curious only though but that might helps to determine one or more of
>> your patches issue.  One of your patch at least is using vif?.N to get
>> its physical ethN if I am not mistaken:
>>
>> https://bugzilla.redhat.com/attachment.cgi?id=492964&action=diff
>
> That patch is now outdated, and was the one with the flaw.
>
>> pdev=`echo $dev | sed -e "s/^vif\(.*\)\.\(.*\)$/peth\2/"`
>>
>> So if $dev is vif?.N here, the pdev will be pethN right?  However,
>> since you are using system network configuration to configure your
>> bridge, your physical ethN remain the same as it is and there
isn''t
>> any pethN here I believe.  Correct me if I am wrong though :p
>
> peth=pethN, yes. At the time though, I was /not/ configuring the xenbrX
> devices in the system, I was letting Xen do it. When Xen does it, that
> is when the ''pethX'' devices were created. Now that I am
using
> system-built bridges, pethX devices no longer exist.
>
>> And your next line is:
>>
>> mtu=`cat /sys/class/net/$pdev/mtu`
>>
>> So if $pdev not set or empty or pethN not found, you will have issue
here... ...
>
> Exactly, however, $pdev was set to ''peth1'' when the vif
was about to be
> connected to xenbr2. The results were the same; The script failed, but
> the cause was different (the xenbr2 <-> peth1 mapping).
>
>> Now let said you are using xen provided to do the bridging instead of
>> using system network configuration... so pethN will be set by xen or
>> rename real ethN to pethN and setup bridge name as ethN if I remember
>> correctly... ... now if vif?.N where N is 1 for your xenbr2/peth2 you
>> will have issue as well :p
>
> Bingo.
>
>> You might want to try setting vifname for your testing domU config to
>> see whether does it resolve your issue?
>>
>> I haven''t look closely for your other patches though and what
I stated
>> above hopefully make sense :p
>>
>> Thanks.
>>
>> Kindest regards,
>> Giam Teck Choon
>
> I was able to make it work with a slight modification to a suggestion
> that Paolo gave me on IRC:
>
> https://bugzilla.redhat.com/attachment.cgi?id=496016&action=diff
>
> The change was that, at this point in the xen-network-common[-bonded].sh
> scripts, the ''vif'' special file didn''t exist, so
I directly grabbed the
> MTU of the bridge instead. So far, this seems to be the winner. Testing
> will tell if there are new/remaining issues.
So can you verify that for your xenbr2 those vif/tap are ending with
{vif,tap}Y.2 instead of {vif,tap}Y.1?  That is why I am asking for
your brctl show output :p

One thing though, if doing anything related to HVM... ... tap/vif
related devices might be created slightly slower (sorry, can''t
remember if is vif or tap)... as xen created vif as well regardless
you are using HVM which using tap devices.  This is something that
caught me when I create patch to support tap devices if antispoofing
is enabled.  (sorry for this off-topic)... ...

Your qemu-ifup patch especially the below line:

pdev=`echo $1 | sed -e "s/^tap\(.*\)$/eth\1/"`

If I read correctly, you are trying to convert tapN to ethN right?  I
can''t remember xen <=3.1 naming for tap/vif devices... if it is
{tap/vif}N then it is fine but for xen4, tap/vif naming is
{tap/vif}Y.N though so your above patch will have issue in xen4 or
xen3.4 I think as you will get tap3.1 to eth3.1 instead of eth1... ...
I think it is {tap/vif}N mostly for xen <=3.1 and sorry I got bad
memory about different version of xen about the naming change in
vif/tap devices.

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users