Steven Maresca
2008-Apr-07 10:31 UTC
[Xen-devel] [RFC] correlation of HVM tapX devices to domain
Hello all.
This message is a request-for-comments regarding a common issue with
fully virtualized domUs. Individuals carbon copied have expressed
interest in the issue recently and in the past. It may eventually be
beneficial to cross-post to xen-users, but I suspect that xen-devel is
the proper venue for now.
Specifically, I am inquiring about the correlation of tapX network
devices to the relevant HVM domains. At present such correlation is
not possible, though I will propose some solutions. If this problem or
any facets have already been addressed in recent changesets, I
apologize, but I do not believe that to be the case. Regardless, this
should remain relevant for those with existing installations.
Following the body of this message, I am including a bash function
with instructions for an accurate workaround in qemu-ifup. A possible
patch to the existing qemu code that creates the tap device is
suggested, seeking comment.
Some summary of behavior:
1.) These devices are generated by qemu-dm, are not managed by the
vif-* scripts, and at present do not have any representation in
xenstore.
2.) Unlike the vifDOMID.IFNUM naming conventions of devices for
paravirtual network interfaces, these emulated devices are simply
tapX, with corresponding loosely to the number of HVM domains and
number of emulated interfaces per domain.
3.) The vifname parameter when specified in a domU configuration
applies only to paravirtual network interfaces; it is not applied to
the tap
The implications include difficulty for firewalling, inability to
accurately monitor domains at an interface level, and divergence of
semantics between paravirtual and HVM domains.
Further (tangentally) complicating the issue are the paths by which
HVM network vifX.Y and tapX interfaces are created, as summarized in
this posting by Dan Berrange:
http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00160.html
: The lack of a type= definition for the vif(s) causes both ioemu and
PV interfaces to be created; the vifname parameter is applied only to
the interface for the paravirtual networking (obviously if two
interfaces are created, the vifname is of little use anyway due to a
lack of uniqueness).
---
With all of that said: In the interest of being noninvasive and
maintaining existing behaviors that some may rely upon, I would
suggest two possible solutions.
1) A workaround using the qemu-ifup script, tested and accurate. It
is called directly by the qemu-dm process, and thus, the $PPID
variable set at runtime in the script directly correlates with the PID
of the qemu-dm process of the related domain. Likewise, the first
field of the $* variable contains the tap interface name. Parsing the
DOMID of the domain from qemu-dm''s arguments, we can then write the
information to xenstore. I currently use the following path:
/local/domain/${DOMID}/device/vif-ioemu . This option would integrate
easily with existing installations and is valid for multiple ioemu
interfaces. Function and usage follows the body of this message.
2) A patch to tools/ioemu/vl.c in the net_tap_init() function which
accomplishes the same as above in a more appropriate place. The same
may be relevant in the vnet code where tap_open() is called. *Provided
that this route is agreeable, I will create the patch and followup to
this message.
Other possible solutions might include patching net_tap_init() to use
the appropriate vif-script to bring behavior closer to paravirtual
machines, but this would touch many places in the code and may not be
desirous. Alternatively or in conjunction, it may be beneficial to
apply the vifname parameter both to the PV vif and to the ioemu tap
device (with an "ioemu" suffix or something similar).
Notes for completeness:
The following thread from October 2007 "How to tell which tapX
interface belongs to which domain" addressed the topic and arrived at
a usable solution, yet is not necessarily ideal given the many
possible sources of error.
http://lists.xensource.com/archives/html/xen-users/2007-10/msg00208.html
The following recently submitted patch attempted to address this issue
supervficially in image.py, but (as it both lacks any xenstore
association and is several layers above the origin of the device
name), I''m not quite sure its the right way to approach the problem:
http://lists.xensource.com/archives/html/xen-devel/2008-03/msg00305.html
---
Comments, criticisms, and suggestions welcomed.
Thanks,
Steven Maresca
Workaround using qemu-ifup. Add the below function to qemu-ifup (or
source a file containing it), and place the following two lines at the
very bottom of the qemu-ifup script.
IFACE=`echo $* | awk ''{print $1}''`
saveHVMif ${PPID} ${IFACE}
#inelegant but accurate
saveHVMif () {
PARENTPID=$1
IF=$2
flagnum=0;
for i in $QEMUARGS;
do
flagnum=`expr $flagnum + 1`
if [ "${i}" == "-domain-name" ]; then
nameflag=0;
nameflag=`expr $flagnum + 1`;
NAME=`echo $QEMUARGS | awk ''{print
$''$nameflag''}''`;
fi;
done;
DOMID=`xm list ${NAME} | grep ${NAME} | awk ''{print
$2}''`
TOSAVE="${IF}"
OLDVAL=`xenstore-read /local/domain/${DOMID}/device/vif-ioemu`
if [ $? -ne 1 ]; then
for val in $OLDVALS; do
if [ "${val}" == "${IF}" ]; then
IF=""
fi
done
if [ "${IF}" != "" ]; then
TOSAVE="${IF},${OLDVAL}"
else
TOSAVE="${OLDVAL}"
fi
fi
xenstore-write /local/domain/${DOMID}/device/vif-ioemu
"${TOSAVE}"
};
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
http://zentific.com - Xen web management, coming soon!
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2008-Apr-07 10:43 UTC
Re: [Xen-devel] [RFC] correlation of HVM tapX devices to domain
I think this is addressed by xen-unstable changeset 17208. -- Keir On 7/4/08 11:31, "Steven Maresca" <lightyear4@gmail.com> wrote:> Hello all. > > This message is a request-for-comments regarding a common issue with > fully virtualized domUs. Individuals carbon copied have expressed > interest in the issue recently and in the past. It may eventually be > beneficial to cross-post to xen-users, but I suspect that xen-devel is > the proper venue for now. > > Specifically, I am inquiring about the correlation of tapX network > devices to the relevant HVM domains. At present such correlation is > not possible, though I will propose some solutions. If this problem or > any facets have already been addressed in recent changesets, I > apologize, but I do not believe that to be the case. Regardless, this > should remain relevant for those with existing installations. > > Following the body of this message, I am including a bash function > with instructions for an accurate workaround in qemu-ifup. A possible > patch to the existing qemu code that creates the tap device is > suggested, seeking comment. > > Some summary of behavior: > 1.) These devices are generated by qemu-dm, are not managed by the > vif-* scripts, and at present do not have any representation in > xenstore. > 2.) Unlike the vifDOMID.IFNUM naming conventions of devices for > paravirtual network interfaces, these emulated devices are simply > tapX, with corresponding loosely to the number of HVM domains and > number of emulated interfaces per domain. > 3.) The vifname parameter when specified in a domU configuration > applies only to paravirtual network interfaces; it is not applied to > the tap > > The implications include difficulty for firewalling, inability to > accurately monitor domains at an interface level, and divergence of > semantics between paravirtual and HVM domains. > > Further (tangentally) complicating the issue are the paths by which > HVM network vifX.Y and tapX interfaces are created, as summarized in > this posting by Dan Berrange: > http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00160.html > : The lack of a type= definition for the vif(s) causes both ioemu and > PV interfaces to be created; the vifname parameter is applied only to > the interface for the paravirtual networking (obviously if two > interfaces are created, the vifname is of little use anyway due to a > lack of uniqueness). > > --- > > With all of that said: In the interest of being noninvasive and > maintaining existing behaviors that some may rely upon, I would > suggest two possible solutions. > > 1) A workaround using the qemu-ifup script, tested and accurate. It > is called directly by the qemu-dm process, and thus, the $PPID > variable set at runtime in the script directly correlates with the PID > of the qemu-dm process of the related domain. Likewise, the first > field of the $* variable contains the tap interface name. Parsing the > DOMID of the domain from qemu-dm''s arguments, we can then write the > information to xenstore. I currently use the following path: > /local/domain/${DOMID}/device/vif-ioemu . This option would integrate > easily with existing installations and is valid for multiple ioemu > interfaces. Function and usage follows the body of this message. > > 2) A patch to tools/ioemu/vl.c in the net_tap_init() function which > accomplishes the same as above in a more appropriate place. The same > may be relevant in the vnet code where tap_open() is called. *Provided > that this route is agreeable, I will create the patch and followup to > this message. > > Other possible solutions might include patching net_tap_init() to use > the appropriate vif-script to bring behavior closer to paravirtual > machines, but this would touch many places in the code and may not be > desirous. Alternatively or in conjunction, it may be beneficial to > apply the vifname parameter both to the PV vif and to the ioemu tap > device (with an "ioemu" suffix or something similar). > > Notes for completeness: > > The following thread from October 2007 "How to tell which tapX > interface belongs to which domain" addressed the topic and arrived at > a usable solution, yet is not necessarily ideal given the many > possible sources of error. > http://lists.xensource.com/archives/html/xen-users/2007-10/msg00208.html > > The following recently submitted patch attempted to address this issue > supervficially in image.py, but (as it both lacks any xenstore > association and is several layers above the origin of the device > name), I''m not quite sure its the right way to approach the problem: > http://lists.xensource.com/archives/html/xen-devel/2008-03/msg00305.html > > --- > > Comments, criticisms, and suggestions welcomed. > > Thanks, > Steven Maresca > > > > > Workaround using qemu-ifup. Add the below function to qemu-ifup (or > source a file containing it), and place the following two lines at the > very bottom of the qemu-ifup script. > > IFACE=`echo $* | awk ''{print $1}''` > saveHVMif ${PPID} ${IFACE} > > > #inelegant but accurate > saveHVMif () { > PARENTPID=$1 > IF=$2 > > flagnum=0; > for i in $QEMUARGS; > do > > flagnum=`expr $flagnum + 1` > if [ "${i}" == "-domain-name" ]; then > nameflag=0; > nameflag=`expr $flagnum + 1`; > NAME=`echo $QEMUARGS | awk ''{print $''$nameflag''}''`; > fi; > done; > DOMID=`xm list ${NAME} | grep ${NAME} | awk ''{print $2}''` > TOSAVE="${IF}" > OLDVAL=`xenstore-read /local/domain/${DOMID}/device/vif-ioemu` > > if [ $? -ne 1 ]; then > for val in $OLDVALS; do > if [ "${val}" == "${IF}" ]; then > IF="" > fi > done > if [ "${IF}" != "" ]; then > TOSAVE="${IF},${OLDVAL}" > else > TOSAVE="${OLDVAL}" > fi > fi > xenstore-write /local/domain/${DOMID}/device/vif-ioemu "${TOSAVE}" > }; > > > > > > ------------------------------------------------------------------------------ > ---- > ------------------------------------------------------------------------------ > ---- > ------------------------------------------------------------------------------ > ---- > http://zentific.com - Xen web management, coming soon! > ------------------------------------------------------------------------------ > ---- > ------------------------------------------------------------------------------ > ---- > ------------------------------------------------------------------------------ > ---- > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Steven Maresca
2008-Apr-07 11:57 UTC
Re: [Xen-devel] [RFC] correlation of HVM tapX devices to domain
Keir,
Many thanks for the rapid reply.
It''s partly a fix, yes. I referenced the original relevant xen-devel
post it in my own; however, I think it falls short of being ideal.
Among other things, it would be beneficial to store such information
in xenstore. At present, no such record (of which I am aware) exists
for ioemu vifs. Additionally, it forces a name for the interface,
whereas some might find it desirable to specify vifname=ABC as is the
case for paravirtual interfaces.
An aside:
Qemu already supports a vifname field; it would be pretty simple to
implement in the same block of code in cs 17208 (though given the
different semantics of type=netfront vs type=ioemu vs no type defined,
some variety of suffix would be needed to keep it unique from a PV vif
if created). Something like:
vifname = devinfo.get(''vifname'', ''tap%d.%d''
% (self.vm.getDomid(), nics-1))
ret.append("tap,vlan=%d,ifname=tap%d.%d,bridge=%s,name=%s" % (nics,
self.vm.getDomid(), nics-1, bridge, vifname))
This sort of behavior (as discussed above and partly expressed in
changeset 17208) is definitely an improvement, though I still feel
that a corresponding entry should be established in xenstore.
-Steve
On Mon, Apr 7, 2008 at 6:43 AM, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:> I think this is addressed by xen-unstable changeset 17208.
>
> -- Keir
>
>
>
> On 7/4/08 11:31, "Steven Maresca" <lightyear4@gmail.com>
wrote:
>
> > Hello all.
> >
> > This message is a request-for-comments regarding a common issue with
> > fully virtualized domUs. Individuals carbon copied have expressed
> > interest in the issue recently and in the past. It may eventually be
> > beneficial to cross-post to xen-users, but I suspect that xen-devel
is
> > the proper venue for now.
> >
> > Specifically, I am inquiring about the correlation of tapX network
> > devices to the relevant HVM domains. At present such correlation is
> > not possible, though I will propose some solutions. If this problem
or
> > any facets have already been addressed in recent changesets, I
> > apologize, but I do not believe that to be the case. Regardless, this
> > should remain relevant for those with existing installations.
> >
> > Following the body of this message, I am including a bash function
> > with instructions for an accurate workaround in qemu-ifup. A possible
> > patch to the existing qemu code that creates the tap device is
> > suggested, seeking comment.
> >
> > Some summary of behavior:
> > 1.) These devices are generated by qemu-dm, are not managed by the
> > vif-* scripts, and at present do not have any representation in
> > xenstore.
> > 2.) Unlike the vifDOMID.IFNUM naming conventions of devices for
> > paravirtual network interfaces, these emulated devices are simply
> > tapX, with corresponding loosely to the number of HVM domains and
> > number of emulated interfaces per domain.
> > 3.) The vifname parameter when specified in a domU configuration
> > applies only to paravirtual network interfaces; it is not applied to
> > the tap
> >
> > The implications include difficulty for firewalling, inability to
> > accurately monitor domains at an interface level, and divergence of
> > semantics between paravirtual and HVM domains.
> >
> > Further (tangentally) complicating the issue are the paths by which
> > HVM network vifX.Y and tapX interfaces are created, as summarized in
> > this posting by Dan Berrange:
> >
http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00160.html
> > : The lack of a type= definition for the vif(s) causes both ioemu and
> > PV interfaces to be created; the vifname parameter is applied only to
> > the interface for the paravirtual networking (obviously if two
> > interfaces are created, the vifname is of little use anyway due to a
> > lack of uniqueness).
> >
> > ---
> >
> > With all of that said: In the interest of being noninvasive and
> > maintaining existing behaviors that some may rely upon, I would
> > suggest two possible solutions.
> >
> > 1) A workaround using the qemu-ifup script, tested and accurate. It
> > is called directly by the qemu-dm process, and thus, the $PPID
> > variable set at runtime in the script directly correlates with the
PID
> > of the qemu-dm process of the related domain. Likewise, the first
> > field of the $* variable contains the tap interface name. Parsing the
> > DOMID of the domain from qemu-dm''s arguments, we can then
write the
> > information to xenstore. I currently use the following path:
> > /local/domain/${DOMID}/device/vif-ioemu . This option would
integrate
> > easily with existing installations and is valid for multiple ioemu
> > interfaces. Function and usage follows the body of this message.
> >
> > 2) A patch to tools/ioemu/vl.c in the net_tap_init() function which
> > accomplishes the same as above in a more appropriate place. The same
> > may be relevant in the vnet code where tap_open() is called.
*Provided
> > that this route is agreeable, I will create the patch and followup to
> > this message.
> >
> > Other possible solutions might include patching net_tap_init() to use
> > the appropriate vif-script to bring behavior closer to paravirtual
> > machines, but this would touch many places in the code and may not be
> > desirous. Alternatively or in conjunction, it may be beneficial to
> > apply the vifname parameter both to the PV vif and to the ioemu tap
> > device (with an "ioemu" suffix or something similar).
> >
> > Notes for completeness:
> >
> > The following thread from October 2007 "How to tell which tapX
> > interface belongs to which domain" addressed the topic and
arrived at
> > a usable solution, yet is not necessarily ideal given the many
> > possible sources of error.
> >
http://lists.xensource.com/archives/html/xen-users/2007-10/msg00208.html
> >
> > The following recently submitted patch attempted to address this
issue
> > supervficially in image.py, but (as it both lacks any xenstore
> > association and is several layers above the origin of the device
> > name), I''m not quite sure its the right way to approach the
problem:
> >
http://lists.xensource.com/archives/html/xen-devel/2008-03/msg00305.html
> >
> > ---
> >
> > Comments, criticisms, and suggestions welcomed.
> >
> > Thanks,
> > Steven Maresca
> >
> >
> >
> >
> > Workaround using qemu-ifup. Add the below function to qemu-ifup (or
> > source a file containing it), and place the following two lines at
the
> > very bottom of the qemu-ifup script.
> >
> > IFACE=`echo $* | awk ''{print $1}''`
> > saveHVMif ${PPID} ${IFACE}
> >
> >
> > #inelegant but accurate
> > saveHVMif () {
> > PARENTPID=$1
> > IF=$2
> >
> > flagnum=0;
> > for i in $QEMUARGS;
> > do
> >
> > flagnum=`expr $flagnum + 1`
> > if [ "${i}" == "-domain-name" ];
then
> > nameflag=0;
> > nameflag=`expr $flagnum + 1`;
> > NAME=`echo $QEMUARGS | awk ''{print
$''$nameflag''}''`;
> > fi;
> > done;
> > DOMID=`xm list ${NAME} | grep ${NAME} | awk ''{print
$2}''`
> > TOSAVE="${IF}"
> > OLDVAL=`xenstore-read
/local/domain/${DOMID}/device/vif-ioemu`
> >
> > if [ $? -ne 1 ]; then
> > for val in $OLDVALS; do
> > if [ "${val}" == "${IF}"
]; then
> > IF=""
> > fi
> > done
> > if [ "${IF}" != "" ]; then
> > TOSAVE="${IF},${OLDVAL}"
> > else
> > TOSAVE="${OLDVAL}"
> > fi
> > fi
> > xenstore-write /local/domain/${DOMID}/device/vif-ioemu
"${TOSAVE}"
> > };
> >
> >
> >
> >
> >
> >
------------------------------------------------------------------------------
> > ----
> >
------------------------------------------------------------------------------
> > ----
> >
------------------------------------------------------------------------------
> > ----
> > http://zentific.com - Xen web management, coming soon!
> >
------------------------------------------------------------------------------
> > ----
> >
------------------------------------------------------------------------------
> > ----
> >
------------------------------------------------------------------------------
> > ----
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel