libvirt users - May 2016 - Re: libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?

Hey folks, I'm working on setting up a Cuckoo sandbox malware lab at
home, and can't seem to get openvswitch and libvirt to work together
as-documented.  I'm not subscribed to the list, so please reply-all. 
Thanks in advance!

I've debugged this, and I think the interfaceid stuff in network XML
isn't doing much besides setting a custom interfaceid in an OpenVSwitch
table that really has no effect on the operation of OpenVSwitch.  I
think the -- set Interface vnet0 "external-ids:iface-id="{uuid
here}"
command run by libvirt doesn't actually set the _uuid of a new
interface. :(  It's just for an external system (e.g. libvirt) to track
an interface *it* creates in OVS.

I've got an OVS switch, "malware0" which is my malware lab
network.  I
made a "monitor0" interface, and told OVS to make that a span/mirror
of
all the traffic on the malware0 switch.

I attach monitor0 to my cuckoo VM, and boot the VM.  Well, actually
libvirt REALLY wants to create the interface itself, so I have to delete
the monitor0 interface from OVS.  I boot the VM, libvirt creates the
monitor0 interface (with a different UUID) and it's not a span/mirror
port any more. :( 

I had all the ports in the cuckoo VM <parameters interfaceid='{uuid
here}'/>, but libvirt keeps deleting/creating ports and the OVS internal
UUID changes, which means any setting applied in OVS (e.g. setting a
port as a mirror) is lost during the add/delete cycle.

With my cuckoo sandbox vm up: (no bfc10d05 UUID in ports, the mirror
port UUID)

# ovs-vsctl list bridge
_uuid               : 16847994-eb75-4e71-a913-50edd8a89252
mirrors             : [bfc10d05-846e-4653-8417-27e1f648da93]
name                : "malware0"
ports               : [3931cb83-1f85-4fc8-880d-ffe299a85857,
6c6e3d97-d55b-4d55-8179-302412242664, 8eb4e13b-b5ea-4994-8e95-87d61db87ba6]

With the cuckoo vm down:  (393, 83b go away)

# ovs-vsctl list bridge
_uuid               : 16847994-eb75-4e71-a913-50edd8a89252
mirrors             : [bfc10d05-846e-4653-8417-27e1f648da93]
name                : "malware0"
ports               : [6c6e3d97-d55b-4d55-8179-302412242664]

... and lets bring up the cuckoo vm again: (only 6c6 remains)

# ovs-vsctl list bridge
_uuid               : 16847994-eb75-4e71-a913-50edd8a89252
mirrors             : [bfc10d05-846e-4653-8417-27e1f648da93]
name                : "malware0"
ports               : [1c09dd43-52d0-449b-81a2-537ddafb4966,
6c6e3d97-d55b-4d55-8179-302412242664, f90820f9-056f-47a3-bd51-c5190ad1df46]

On 05/13/2016 09:57 AM, Richard Harman wrote:
> Hey folks, I'm working on setting up a Cuckoo sandbox malware lab at
> home, and can't seem to get openvswitch and libvirt to work together
> as-documented.  I'm not subscribed to the list, so please reply-all.
> Thanks in advance!
>
> I've debugged this, and I think the interfaceid stuff in network XML
> isn't doing much besides setting a custom interfaceid in an OpenVSwitch
> table that really has no effect on the operation of OpenVSwitch.  I
> think the -- set Interface vnet0 "external-ids:iface-id="{uuid
here}"
> command run by libvirt doesn't actually set the _uuid of a new
> interface. :(
I don't use Open vSwitch, but I was around when the support was added, 
and reviewed the patches (which were authored by Kyle Mestery, whose 
email has changed since then, so I'm not sure how to Cc him). My 
understanding at the time was that the interfaceid was intended for 
exactly the purpose that you say doesn't work (i.e. so that OVS itself 
could recognize that interface as it appeared and disappeared at 
different locations on the network). I would classify that as a bug, but 
that's just my understanding as an OVS outsider.

Anyone who is following OVS more closely?
>   It's just for an external system (e.g. libvirt) to track
> an interface *it* creates in OVS.
>
> I've got an OVS switch, "malware0" which is my malware lab
network.  I
> made a "monitor0" interface, and told OVS to make that a
span/mirror of
> all the traffic on the malware0 switch.
>
> I attach monitor0 to my cuckoo VM, and boot the VM.  Well, actually
> libvirt REALLY wants to create the interface itself, so I have to delete
> the monitor0 interface from OVS.  I boot the VM, libvirt creates the
> monitor0 interface (with a different UUID) and it's not a span/mirror
> port any more. :(
>
> I had all the ports in the cuckoo VM <parameters interfaceid='{uuid
> here}'/>, but libvirt keeps deleting/creating ports and the OVS
internal
> UUID changes, which means any setting applied in OVS (e.g. setting a
> port as a mirror) is lost during the add/delete cycle.
>
> With my cuckoo sandbox vm up: (no bfc10d05 UUID in ports, the mirror
> port UUID)
>
> # ovs-vsctl list bridge
> _uuid               : 16847994-eb75-4e71-a913-50edd8a89252
> mirrors             : [bfc10d05-846e-4653-8417-27e1f648da93]
> name                : "malware0"
> ports               : [3931cb83-1f85-4fc8-880d-ffe299a85857,
> 6c6e3d97-d55b-4d55-8179-302412242664, 8eb4e13b-b5ea-4994-8e95-87d61db87ba6]
>
> With the cuckoo vm down:  (393, 83b go away)
>
> # ovs-vsctl list bridge
> _uuid               : 16847994-eb75-4e71-a913-50edd8a89252
> mirrors             : [bfc10d05-846e-4653-8417-27e1f648da93]
> name                : "malware0"
> ports               : [6c6e3d97-d55b-4d55-8179-302412242664]
>
> ... and lets bring up the cuckoo vm again: (only 6c6 remains)
>
> # ovs-vsctl list bridge
> _uuid               : 16847994-eb75-4e71-a913-50edd8a89252
> mirrors             : [bfc10d05-846e-4653-8417-27e1f648da93]
> name                : "malware0"
> ports               : [1c09dd43-52d0-449b-81a2-537ddafb4966,
> 6c6e3d97-d55b-4d55-8179-302412242664, f90820f9-056f-47a3-bd51-c5190ad1df46]
>
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>

On 05/13/16 20:04, Laine Stump wrote:
> On 05/13/2016 09:57 AM, Richard Harman wrote:
>> I've debugged this, and I think the interfaceid stuff in network
XML
>> isn't doing much besides setting a custom interfaceid in an
OpenVSwitch
>> table that really has no effect on the operation of OpenVSwitch.  I
>> think the -- set Interface vnet0
"external-ids:iface-id="{uuid here}"
>> command run by libvirt doesn't actually set the _uuid of a new
>> interface. :(
>
> I don't use Open vSwitch, but I was around when the support was added,
> and reviewed the patches (which were authored by Kyle Mestery, whose
> email has changed since then, so I'm not sure how to Cc him). My
> understanding at the time was that the interfaceid was intended for
> exactly the purpose that you say doesn't work (i.e. so that OVS itself
> could recognize that interface as it appeared and disappeared at
> different locations on the network). I would classify that as a bug,
> but that's just my understanding as an OVS outsider.
>
Further evidence, now that I figured out how to dump the external-ids
stuff in OVS:

Here's the relevant ports of the domain XML:

<domain type='kvm' id='5'>
  <name>cuckoo</name>
  <uuid>cb4903aa-eaf4-432d-aa57-e9f74f1c8b79</uuid>
....
    <interface type='bridge'>
      <mac address='52:54:00:f5:29:72'/>
      <source network='malware' bridge='malware0'/>
      <virtualport type='openvswitch'>
        <parameters
interfaceid='23e982f3-f344-48a2-8bed-b584acb95616'/>
      </virtualport>
      <target dev='monitor0'/>
      <model type='virtio'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x0'/>
    </interface>
....

cuckoo vm up: monitor0 _uuid is 969ef76c...

# ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3
monitor0
name                : "monitor0"
_uuid               : 969ef76c-c802-4e9f-97e0-9a06ce133252
external_ids        : {attached-mac="52:54:00:f5:29:72",
iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active,
vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"}
#

cuckoo vm down: (no output, the interface has been deleted by libvirt)

# ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3
monitor0
#

I found this in my log, showing libvirt deleted the interface:
ovs-vsctl[4590]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 --
--if-exists del-port monitor0

cuckoo vm up again: uuid is now fb025df0...

# ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3
monitor0
name                : "monitor0"
_uuid               : fb025df0-d934-46d6-b9ed-314bc1de5ef3
external_ids        : {attached-mac="52:54:00:f5:29:72",
iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active,
vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"}
#

Anyway, thanks for reading!  Hopefully we'll get to the bottom of this,
where I can keep my span port. :)  In the mean time I'm going to go
digging through the OVS source, specifically the ovs-vsctl command to
see if an argument can be used to forcibly set the internal _uuid in OVS.

Richard