Richard Harman
2016-May-13 13:57 UTC
[libvirt-users] libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?
Hey folks, I'm working on setting up a Cuckoo sandbox malware lab at home, and can't seem to get openvswitch and libvirt to work together as-documented. I'm not subscribed to the list, so please reply-all. Thanks in advance! I've debugged this, and I think the interfaceid stuff in network XML isn't doing much besides setting a custom interfaceid in an OpenVSwitch table that really has no effect on the operation of OpenVSwitch. I think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" command run by libvirt doesn't actually set the _uuid of a new interface. :( It's just for an external system (e.g. libvirt) to track an interface *it* creates in OVS. I've got an OVS switch, "malware0" which is my malware lab network. I made a "monitor0" interface, and told OVS to make that a span/mirror of all the traffic on the malware0 switch. I attach monitor0 to my cuckoo VM, and boot the VM. Well, actually libvirt REALLY wants to create the interface itself, so I have to delete the monitor0 interface from OVS. I boot the VM, libvirt creates the monitor0 interface (with a different UUID) and it's not a span/mirror port any more. :( I had all the ports in the cuckoo VM <parameters interfaceid='{uuid here}'/>, but libvirt keeps deleting/creating ports and the OVS internal UUID changes, which means any setting applied in OVS (e.g. setting a port as a mirror) is lost during the add/delete cycle. With my cuckoo sandbox vm up: (no bfc10d05 UUID in ports, the mirror port UUID) # ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [3931cb83-1f85-4fc8-880d-ffe299a85857, 6c6e3d97-d55b-4d55-8179-302412242664, 8eb4e13b-b5ea-4994-8e95-87d61db87ba6] With the cuckoo vm down: (393, 83b go away) # ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [6c6e3d97-d55b-4d55-8179-302412242664] ... and lets bring up the cuckoo vm again: (only 6c6 remains) # ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [1c09dd43-52d0-449b-81a2-537ddafb4966, 6c6e3d97-d55b-4d55-8179-302412242664, f90820f9-056f-47a3-bd51-c5190ad1df46]
Laine Stump
2016-May-14 00:04 UTC
Re: [libvirt-users] libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?
On 05/13/2016 09:57 AM, Richard Harman wrote:> Hey folks, I'm working on setting up a Cuckoo sandbox malware lab at > home, and can't seem to get openvswitch and libvirt to work together > as-documented. I'm not subscribed to the list, so please reply-all. > Thanks in advance! > > I've debugged this, and I think the interfaceid stuff in network XML > isn't doing much besides setting a custom interfaceid in an OpenVSwitch > table that really has no effect on the operation of OpenVSwitch. I > think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" > command run by libvirt doesn't actually set the _uuid of a new > interface. :(I don't use Open vSwitch, but I was around when the support was added, and reviewed the patches (which were authored by Kyle Mestery, whose email has changed since then, so I'm not sure how to Cc him). My understanding at the time was that the interfaceid was intended for exactly the purpose that you say doesn't work (i.e. so that OVS itself could recognize that interface as it appeared and disappeared at different locations on the network). I would classify that as a bug, but that's just my understanding as an OVS outsider. Anyone who is following OVS more closely?> It's just for an external system (e.g. libvirt) to track > an interface *it* creates in OVS. > > I've got an OVS switch, "malware0" which is my malware lab network. I > made a "monitor0" interface, and told OVS to make that a span/mirror of > all the traffic on the malware0 switch. > > I attach monitor0 to my cuckoo VM, and boot the VM. Well, actually > libvirt REALLY wants to create the interface itself, so I have to delete > the monitor0 interface from OVS. I boot the VM, libvirt creates the > monitor0 interface (with a different UUID) and it's not a span/mirror > port any more. :( > > I had all the ports in the cuckoo VM <parameters interfaceid='{uuid > here}'/>, but libvirt keeps deleting/creating ports and the OVS internal > UUID changes, which means any setting applied in OVS (e.g. setting a > port as a mirror) is lost during the add/delete cycle. > > With my cuckoo sandbox vm up: (no bfc10d05 UUID in ports, the mirror > port UUID) > > # ovs-vsctl list bridge > _uuid : 16847994-eb75-4e71-a913-50edd8a89252 > mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] > name : "malware0" > ports : [3931cb83-1f85-4fc8-880d-ffe299a85857, > 6c6e3d97-d55b-4d55-8179-302412242664, 8eb4e13b-b5ea-4994-8e95-87d61db87ba6] > > With the cuckoo vm down: (393, 83b go away) > > # ovs-vsctl list bridge > _uuid : 16847994-eb75-4e71-a913-50edd8a89252 > mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] > name : "malware0" > ports : [6c6e3d97-d55b-4d55-8179-302412242664] > > ... and lets bring up the cuckoo vm again: (only 6c6 remains) > > # ovs-vsctl list bridge > _uuid : 16847994-eb75-4e71-a913-50edd8a89252 > mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] > name : "malware0" > ports : [1c09dd43-52d0-449b-81a2-537ddafb4966, > 6c6e3d97-d55b-4d55-8179-302412242664, f90820f9-056f-47a3-bd51-c5190ad1df46] > > > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users >
Richard Harman
2016-May-14 00:24 UTC
Re: [libvirt-users] libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?
On 05/13/16 20:04, Laine Stump wrote:> On 05/13/2016 09:57 AM, Richard Harman wrote: >> I've debugged this, and I think the interfaceid stuff in network XML >> isn't doing much besides setting a custom interfaceid in an OpenVSwitch >> table that really has no effect on the operation of OpenVSwitch. I >> think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" >> command run by libvirt doesn't actually set the _uuid of a new >> interface. :( > > I don't use Open vSwitch, but I was around when the support was added, > and reviewed the patches (which were authored by Kyle Mestery, whose > email has changed since then, so I'm not sure how to Cc him). My > understanding at the time was that the interfaceid was intended for > exactly the purpose that you say doesn't work (i.e. so that OVS itself > could recognize that interface as it appeared and disappeared at > different locations on the network). I would classify that as a bug, > but that's just my understanding as an OVS outsider. >Further evidence, now that I figured out how to dump the external-ids stuff in OVS: Here's the relevant ports of the domain XML: <domain type='kvm' id='5'> <name>cuckoo</name> <uuid>cb4903aa-eaf4-432d-aa57-e9f74f1c8b79</uuid> .... <interface type='bridge'> <mac address='52:54:00:f5:29:72'/> <source network='malware' bridge='malware0'/> <virtualport type='openvswitch'> <parameters interfaceid='23e982f3-f344-48a2-8bed-b584acb95616'/> </virtualport> <target dev='monitor0'/> <model type='virtio'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface> .... cuckoo vm up: monitor0 _uuid is 969ef76c... # ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 name : "monitor0" _uuid : 969ef76c-c802-4e9f-97e0-9a06ce133252 external_ids : {attached-mac="52:54:00:f5:29:72", iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active, vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"} # cuckoo vm down: (no output, the interface has been deleted by libvirt) # ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 # I found this in my log, showing libvirt deleted the interface: ovs-vsctl[4590]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port monitor0 cuckoo vm up again: uuid is now fb025df0... # ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 name : "monitor0" _uuid : fb025df0-d934-46d6-b9ed-314bc1de5ef3 external_ids : {attached-mac="52:54:00:f5:29:72", iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active, vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"} # Anyway, thanks for reading! Hopefully we'll get to the bottom of this, where I can keep my span port. :) In the mean time I'm going to go digging through the OVS source, specifically the ovs-vsctl command to see if an argument can be used to forcibly set the internal _uuid in OVS. Richard
Possibly Parallel Threads
- libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?
- Re: libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?
- ubuntu virsh snapshot-create-as gives Error -22 while writing VM
- Re: ubuntu virsh snapshot-create-as gives Error -22 while writing VM
- A new era in Radio! Don't miss The Experimental Beat! (fwd)