libvirt users - May 2018 - Re: VF MAC not reverted to all zero MAC/domain xml MAC on VM restart

Hi Laine,

Yes we are setting the names as GE0-0 manually. We have turned trust ON for host
IGB driver.
[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
9216<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:be, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
That’s why we are able to change MAC on VF from Guest.

When we are using bonding on the guest VM, we need the host driver(IGB) to allow
setting the guest mac on VF and to do that we need to enable VF trust on.(using
the command “ip link set GE0-0 vf 0 trust on”)
So when VM reboots, as you mentioned guest VF driver is getting reloaded and we
expect libvirt should revert VF’s MAC to the domain xml MAC and that’s not
happening.

Thanks
Thiru.

From: sendmail <justsendmailnothingelse@gmail.com> On Behalf Of Laine
Stump
Sent: Friday, May 4, 2018 6:48 PM
To: Chanda Mendon (cmendon) <cmendon@cisco.com>
Cc: Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at
Cisco) <tsengalv@cisco.com>; libvirt-users@redhat.com
Subject: Re: VF MAC not reverted to all zero MAC/domain xml MAC on VM restart

On 05/04/2018 06:25 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
TECHNOLOGIES LIMITED at Cisco) wrote:
Hi Laine,

Thanks for taking the time to respond to my question. I think I have not
described my problem clearly.

Let me explain my issue below with the information that you had requested.

My assumption according to the information you gave me is that the admin MAC and
VF MAC are the same in my case.
I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as you
mentioned in your email.

If the VF has no separate netdev visible on the host, then it is not being bound
to the VF net driver for some reason. This should cause no ill effect - it just
means there is no "original" VF mac to restore.



* Given this assumption, when the host is booted, the admin MAC and VF MAC are
both 00:00:00:00:00:00.

Well, if there is no VF netdev on the host, there is no "vf mac" (and
the igb driver wouldn't allow it to be 00:00:00:00:00:00). But again, not
important



[root@nfvis ~]# ip link show GE0-0

"GEO-0"? That's not a normal name for an igb PF. Are you manually
selecting this name?


3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
9216<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
[root@nfvis ~]#

* the VF is assigned to a guest, so the admin MAC is set to the configured value
(52:54:00:29:3c:bf) as shown below
[root@nfvis ~]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
9216<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:bf, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
[root@nfvis ~]#

* Configure bond interface on the guest and added member interface to the bond
   On doing the above step, bond interface mac (52:54:00:29:3c:be) gets assigned
to VF
[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
9216<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:be, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
[root@nfvis libvirt]#

Okay, this part makes no sense, for 2 reasons:

1) once the admin MAC has been set by the host, a flag is set in the PF driver
marking this VF as having an "administratively set" MAC (that's
why I call it the "admin mac"), and after that point it should not be
possible for a guest to modify the MAC address. If your guest is successfully
setting the MAC of the VF, then either you don't have a device that uses the
igb driver, or there is a bug in the driver.

2) setting the MAC address of the VF shouldn't update the admin mac for that
VF - they are separate entities, and only synched up when the VF driver is
reloaded (and in that case it is the *admin mac* that is used as the master copy
to set both of them)




* The next step would be to restart the VM from within the VM console (* we are
not doing a shut down and start from the host *).

Okay, so that reloads the VF driver in the guest, which would set it to the
admin mac (which you have said showed to be ....:be just before the reset.


   At this stage, the admin MAC is not reverted to the domain xml
mac(52:54:00:29:3c:bf)
   Instead it retains the same bond mac (52:54:00:29:3c:be) which was set before
the VM was restarted from the console.

[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
9216<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:be, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on

So the actual problem we face here, is after the VM is restarted and after the
members of bond interface are released, VF mac is not getting reverted back to
domain xml mac(52:54:00:29:3c:bf).
Instead the VF mac and admin mac are set to the bond mac “52:54:00:29:3c:be” and
leading to a duplicate mac issue.
My expectation was that after the VM is restarted from the console, libvirt
should revert the VF’s MAC to the original MAC,
which in our case was the domain xml MAC, which is not happening in my case.

My expectation was that the guest would not be allowed to modify the MAC address
of the VF at all, and certainly that even if a change to the guest MAC was
allowed, that it wouldn't propagate to the PF's list of admin mac
addresses for the VFs. So we've both been disappointed :-)

Your VF and PF drivers are behaving strangely. But if you completely shutdown
the guest, then restart it, you should get the original "...:bf" back.
Even if that works for you, I would not count on the ability of the guest to
modify the VF's MAC address - that is not the defined behavior and thus is
likely to change in the future.




Attached is the log file as requested.

Interface related contents from domain xml:
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:29:3c:bf'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x02'
slot='0x10' function='0x0'/>
</source>
<target dev='vnic1'/>
<model type='virtio'/>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04' function='0x0'/>
</interface>

Thanks
Thiru.
From: sendmail
<justsendmailnothingelse@gmail.com><mailto:justsendmailnothingelse@gmail.com>
On Behalf Of Laine Stump
Sent: Wednesday, May 2, 2018 10:01 AM
To: Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at
Cisco) <tsengalv@cisco.com><mailto:tsengalv@cisco.com>
Cc: Chanda Mendon (cmendon)
<cmendon@cisco.com><mailto:cmendon@cisco.com>
Subject: Re: VF MAC not reverted to all zero MAC on VM restart

On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
TECHNOLOGIES LIMITED at Cisco) wrote:
Hi Laine Stump,

We are running linux based VM on KVM based Hypervisor and facing issue with
respect to VF MAC.
Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and VF driver IGBVF
version 2.3.7.1

Description of problem:
When passing a VF to a guest, libvirt sets its MAC according to the domain xml.
On shutting down the VM or power off the VM, libvirt attempts to restore its
original MAC(all-zero mac). There is no issue with this scenario.
When we restart the VM, there is no attempt to restore/revert its original
MAC(all zero mac).
This problem leads to duplicate MAC issue on the guest (if configured with
portchannel).

Could you please help us in resolving the issue?

Questions like this should be sent to
libvirt-users@redhat.com<mailto:libvirt-users@redhat.com> rather than to
an individual's email address. This makes the likelyhood of getting an
answer much higher, and also creates an archive of the problem and eventual
solution, which may help others in the future. I'm Cc'ing this response
to that list so that any further communication will happen there.

Before reading any of the following, it's useful to know this - for SRIOV
network devices that *properly* support SRIOV (and I consider those using the
igb driver to be in this category) each VF has two MAC addresses that need to be
discussed:

1) the VF netdev MAC address (which I will just call the "mac address"
- this is the MAC that is known to the VF net driver, and displayed in the
output of "ip link show dev $vfnetdev".

2) the MAC address that is stored in the PF driver for each VF, displayed in the
"VF" lines immediately following the "ip link" info of the
*PF*, and which will be used to initialize the VF's own MAC address when its
driver is re-initialized (I will call this the "admin mac")

Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the Cisco enic
driver) and some initialize it to a random number (e.g. igb and all the other
Intel VF net drivers). Likewise, some PF drivers initialize the admin macs to
00:00:00:00:00:00 and some to random numbers (as a matter of fact, this behavior
changes between different versions of the same driver in at least one case!).

Beyond this, some PF and VF drivers allow setting the mac/admin mac to
00:00:00:00:00:00, and some prohibit one or the other. (in many cases, a driver
that itself initializes the mac/admin mac to 00:00:00:00:00:00 also prohibits
setting it back to that same value once it has been changed!)

When libvirt sets up a VF to be used by a guest (either for vfio device
assignment, or for macvtap passthrough), it saves both the mac and the admin mac
with the intent of restoring them later.

Once these values have been saved, libvirt will set the mac (in the case of
macvtap passthrough) or the admin mac (in the case of vfio device assignment),
and send the device on to qemu to be used by the guest.

After the guest is finished running (or the device is detached), libvirt
attempts to restore the settings it had saved at the beginning. Trouble comes up
in 2 ways though - 1) once the admin mac has been set for a VF (via the PF), it
is no longer possible to directly set the mac (in the VF) (this is done for
security reasons, to prevent the guest from changing its mac address), and 2) as
said above, many drivers don't allow setting 00:00:00:00:00:00, but in many
cases that was the original setting.

As of libvirt 3.2.0, libvirt uses the following strategies to work around these
problems:

a) if setting a non-0 mac (to the VF) fails, then libvirt will try setting the
*admin MAC* to that value, then detaching/re-attaching the VF net driver. This
will cause the PF to reinitialize the mac in the VF driver. (NB: if we do this,
then we don't bother trying to later re-set the admin MAC to its own
original value; it's really not important).

b) if the failure was in setting the MAC to 00:00:00:00:00:00, then libvirt will
set the mac/admin mac to 02:00:00:00:00:00 (which should be a legal value for
any driver, but also should not conflict with and "real" mac on the
network).

Okay, enough background. On to your problem.


To make sure I understand your situation correctly:

* When the host is booted, the admin MAC is 00:00:00:00:00:00, and VF mac is
random (call it rr:rr:rr:rr:rr:rr).

* the VF is assigned to a guest (with <interface type='hostdev'> I
guess?), so the admin MAC is set to the configured value (let's call it
gg:gg:gg:gg:gg:gg)

* the guest is shutdown, which causes *both* mac and admin mac to be set to
rr:rr:rr:rr:rr:rr

* the next time the guest is start *its mac is not changed*?

There must be something I'm not getting. Can you maybe perform this test and
run "ip link show $PF; ip link show $VF" (substituting the netdev
names of the PF and VF for $PF and $VF) after each step to illustrate the
problem? Also, it would be helpful to add this to /etc/libvirt/libvirtd.conf:

  log_filters="1:util.netdev"
 
log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log<file://var/log/libvirt/libvirtd-netdev.log>"

then restart libvirtd (systemctl restart libvirtd.service), and post the
contents of libvirtd-netdev.log somewhere and reference it in your next reply.
(don't forget to remove those lines and restart libvirtd again after your
testing is done!)

Finally, you should include the contents of the <interface ...> section of
your config in your next response.

(BTW, since you're using libvirt-3.2.0, I assume that you are using either
RHEL 7.4 or CentOS 7.4. If you are using RHEL, you should open a customer case
with Red Hat so that your problem is appropriately prioritized).

(I should have said this initially - please don't top-post your replies
on this list. Put the answers/responses inline in context with the
questions. It makes it easier to follow the conversation).

On 05/06/2018 05:50 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
TECHNOLOGIES LIMITED at Cisco) wrote:
> Hi Laine,
> 
>  
> 
> Yes we are setting the names as GE0-0 manually.
Okay, that's explained.
> We have turned trust ON for host IGB driver.
Interesting. I hadn't known that the igb driver supported trust=on - the
last time I tried it, it didn't.
> [root@nfvis libvirt]# ip link show GE0-0
> 
> 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
<tel:9216>
> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000 <tel:1000>
> 
> link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
> vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, *trust
on*
> 
> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, *trust on*
> 
> That’s why we are able to change MAC on VF from Guest.
Okay, so now that part makes sense.
> 
> When we are using bonding on the guest VM, we need the host driver(IGB)
> to allow setting the guest mac on VF and to do that we need to enable VF
> trust on.(using the command “ip link set GE0-0 vf 0 trust on”)
> 
> So when VM reboots, as you mentioned guest VF driver is getting reloaded
> and we expect libvirt should revert VF’s MAC to the domain xml MAC and
> that’s not happening.

libvirt has no control over what happens when the guest OS reboots and
reloads the vf net driver. As a matter of fact, libvirt has no control
over what happens after the device is given to qemu for assignment until
the time that the qemu process is *terminated*, and it cleans up all the
devices that had been used by the guest.

When a guest OS reboots, this is just the guest OS reloading, so as far
as libvirt is concerned, the guest is just continuing to use the VF, so
no reason to reload any MAC address.

It appears that the problem is that when you have "trust=on" set for
the
VF, then the PF driver in the host sees the new mac set for the VF and
sets the admin mac to that value. So the next time the VF driver is
reloaded, that new admin mac (aka the vf's mac set in the guest) is what
the vf mac gets set to.

In the past (when "trust=on" didn't exist) this wasn't a
problem. Maybe
we need to explicitly support setting trust=on, as well as
re-initializing the MAC address when we get the RESET event from qemu;
it seems like this might be something that wouldn't always be desired,
so it may need to be configurable.

If you're using RHEL, can you open a support case asking for this
behavior - this will cause a properly prioritized bugzilla record to be
created and tracked. If you're not using RHEL, then you can either open
an upstream bugzilla (there is information on how to do this somewhere
on libvirt.org), or (even better) you could submit a patch to do it.

In the meantime, I'm fairly certain you can get the behavior you desire
by powering down the guest, then restarting it. This will cause libvirt
to reinitialize the admin mac with the original value set in libvirt's
config.
> 
>  
> 
> Thanks
> 
> Thiru.
> 
>  
> 
> *From:*sendmail <justsendmailnothingelse@gmail.com> *On Behalf Of
*Laine
> Stump
> *Sent:* Friday, May 4, 2018 6:48 PM
> *To:* Chanda Mendon (cmendon) <cmendon@cisco.com>
> *Cc:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES
> LIMITED at Cisco) <tsengalv@cisco.com>; libvirt-users@redhat.com
> *Subject:* Re: VF MAC not reverted to all zero MAC/domain xml MAC on VM
> restart
> 
>  
> 
> On 05/04/2018 06:25 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
> TECHNOLOGIES LIMITED at Cisco) wrote:
> 
>     Hi Laine,
> 
>      
> 
>     Thanks for taking the time to respond to my question. I think I have
>     not described my problem clearly.
> 
>      
> 
>     Let me explain my issue below with the information that you had
>     requested.
> 
>      
> 
>     My assumption according to the information you gave me is that the
>     admin MAC and VF MAC are the same in my case.
> 
>     I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as
>     you mentioned in your email.
> 
> 
> If the VF has no separate netdev visible on the host, then it is not
> being bound to the VF net driver for some reason. This should cause no
> ill effect - it just means there is no "original" VF mac to
restore.
> 
> 
>      
> 
>     * Given this assumption, when the host is booted, the admin MAC and
>     VF MAC are both*00:00:00:00:00:00. *
> 
> 
> Well, if there is no VF netdev on the host, there is no "vf mac"
(and
> the igb driver wouldn't allow it to be 00:00:00:00:00:00). But again,
> not important
> 
> 
> 
>     [root@nfvis ~]# ip link show GE0-0
> 
> 
> "GEO-0"? That's not a normal name for an igb PF. Are you
manually
> selecting this name?
> 
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
>     trust off
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
>     trust off
> 
>     [root@nfvis ~]#
> 
>      
> 
>     * the VF is assigned to a guest, so the admin MAC is set to the
>     configured value *(52:54:00:29:3c:bf) *as shown below
> 
>     [root@nfvis ~]# ip link show GE0-0
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC *52:54:00:29:3c:bf*, spoof checking on, link-state auto,
>     trust on
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust
on
> 
>     [root@nfvis ~]#
> 
>      
> 
>     * Configure bond interface on the guest and added member interface
>     to the bond
> 
>        On doing the above step, bond interface mac*(52:54:00:29:3c:be)*
>     gets assigned to VF
> 
>     [root@nfvis libvirt]# ip link show GE0-0
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto,
>     trust on
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust
on
> 
>     [root@nfvis libvirt]#
> 
> 
> Okay, this part makes no sense, for 2 reasons:
> 
> 1) once the admin MAC has been set by the host, a flag is set in the PF
> driver marking this VF as having an "administratively set" MAC
(that's
> why I call it the "admin mac"), and after that point it should
not be
> possible for a guest to modify the MAC address. If your guest is
> successfully setting the MAC of the VF, then either you don't have a
> device that uses the igb driver, or there is a bug in the driver.
> 
> 2) setting the MAC address of the VF shouldn't update the admin mac for
> that VF - they are separate entities, and only synched up when the VF
> driver is reloaded (and in that case it is the *admin mac* that is used
> as the master copy to set both of them)
> 
> 
> 
>      
> 
>     * The next step would be to restart the VM from within the VM
>     console (* we are not doing a shut down and start from the host *).
> 
> 
> Okay, so that reloads the VF driver in the guest, which would set it to
> the admin mac (which you have said showed to be ....:be just before the
> reset.
> 
> 
>        At this stage, the admin MAC is not reverted to the domain xml
>     mac*(52:54:00:29:3c:bf)*
> 
>        Instead it retains the same bond mac *(52:54:00:29:3c:be)* which
>     was set before the VM was restarted from the console.
> 
>      
> 
>     [root@nfvis libvirt]# ip link show GE0-0
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto,
>     trust on
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust
on
> 
>      
> 
>     So the actual problem we face here, is after the VM is restarted and
>     after the members of bond interface are released, VF mac is not
>     getting reverted back to domain xml mac(*52:54:00:29:3c:bf*).
> 
>     Instead the VF mac and admin mac are set to the bond mac
>     “*52:54:00:29:3c:be*” and leading to a duplicate mac issue.
> 
>     My expectation was that after the VM is restarted from the console,
>     libvirt should revert the VF’s MAC to the original MAC,
> 
>     which in our case was the domain xml MAC, which is not happening in
>     my case.
> 
> 
> My expectation was that the guest would not be allowed to modify the MAC
> address of the VF at all, and certainly that even if a change to the
> guest MAC was allowed, that it wouldn't propagate to the PF's list
of
> admin mac addresses for the VFs. So we've both been disappointed :-)
> 
> Your VF and PF drivers are behaving strangely. But if you completely
> shutdown the guest, then restart it, you should get the original
> "...:bf" back. Even if that works for you, I would not count on
the
> ability of the guest to modify the VF's MAC address - that is not the
> defined behavior and thus is likely to change in the future.
> 
> 
>      
> 
>      
> 
>     Attached is the log file as requested.
> 
>      
> 
>     *_Interface related contents from domain xml: _*
> 
>     <interface type='hostdev' managed='yes'>
> 
>     <mac address='52:54:00:29:3c:bf'/>
> 
>     <driver name='vfio'/>
> 
>     <source>
> 
>     <address type='pci' domain='0x0000'
bus='0x02' slot='0x10'
>     function='0x0'/>
> 
>     </source>
> 
>     <target dev='vnic1'/>
> 
>     <model type='virtio'/>
> 
>     <alias name='hostdev0'/>
> 
>     <address type='pci' domain='0x0000'
bus='0x00' slot='0x04'
>     function='0x0'/>
> 
>     </interface>
> 
>      
> 
>     Thanks
> 
>     Thiru.
> 
>     *From:*sendmail <justsendmailnothingelse@gmail.com>
>     <mailto:justsendmailnothingelse@gmail.com> *On Behalf Of *Laine
Stump
>     *Sent:* Wednesday, May 2, 2018 10:01 AM
>     *To:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES
>     LIMITED at Cisco) <tsengalv@cisco.com>
<mailto:tsengalv@cisco.com>
>     *Cc:* Chanda Mendon (cmendon) <cmendon@cisco.com>
>     <mailto:cmendon@cisco.com>
>     *Subject:* Re: VF MAC not reverted to all zero MAC on VM restart
> 
>      
> 
>     On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv -
>     HCL TECHNOLOGIES LIMITED at Cisco) wrote:
> 
>         Hi Laine Stump,
> 
>          
> 
>         We are running linux based VM on KVM based Hypervisor and facing
>         issue with respect to VF MAC.
> 
>         Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and
>         VF driver IGBVF version 2.3.7.1
> 
>          
> 
>         *_Description of problem:_*
> 
>         When passing a VF to a guest, libvirt sets its MAC according to
>         the domain xml.
> 
>         On shutting down the VM or power off the VM, libvirt attempts to
>         restore its original MAC(all-zero mac). There is no issue with
>         this scenario.
> 
>         *When we restart the VM, there is no attempt to restore/revert
>         its original MAC(all zero mac). *
> 
>         This problem leads to duplicate MAC issue on the guest (if
>         configured with portchannel).
> 
>          
> 
>         Could you please help us in resolving the issue?
> 
> 
>     Questions like this should be sent to libvirt-users@redhat.com
>     <mailto:libvirt-users@redhat.com>rather than to an
individual's
>     email address. This makes the likelyhood of getting an answer much
>     higher, and also creates an archive of the problem and eventual
>     solution, which may help others in the future. I'm Cc'ing this
>     response to that list so that any further communication will happen
>     there.
> 
>     Before reading any of the following, it's useful to know this - for
>     SRIOV network devices that *properly* support SRIOV (and I consider
>     those using the igb driver to be in this category) each VF has two
>     MAC addresses that need to be discussed:
> 
>     1) the VF netdev MAC address (which I will just call the "mac
>     address" - this is the MAC that is known to the VF net driver, and
>     displayed in the output of "ip link show dev $vfnetdev".
> 
>     2) the MAC address that is stored in the PF driver for each VF,
>     displayed in the "VF" lines immediately following the
"ip link" info
>     of the *PF*, and which will be used to initialize the VF's own MAC
>     address when its driver is re-initialized (I will call this the
>     "admin mac")
> 
>     Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the
>     Cisco enic driver) and some initialize it to a random number (e.g.
>     igb and all the other Intel VF net drivers). Likewise, some PF
>     drivers initialize the admin macs to 00:00:00:00:00:00 and some to
>     random numbers (as a matter of fact, this behavior changes between
>     different versions of the same driver in at least one case!).
> 
>     Beyond this, some PF and VF drivers allow setting the mac/admin mac
>     to 00:00:00:00:00:00, and some prohibit one or the other. (in many
>     cases, a driver that itself initializes the mac/admin mac to
>     00:00:00:00:00:00 also prohibits setting it back to that same value
>     once it has been changed!)
> 
>     When libvirt sets up a VF to be used by a guest (either for vfio
>     device assignment, or for macvtap passthrough), it saves both the
>     mac and the admin mac with the intent of restoring them later.
> 
>     Once these values have been saved, libvirt will set the mac (in the
>     case of macvtap passthrough) or the admin mac (in the case of vfio
>     device assignment), and send the device on to qemu to be used by the
>     guest.
> 
>     After the guest is finished running (or the device is detached),
>     libvirt attempts to restore the settings it had saved at the
>     beginning. Trouble comes up in 2 ways though - 1) once the admin mac
>     has been set for a VF (via the PF), it is no longer possible to
>     directly set the mac (in the VF) (this is done for security reasons,
>     to prevent the guest from changing its mac address), and 2) as said
>     above, many drivers don't allow setting 00:00:00:00:00:00, but in
>     many cases that was the original setting.
> 
>     As of libvirt 3.2.0, libvirt uses the following strategies to work
>     around these problems:
> 
>     a) if setting a non-0 mac (to the VF) fails, then libvirt will try
>     setting the *admin MAC* to that value, then detaching/re-attaching
>     the VF net driver. This will cause the PF to reinitialize the mac in
>     the VF driver. (NB: if we do this, then we don't bother trying to
>     later re-set the admin MAC to its own original value; it's really
>     not important).
> 
>     b) if the failure was in setting the MAC to 00:00:00:00:00:00, then
>     libvirt will set the mac/admin mac to 02:00:00:00:00:00 (which
>     should be a legal value for any driver, but also should not conflict
>     with and "real" mac on the network).
> 
>     Okay, enough background. On to your problem.
> 
> 
>     To make sure I understand your situation correctly:
> 
>     * When the host is booted, the admin MAC is 00:00:00:00:00:00, and
>     VF mac is random (call it rr:rr:rr:rr:rr:rr).
> 
>     * the VF is assigned to a guest (with <interface
type='hostdev'> I
>     guess?), so the admin MAC is set to the configured value (let's
call
>     it gg:gg:gg:gg:gg:gg)
> 
>     * the guest is shutdown, which causes *both* mac and admin mac to be
>     set to rr:rr:rr:rr:rr:rr
> 
>     * the next time the guest is start *its mac is not changed*?
> 
>     There must be something I'm not getting. Can you maybe perform this
>     test and run "ip link show $PF; ip link show $VF"
(substituting the
>     netdev names of the PF and VF for $PF and $VF) after each step to
>     illustrate the problem? Also, it would be helpful to add this to
>     /etc/libvirt/libvirtd.conf:
> 
>       log_filters="1:util.netdev"
>       log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log
>     <file://var/log/libvirt/libvirtd-netdev.log>"
> 
>     then restart libvirtd (systemctl restart libvirtd.service), and post
>     the contents of libvirtd-netdev.log somewhere and reference it in
>     your next reply. (don't forget to remove those lines and restart
>     libvirtd again after your testing is done!)
> 
>     Finally, you should include the contents of the <interface ...>
>     section of your config in your next response.
> 
>     (BTW, since you're using libvirt-3.2.0, I assume that you are using
>     either RHEL 7.4 or CentOS 7.4. If you are using RHEL, you should
>     open a customer case with Red Hat so that your problem is
>     appropriately prioritized).
> 
>  
> 
> 
> 
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>

Hi Laine,

Thanks for your detailed response.
I have replied Inline.

Thanks
Thiru.


-----Original Message-----
From: Laine Stump <laine@redhat.com> 
Sent: Sunday, May 6, 2018 3:46 PM
To: libvirt-users@redhat.com
Cc: Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at
Cisco) <tsengalv@cisco.com>; Chanda Mendon (cmendon)
<cmendon@cisco.com>
Subject: Re: [libvirt-users] VF MAC not reverted to all zero MAC/domain xml MAC
on VM restart

(I should have said this initially - please don't top-post your replies on
this list. Put the answers/responses inline in context with the questions. It
makes it easier to follow the conversation).

On 05/06/2018 05:50 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
TECHNOLOGIES LIMITED at Cisco) wrote:
> Hi Laine,
> 
>  
> 
> Yes we are setting the names as GE0-0 manually.
Okay, that's explained.
> We have turned trust ON for host IGB driver.
Interesting. I hadn't known that the igb driver supported trust=on - the
last time I tried it, it didn't.
> [root@nfvis libvirt]# ip link show GE0-0
> 
> 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216 
> <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
> <tel:1000>
> 
> link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
> vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, 
> *trust on*
> 
> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, *trust 
> on*
> 
> That’s why we are able to change MAC on VF from Guest.
Okay, so now that part makes sense.
> 
> When we are using bonding on the guest VM, we need the host 
> driver(IGB) to allow setting the guest mac on VF and to do that we 
> need to enable VF trust on.(using the command “ip link set GE0-0 vf 0 
> trust on”)
> 
> So when VM reboots, as you mentioned guest VF driver is getting 
> reloaded and we expect libvirt should revert VF’s MAC to the domain 
> xml MAC and that’s not happening.

libvirt has no control over what happens when the guest OS reboots and reloads
the vf net driver. As a matter of fact, libvirt has no control over what happens
after the device is given to qemu for assignment until the time that the qemu
process is *terminated*, and it cleans up all the devices that had been used by
the guest.

When a guest OS reboots, this is just the guest OS reloading, so as far as
libvirt is concerned, the guest is just continuing to use the VF, so no reason
to reload any MAC address.

It appears that the problem is that when you have "trust=on" set for
the VF, then the PF driver in the host sees the new mac set for the VF and sets
the admin mac to that value. So the next time the VF driver is reloaded, that
new admin mac (aka the vf's mac set in the guest) is what the vf mac gets
set to.

In the past (when "trust=on" didn't exist) this wasn't a
problem. Maybe we need to explicitly support setting trust=on, as well as
re-initializing the MAC address when we get the RESET event from qemu; it seems
like this might be something that wouldn't always be desired, so it may need
to be configurable.

If you're using RHEL, can you open a support case asking for this behavior -
this will cause a properly prioritized bugzilla record to be created and
tracked. If you're not using RHEL, then you can either open an upstream
bugzilla (there is information on how to do this somewhere on libvirt.org), or
(even better) you could submit a patch to do it.

In the meantime, I'm fairly certain you can get the behavior you desire by
powering down the guest, then restarting it. This will cause libvirt to
reinitialize the admin mac with the original value set in libvirt's config.

<Thiru>  I understand that you agree to do necessary changes in libvirt
which will reinitialize the MAC address on VM reboot.
	 Yes, we are using RHEL. I would open a case asking for this behavior change.
	           
> 
>  
> 
> Thanks
> 
> Thiru.
> 
>  
> 
> *From:*sendmail <justsendmailnothingelse@gmail.com> *On Behalf Of 
> *Laine Stump
> *Sent:* Friday, May 4, 2018 6:48 PM
> *To:* Chanda Mendon (cmendon) <cmendon@cisco.com>
> *Cc:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES 
> LIMITED at Cisco) <tsengalv@cisco.com>; libvirt-users@redhat.com
> *Subject:* Re: VF MAC not reverted to all zero MAC/domain xml MAC on 
> VM restart
> 
>  
> 
> On 05/04/2018 06:25 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - 
> HCL TECHNOLOGIES LIMITED at Cisco) wrote:
> 
>     Hi Laine,
> 
>      
> 
>     Thanks for taking the time to respond to my question. I think I have
>     not described my problem clearly.
> 
>      
> 
>     Let me explain my issue below with the information that you had
>     requested.
> 
>      
> 
>     My assumption according to the information you gave me is that the
>     admin MAC and VF MAC are the same in my case.
> 
>     I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as
>     you mentioned in your email.
> 
> 
> If the VF has no separate netdev visible on the host, then it is not 
> being bound to the VF net driver for some reason. This should cause no 
> ill effect - it just means there is no "original" VF mac to
restore.
> 
> 
>      
> 
>     * Given this assumption, when the host is booted, the admin MAC and
>     VF MAC are both*00:00:00:00:00:00. *
> 
> 
> Well, if there is no VF netdev on the host, there is no "vf mac"
(and
> the igb driver wouldn't allow it to be 00:00:00:00:00:00). But again, 
> not important
> 
> 
> 
>     [root@nfvis ~]# ip link show GE0-0
> 
> 
> "GEO-0"? That's not a normal name for an igb PF. Are you
manually
> selecting this name?
> 
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
>     trust off
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
>     trust off
> 
>     [root@nfvis ~]#
> 
>      
> 
>     * the VF is assigned to a guest, so the admin MAC is set to the
>     configured value *(52:54:00:29:3c:bf) *as shown below
> 
>     [root@nfvis ~]# ip link show GE0-0
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC *52:54:00:29:3c:bf*, spoof checking on, link-state auto,
>     trust on
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, 
> trust on
> 
>     [root@nfvis ~]#
> 
>      
> 
>     * Configure bond interface on the guest and added member interface
>     to the bond
> 
>        On doing the above step, bond interface mac*(52:54:00:29:3c:be)*
>     gets assigned to VF
> 
>     [root@nfvis libvirt]# ip link show GE0-0
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto,
>     trust on
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, 
> trust on
> 
>     [root@nfvis libvirt]#
> 
> 
> Okay, this part makes no sense, for 2 reasons:
> 
> 1) once the admin MAC has been set by the host, a flag is set in the 
> PF driver marking this VF as having an "administratively set" MAC
> (that's why I call it the "admin mac"), and after that point
it should
> not be possible for a guest to modify the MAC address. If your guest 
> is successfully setting the MAC of the VF, then either you don't have 
> a device that uses the igb driver, or there is a bug in the driver.
> 
> 2) setting the MAC address of the VF shouldn't update the admin mac 
> for that VF - they are separate entities, and only synched up when the 
> VF driver is reloaded (and in that case it is the *admin mac* that is 
> used as the master copy to set both of them)
> 
> 
> 
>      
> 
>     * The next step would be to restart the VM from within the VM
>     console (* we are not doing a shut down and start from the host *).
> 
> 
> Okay, so that reloads the VF driver in the guest, which would set it 
> to the admin mac (which you have said showed to be ....:be just before 
> the reset.
> 
> 
>        At this stage, the admin MAC is not reverted to the domain xml
>     mac*(52:54:00:29:3c:bf)*
> 
>        Instead it retains the same bond mac *(52:54:00:29:3c:be)* which
>     was set before the VM was restarted from the console.
> 
>      
> 
>     [root@nfvis libvirt]# ip link show GE0-0
> 
>     3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
>     <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen
>     1000 <tel:1000>
> 
>     link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
> 
>     vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto,
>     trust on
> 
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, 
> trust on
> 
>      
> 
>     So the actual problem we face here, is after the VM is restarted and
>     after the members of bond interface are released, VF mac is not
>     getting reverted back to domain xml mac(*52:54:00:29:3c:bf*).
> 
>     Instead the VF mac and admin mac are set to the bond mac
>     “*52:54:00:29:3c:be*” and leading to a duplicate mac issue.
> 
>     My expectation was that after the VM is restarted from the console,
>     libvirt should revert the VF’s MAC to the original MAC,
> 
>     which in our case was the domain xml MAC, which is not happening in
>     my case.
> 
> 
> My expectation was that the guest would not be allowed to modify the 
> MAC address of the VF at all, and certainly that even if a change to 
> the guest MAC was allowed, that it wouldn't propagate to the PF's
list
> of admin mac addresses for the VFs. So we've both been disappointed 
> :-)
> 
> Your VF and PF drivers are behaving strangely. But if you completely 
> shutdown the guest, then restart it, you should get the original 
> "...:bf" back. Even if that works for you, I would not count on
the
> ability of the guest to modify the VF's MAC address - that is not the 
> defined behavior and thus is likely to change in the future.
> 
> 
>      
> 
>      
> 
>     Attached is the log file as requested.
> 
>      
> 
>     *_Interface related contents from domain xml: _*
> 
>     <interface type='hostdev' managed='yes'>
> 
>     <mac address='52:54:00:29:3c:bf'/>
> 
>     <driver name='vfio'/>
> 
>     <source>
> 
>     <address type='pci' domain='0x0000'
bus='0x02' slot='0x10'
>     function='0x0'/>
> 
>     </source>
> 
>     <target dev='vnic1'/>
> 
>     <model type='virtio'/>
> 
>     <alias name='hostdev0'/>
> 
>     <address type='pci' domain='0x0000'
bus='0x00' slot='0x04'
>     function='0x0'/>
> 
>     </interface>
> 
>      
> 
>     Thanks
> 
>     Thiru.
> 
>     *From:*sendmail <justsendmailnothingelse@gmail.com>
>     <mailto:justsendmailnothingelse@gmail.com> *On Behalf Of *Laine
Stump
>     *Sent:* Wednesday, May 2, 2018 10:01 AM
>     *To:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES
>     LIMITED at Cisco) <tsengalv@cisco.com>
<mailto:tsengalv@cisco.com>
>     *Cc:* Chanda Mendon (cmendon) <cmendon@cisco.com>
>     <mailto:cmendon@cisco.com>
>     *Subject:* Re: VF MAC not reverted to all zero MAC on VM restart
> 
>      
> 
>     On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv -
>     HCL TECHNOLOGIES LIMITED at Cisco) wrote:
> 
>         Hi Laine Stump,
> 
>          
> 
>         We are running linux based VM on KVM based Hypervisor and facing
>         issue with respect to VF MAC.
> 
>         Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and
>         VF driver IGBVF version 2.3.7.1
> 
>          
> 
>         *_Description of problem:_*
> 
>         When passing a VF to a guest, libvirt sets its MAC according to
>         the domain xml.
> 
>         On shutting down the VM or power off the VM, libvirt attempts to
>         restore its original MAC(all-zero mac). There is no issue with
>         this scenario.
> 
>         *When we restart the VM, there is no attempt to restore/revert
>         its original MAC(all zero mac). *
> 
>         This problem leads to duplicate MAC issue on the guest (if
>         configured with portchannel).
> 
>          
> 
>         Could you please help us in resolving the issue?
> 
> 
>     Questions like this should be sent to libvirt-users@redhat.com
>     <mailto:libvirt-users@redhat.com>rather than to an
individual's
>     email address. This makes the likelyhood of getting an answer much
>     higher, and also creates an archive of the problem and eventual
>     solution, which may help others in the future. I'm Cc'ing this
>     response to that list so that any further communication will happen
>     there.
> 
>     Before reading any of the following, it's useful to know this - for
>     SRIOV network devices that *properly* support SRIOV (and I consider
>     those using the igb driver to be in this category) each VF has two
>     MAC addresses that need to be discussed:
> 
>     1) the VF netdev MAC address (which I will just call the "mac
>     address" - this is the MAC that is known to the VF net driver, and
>     displayed in the output of "ip link show dev $vfnetdev".
> 
>     2) the MAC address that is stored in the PF driver for each VF,
>     displayed in the "VF" lines immediately following the
"ip link" info
>     of the *PF*, and which will be used to initialize the VF's own MAC
>     address when its driver is re-initialized (I will call this the
>     "admin mac")
> 
>     Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the
>     Cisco enic driver) and some initialize it to a random number (e.g.
>     igb and all the other Intel VF net drivers). Likewise, some PF
>     drivers initialize the admin macs to 00:00:00:00:00:00 and some to
>     random numbers (as a matter of fact, this behavior changes between
>     different versions of the same driver in at least one case!).
> 
>     Beyond this, some PF and VF drivers allow setting the mac/admin mac
>     to 00:00:00:00:00:00, and some prohibit one or the other. (in many
>     cases, a driver that itself initializes the mac/admin mac to
>     00:00:00:00:00:00 also prohibits setting it back to that same value
>     once it has been changed!)
> 
>     When libvirt sets up a VF to be used by a guest (either for vfio
>     device assignment, or for macvtap passthrough), it saves both the
>     mac and the admin mac with the intent of restoring them later.
> 
>     Once these values have been saved, libvirt will set the mac (in the
>     case of macvtap passthrough) or the admin mac (in the case of vfio
>     device assignment), and send the device on to qemu to be used by the
>     guest.
> 
>     After the guest is finished running (or the device is detached),
>     libvirt attempts to restore the settings it had saved at the
>     beginning. Trouble comes up in 2 ways though - 1) once the admin mac
>     has been set for a VF (via the PF), it is no longer possible to
>     directly set the mac (in the VF) (this is done for security reasons,
>     to prevent the guest from changing its mac address), and 2) as said
>     above, many drivers don't allow setting 00:00:00:00:00:00, but in
>     many cases that was the original setting.
> 
>     As of libvirt 3.2.0, libvirt uses the following strategies to work
>     around these problems:
> 
>     a) if setting a non-0 mac (to the VF) fails, then libvirt will try
>     setting the *admin MAC* to that value, then detaching/re-attaching
>     the VF net driver. This will cause the PF to reinitialize the mac in
>     the VF driver. (NB: if we do this, then we don't bother trying to
>     later re-set the admin MAC to its own original value; it's really
>     not important).
> 
>     b) if the failure was in setting the MAC to 00:00:00:00:00:00, then
>     libvirt will set the mac/admin mac to 02:00:00:00:00:00 (which
>     should be a legal value for any driver, but also should not conflict
>     with and "real" mac on the network).
> 
>     Okay, enough background. On to your problem.
> 
> 
>     To make sure I understand your situation correctly:
> 
>     * When the host is booted, the admin MAC is 00:00:00:00:00:00, and
>     VF mac is random (call it rr:rr:rr:rr:rr:rr).
> 
>     * the VF is assigned to a guest (with <interface
type='hostdev'> I
>     guess?), so the admin MAC is set to the configured value (let's
call
>     it gg:gg:gg:gg:gg:gg)
> 
>     * the guest is shutdown, which causes *both* mac and admin mac to be
>     set to rr:rr:rr:rr:rr:rr
> 
>     * the next time the guest is start *its mac is not changed*?
> 
>     There must be something I'm not getting. Can you maybe perform this
>     test and run "ip link show $PF; ip link show $VF"
(substituting the
>     netdev names of the PF and VF for $PF and $VF) after each step to
>     illustrate the problem? Also, it would be helpful to add this to
>     /etc/libvirt/libvirtd.conf:
> 
>       log_filters="1:util.netdev"
>       log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log
>     <file://var/log/libvirt/libvirtd-netdev.log>"
> 
>     then restart libvirtd (systemctl restart libvirtd.service), and post
>     the contents of libvirtd-netdev.log somewhere and reference it in
>     your next reply. (don't forget to remove those lines and restart
>     libvirtd again after your testing is done!)
> 
>     Finally, you should include the contents of the <interface ...>
>     section of your config in your next response.
> 
>     (BTW, since you're using libvirt-3.2.0, I assume that you are using
>     either RHEL 7.4 or CentOS 7.4. If you are using RHEL, you should
>     open a customer case with Red Hat so that your problem is
>     appropriately prioritized).
> 
>  
> 
> 
> 
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>