thr3ads.net - Linux Virtualization - vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot [Feb 2020]

If this information is useful, please help other people find it:
Share via:

Michael S. Tsirkin

2020-Feb-07 07:58 UTC

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger
wrote:> Also adding Cornelia.
> 
> 
> On 06.02.20 23:17, Michael S. Tsirkin wrote:
> > On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.02.20 15:22, eperezma at redhat.com wrote:
> >>> Hi Christian.
> >>>
> >>> Could you try this patch on top of ("38ced0208491 vhost:
use batched version by default")?
> >>>
> >>> It will not solve your first random crash but it should help
with the lost of network connectivity.
> >>>
> >>> Please let me know how does it goes.
> >>
> >>
> >> 38ced0208491 + this seem to be ok.
> >>
> >> Not sure if you can make out anything of this (and the previous
git bisect log)
> > 
> > Yes it does - that this is just bad split-up of patches, and
there's
> > still a real bug that caused worse crashes :)
> > 
> > So I just pushed batch-v4.
> > I expect that will fail, and bisect to give us
> >     vhost: batching fetches
> > Can you try that please?
> > 
> 
> yes.
> 
> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> Author: Michael S. Tsirkin <mst at redhat.com>
> Date:   Mon Oct 7 06:11:18 2019 -0400
> 
>     vhost: batching fetches
>     
>     With this patch applied, new and old code perform identically.
>     
>     Lots of extra optimizations are now possible, e.g.
>     we can fetch multiple heads with copy_from/to_user now.
>     We can get rid of maintaining the log array.  Etc etc.
>     
>     Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
> 
>  drivers/vhost/test.c  |  2 +-
>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>  drivers/vhost/vhost.h |  4 +++-
>  3 files changed, 38 insertions(+), 7 deletions(-)
> 

And the symptom is still the same - random crashes
after a bit of traffic, right?
> > 
>

Christian Borntraeger

2020-Feb-07 08:13 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 07.02.20 08:58, Michael S. Tsirkin wrote:> On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
>> Also adding Cornelia.
>>
>>
>> On 06.02.20 23:17, Michael S. Tsirkin wrote:
>>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger
wrote:
>>>>
>>>>
>>>> On 06.02.20 15:22, eperezma at redhat.com wrote:
>>>>> Hi Christian.
>>>>>
>>>>> Could you try this patch on top of ("38ced0208491
vhost: use batched version by default")?
>>>>>
>>>>> It will not solve your first random crash but it should
help with the lost of network connectivity.
>>>>>
>>>>> Please let me know how does it goes.
>>>>
>>>>
>>>> 38ced0208491 + this seem to be ok.
>>>>
>>>> Not sure if you can make out anything of this (and the previous
git bisect log)
>>>
>>> Yes it does - that this is just bad split-up of patches, and
there's
>>> still a real bug that caused worse crashes :)
>>>
>>> So I just pushed batch-v4.
>>> I expect that will fail, and bisect to give us
>>>     vhost: batching fetches
>>> Can you try that please?
>>>
>>
>> yes.
>>
>> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
>> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
>> Author: Michael S. Tsirkin <mst at redhat.com>
>> Date:   Mon Oct 7 06:11:18 2019 -0400
>>
>>     vhost: batching fetches
>>     
>>     With this patch applied, new and old code perform identically.
>>     
>>     Lots of extra optimizations are now possible, e.g.
>>     we can fetch multiple heads with copy_from/to_user now.
>>     We can get rid of maintaining the log array.  Etc etc.
>>     
>>     Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
>>
>>  drivers/vhost/test.c  |  2 +-
>>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>>  drivers/vhost/vhost.h |  4 +++-
>>  3 files changed, 38 insertions(+), 7 deletions(-)
>>
> 
> 
> And the symptom is still the same - random crashes
> after a bit of traffic, right?
random guest crashes after a reboot of the guests. As if vhost would still
write into now stale buffers.

Cornelia Huck

2020-Feb-07 08:53 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On Fri, 7 Feb 2020 09:13:14 +0100
Christian Borntraeger <borntraeger at de.ibm.com> wrote:
> On 07.02.20 08:58, Michael S. Tsirkin wrote:
> > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
> >> Also adding Cornelia.
> >>
> >>
> >> On 06.02.20 23:17, Michael S. Tsirkin wrote:  
> >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian
Borntraeger wrote:
> >>>>
> >>>>
> >>>> On 06.02.20 15:22, eperezma at redhat.com wrote:  
> >>>>> Hi Christian.
> >>>>>
> >>>>> Could you try this patch on top of ("38ced0208491
vhost: use batched version by default")?
> >>>>>
> >>>>> It will not solve your first random crash but it
should help with the lost of network connectivity.
> >>>>>
> >>>>> Please let me know how does it goes.  
> >>>>
> >>>>
> >>>> 38ced0208491 + this seem to be ok.
> >>>>
> >>>> Not sure if you can make out anything of this (and the
previous git bisect log)
> >>>
> >>> Yes it does - that this is just bad split-up of patches, and
there's
> >>> still a real bug that caused worse crashes :)
> >>>
> >>> So I just pushed batch-v4.
> >>> I expect that will fail, and bisect to give us
> >>>     vhost: batching fetches
> >>> Can you try that please?
> >>>  
> >>
> >> yes.
> >>
> >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> >> Author: Michael S. Tsirkin <mst at redhat.com>
> >> Date:   Mon Oct 7 06:11:18 2019 -0400
> >>
> >>     vhost: batching fetches
> >>     
> >>     With this patch applied, new and old code perform identically.
> >>     
> >>     Lots of extra optimizations are now possible, e.g.
> >>     we can fetch multiple heads with copy_from/to_user now.
> >>     We can get rid of maintaining the log array.  Etc etc.
> >>     
> >>     Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
> >>
> >>  drivers/vhost/test.c  |  2 +-
> >>  drivers/vhost/vhost.c | 39
++++++++++++++++++++++++++++++++++-----
> >>  drivers/vhost/vhost.h |  4 +++-
> >>  3 files changed, 38 insertions(+), 7 deletions(-)
> >>  
> > 
> > 
> > And the symptom is still the same - random crashes
> > after a bit of traffic, right?  
> 
> random guest crashes after a reboot of the guests. As if vhost would still
> write into now stale buffers.
> 
I'm late to the party; but where is that commit located? Or has it been
dropped again already?

Christian Borntraeger

2020-Feb-10 10:09 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 10.02.20 10:40, Eugenio Perez Martin wrote:> Hi Christian.
> 
> I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data? Your
configuration (libvirt or qemu line), and host's dmesg output if any?
I do the following in the guest:
ping -c 200 -f somevalidip; reboot
sometimes I need to do that multiple times and sometimes I do not get a guest
crash but host dmesg like

Guest moved used index from 0 to 292

xml is pretty simple

    <interface type='direct'>
      <mac address='52:54:00:7c:2c:f3'/>
      <source dev='encbd00' mode='bridge'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <address type='ccw' cssid='0xfe' ssid='0x0'
devno='0x0001'/>
    </interface>


Reverting this patch seems to make both problems go away.

> 
> Thanks!
> 
> On Fri, Feb 7, 2020 at 9:13 AM Christian Borntraeger <borntraeger at
de.ibm.com <mailto:borntraeger at de.ibm.com>> wrote:
> 
> 
> 
>     On 07.02.20 08:58, Michael S. Tsirkin wrote:
>     > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger
wrote:
>     >> Also adding Cornelia.
>     >>
>     >>
>     >> On 06.02.20 23:17, Michael S. Tsirkin wrote:
>     >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian
Borntraeger wrote:
>     >>>>
>     >>>>
>     >>>> On 06.02.20 15:22, eperezma at redhat.com
<mailto:eperezma at redhat.com> wrote:
>     >>>>> Hi Christian.
>     >>>>>
>     >>>>> Could you try this patch on top of
("38ced0208491 vhost: use batched version by default")?
>     >>>>>
>     >>>>> It will not solve your first random crash but it
should help with the lost of network connectivity.
>     >>>>>
>     >>>>> Please let me know how does it goes.
>     >>>>
>     >>>>
>     >>>> 38ced0208491 + this seem to be ok.
>     >>>>
>     >>>> Not sure if you can make out anything of this (and the
previous git bisect log)
>     >>>
>     >>> Yes it does - that this is just bad split-up of patches,
and there's
>     >>> still a real bug that caused worse crashes :)
>     >>>
>     >>> So I just pushed batch-v4.
>     >>> I expect that will fail, and bisect to give us
>     >>>? ? ?vhost: batching fetches
>     >>> Can you try that please?
>     >>>
>     >>
>     >> yes.
>     >>
>     >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad
commit
>     >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
>     >> Author: Michael S. Tsirkin <mst at redhat.com
<mailto:mst at redhat.com>>
>     >> Date:? ?Mon Oct 7 06:11:18 2019 -0400
>     >>
>     >>? ? ?vhost: batching fetches
>     >>? ? ?
>     >>? ? ?With this patch applied, new and old code perform
identically.
>     >>? ? ?
>     >>? ? ?Lots of extra optimizations are now possible, e.g.
>     >>? ? ?we can fetch multiple heads with copy_from/to_user now.
>     >>? ? ?We can get rid of maintaining the log array.? Etc etc.
>     >>? ? ?
>     >>? ? ?Signed-off-by: Michael S. Tsirkin <mst at redhat.com
<mailto:mst at redhat.com>>
>     >>
>     >>? drivers/vhost/test.c? |? 2 +-
>     >>? drivers/vhost/vhost.c | 39
++++++++++++++++++++++++++++++++++-----
>     >>? drivers/vhost/vhost.h |? 4 +++-
>     >>? 3 files changed, 38 insertions(+), 7 deletions(-)
>     >>
>     >
>     >
>     > And the symptom is still the same - random crashes
>     > after a bit of traffic, right?
> 
>     random guest crashes after a reboot of the guests. As if vhost would
still
>     write into now stale buffers.
>

Christian Borntraeger

2020-Feb-10 11:01 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 10.02.20 10:47, Eugenio Perez Martin wrote:> Hi Christian.
> 
> I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data? Your
configuration (libvirt or qemu line), and host's dmesg output if any?
> 
> Thanks!
If it was not obvious, this is on s390x, a big endian system.

Christian Borntraeger

2020-Feb-11 09:56 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 11.02.20 10:33, Eugenio P?rez wrote:> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>
>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>> Hi Christian.
>>>
>>> I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>> Your configuration (libvirt or qemu line), and host's dmesg
output if any?
>>>
>>> Thanks!
>>
>> If it was not obvious, this is on s390x, a big endian system.
>>
> 
> Hi Christian. Thank you very much for the hints.
> 
> Could we add some debug traces? Something like the inline patch should give
us some clues.
attached. this time the guest crashed (and no guest moved message)
-------------- next part --------------
[    0.140906] Linux version 5.5.0+ (cborntra at m83lp52.lnxne.boe) (gcc version
9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #18 SMP Tue Feb 11 10:51:14 CET 2020
[    0.140908] setup: Linux is running natively in 64-bit mode
[    0.140950] setup: The maximum memory size is 131072MB
[    0.140955] setup: Reserving 1024MB of memory at 130048MB for crashkernel
(System RAM: 130048MB)
[    0.140967] numa: NUMA mode: plain
[    0.141060] cpu: 64 configured CPUs, 0 standby CPUs
[    0.141107] cpu: The CPU configuration topology of the machine is: 0 0 4 2 3
10 / 4
[    0.141626] Write protected kernel read-only data: 13420k
[    0.142408] Zone ranges:
[    0.142409]   DMA      [mem 0x0000000000000000-0x000000007fffffff]
[    0.142411]   Normal   [mem 0x0000000080000000-0x0000001fffffffff]
[    0.142412] Movable zone start for each node
[    0.142413] Early memory node ranges
[    0.142414]   node   0: [mem 0x0000000000000000-0x0000001fffffffff]
[    0.142421] Initmem setup node 0 [mem 0x0000000000000000-0x0000001fffffffff]
[    0.142422] On node 0 totalpages: 33554432
[    0.142423]   DMA zone: 8192 pages used for memmap
[    0.142423]   DMA zone: 0 pages reserved
[    0.142424]   DMA zone: 524288 pages, LIFO batch:63
[    0.159790]   Normal zone: 516096 pages used for memmap
[    0.159791]   Normal zone: 33030144 pages, LIFO batch:63
[    0.175978] percpu: Embedded 33 pages/cpu s97024 r8192 d29952 u135168
[    0.175986] pcpu-alloc: s97024 r8192 d29952 u135168 alloc=33*4096
[    0.175987] pcpu-alloc: [0] 000 [0] 001 [0] 002 [0] 003 
[    0.175988] pcpu-alloc: [0] 004 [0] 005 [0] 006 [0] 007 
[    0.175989] pcpu-alloc: [0] 008 [0] 009 [0] 010 [0] 011 
[    0.175991] pcpu-alloc: [0] 012 [0] 013 [0] 014 [0] 015 
[    0.175992] pcpu-alloc: [0] 016 [0] 017 [0] 018 [0] 019 
[    0.175993] pcpu-alloc: [0] 020 [0] 021 [0] 022 [0] 023 
[    0.175994] pcpu-alloc: [0] 024 [0] 025 [0] 026 [0] 027 
[    0.175996] pcpu-alloc: [0] 028 [0] 029 [0] 030 [0] 031 
[    0.175997] pcpu-alloc: [0] 032 [0] 033 [0] 034 [0] 035 
[    0.175998] pcpu-alloc: [0] 036 [0] 037 [0] 038 [0] 039 
[    0.175999] pcpu-alloc: [0] 040 [0] 041 [0] 042 [0] 043 
[    0.176000] pcpu-alloc: [0] 044 [0] 045 [0] 046 [0] 047 
[    0.176001] pcpu-alloc: [0] 048 [0] 049 [0] 050 [0] 051 
[    0.176003] pcpu-alloc: [0] 052 [0] 053 [0] 054 [0] 055 
[    0.176004] pcpu-alloc: [0] 056 [0] 057 [0] 058 [0] 059 
[    0.176005] pcpu-alloc: [0] 060 [0] 061 [0] 062 [0] 063 
[    0.176006] pcpu-alloc: [0] 064 [0] 065 [0] 066 [0] 067 
[    0.176007] pcpu-alloc: [0] 068 [0] 069 [0] 070 [0] 071 
[    0.176008] pcpu-alloc: [0] 072 [0] 073 [0] 074 [0] 075 
[    0.176010] pcpu-alloc: [0] 076 [0] 077 [0] 078 [0] 079 
[    0.176011] pcpu-alloc: [0] 080 [0] 081 [0] 082 [0] 083 
[    0.176012] pcpu-alloc: [0] 084 [0] 085 [0] 086 [0] 087 
[    0.176013] pcpu-alloc: [0] 088 [0] 089 [0] 090 [0] 091 
[    0.176015] pcpu-alloc: [0] 092 [0] 093 [0] 094 [0] 095 
[    0.176016] pcpu-alloc: [0] 096 [0] 097 [0] 098 [0] 099 
[    0.176017] pcpu-alloc: [0] 100 [0] 101 [0] 102 [0] 103 
[    0.176019] pcpu-alloc: [0] 104 [0] 105 [0] 106 [0] 107 
[    0.176020] pcpu-alloc: [0] 108 [0] 109 [0] 110 [0] 111 
[    0.176021] pcpu-alloc: [0] 112 [0] 113 [0] 114 [0] 115 
[    0.176022] pcpu-alloc: [0] 116 [0] 117 [0] 118 [0] 119 
[    0.176024] pcpu-alloc: [0] 120 [0] 121 [0] 122 [0] 123 
[    0.176025] pcpu-alloc: [0] 124 [0] 125 [0] 126 [0] 127 
[    0.176026] pcpu-alloc: [0] 128 [0] 129 [0] 130 [0] 131 
[    0.176028] pcpu-alloc: [0] 132 [0] 133 [0] 134 [0] 135 
[    0.176029] pcpu-alloc: [0] 136 [0] 137 [0] 138 [0] 139 
[    0.176030] pcpu-alloc: [0] 140 [0] 141 [0] 142 [0] 143 
[    0.176031] pcpu-alloc: [0] 144 [0] 145 [0] 146 [0] 147 
[    0.176033] pcpu-alloc: [0] 148 [0] 149 [0] 150 [0] 151 
[    0.176034] pcpu-alloc: [0] 152 [0] 153 [0] 154 [0] 155 
[    0.176035] pcpu-alloc: [0] 156 [0] 157 [0] 158 [0] 159 
[    0.176036] pcpu-alloc: [0] 160 [0] 161 [0] 162 [0] 163 
[    0.176037] pcpu-alloc: [0] 164 [0] 165 [0] 166 [0] 167 
[    0.176039] pcpu-alloc: [0] 168 [0] 169 [0] 170 [0] 171 
[    0.176040] pcpu-alloc: [0] 172 [0] 173 [0] 174 [0] 175 
[    0.176041] pcpu-alloc: [0] 176 [0] 177 [0] 178 [0] 179 
[    0.176042] pcpu-alloc: [0] 180 [0] 181 [0] 182 [0] 183 
[    0.176044] pcpu-alloc: [0] 184 [0] 185 [0] 186 [0] 187 
[    0.176045] pcpu-alloc: [0] 188 [0] 189 [0] 190 [0] 191 
[    0.176046] pcpu-alloc: [0] 192 [0] 193 [0] 194 [0] 195 
[    0.176047] pcpu-alloc: [0] 196 [0] 197 [0] 198 [0] 199 
[    0.176048] pcpu-alloc: [0] 200 [0] 201 [0] 202 [0] 203 
[    0.176049] pcpu-alloc: [0] 204 [0] 205 [0] 206 [0] 207 
[    0.176051] pcpu-alloc: [0] 208 [0] 209 [0] 210 [0] 211 
[    0.176052] pcpu-alloc: [0] 212 [0] 213 [0] 214 [0] 215 
[    0.176053] pcpu-alloc: [0] 216 [0] 217 [0] 218 [0] 219 
[    0.176054] pcpu-alloc: [0] 220 [0] 221 [0] 222 [0] 223 
[    0.176055] pcpu-alloc: [0] 224 [0] 225 [0] 226 [0] 227 
[    0.176056] pcpu-alloc: [0] 228 [0] 229 [0] 230 [0] 231 
[    0.176058] pcpu-alloc: [0] 232 [0] 233 [0] 234 [0] 235 
[    0.176059] pcpu-alloc: [0] 236 [0] 237 [0] 238 [0] 239 
[    0.176060] pcpu-alloc: [0] 240 [0] 241 [0] 242 [0] 243 
[    0.176061] pcpu-alloc: [0] 244 [0] 245 [0] 246 [0] 247 
[    0.176062] pcpu-alloc: [0] 248 [0] 249 [0] 250 [0] 251 
[    0.176064] pcpu-alloc: [0] 252 [0] 253 [0] 254 [0] 255 
[    0.176065] pcpu-alloc: [0] 256 [0] 257 [0] 258 [0] 259 
[    0.176066] pcpu-alloc: [0] 260 [0] 261 [0] 262 [0] 263 
[    0.176067] pcpu-alloc: [0] 264 [0] 265 [0] 266 [0] 267 
[    0.176068] pcpu-alloc: [0] 268 [0] 269 [0] 270 [0] 271 
[    0.176069] pcpu-alloc: [0] 272 [0] 273 [0] 274 [0] 275 
[    0.176071] pcpu-alloc: [0] 276 [0] 277 [0] 278 [0] 279 
[    0.176072] pcpu-alloc: [0] 280 [0] 281 [0] 282 [0] 283 
[    0.176073] pcpu-alloc: [0] 284 [0] 285 [0] 286 [0] 287 
[    0.176074] pcpu-alloc: [0] 288 [0] 289 [0] 290 [0] 291 
[    0.176075] pcpu-alloc: [0] 292 [0] 293 [0] 294 [0] 295 
[    0.176077] pcpu-alloc: [0] 296 [0] 297 [0] 298 [0] 299 
[    0.176078] pcpu-alloc: [0] 300 [0] 301 [0] 302 [0] 303 
[    0.176079] pcpu-alloc: [0] 304 [0] 305 [0] 306 [0] 307 
[    0.176080] pcpu-alloc: [0] 308 [0] 309 [0] 310 [0] 311 
[    0.176081] pcpu-alloc: [0] 312 [0] 313 [0] 314 [0] 315 
[    0.176082] pcpu-alloc: [0] 316 [0] 317 [0] 318 [0] 319 
[    0.176084] pcpu-alloc: [0] 320 [0] 321 [0] 322 [0] 323 
[    0.176085] pcpu-alloc: [0] 324 [0] 325 [0] 326 [0] 327 
[    0.176086] pcpu-alloc: [0] 328 [0] 329 [0] 330 [0] 331 
[    0.176087] pcpu-alloc: [0] 332 [0] 333 [0] 334 [0] 335 
[    0.176088] pcpu-alloc: [0] 336 [0] 337 [0] 338 [0] 339 
[    0.176109] Built 1 zonelists, mobility grouping on.  Total pages: 33030144
[    0.176110] Policy zone: Normal
[    0.176111] Kernel command line: root=/dev/disk/by-path/ccw-0.0.3318-part1
rd.dasd=0.0.3318 cio_ignore=all,!condev
rd.znet=qeth,0.0.bd00,0.0.bd01,0.0.bd02,layer2=1,portno=0,portname=OSAPORT
zfcp.allow_lun_scan=0 BOOT_IMAGE=0 crashkernel=1G BOOT_IMAGE=0
[    0.177105] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.177106] printk: log_buf_len total cpu_extra contributions: 1388544 bytes
[    0.177106] printk: log_buf_len min size: 131072 bytes
[    0.177403] printk: log_buf_len: 2097152 bytes
[    0.177404] printk: early log buf free: 123868(94%)
[    0.186473] Dentry cache hash table entries: 8388608 (order: 14, 67108864
bytes, linear)
[    0.190998] Inode-cache hash table entries: 4194304 (order: 13, 33554432
bytes, linear)
[    0.191000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.223325] Memory: 2314920K/134217728K available (10364K kernel code, 1904K
rwdata, 3056K rodata, 3912K init, 856K bss, 3355756K reserved, 0K cma-reserved)
[    0.223685] SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=340, Nodes=1
[    0.223724] ftrace: allocating 31322 entries in 123 pages
[    0.228303] ftrace: allocated 123 pages with 6 groups
[    0.229076] rcu: Hierarchical RCU implementation.
[    0.229076] rcu: 	RCU event tracing is enabled.
[    0.229077] rcu: 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=340.
[    0.229077] 	Tasks RCU enabled.
[    0.229078] rcu: RCU calculated value of scheduler-enlistment delay is 11
jiffies.
[    0.229079] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=340
[    0.231937] NR_IRQS: 3, nr_irqs: 3, preallocated irqs: 3
[    0.232042] clocksource: tod: mask: 0xffffffffffffffff max_cycles:
0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
[    0.232242] Console: colour dummy device 80x25
[    0.410136] printk: console [ttyS0] enabled
[    0.503299] Calibrating delay loop (skipped)... 21881.00 BogoMIPS preset
[    0.503300] pid_max: default: 348160 minimum: 2720
[    0.503432] LSM: Security Framework initializing
[    0.503451] SELinux:  Initializing.
[    0.503695] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes,
linear)
[    0.503841] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576
bytes, linear)
[    0.504709] rcu: Hierarchical SRCU implementation.
[    0.507419] smp: Bringing up secondary CPUs ...
[    0.520947] smp: Brought up 1 node, 64 CPUs
[    1.416671] node 0 initialised, 32136763 pages in 890ms
[    1.443532] devtmpfs: initialized
[    1.444269] random: get_random_u32 called from
bucket_table_alloc.isra.0+0x82/0x120 with crng_init=0
[    1.444808] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
max_idle_ns: 19112604462750000 ns
[    1.445430] futex hash table entries: 131072 (order: 13, 33554432 bytes,
vmalloc)
[    1.449946] xor: automatically using best checksumming function   xc        
[    1.450107] NET: Registered protocol family 16
[    1.450143] audit: initializing netlink subsys (disabled)
[    1.450207] audit: type=2000 audit(1581414814.421:1): state=initialized
audit_enabled=0 res=1
[    1.450331] Spectre V2 mitigation: etokens
[    1.451141] random: fast init done
[    1.480154] HugeTLB registered 1.00 MiB page size, pre-allocated 0 pages
[    1.722111] raid6: vx128x8  gen() 23433 MB/s
[    1.892050] raid6: vx128x8  xor() 13777 MB/s
[    1.892052] raid6: using algorithm vx128x8 gen() 23433 MB/s
[    1.892053] raid6: .... xor() 13777 MB/s, rmw enabled
[    1.892054] raid6: using s390xc recovery algorithm
[    1.892402] iommu: Default domain type: Translated 
[    1.892573] SCSI subsystem initialized
[    1.960177] PCI host bridge to bus 0000:00
[    1.960183] pci_bus 0000:00: root bus resource [mem
0x8000000000000000-0x8000000007ffffff 64bit pref]
[    1.960185] pci_bus 0000:00: No busn resource found for root bus, will use
[bus 00-ff]
[    1.960246] pci 0000:00:00.0: [1014:044b] type 00 class 0x120000
[    1.960296] pci 0000:00:00.0: reg 0x10: [mem
0xffffd80008000000-0xffffd8000fffffff 64bit pref]
[    1.960563] pci 0000:00:00.0: 0.000 Gb/s available PCIe bandwidth, limited by
Unknown speed x0 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5 GT/s x8
link)
[    1.960599] pci 0000:00:00.0: Adding to iommu group 0
[    1.960608] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[    1.962518] PCI host bridge to bus 0001:00
[    1.962520] pci_bus 0001:00: root bus resource [mem
0x8001000000000000-0x80010000000fffff 64bit pref]
[    1.962522] pci_bus 0001:00: No busn resource found for root bus, will use
[bus 00-ff]
[    1.962612] pci 0001:00:00.0: [15b3:1016] type 00 class 0x020000
[    1.962713] pci 0001:00:00.0: reg 0x10: [mem
0xffffd40002000000-0xffffd400020fffff 64bit pref]
[    1.962865] pci 0001:00:00.0: enabling Extended Tags
[    1.963360] pci 0001:00:00.0: 0.000 Gb/s available PCIe bandwidth, limited by
Unknown speed x0 link at 0001:00:00.0 (capable of 63.008 Gb/s with 8 GT/s x8
link)
[    1.963410] pci 0001:00:00.0: Adding to iommu group 1
[    1.963418] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to 00
[    1.965352] PCI host bridge to bus 0002:00
[    1.965354] pci_bus 0002:00: root bus resource [mem
0x8002000000000000-0x80020000000fffff 64bit pref]
[    1.965356] pci_bus 0002:00: No busn resource found for root bus, will use
[bus 00-ff]
[    1.965443] pci 0002:00:00.0: [15b3:1016] type 00 class 0x020000
[    1.965541] pci 0002:00:00.0: reg 0x10: [mem
0xffffd40008000000-0xffffd400080fffff 64bit pref]
[    1.965692] pci 0002:00:00.0: enabling Extended Tags
[    1.966190] pci 0002:00:00.0: 0.000 Gb/s available PCIe bandwidth, limited by
Unknown speed x0 link at 0002:00:00.0 (capable of 63.008 Gb/s with 8 GT/s x8
link)
[    1.966223] pci 0002:00:00.0: Adding to iommu group 2
[    1.966230] pci_bus 0002:00: busn_res: [bus 00-ff] end is updated to 00
[    2.455601] VFS: Disk quotas dquot_6.6.0
[    2.455644] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.457224] NET: Registered protocol family 2
[    2.457763] random: crng init done
[    2.458056] tcp_listen_portaddr_hash hash table entries: 65536 (order: 8,
1048576 bytes, linear)
[    2.458607] TCP established hash table entries: 524288 (order: 10, 4194304
bytes, vmalloc)
[    2.460722] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes,
linear)
[    2.461169] TCP: Hash tables configured (established 524288 bind 65536)
[    2.461517] UDP hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[    2.462534] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes,
vmalloc)
[    2.463929] NET: Registered protocol family 1
[    2.464136] Trying to unpack rootfs image as initramfs...
[    3.031779] Freeing initrd memory: 43772K
[    3.032996] alg: No test for crc32be (crc32be-vx)
[    3.037140] Initialise system trusted keyrings
[    3.037188] workingset: timestamp_bits=42 max_order=25 bucket_order=0
[    3.038293] fuse: init (API version 7.31)
[    3.038367] SGI XFS with ACLs, security attributes, realtime, quota, no debug
enabled
[    3.046520] Key type asymmetric registered
[    3.046522] Asymmetric key parser 'x509' registered
[    3.046529] Block layer SCSI generic (bsg) driver version 0.4 loaded (major
251)
[    3.046738] io scheduler mq-deadline registered
[    3.046740] io scheduler kyber registered
[    3.046761] io scheduler bfq registered
[    3.047604] atomic64_test: passed
[    3.047665] hvc_iucv: The z/VM IUCV HVC device driver cannot be used without
z/VM
[    3.053492] brd: module loaded
[    3.053875] cio: Channel measurement facility initialized using format
extended (mode autodetected)
[    3.054146] Discipline DIAG cannot be used without z/VM
[    4.874424] sclp_sd: No data is available for the config data entity
[    5.345780] qeth: loading core functions
[    5.345842] qeth: register layer 2 discipline
[    5.345843] qeth: register layer 3 discipline
[    5.346294] NET: Registered protocol family 10
[    5.347206] Segment Routing with IPv6
[    5.347223] NET: Registered protocol family 17
[    5.347237] Key type dns_resolver registered
[    5.347314] registered taskstats version 1
[    5.347319] Loading compiled-in X.509 certificates
[    5.387008] Loaded X.509 cert 'Build time autogenerated kernel key:
c46ba92ee388c82c5891ee836c9c20b752cdfac5'
[    5.387643] zswap: default zpool zbud not available
[    5.387645] zswap: pool creation failed
[    5.388151] Key type ._fscrypt registered
[    5.388153] Key type .fscrypt registered
[    5.388421] Btrfs loaded, crc32c=crc32c-vx
[    5.392827] Key type big_key registered
[    5.392833] ima: No TPM chip found, activating TPM-bypass!
[    5.392837] ima: Allocated hash algorithm: sha256
[    5.392845] ima: No architecture policies found
[    5.393985] Freeing unused kernel memory: 3912K
[    5.492121] Write protected read-only-after-init data: 68k
[    5.492123] Run /init as init process
[    5.506237] systemd[1]: Inserted module 'autofs4'
[    5.507585] systemd[1]: systemd v243.5-1.fc31 running in system mode. (+PAM
+AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT
+GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2
default-hierarchy=unified)
[    5.507985] systemd[1]: Detected architecture s390x.
[    5.507987] systemd[1]: Running in initial RAM disk.
[    5.508036] systemd[1]: Set hostname to <m83lp52.lnxne.boe>.
[    5.509031] systemd[1]: Couldn't move remaining userspace processes,
ignoring: Input/output error
[    5.542708] systemd[1]: Reached target Local File Systems.
[    5.542751] systemd[1]: Reached target Slices.
[    5.542770] systemd[1]: Reached target Swap.
[    5.542787] systemd[1]: Reached target Timers.
[    5.542868] systemd[1]: Listening on Journal Audit Socket.
[    5.812348] audit: type=1130 audit(1581414818.791:2): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    5.819297] audit: type=1130 audit(1581414818.791:3): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    6.765404] audit: type=1130 audit(1581414819.741:4): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    6.780267] audit: type=1130 audit(1581414819.751:5): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    7.005557] audit: type=1130 audit(1581414819.981:6): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    7.009701] qeth 0.0.bd00: Priority Queueing not supported
[    7.010265] qeth 0.0.bd00: portname is deprecated and is ignored
[    7.011335] dasd-eckd 0.0.3319: A channel path to the device has become
operational
[    7.011795] dasd-eckd 0.0.3318: A channel path to the device has become
operational
[    7.012163] dasd-eckd 0.0.331a: A channel path to the device has become
operational
[    7.012327] dasd-eckd 0.0.331b: A channel path to the device has become
operational
[    7.015361] qdio: 0.0.bd02 OSA on SC 159b using AI:1 QEBSM:0 PRI:1 TDD:1
SIGA: W AP
[    7.021855] dasd-eckd 0.0.3319: New DASD 3390/0E (CU 3990/01) with 262668
cylinders, 15 heads, 224 sectors
[    7.024637] dasd-eckd 0.0.3319: DASD with 4 KB/block, 189120960 KB total
size, 48 KB/track, compatible disk layout
[    7.025745]  dasda:VOL1/  0X3319: dasda1
[    7.026758] dasd-eckd 0.0.3318: New DASD 3390/0E (CU 3990/01) with 262668
cylinders, 15 heads, 224 sectors
[    7.029512] dasd-eckd 0.0.3318: DASD with 4 KB/block, 189120960 KB total
size, 48 KB/track, compatible disk layout
[    7.030552]  dasdb:VOL1/  0X3318: dasdb1
[    7.031661] dasd-eckd 0.0.331a: New DASD 3390/0C (CU 3990/01) with 30051
cylinders, 15 heads, 224 sectors
[    7.034797] dasd-eckd 0.0.331a: DASD with 4 KB/block, 21636720 KB total size,
48 KB/track, compatible disk layout
[    7.035495]  dasdc:VOL1/  0X331A:
[    7.036623] dasd-eckd 0.0.331b: New DASD 3390/0C (CU 3990/01) with 30051
cylinders, 15 heads, 224 sectors
[    7.039286] dasd-eckd 0.0.331b: DASD with 4 KB/block, 21636720 KB total size,
48 KB/track, compatible disk layout
[    7.039991]  dasdd:VOL1/  0X331B:
[    7.046991] audit: type=1130 audit(1581414820.021:7): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    7.055127] qeth 0.0.bd00: QDIO data connection isolation is deactivated
[    7.055590] qeth 0.0.bd00: The device represents a Bridge Capable Port
[    7.058856] qeth 0.0.bd00: MAC address 0e:87:75:01:5c:5b successfully
registered
[    7.059383] qeth 0.0.bd00: Device is a OSD Express card (level: 0198)
               with link type OSD_10GIG.
[    7.062125] audit: type=1130 audit(1581414820.041:8): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=plymouth-start
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    7.089046] qeth 0.0.bd00: MAC address de:45:d7:61:c4:13 successfully
registered
[    7.091260] qeth 0.0.bd00 encbd00: renamed from eth0
[    7.273398] mlx5_core 0001:00:00.0: enabling device (0000 -> 0002)
[    7.273485] mlx5_core 0001:00:00.0: firmware version: 14.23.1020
[    7.340188] dasdconf.sh Warning: 0.0.331b is already online, not configuring
[    7.353716] dasdconf.sh Warning: 0.0.3319 is already online, not configuring
[    7.353908] dasdconf.sh Warning: 0.0.3318 is already online, not configuring
[    7.487167] dasdconf.sh Warning: 0.0.331a is already online, not configuring
[    7.716124] mlx5_core 0002:00:00.0: enabling device (0000 -> 0002)
[    7.716210] mlx5_core 0002:00:00.0: firmware version: 14.23.1020
[    8.163850] mlx5_core 0001:00:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256)
RxCqeCmprss(0)
[    8.312389] mlx5_core 0002:00:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256)
RxCqeCmprss(0)
[    8.468505] mlx5_core 0001:00:00.0 enP1s519: renamed from eth0
[    8.782875] mlx5_core 0002:00:00.0 enP2s564: renamed from eth1
[    8.973475] audit: type=1130 audit(1581414821.951:9): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    8.990598] audit: type=1130 audit(1581414821.961:10): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    9.000284] EXT4-fs (dasdb1): mounted filesystem with ordered data mode.
Opts: (null)
[    9.107788] audit: type=1130 audit(1581414822.081:11): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    9.107792] audit: type=1131 audit(1581414822.081:12): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    9.152972] audit: type=1130 audit(1581414822.131:13): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?
addr=? terminal=? res=success'
[    9.356930] systemd-journald[542]: Received SIGTERM from PID 1 (systemd).
[    9.371415] printk: systemd: 20 output lines suppressed due to ratelimiting
[    9.689903] SELinux:  Permission watch in class filesystem not defined in
policy.
[    9.689908] SELinux:  Permission watch in class file not defined in policy.
[    9.689909] SELinux:  Permission watch_mount in class file not defined in
policy.
[    9.689910] SELinux:  Permission watch_sb in class file not defined in
policy.
[    9.689911] SELinux:  Permission watch_with_perm in class file not defined in
policy.
[    9.689912] SELinux:  Permission watch_reads in class file not defined in
policy.
[    9.689915] SELinux:  Permission watch in class dir not defined in policy.
[    9.689915] SELinux:  Permission watch_mount in class dir not defined in
policy.
[    9.689916] SELinux:  Permission watch_sb in class dir not defined in policy.
[    9.689917] SELinux:  Permission watch_with_perm in class dir not defined in
policy.
[    9.689918] SELinux:  Permission watch_reads in class dir not defined in
policy.
[    9.689921] SELinux:  Permission watch in class lnk_file not defined in
policy.
[    9.689922] SELinux:  Permission watch_mount in class lnk_file not defined in
policy.
[    9.689923] SELinux:  Permission watch_sb in class lnk_file not defined in
policy.
[    9.689924] SELinux:  Permission watch_with_perm in class lnk_file not
defined in policy.
[    9.689925] SELinux:  Permission watch_reads in class lnk_file not defined in
policy.
[    9.689927] SELinux:  Permission watch in class chr_file not defined in
policy.
[    9.689941] SELinux:  Permission watch_mount in class chr_file not defined in
policy.
[    9.689942] SELinux:  Permission watch_sb in class chr_file not defined in
policy.
[    9.689943] SELinux:  Permission watch_with_perm in class chr_file not
defined in policy.
[    9.689944] SELinux:  Permission watch_reads in class chr_file not defined in
policy.
[    9.689946] SELinux:  Permission watch in class blk_file not defined in
policy.
[    9.689947] SELinux:  Permission watch_mount in class blk_file not defined in
policy.
[    9.689948] SELinux:  Permission watch_sb in class blk_file not defined in
policy.
[    9.689949] SELinux:  Permission watch_with_perm in class blk_file not
defined in policy.
[    9.689950] SELinux:  Permission watch_reads in class blk_file not defined in
policy.
[    9.689952] SELinux:  Permission watch in class sock_file not defined in
policy.
[    9.689953] SELinux:  Permission watch_mount in class sock_file not defined
in policy.
[    9.689954] SELinux:  Permission watch_sb in class sock_file not defined in
policy.
[    9.689955] SELinux:  Permission watch_with_perm in class sock_file not
defined in policy.
[    9.689956] SELinux:  Permission watch_reads in class sock_file not defined
in policy.
[    9.689958] SELinux:  Permission watch in class fifo_file not defined in
policy.
[    9.689959] SELinux:  Permission watch_mount in class fifo_file not defined
in policy.
[    9.689961] SELinux:  Permission watch_sb in class fifo_file not defined in
policy.
[    9.689962] SELinux:  Permission watch_with_perm in class fifo_file not
defined in policy.
[    9.689963] SELinux:  Permission watch_reads in class fifo_file not defined
in policy.
[    9.690042] SELinux:  Class perf_event not defined in policy.
[    9.690043] SELinux: the above unknown classes and permissions will be
allowed
[    9.690045] SELinux:  policy capability network_peer_controls=1
[    9.690046] SELinux:  policy capability open_perms=1
[    9.690047] SELinux:  policy capability extended_socket_class=1
[    9.690048] SELinux:  policy capability always_check_network=0
[    9.690048] SELinux:  policy capability cgroup_seclabel=1
[    9.690049] SELinux:  policy capability nnp_nosuid_transition=1
[    9.766488] systemd[1]: Successfully loaded SELinux policy in 336.854ms.
[    9.805966] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in
12.884ms.
[    9.808232] systemd[1]: systemd v243.5-1.fc31 running in system mode. (+PAM
+AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT
+GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2
default-hierarchy=unified)
[    9.808649] systemd[1]: Detected architecture s390x.
[    9.810801] systemd[1]: Set hostname to <m83lp52.lnxne.boe>.
[    9.813360] systemd[1]: Couldn't move remaining userspace processes,
ignoring: Input/output error
[    9.907681] systemd[1]: /usr/lib/systemd/system/sssd.service:12: PIDFile=
references a path below legacy directory /var/run/, updating /var/run/sssd.pid ?
/run/sssd.pid; please update the unit file accordingly.
[    9.911744] systemd[1]: /usr/lib/systemd/system/iscsid.service:11: PIDFile=
references a path below legacy directory /var/run/, updating /var/run/iscsid.pid
? /run/iscsid.pid; please update the unit file accordingly.
[    9.911947] systemd[1]: /usr/lib/systemd/system/iscsiuio.service:13: PIDFile=
references a path below legacy directory /var/run/, updating
/var/run/iscsiuio.pid ? /run/iscsiuio.pid; please update the unit file
accordingly.
[    9.939442] systemd[1]: /usr/lib/systemd/system/sssd-kcm.socket:7:
ListenStream= references a path below legacy directory /var/run/, updating
/var/run/.heim_org.h5l.kcm-socket ? /run/.heim_org.h5l.kcm-socket; please update
the unit file accordingly.
[    9.966989] systemd[1]: initrd-switch-root.service: Succeeded.
[    9.967102] systemd[1]: Stopped Switch Root.
[    9.992173] EXT4-fs (dasdb1): re-mounted. Opts: (null)
[   10.269295] systemd-journald[1086]: Received client request to flush runtime
journal.
[   10.448707] VFIO - User Level meta-driver version: 0.3
[   10.487288] dasdconf.sh Warning: 0.0.3319 is already online, not configuring
[   10.488092] dasdconf.sh Warning: 0.0.3318 is already online, not configuring
[   10.488683] dasdconf.sh Warning: 0.0.331a is already online, not configuring
[   10.489424] genwqe 0000:00:00.0: enabling device (0000 -> 0002)
[   10.489591] dasdconf.sh Warning: 0.0.331b is already online, not configuring
[   10.756189] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0
[   10.882438] RPC: Registered named UNIX socket transport module.
[   10.882441] RPC: Registered udp transport module.
[   10.882443] RPC: Registered tcp transport module.
[   10.882443] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   10.912247] RPC: Registered rdma transport module.
[   10.912249] RPC: Registered rdma backchannel transport module.
[   10.975284] XFS (dasda1): Mounting V5 Filesystem
[   10.998984] XFS (dasda1): Ending clean mount
[   11.001618] xfs filesystem being mounted at /home supports timestamps until
2038 (0x7fffffff)
[   11.958825] mlx5_core 0001:00:00.0 enP1s519: Link up
[   11.960449] IPv6: ADDRCONF(NETDEV_CHANGE): enP1s519: link becomes ready
[   12.069226] mlx5_core 0002:00:00.0 enP2s564: Link up
[   12.132744] bridge: filtering via arp/ip/ip6tables is no longer available by
default. Update your scripts to load br_netfilter if you need this.
[   12.140433] tun: Universal TUN/TAP device driver, 1.6
[   12.140993] virbr0: port 1(virbr0-nic) entered blocking state
[   12.140995] virbr0: port 1(virbr0-nic) entered disabled state
[   12.141059] device virbr0-nic entered promiscuous mode
[   12.836770] virbr0: port 1(virbr0-nic) entered blocking state
[   12.836773] virbr0: port 1(virbr0-nic) entered listening state
[   12.860218] virbr0: port 1(virbr0-nic) entered disabled state
[   13.032109] IPv6: ADDRCONF(NETDEV_CHANGE): enP2s564: link becomes ready
[   43.323371] [vq=00000000abd097cd][vq->ndescs=0]
[   43.323373] [vq=000000008be108ff][vq->ndescs=0]
[   44.331333] [vq=00000000abd097cd][vq->ndescs=65]
[   44.331336] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   44.331339] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   44.331341] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   44.331342] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   44.331344] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   44.331346] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   44.331347] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   44.331349] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   44.331351] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   44.331353] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   44.331355] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   44.331357] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   44.331358] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   44.331360] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   44.331361] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   44.331363] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   44.331365] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   44.331367] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   44.331369] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   44.331370] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   44.331372] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   44.331373] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   44.331375] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   44.331376] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   44.331378] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   44.331393] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   44.331395] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   44.331396] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   44.331398] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   44.331399] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   44.331401] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   44.331402] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   44.331403] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   44.331405] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   44.331406] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   44.331407] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   44.331408] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   44.331410] [vq=00000000abd097cd][start=0][count=37]
[   44.348284] [vq=000000008be108ff][vq->ndescs=1]
[   44.348287] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.348290] [vq=000000008be108ff][vq->ndescs=0]
[   44.348297] [vq=000000008be108ff][start=0][count=1]
[   44.362505] [vq=000000008be108ff][vq->ndescs=1]
[   44.362506] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.362507] [vq=000000008be108ff][vq->ndescs=0]
[   44.362510] [vq=000000008be108ff][start=1][count=1]
[   44.363079] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   44.363081] [vq=00000000abd097cd][start=37][count=1]
[   44.363102] [vq=000000008be108ff][vq->ndescs=2]
[   44.363104] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=1]
[   44.363105] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.363107] [vq=000000008be108ff][vq->ndescs=0]
[   44.363110] [vq=000000008be108ff][start=2][count=2]
[   44.365821] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   44.365823] [vq=00000000abd097cd][start=38][count=1]
[   44.365946] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   44.365947] [vq=00000000abd097cd][start=39][count=1]
[   44.372309] [vq=000000008be108ff][vq->ndescs=1]
[   44.372311] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.372312] [vq=000000008be108ff][vq->ndescs=0]
[   44.372316] [vq=000000008be108ff][start=4][count=1]
[   44.422280] [vq=000000008be108ff][vq->ndescs=1]
[   44.422282] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.422283] [vq=000000008be108ff][vq->ndescs=0]
[   44.422286] [vq=000000008be108ff][start=5][count=1]
[   44.569329] [vq=000000008be108ff][vq->ndescs=2]
[   44.569330] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.569332] [vq=000000008be108ff][vq->ndescs=0]
[   44.569334] [vq=000000008be108ff][start=6][count=1]
[   44.582286] [vq=000000008be108ff][vq->ndescs=1]
[   44.582287] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.582288] [vq=000000008be108ff][vq->ndescs=0]
[   44.582291] [vq=000000008be108ff][start=7][count=1]
[   44.582531] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   44.582533] [vq=00000000abd097cd][start=40][count=1]
[   44.633175] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   44.633177] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   44.633178] [vq=00000000abd097cd][start=41][count=2]
[   44.701286] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   44.701287] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   44.701289] [vq=00000000abd097cd][start=43][count=2]
[   44.701557] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   44.701560] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   44.701562] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   44.701563] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   44.701564] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   44.701565] [vq=00000000abd097cd][start=45][count=5]
[   44.701856] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   44.701858] [vq=00000000abd097cd][start=50][count=1]
[   44.773183] [vq=000000008be108ff][vq->ndescs=2]
[   44.773185] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.773187] [vq=000000008be108ff][vq->ndescs=0]
[   44.773197] [vq=000000008be108ff][start=8][count=1]
[   44.801919] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   44.801923] [vq=00000000abd097cd][start=51][count=1]
[   44.911482] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   44.911487] [vq=00000000abd097cd][start=52][count=1]
[   44.919072] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   44.919074] [vq=00000000abd097cd][start=53][count=1]
[   44.953778] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   44.953780] [vq=00000000abd097cd][start=54][count=1]
[   44.974158] [vq=000000008be108ff][vq->ndescs=2]
[   44.974159] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.974161] [vq=000000008be108ff][vq->ndescs=0]
[   44.974169] [vq=000000008be108ff][start=9][count=1]
[   45.012038] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   45.012040] [vq=00000000abd097cd][start=55][count=1]
[   45.129882] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   45.129884] [vq=00000000abd097cd][start=56][count=1]
[   45.178065] [vq=000000008be108ff][vq->ndescs=2]
[   45.178066] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   45.178067] [vq=000000008be108ff][vq->ndescs=0]
[   45.178070] [vq=000000008be108ff][start=10][count=1]
[   45.192279] [vq=000000008be108ff][vq->ndescs=1]
[   45.192280] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   45.192282] [vq=000000008be108ff][vq->ndescs=0]
[   45.192286] [vq=000000008be108ff][start=11][count=1]
[   45.217428] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   45.217430] [vq=00000000abd097cd][start=57][count=1]
[   45.427294] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   45.427296] [vq=00000000abd097cd][start=58][count=1]
[   45.702447] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   45.702449] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   45.702450] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   45.702451] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   45.702453] [vq=00000000abd097cd][start=59][count=4]
[   45.702797] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   45.702799] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   45.702803] [vq=00000000abd097cd][vq->ndescs=65]
[   45.702804] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   45.702806] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   45.702807] [vq=00000000abd097cd][start=63][count=4]
[   45.736690] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   45.736692] [vq=00000000abd097cd][start=67][count=1]
[   45.772319] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   45.772320] [vq=00000000abd097cd][start=68][count=1]
[   45.952427] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   45.952430] [vq=00000000abd097cd][start=69][count=1]
[   45.987279] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   45.987281] [vq=00000000abd097cd][start=70][count=1]
[   46.232287] [vq=000000008be108ff][vq->ndescs=1]
[   46.232289] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.232290] [vq=000000008be108ff][vq->ndescs=0]
[   46.232295] [vq=000000008be108ff][start=12][count=1]
[   46.262277] [vq=000000008be108ff][vq->ndescs=1]
[   46.262278] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.262279] [vq=000000008be108ff][vq->ndescs=0]
[   46.262282] [vq=000000008be108ff][start=13][count=1]
[   46.312265] [vq=000000008be108ff][vq->ndescs=1]
[   46.312266] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.312267] [vq=000000008be108ff][vq->ndescs=0]
[   46.312270] [vq=000000008be108ff][start=14][count=1]
[   46.348301] [vq=000000008be108ff][vq->ndescs=1]
[   46.348302] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.348303] [vq=000000008be108ff][vq->ndescs=0]
[   46.348306] [vq=000000008be108ff][start=15][count=1]
[   46.464029] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   46.464031] [vq=00000000abd097cd][start=71][count=1]
[   46.602312] [vq=000000008be108ff][vq->ndescs=1]
[   46.602313] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.602314] [vq=000000008be108ff][vq->ndescs=0]
[   46.602318] [vq=000000008be108ff][start=16][count=1]
[   46.609109] [vq=000000008be108ff][vq->ndescs=2]
[   46.609110] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   46.609111] [vq=000000008be108ff][vq->ndescs=0]
[   46.609114] [vq=000000008be108ff][start=17][count=1]
[   46.625327] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   46.625328] [vq=00000000abd097cd][start=72][count=1]
[   46.652313] [vq=000000008be108ff][vq->ndescs=1]
[   46.652315] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.652316] [vq=000000008be108ff][vq->ndescs=0]
[   46.652319] [vq=000000008be108ff][start=18][count=1]
[   46.703259] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   46.703260] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   46.703265] [vq=00000000abd097cd][start=73][count=2]
[   46.703685] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   46.703688] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   46.703690] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   46.703691] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   46.703692] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   46.703693] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   46.703695] [vq=00000000abd097cd][start=75][count=6]
[   46.776740] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   46.776743] [vq=00000000abd097cd][start=81][count=1]
[   46.833623] [vq=000000008be108ff][vq->ndescs=2]
[   46.833624] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   46.833626] [vq=000000008be108ff][vq->ndescs=0]
[   46.833631] [vq=000000008be108ff][start=19][count=1]
[   46.858225] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   46.858227] [vq=00000000abd097cd][start=82][count=1]
[   46.991350] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   46.991352] [vq=00000000abd097cd][start=83][count=1]
[   47.027160] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   47.027162] [vq=00000000abd097cd][start=84][count=1]
[   47.051619] [vq=000000008be108ff][vq->ndescs=2]
[   47.051621] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   47.051623] [vq=000000008be108ff][vq->ndescs=0]
[   47.051627] [vq=000000008be108ff][start=20][count=1]
[   47.091535] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   47.091537] [vq=00000000abd097cd][start=85][count=1]
[   47.254867] [vq=000000008be108ff][vq->ndescs=2]
[   47.254868] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   47.254870] [vq=000000008be108ff][vq->ndescs=0]
[   47.254874] [vq=000000008be108ff][start=21][count=1]
[   47.291144] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   47.291146] [vq=00000000abd097cd][start=86][count=1]
[   47.503996] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   47.503998] [vq=00000000abd097cd][start=87][count=1]
[   47.704153] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   47.704155] [vq=00000000abd097cd][start=88][count=1]
[   47.704433] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   47.704436] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   47.704437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   47.704439] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   47.704440] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   47.704441] [vq=00000000abd097cd][start=89][count=5]
[   47.704582] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   47.704584] [vq=00000000abd097cd][start=94][count=1]
[   47.748063] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   47.748065] [vq=00000000abd097cd][start=95][count=1]
[   47.912449] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   47.912452] [vq=00000000abd097cd][start=96][count=1]
[   48.031357] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   48.031358] [vq=00000000abd097cd][start=97][count=1]
[   48.348365] [vq=000000008be108ff][vq->ndescs=1]
[   48.348366] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   48.348368] [vq=000000008be108ff][vq->ndescs=0]
[   48.348373] [vq=000000008be108ff][start=22][count=1]
[   48.660191] [vq=000000008be108ff][vq->ndescs=2]
[   48.660193] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   48.660194] [vq=000000008be108ff][vq->ndescs=0]
[   48.660197] [vq=000000008be108ff][start=23][count=1]
[   48.681124] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   48.681126] [vq=00000000abd097cd][start=98][count=1]
[   48.703279] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   48.703281] [vq=00000000abd097cd][start=99][count=1]
[   48.705144] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   48.705146] [vq=00000000abd097cd][start=100][count=1]
[   48.705468] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   48.705469] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   48.705471] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   48.705472] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   48.705473] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   48.705475] [vq=00000000abd097cd][start=101][count=5]
[   48.705657] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   48.705658] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   48.705660] [vq=00000000abd097cd][start=106][count=2]
[   48.863171] [vq=000000008be108ff][vq->ndescs=2]
[   48.863173] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   48.863175] [vq=000000008be108ff][vq->ndescs=0]
[   48.863182] [vq=000000008be108ff][start=24][count=1]
[   48.888847] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   48.888858] [vq=00000000abd097cd][start=108][count=1]
[   49.117382] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   49.117386] [vq=00000000abd097cd][start=109][count=1]
[   49.119562] [vq=000000008be108ff][vq->ndescs=2]
[   49.119563] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   49.119565] [vq=000000008be108ff][vq->ndescs=0]
[   49.119572] [vq=000000008be108ff][start=25][count=1]
[   49.157488] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   49.157490] [vq=00000000abd097cd][start=110][count=1]
[   49.321369] [vq=000000008be108ff][vq->ndescs=2]
[   49.321371] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   49.321372] [vq=000000008be108ff][vq->ndescs=0]
[   49.321375] [vq=000000008be108ff][start=26][count=1]
[   49.353755] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   49.353758] [vq=00000000abd097cd][start=111][count=1]
[   49.370006] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   49.370008] [vq=00000000abd097cd][start=112][count=1]
[   49.370033] [vq=000000008be108ff][vq->ndescs=1]
[   49.370035] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   49.370036] [vq=000000008be108ff][vq->ndescs=0]
[   49.370040] [vq=000000008be108ff][start=27][count=1]
[   49.429924] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   49.429927] [vq=00000000abd097cd][start=113][count=1]
[   49.706234] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   49.706239] [vq=00000000abd097cd][start=114][count=1]
[   49.706554] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   49.706556] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   49.706558] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   49.706559] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   49.706561] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   49.706563] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   49.706564] [vq=00000000abd097cd][start=115][count=6]
[   49.706714] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   49.706716] [vq=00000000abd097cd][start=121][count=1]
[   49.736717] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   49.736719] [vq=00000000abd097cd][start=122][count=1]
[   49.855886] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   49.855890] [vq=00000000abd097cd][start=123][count=1]
[   50.157344] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   50.157348] [vq=00000000abd097cd][start=124][count=1]
[   50.464012] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   50.464014] [vq=00000000abd097cd][start=125][count=1]
[   50.646328] [vq=000000008be108ff][vq->ndescs=1]
[   50.646330] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   50.646331] [vq=000000008be108ff][vq->ndescs=0]
[   50.646337] [vq=000000008be108ff][start=28][count=1]
[   50.703164] [vq=000000008be108ff][vq->ndescs=2]
[   50.703165] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   50.703166] [vq=000000008be108ff][vq->ndescs=0]
[   50.703169] [vq=000000008be108ff][start=29][count=1]
[   50.707140] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   50.707142] [vq=00000000abd097cd][start=126][count=1]
[   50.707471] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   50.707473] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   50.707475] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   50.707480] [vq=00000000abd097cd][vq->ndescs=65]
[   50.707481] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   50.707483] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   50.707484] [vq=00000000abd097cd][start=127][count=5]
[   50.707633] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   50.707635] [vq=00000000abd097cd][start=132][count=1]
[   50.720693] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   50.720695] [vq=00000000abd097cd][start=133][count=1]
[   50.776744] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   50.776746] [vq=00000000abd097cd][start=134][count=1]
[   50.821119] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   50.821121] [vq=00000000abd097cd][start=135][count=1]
[   50.829012] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   50.829014] [vq=00000000abd097cd][start=136][count=1]
[   50.904533] [vq=000000008be108ff][vq->ndescs=2]
[   50.904535] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   50.904536] [vq=000000008be108ff][vq->ndescs=0]
[   50.904540] [vq=000000008be108ff][start=30][count=1]
[   50.921538] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   50.921540] [vq=00000000abd097cd][start=137][count=1]
[   50.930125] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   50.930127] [vq=00000000abd097cd][start=138][count=1]
[   51.180313] [vq=000000008be108ff][vq->ndescs=2]
[   51.180315] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   51.180316] [vq=000000008be108ff][vq->ndescs=0]
[   51.180320] [vq=000000008be108ff][start=31][count=1]
[   51.197225] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   51.197226] [vq=00000000abd097cd][start=139][count=1]
[   51.218861] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   51.218863] [vq=00000000abd097cd][start=140][count=1]
[   51.382675] [vq=000000008be108ff][vq->ndescs=2]
[   51.382676] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   51.382680] [vq=000000008be108ff][vq->ndescs=0]
[   51.382682] [vq=000000008be108ff][start=32][count=1]
[   51.410246] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   51.410247] [vq=00000000abd097cd][start=141][count=1]
[   51.503908] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   51.503910] [vq=00000000abd097cd][start=142][count=1]
[   51.626662] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   51.626666] [vq=00000000abd097cd][start=143][count=1]
[   51.708185] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   51.708187] [vq=00000000abd097cd][start=144][count=1]
[   51.708530] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   51.708532] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   51.708534] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   51.708536] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   51.708538] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   51.708540] [vq=00000000abd097cd][start=145][count=5]
[   51.708676] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   51.708678] [vq=00000000abd097cd][start=150][count=1]
[   51.863910] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   51.863912] [vq=00000000abd097cd][start=151][count=1]
[   51.951289] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   51.951291] [vq=00000000abd097cd][start=152][count=1]
[   52.561591] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   52.561594] [vq=00000000abd097cd][start=153][count=1]
[   52.564073] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   52.564076] [vq=00000000abd097cd][start=154][count=1]
[   52.705924] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   52.705926] [vq=00000000abd097cd][start=155][count=1]
[   52.709187] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   52.709191] [vq=00000000abd097cd][start=156][count=1]
[   52.709509] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   52.709511] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   52.709513] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   52.709515] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   52.709516] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   52.709518] [vq=00000000abd097cd][start=157][count=5]
[   52.709666] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   52.709668] [vq=00000000abd097cd][start=162][count=1]
[   52.783101] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   52.783103] [vq=00000000abd097cd][start=163][count=1]
[   52.992797] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   52.992801] [vq=00000000abd097cd][start=164][count=1]
[   53.433538] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   53.433542] [vq=00000000abd097cd][start=165][count=1]
[   53.710292] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   53.710295] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   53.710298] [vq=00000000abd097cd][start=166][count=2]
[   53.710609] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   53.710611] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   53.710613] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   53.710615] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   53.710617] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   53.710618] [vq=00000000abd097cd][start=168][count=5]
[   53.736687] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   53.736689] [vq=00000000abd097cd][start=173][count=1]
[   53.802369] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   53.802372] [vq=00000000abd097cd][start=174][count=1]
[   54.014274] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   54.014277] [vq=00000000abd097cd][start=175][count=1]
[   54.041379] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   54.041381] [vq=00000000abd097cd][start=176][count=1]
[   54.463998] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   54.464001] [vq=00000000abd097cd][start=177][count=1]
[   54.465467] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   54.465469] [vq=00000000abd097cd][start=178][count=1]
[   54.646933] [vq=000000008be108ff][vq->ndescs=1]
[   54.646934] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   54.646936] [vq=000000008be108ff][vq->ndescs=0]
[   54.646942] [vq=000000008be108ff][start=33][count=1]
[   54.711370] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   54.711373] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   54.711375] [vq=00000000abd097cd][start=179][count=2]
[   54.711633] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   54.711634] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   54.711636] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   54.711637] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   54.711639] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   54.711640] [vq=00000000abd097cd][start=181][count=5]
[   54.776734] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   54.776736] [vq=00000000abd097cd][start=186][count=1]
[   55.081427] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   55.081430] [vq=00000000abd097cd][start=187][count=1]
[   55.121658] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   55.121661] [vq=00000000abd097cd][start=188][count=1]
[   55.503925] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   55.503927] [vq=00000000abd097cd][start=189][count=1]
[   55.712305] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   55.712309] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   55.712311] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   55.712313] [vq=00000000abd097cd][start=190][count=3]
[   55.712437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   55.712438] [vq=00000000abd097cd][start=193][count=1]
[   56.034990] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   56.034993] [vq=00000000abd097cd][start=194][count=1]
[   56.308036] [vq=00000000abd097cd][vq->ndescs=61]
[   56.308037] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=1]
[   56.308040] [vq=00000000abd097cd][start=195][count=1]
[   56.512620] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=2]
[   56.512622] [vq=00000000abd097cd][start=196][count=1]
[   56.707754] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=3]
[   56.707758] [vq=00000000abd097cd][start=197][count=1]
[   56.713242] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=4]
[   56.713244] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=5]
[   56.713246] [vq=00000000abd097cd][start=198][count=2]
[   56.713463] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=6]
[   56.713465] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=7]
[   56.713467] [vq=00000000abd097cd][start=200][count=2]
[   57.238741] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=8]
[   57.238747] [vq=00000000abd097cd][start=202][count=1]
[   57.281662] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=9]
[   57.281664] [vq=00000000abd097cd][start=203][count=1]
[   57.347235] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=10]
[   57.347237] [vq=00000000abd097cd][start=204][count=1]
[   57.436044] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=11]
[   57.436047] [vq=00000000abd097cd][start=205][count=1]
[   57.714242] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=12]
[   57.714245] [vq=00000000abd097cd][start=206][count=1]
[   57.714470] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=13]
[   57.714472] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=14]
[   57.714474] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=15]
[   57.714476] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=16]
[   57.714478] [vq=00000000abd097cd][start=207][count=4]
[   57.736863] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=17]
[   57.736865] [vq=00000000abd097cd][start=211][count=1]
[   58.369315] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=18]
[   58.369319] [vq=00000000abd097cd][start=212][count=1]
[   58.387247] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=19]
[   58.387249] [vq=00000000abd097cd][start=213][count=1]
[   58.464066] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=20]
[   58.464071] [vq=00000000abd097cd][start=214][count=1]
[   58.715327] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=21]
[   58.715330] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=22]
[   58.715333] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=23]
[   58.715335] [vq=00000000abd097cd][start=215][count=3]
[   58.715525] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=24]
[   58.715527] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=25]
[   58.715529] [vq=00000000abd097cd][start=218][count=2]
[   58.776675] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=26]
[   58.776677] [vq=00000000abd097cd][start=220][count=1]
[   59.427685] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=27]
[   59.427689] [vq=00000000abd097cd][start=221][count=1]
[   59.473816] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=28]
[   59.473819] [vq=00000000abd097cd][start=222][count=1]
[   59.503898] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=29]
[   59.503901] [vq=00000000abd097cd][start=223][count=1]
[   59.646298] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=30]
[   59.646302] [vq=00000000abd097cd][start=224][count=1]
[   59.716224] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=31]
[   59.716226] [vq=00000000abd097cd][start=225][count=1]
[   59.716457] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=32]
[   59.716459] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=33]
[   59.716461] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=34]
[   59.716463] [vq=00000000abd097cd][start=226][count=3]
[   60.467226] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=35]
[   60.467229] [vq=00000000abd097cd][start=229][count=1]
[   60.710063] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=36]
[   60.710066] [vq=00000000abd097cd][start=230][count=1]
[   60.717198] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=37]
[   60.717200] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=38]
[   60.717201] [vq=00000000abd097cd][start=231][count=2]
[   60.717438] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=39]
[   60.717440] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=40]
[   60.717442] [vq=00000000abd097cd][start=233][count=2]
[   61.001117] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=41]
[   61.001119] [vq=00000000abd097cd][start=235][count=1]
[   61.057080] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=42]
[   61.057082] [vq=00000000abd097cd][start=236][count=1]
[   61.382703] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=43]
[   61.382707] [vq=00000000abd097cd][start=237][count=1]
[   61.438023] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=44]
[   61.438025] [vq=00000000abd097cd][start=238][count=1]
[   61.507231] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=45]
[   61.507234] [vq=00000000abd097cd][start=239][count=1]
[   61.718344] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=46]
[   61.718348] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=47]
[   61.718349] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=48]
[   61.718351] [vq=00000000abd097cd][start=240][count=3]
[   61.718483] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=49]
[   61.718484] [vq=00000000abd097cd][start=243][count=1]
[   61.736698] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=50]
[   61.736700] [vq=00000000abd097cd][start=244][count=1]
[   61.953423] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=51]
[   61.953425] [vq=00000000abd097cd][start=245][count=1]
[   62.464045] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=52]
[   62.464049] [vq=00000000abd097cd][start=246][count=1]
[   62.719113] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=53]
[   62.719115] [vq=00000000abd097cd][start=247][count=1]
[   62.719328] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=54]
[   62.719330] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=55]
[   62.719331] [vq=00000000abd097cd][start=248][count=2]
[   62.776707] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=56]
[   62.776709] [vq=00000000abd097cd][start=250][count=1]
[   63.503845] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=57]
[   63.503849] [vq=00000000abd097cd][start=251][count=1]
[   63.720279] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=58]
[   63.720282] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=59]
[   63.720283] [vq=00000000abd097cd][start=252][count=2]
[   63.720442] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=60]
[   63.720444] [vq=00000000abd097cd][start=254][count=1]
[   64.620475] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=61]
[   64.620479] [vq=00000000abd097cd][start=255][count=1]
[   64.711456] [vq=00000000abd097cd][vq->ndescs=65]
[   64.711461] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   64.711466] [vq=00000000abd097cd][start=0][count=1]
[   64.721336] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   64.721338] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   64.721340] [vq=00000000abd097cd][start=1][count=2]
[   65.440290] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   65.440295] [vq=00000000abd097cd][start=3][count=1]
[   65.493920] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   65.493922] [vq=00000000abd097cd][start=4][count=1]
[   65.722216] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   65.722221] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   65.722223] [vq=00000000abd097cd][start=5][count=2]
[   65.736673] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   65.736675] [vq=00000000abd097cd][start=7][count=1]
[   65.812061] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   65.812063] [vq=00000000abd097cd][start=8][count=1]
[   65.954015] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   65.954017] [vq=00000000abd097cd][start=9][count=1]
[   66.463948] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   66.463953] [vq=00000000abd097cd][start=10][count=1]
[   66.723295] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   66.723297] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   66.723298] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   66.723300] [vq=00000000abd097cd][start=11][count=3]
[   66.776726] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   66.776728] [vq=00000000abd097cd][start=14][count=1]
[   66.831446] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   66.831449] [vq=00000000abd097cd][start=15][count=1]
[   67.154905] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   67.154908] [vq=00000000abd097cd][start=16][count=1]
[   67.202734] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   67.202737] [vq=00000000abd097cd][start=17][count=1]
[   67.299794] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   67.299797] [vq=00000000abd097cd][start=18][count=1]
[   67.503931] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   67.503934] [vq=00000000abd097cd][start=19][count=1]
[   67.724138] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   67.724140] [vq=00000000abd097cd][start=20][count=1]
[   67.724307] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   67.724308] [vq=00000000abd097cd][start=21][count=1]
[   67.871496] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   67.871499] [vq=00000000abd097cd][start=22][count=1]
[   68.227397] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   68.227399] [vq=00000000abd097cd][start=23][count=1]
[   68.625622] [vq=000000008be108ff][vq->ndescs=1]
[   68.625624] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.625627] [vq=000000008be108ff][vq->ndescs=0]
[   68.625632] [vq=000000008be108ff][start=34][count=1]
[   68.625897] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   68.625899] [vq=00000000abd097cd][start=24][count=1]
[   68.625920] [vq=000000008be108ff][vq->ndescs=1]
[   68.625921] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.625923] [vq=000000008be108ff][vq->ndescs=0]
[   68.625925] [vq=000000008be108ff][start=35][count=1]
[   68.626108] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   68.626109] [vq=00000000abd097cd][start=25][count=1]
[   68.626163] [vq=000000008be108ff][vq->ndescs=1]
[   68.626164] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626165] [vq=000000008be108ff][vq->ndescs=0]
[   68.626168] [vq=000000008be108ff][start=36][count=1]
[   68.626309] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   68.626311] [vq=00000000abd097cd][start=26][count=1]
[   68.626348] [vq=000000008be108ff][vq->ndescs=1]
[   68.626349] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626351] [vq=000000008be108ff][vq->ndescs=0]
[   68.626353] [vq=000000008be108ff][start=37][count=1]
[   68.626489] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   68.626491] [vq=00000000abd097cd][start=27][count=1]
[   68.626527] [vq=000000008be108ff][vq->ndescs=1]
[   68.626528] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626529] [vq=000000008be108ff][vq->ndescs=0]
[   68.626531] [vq=000000008be108ff][start=38][count=1]
[   68.626668] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   68.626669] [vq=00000000abd097cd][start=28][count=1]
[   68.626705] [vq=000000008be108ff][vq->ndescs=1]
[   68.626706] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626707] [vq=000000008be108ff][vq->ndescs=0]
[   68.626710] [vq=000000008be108ff][start=39][count=1]
[   68.626848] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   68.626850] [vq=00000000abd097cd][start=29][count=1]
[   68.626905] [vq=000000008be108ff][vq->ndescs=1]
[   68.626906] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626907] [vq=000000008be108ff][vq->ndescs=0]
[   68.626910] [vq=000000008be108ff][start=40][count=1]
[   68.627046] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   68.627048] [vq=00000000abd097cd][start=30][count=1]
[   68.627083] [vq=000000008be108ff][vq->ndescs=1]
[   68.627084] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627086] [vq=000000008be108ff][vq->ndescs=0]
[   68.627088] [vq=000000008be108ff][start=41][count=1]
[   68.627225] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   68.627227] [vq=00000000abd097cd][start=31][count=1]
[   68.627263] [vq=000000008be108ff][vq->ndescs=1]
[   68.627264] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627265] [vq=000000008be108ff][vq->ndescs=0]
[   68.627267] [vq=000000008be108ff][start=42][count=1]
[   68.627412] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   68.627413] [vq=00000000abd097cd][start=32][count=1]
[   68.627455] [vq=000000008be108ff][vq->ndescs=1]
[   68.627456] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627458] [vq=000000008be108ff][vq->ndescs=0]
[   68.627461] [vq=000000008be108ff][start=43][count=1]
[   68.627597] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   68.627598] [vq=00000000abd097cd][start=33][count=1]
[   68.627632] [vq=000000008be108ff][vq->ndescs=1]
[   68.627633] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627634] [vq=000000008be108ff][vq->ndescs=0]
[   68.627636] [vq=000000008be108ff][start=44][count=1]
[   68.627788] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   68.627791] [vq=00000000abd097cd][start=34][count=1]
[   68.627840] [vq=000000008be108ff][vq->ndescs=1]
[   68.627841] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627843] [vq=000000008be108ff][vq->ndescs=0]
[   68.627845] [vq=000000008be108ff][start=45][count=1]
[   68.627976] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   68.627978] [vq=00000000abd097cd][start=35][count=1]
[   68.628011] [vq=000000008be108ff][vq->ndescs=1]
[   68.628012] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628013] [vq=000000008be108ff][vq->ndescs=0]
[   68.628016] [vq=000000008be108ff][start=46][count=1]
[   68.628145] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   68.628147] [vq=00000000abd097cd][start=36][count=1]
[   68.628178] [vq=000000008be108ff][vq->ndescs=1]
[   68.628179] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628181] [vq=000000008be108ff][vq->ndescs=0]
[   68.628183] [vq=000000008be108ff][start=47][count=1]
[   68.628311] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   68.628313] [vq=00000000abd097cd][start=37][count=1]
[   68.628348] [vq=000000008be108ff][vq->ndescs=1]
[   68.628349] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628350] [vq=000000008be108ff][vq->ndescs=0]
[   68.628353] [vq=000000008be108ff][start=48][count=1]
[   68.628481] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   68.628483] [vq=00000000abd097cd][start=38][count=1]
[   68.628517] [vq=000000008be108ff][vq->ndescs=1]
[   68.628518] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628520] [vq=000000008be108ff][vq->ndescs=0]
[   68.628522] [vq=000000008be108ff][start=49][count=1]
[   68.628648] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   68.628650] [vq=00000000abd097cd][start=39][count=1]
[   68.628683] [vq=000000008be108ff][vq->ndescs=1]
[   68.628684] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628686] [vq=000000008be108ff][vq->ndescs=0]
[   68.628688] [vq=000000008be108ff][start=50][count=1]
[   68.628814] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   68.628816] [vq=00000000abd097cd][start=40][count=1]
[   68.628847] [vq=000000008be108ff][vq->ndescs=1]
[   68.628848] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628849] [vq=000000008be108ff][vq->ndescs=0]
[   68.628851] [vq=000000008be108ff][start=51][count=1]
[   68.628977] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   68.628979] [vq=00000000abd097cd][start=41][count=1]
[   68.629012] [vq=000000008be108ff][vq->ndescs=1]
[   68.629013] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.629014] [vq=000000008be108ff][vq->ndescs=0]
[   68.629017] [vq=000000008be108ff][start=52][count=1]
[   68.629143] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   68.629144] [vq=00000000abd097cd][start=42][count=1]
[   68.629177] [vq=000000008be108ff][vq->ndescs=1]
[   68.629178] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.629179] [vq=000000008be108ff][vq->ndescs=0]
[   68.629182] [vq=000000008be108ff][start=53][count=1]
[   68.629310] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   68.629311] [vq=00000000abd097cd][start=43][count=1]
[   68.629344] [vq=000000008be108ff][vq->ndescs=1]
[   68.629345] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.629346] [vq=000000008be108ff][vq->ndescs=0]
[   68.629349] [vq=000000008be108ff][start=54][count=1]
[   68.629490] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   68.629495] [vq=00000000abd097cd][start=44][count=1]
[   68.713536] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   68.713539] [vq=00000000abd097cd][start=45][count=1]
[   68.725293] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   68.725295] [vq=00000000abd097cd][start=46][count=1]
[   68.748355] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   68.748359] [vq=00000000abd097cd][start=47][count=1]
[   68.805702] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   68.805704] [vq=00000000abd097cd][start=48][count=1]
[   68.911555] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   68.911558] [vq=00000000abd097cd][start=49][count=1]
[   69.133394] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   69.133397] [vq=00000000abd097cd][start=50][count=1]
[   69.267403] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   69.267405] [vq=00000000abd097cd][start=51][count=1]
[   69.442123] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   69.442127] [vq=00000000abd097cd][start=52][count=1]
[   69.458741] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   69.458743] [vq=00000000abd097cd][start=53][count=1]
[   69.726266] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   69.726270] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   69.726272] [vq=00000000abd097cd][start=54][count=2]
[   69.736761] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   69.736764] [vq=00000000abd097cd][start=56][count=1]
[   69.826013] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   69.826014] [vq=00000000abd097cd][start=57][count=1]
[   69.951440] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   69.951442] [vq=00000000abd097cd][start=58][count=1]
[   70.110704] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   70.110706] [vq=00000000abd097cd][start=59][count=1]
[   70.317450] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   70.317452] [vq=00000000abd097cd][start=60][count=1]
[   70.463806] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   70.463808] [vq=00000000abd097cd][start=61][count=1]
[   70.491573] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   70.491575] [vq=00000000abd097cd][start=62][count=1]
[   70.727266] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   70.727269] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   70.727271] [vq=00000000abd097cd][start=63][count=2]
[   70.776670] [vq=00000000abd097cd][vq->ndescs=65]
[   70.776671] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   70.776673] [vq=00000000abd097cd][start=65][count=1]
[   70.991537] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   70.991539] [vq=00000000abd097cd][start=66][count=1]
[   71.208420] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   71.208423] [vq=00000000abd097cd][start=67][count=1]
[   71.347437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   71.347439] [vq=00000000abd097cd][start=68][count=1]
[   71.503869] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   71.503871] [vq=00000000abd097cd][start=69][count=1]
[   71.728241] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   71.728244] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   71.728246] [vq=00000000abd097cd][start=70][count=2]
[   74.661409] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   74.661417] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   74.661419] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   74.661421] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   74.661423] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   74.661424] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   74.661425] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   74.661427] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   74.661429] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   74.661432] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   74.661433] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   74.661435] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   74.661437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   74.661438] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   74.661440] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   74.661442] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   74.661444] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   74.661446] [vq=00000000abd097cd][start=0][count=17]

Christian Borntraeger

2020-Feb-11 10:07 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 11.02.20 10:56, Christian Borntraeger wrote:> 
> 
> On 11.02.20 10:33, Eugenio P?rez wrote:
>> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>>
>>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>>> Hi Christian.
>>>>
>>>> I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>>> Your configuration (libvirt or qemu line), and host's dmesg
output if any?
>>>>
>>>> Thanks!
>>>
>>> If it was not obvious, this is on s390x, a big endian system.
>>>
>>
>> Hi Christian. Thank you very much for the hints.
>>
>> Could we add some debug traces? Something like the inline patch should
give us some clues.
> 
> attached. this time the guest crashed (and no guest moved message)
> 
And here with Guest moved vq after some reboots.
-------------- next part --------------
[   74.661450] [vq=000000008be108ff][vq->ndescs=0]
[  757.654136] [vq=00000000201c16ab][vq->ndescs=0]
[  757.654142] [vq=00000000465a3846][vq->ndescs=0]
[  758.283378] [vq=00000000201c16ab][vq->ndescs=65]
[  758.283381] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  758.283385] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  758.283386] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  758.283388] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  758.283390] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  758.283391] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  758.283393] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  758.283394] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  758.283397] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  758.283399] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  758.283401] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  758.283403] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  758.283406] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  758.283407] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  758.283409] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  758.283410] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  758.283413] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  758.283414] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  758.283417] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  758.283418] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  758.283420] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  758.283422] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  758.283423] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  758.283425] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  758.283426] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  758.283428] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  758.283429] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  758.283431] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  758.283433] [vq=00000000201c16ab][start=0][count=28]
[  758.344947] [vq=00000000465a3846][vq->ndescs=1]
[  758.344948] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.344951] [vq=00000000465a3846][vq->ndescs=0]
[  758.344962] [vq=00000000465a3846][start=0][count=1]
[  758.366414] [vq=00000000465a3846][vq->ndescs=1]
[  758.366415] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.366417] [vq=00000000465a3846][vq->ndescs=0]
[  758.366420] [vq=00000000465a3846][start=1][count=1]
[  758.366913] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  758.366915] [vq=00000000201c16ab][start=28][count=1]
[  758.366917] [vq=00000000465a3846][vq->ndescs=1]
[  758.366918] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.366919] [vq=00000000465a3846][vq->ndescs=0]
[  758.366924] [vq=00000000465a3846][start=2][count=1]
[  758.366943] [vq=00000000465a3846][vq->ndescs=2]
[  758.366944] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  758.366946] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.366947] [vq=00000000465a3846][vq->ndescs=0]
[  758.366951] [vq=00000000465a3846][start=3][count=2]
[  758.369847] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  758.369852] [vq=00000000201c16ab][start=29][count=1]
[  758.382143] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  758.382145] [vq=00000000201c16ab][start=30][count=1]
[  758.421531] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  758.421534] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  758.421536] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  758.421537] [vq=00000000201c16ab][start=31][count=3]
[  758.480414] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  758.480418] [vq=00000000201c16ab][start=34][count=1]
[  758.584968] [vq=00000000465a3846][vq->ndescs=2]
[  758.584973] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.584975] [vq=00000000465a3846][vq->ndescs=0]
[  758.584987] [vq=00000000465a3846][start=5][count=1]
[  758.629880] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  758.629883] [vq=00000000201c16ab][start=35][count=1]
[  758.696982] [vq=00000000465a3846][vq->ndescs=1]
[  758.696985] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.696987] [vq=00000000465a3846][vq->ndescs=0]
[  758.696994] [vq=00000000465a3846][start=6][count=1]
[  758.788476] [vq=00000000465a3846][vq->ndescs=2]
[  758.788478] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.788480] [vq=00000000465a3846][vq->ndescs=0]
[  758.788488] [vq=00000000465a3846][start=7][count=1]
[  758.818471] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  758.818475] [vq=00000000201c16ab][start=36][count=1]
[  758.864048] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  758.864051] [vq=00000000201c16ab][start=37][count=1]
[  758.912219] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  758.912221] [vq=00000000201c16ab][start=38][count=1]
[  758.916937] [vq=00000000465a3846][vq->ndescs=1]
[  758.916939] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.916940] [vq=00000000465a3846][vq->ndescs=0]
[  758.916949] [vq=00000000465a3846][start=8][count=1]
[  758.992422] [vq=00000000465a3846][vq->ndescs=2]
[  758.992425] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.992426] [vq=00000000465a3846][vq->ndescs=0]
[  758.992431] [vq=00000000465a3846][start=9][count=1]
[  759.026942] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  759.026944] [vq=00000000201c16ab][start=39][count=1]
[  759.106290] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  759.106293] [vq=00000000201c16ab][start=40][count=1]
[  759.176843] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  759.176846] [vq=00000000201c16ab][start=41][count=1]
[  759.192881] [vq=00000000465a3846][vq->ndescs=2]
[  759.192884] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  759.192886] [vq=00000000465a3846][vq->ndescs=0]
[  759.192896] [vq=00000000465a3846][start=10][count=1]
[  759.215142] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  759.215144] [vq=00000000201c16ab][start=42][count=1]
[  759.277701] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  759.277704] [vq=00000000201c16ab][start=43][count=1]
[  759.422515] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  759.422518] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  759.422520] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  759.422521] [vq=00000000201c16ab][start=44][count=3]
[  759.422792] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  759.422794] [vq=00000000201c16ab][start=47][count=1]
[  759.626994] [vq=00000000465a3846][vq->ndescs=1]
[  759.626997] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  759.626999] [vq=00000000465a3846][vq->ndescs=0]
[  759.627008] [vq=00000000465a3846][start=11][count=1]
[  759.904224] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  759.904232] [vq=00000000201c16ab][start=48][count=1]
[  759.945942] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  759.945944] [vq=00000000201c16ab][start=49][count=1]
[  759.951344] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  759.951346] [vq=00000000201c16ab][start=50][count=1]
[  760.307838] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  760.307841] [vq=00000000201c16ab][start=51][count=1]
[  760.344981] [vq=00000000465a3846][vq->ndescs=1]
[  760.344983] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.344985] [vq=00000000465a3846][vq->ndescs=0]
[  760.344994] [vq=00000000465a3846][start=12][count=1]
[  760.351941] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  760.351943] [vq=00000000201c16ab][start=52][count=1]
[  760.423655] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  760.423657] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  760.423658] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  760.423659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  760.423661] [vq=00000000201c16ab][start=53][count=4]
[  760.423903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  760.423905] [vq=00000000201c16ab][start=57][count=1]
[  760.633527] [vq=00000000465a3846][vq->ndescs=2]
[  760.633529] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  760.633530] [vq=00000000465a3846][vq->ndescs=0]
[  760.633535] [vq=00000000465a3846][start=13][count=1]
[  760.666987] [vq=00000000465a3846][vq->ndescs=1]
[  760.666988] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.666990] [vq=00000000465a3846][vq->ndescs=0]
[  760.666993] [vq=00000000465a3846][start=14][count=1]
[  760.672771] [vq=00000000465a3846][vq->ndescs=1]
[  760.672772] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.672773] [vq=00000000465a3846][vq->ndescs=0]
[  760.672777] [vq=00000000465a3846][start=15][count=1]
[  760.673253] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  760.673254] [vq=00000000201c16ab][start=58][count=1]
[  760.673277] [vq=00000000465a3846][vq->ndescs=1]
[  760.673278] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673279] [vq=00000000465a3846][vq->ndescs=0]
[  760.673282] [vq=00000000465a3846][start=16][count=1]
[  760.673475] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  760.673476] [vq=00000000201c16ab][start=59][count=1]
[  760.673592] [vq=00000000465a3846][vq->ndescs=1]
[  760.673593] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673594] [vq=00000000465a3846][vq->ndescs=0]
[  760.673597] [vq=00000000465a3846][start=17][count=1]
[  760.673751] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  760.673753] [vq=00000000201c16ab][start=60][count=1]
[  760.673800] [vq=00000000465a3846][vq->ndescs=1]
[  760.673801] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673803] [vq=00000000465a3846][vq->ndescs=0]
[  760.673805] [vq=00000000465a3846][start=18][count=1]
[  760.673944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  760.673946] [vq=00000000201c16ab][start=61][count=1]
[  760.673986] [vq=00000000465a3846][vq->ndescs=1]
[  760.673987] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673989] [vq=00000000465a3846][vq->ndescs=0]
[  760.673992] [vq=00000000465a3846][start=19][count=1]
[  760.674132] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  760.674133] [vq=00000000201c16ab][start=62][count=1]
[  760.674175] [vq=00000000465a3846][vq->ndescs=1]
[  760.674176] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674177] [vq=00000000465a3846][vq->ndescs=0]
[  760.674180] [vq=00000000465a3846][start=20][count=1]
[  760.674316] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  760.674318] [vq=00000000201c16ab][start=63][count=1]
[  760.674358] [vq=00000000465a3846][vq->ndescs=1]
[  760.674359] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674360] [vq=00000000465a3846][vq->ndescs=0]
[  760.674363] [vq=00000000465a3846][start=21][count=1]
[  760.674502] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  760.674504] [vq=00000000201c16ab][start=64][count=1]
[  760.674548] [vq=00000000465a3846][vq->ndescs=1]
[  760.674549] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674550] [vq=00000000465a3846][vq->ndescs=0]
[  760.674553] [vq=00000000465a3846][start=22][count=1]
[  760.674692] [vq=00000000201c16ab][vq->ndescs=65]
[  760.674693] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  760.674694] [vq=00000000201c16ab][start=65][count=1]
[  760.674734] [vq=00000000465a3846][vq->ndescs=1]
[  760.674735] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674737] [vq=00000000465a3846][vq->ndescs=0]
[  760.674739] [vq=00000000465a3846][start=23][count=1]
[  760.674877] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  760.674878] [vq=00000000201c16ab][start=66][count=1]
[  760.674926] [vq=00000000465a3846][vq->ndescs=1]
[  760.674927] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674929] [vq=00000000465a3846][vq->ndescs=0]
[  760.674931] [vq=00000000465a3846][start=24][count=1]
[  760.675082] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  760.675083] [vq=00000000201c16ab][start=67][count=1]
[  760.675124] [vq=00000000465a3846][vq->ndescs=1]
[  760.675125] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675127] [vq=00000000465a3846][vq->ndescs=0]
[  760.675129] [vq=00000000465a3846][start=25][count=1]
[  760.675266] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  760.675268] [vq=00000000201c16ab][start=68][count=1]
[  760.675320] [vq=00000000465a3846][vq->ndescs=1]
[  760.675321] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675322] [vq=00000000465a3846][vq->ndescs=0]
[  760.675325] [vq=00000000465a3846][start=26][count=1]
[  760.675465] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  760.675466] [vq=00000000201c16ab][start=69][count=1]
[  760.675508] [vq=00000000465a3846][vq->ndescs=1]
[  760.675509] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675510] [vq=00000000465a3846][vq->ndescs=0]
[  760.675513] [vq=00000000465a3846][start=27][count=1]
[  760.675659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  760.675661] [vq=00000000201c16ab][start=70][count=1]
[  760.675703] [vq=00000000465a3846][vq->ndescs=1]
[  760.675704] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675706] [vq=00000000465a3846][vq->ndescs=0]
[  760.675708] [vq=00000000465a3846][start=28][count=1]
[  760.675845] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  760.675846] [vq=00000000201c16ab][start=71][count=1]
[  760.675888] [vq=00000000465a3846][vq->ndescs=1]
[  760.675889] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675890] [vq=00000000465a3846][vq->ndescs=0]
[  760.675893] [vq=00000000465a3846][start=29][count=1]
[  760.676031] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  760.676032] [vq=00000000201c16ab][start=72][count=1]
[  760.676073] [vq=00000000465a3846][vq->ndescs=1]
[  760.676074] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676076] [vq=00000000465a3846][vq->ndescs=0]
[  760.676079] [vq=00000000465a3846][start=30][count=1]
[  760.676220] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  760.676221] [vq=00000000201c16ab][start=73][count=1]
[  760.676261] [vq=00000000465a3846][vq->ndescs=1]
[  760.676262] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676263] [vq=00000000465a3846][vq->ndescs=0]
[  760.676266] [vq=00000000465a3846][start=31][count=1]
[  760.676406] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  760.676407] [vq=00000000201c16ab][start=74][count=1]
[  760.676447] [vq=00000000465a3846][vq->ndescs=1]
[  760.676448] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676450] [vq=00000000465a3846][vq->ndescs=0]
[  760.676452] [vq=00000000465a3846][start=32][count=1]
[  760.676590] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  760.676591] [vq=00000000201c16ab][start=75][count=1]
[  760.676631] [vq=00000000465a3846][vq->ndescs=1]
[  760.676632] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676634] [vq=00000000465a3846][vq->ndescs=0]
[  760.676636] [vq=00000000465a3846][start=33][count=1]
[  760.676775] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  760.676777] [vq=00000000201c16ab][start=76][count=1]
[  760.676817] [vq=00000000465a3846][vq->ndescs=1]
[  760.676818] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676819] [vq=00000000465a3846][vq->ndescs=0]
[  760.676822] [vq=00000000465a3846][start=34][count=1]
[  760.676961] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  760.676963] [vq=00000000201c16ab][start=77][count=1]
[  760.677005] [vq=00000000465a3846][vq->ndescs=1]
[  760.677006] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.677007] [vq=00000000465a3846][vq->ndescs=0]
[  760.677011] [vq=00000000465a3846][start=35][count=1]
[  760.677152] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  760.677153] [vq=00000000201c16ab][start=78][count=1]
[  760.678377] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  760.678379] [vq=00000000201c16ab][start=79][count=1]
[  760.696916] [vq=00000000465a3846][vq->ndescs=1]
[  760.696918] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.696919] [vq=00000000465a3846][vq->ndescs=0]
[  760.696922] [vq=00000000465a3846][start=36][count=1]
[  760.731769] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  760.731771] [vq=00000000201c16ab][start=80][count=1]
[  760.906976] [vq=00000000465a3846][vq->ndescs=1]
[  760.906980] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.906982] [vq=00000000465a3846][vq->ndescs=0]
[  760.906992] [vq=00000000465a3846][start=37][count=1]
[  761.001377] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  761.001383] [vq=00000000201c16ab][start=81][count=1]
[  761.025300] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  761.025304] [vq=00000000201c16ab][start=82][count=1]
[  761.088194] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  761.088196] [vq=00000000201c16ab][start=83][count=1]
[  761.096984] [vq=00000000465a3846][vq->ndescs=1]
[  761.096985] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  761.096987] [vq=00000000465a3846][vq->ndescs=0]
[  761.096995] [vq=00000000465a3846][start=38][count=1]
[  761.347842] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  761.347846] [vq=00000000201c16ab][start=84][count=1]
[  761.424800] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  761.424803] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  761.424805] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  761.424807] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  761.424809] [vq=00000000201c16ab][start=85][count=4]
[  761.848308] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  761.848310] [vq=00000000201c16ab][start=89][count=1]
[  761.926827] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  761.926829] [vq=00000000201c16ab][start=90][count=1]
[  761.960724] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  761.960727] [vq=00000000201c16ab][start=91][count=1]
[  762.136662] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  762.136664] [vq=00000000201c16ab][start=92][count=1]
[  762.387928] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  762.387931] [vq=00000000201c16ab][start=93][count=1]
[  762.425545] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  762.425547] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  762.425548] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  762.425550] [vq=00000000201c16ab][start=94][count=3]
[  762.425829] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  762.425830] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  762.425832] [vq=00000000201c16ab][start=97][count=2]
[  762.554053] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  762.554055] [vq=00000000201c16ab][start=99][count=1]
[  762.804248] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  762.804250] [vq=00000000201c16ab][start=100][count=1]
[  762.863934] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  762.863936] [vq=00000000201c16ab][start=101][count=1]
[  763.176763] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  763.176766] [vq=00000000201c16ab][start=102][count=1]
[  763.220580] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  763.220582] [vq=00000000201c16ab][start=103][count=1]
[  763.371347] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  763.371350] [vq=00000000201c16ab][start=104][count=1]
[  763.371376] [vq=00000000465a3846][vq->ndescs=1]
[  763.371379] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  763.371380] [vq=00000000465a3846][vq->ndescs=0]
[  763.371384] [vq=00000000465a3846][start=39][count=1]
[  763.426476] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  763.426479] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  763.426481] [vq=00000000201c16ab][start=105][count=2]
[  763.426783] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  763.426785] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  763.426786] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  763.426788] [vq=00000000201c16ab][start=107][count=3]
[  763.784998] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  763.785001] [vq=00000000201c16ab][start=110][count=1]
[  763.871337] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  763.871340] [vq=00000000201c16ab][start=111][count=1]
[  763.903944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  763.903946] [vq=00000000201c16ab][start=112][count=1]
[  766.596032] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  766.596036] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  766.596038] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  766.596040] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  766.596041] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  766.596043] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  766.596044] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  766.596046] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  766.596047] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  766.596049] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  766.596051] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  766.596053] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  766.596054] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  766.596056] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  766.596058] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  766.596059] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  766.596061] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  766.596063] [vq=00000000201c16ab][vq->ndescs=0]
[  766.596064] [vq=00000000201c16ab][start=0][count=17]
[  766.596067] [vq=00000000465a3846][vq->ndescs=0]
[  767.304752] [vq=00000000201c16ab][vq->ndescs=65]
[  767.304754] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  767.304756] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  767.304757] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  767.304760] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  767.304762] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  767.304763] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  767.304765] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  767.304767] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  767.304768] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  767.304770] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  767.304771] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  767.304773] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  767.304774] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  767.304776] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  767.304777] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  767.304779] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  767.304781] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  767.304782] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  767.304784] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  767.304785] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  767.304787] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  767.304789] [vq=00000000201c16ab][start=17][count=21]
[  767.345024] [vq=00000000465a3846][vq->ndescs=1]
[  767.345025] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.345027] [vq=00000000465a3846][vq->ndescs=0]
[  767.345035] [vq=00000000465a3846][start=0][count=1]
[  767.364564] [vq=00000000465a3846][vq->ndescs=1]
[  767.364566] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.364567] [vq=00000000465a3846][vq->ndescs=0]
[  767.364571] [vq=00000000465a3846][start=1][count=1]
[  767.365202] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  767.365209] [vq=00000000201c16ab][start=38][count=1]
[  767.365235] [vq=00000000465a3846][vq->ndescs=2]
[  767.365237] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  767.365239] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.365241] [vq=00000000465a3846][vq->ndescs=0]
[  767.365249] [vq=00000000465a3846][start=2][count=2]
[  767.365433] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  767.365435] [vq=00000000201c16ab][start=39][count=1]
[  767.365448] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  767.365449] [vq=00000000201c16ab][start=40][count=1]
[  767.370391] [vq=00000000465a3846][vq->ndescs=1]
[  767.370392] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.370394] [vq=00000000465a3846][vq->ndescs=0]
[  767.370398] [vq=00000000465a3846][start=4][count=1]
[  767.430616] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  767.430620] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  767.430622] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  767.430624] [vq=00000000201c16ab][start=41][count=3]
[  767.430875] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  767.430876] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  767.430878] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  767.430879] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  767.430880] [vq=00000000201c16ab][start=44][count=4]
[  767.571839] [vq=00000000465a3846][vq->ndescs=2]
[  767.571842] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.571844] [vq=00000000465a3846][vq->ndescs=0]
[  767.571851] [vq=00000000465a3846][start=5][count=1]
[  767.597747] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  767.597750] [vq=00000000201c16ab][start=48][count=1]
[  767.650396] [vq=00000000465a3846][vq->ndescs=1]
[  767.650398] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.650399] [vq=00000000465a3846][vq->ndescs=0]
[  767.650406] [vq=00000000465a3846][start=6][count=1]
[  767.749947] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  767.749952] [vq=00000000201c16ab][start=49][count=1]
[  767.773664] [vq=00000000465a3846][vq->ndescs=2]
[  767.773666] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.773667] [vq=00000000465a3846][vq->ndescs=0]
[  767.773673] [vq=00000000465a3846][start=7][count=1]
[  767.807483] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  767.807485] [vq=00000000201c16ab][start=50][count=1]
[  767.903813] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  767.903816] [vq=00000000201c16ab][start=51][count=1]
[  767.975773] [vq=00000000465a3846][vq->ndescs=2]
[  767.975774] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.975776] [vq=00000000465a3846][vq->ndescs=0]
[  767.975781] [vq=00000000465a3846][start=8][count=1]
[  768.007409] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  768.007411] [vq=00000000201c16ab][start=52][count=1]
[  768.031512] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  768.031514] [vq=00000000201c16ab][start=53][count=1]
[  768.067894] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  768.067896] [vq=00000000201c16ab][start=54][count=1]
[  768.075648] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  768.075650] [vq=00000000201c16ab][start=55][count=1]
[  768.090397] [vq=00000000465a3846][vq->ndescs=1]
[  768.090398] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  768.090400] [vq=00000000465a3846][vq->ndescs=0]
[  768.090406] [vq=00000000465a3846][start=9][count=1]
[  768.140369] [vq=00000000465a3846][vq->ndescs=1]
[  768.140371] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  768.140373] [vq=00000000465a3846][vq->ndescs=0]
[  768.140378] [vq=00000000465a3846][start=10][count=1]
[  768.176600] [vq=00000000465a3846][vq->ndescs=2]
[  768.176601] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  768.176603] [vq=00000000465a3846][vq->ndescs=0]
[  768.176606] [vq=00000000465a3846][start=11][count=1]
[  768.217108] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  768.217111] [vq=00000000201c16ab][start=56][count=1]
[  768.431553] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  768.431560] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  768.431563] [vq=00000000201c16ab][start=57][count=2]
[  768.431869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  768.431870] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  768.431871] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  768.431872] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  768.431874] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  768.431875] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  768.431876] [vq=00000000201c16ab][start=59][count=6]
[  768.858404] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  768.858412] [vq=00000000201c16ab][start=65][count=1]
[  769.065857] [vq=00000000465a3846][vq->ndescs=1]
[  769.065859] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.065862] [vq=00000000465a3846][vq->ndescs=0]
[  769.065871] [vq=00000000465a3846][start=12][count=1]
[  769.066299] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  769.066301] [vq=00000000201c16ab][start=66][count=1]
[  769.066329] [vq=00000000465a3846][vq->ndescs=1]
[  769.066330] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066331] [vq=00000000465a3846][vq->ndescs=0]
[  769.066335] [vq=00000000465a3846][start=13][count=1]
[  769.066514] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  769.066515] [vq=00000000201c16ab][start=67][count=1]
[  769.066617] [vq=00000000465a3846][vq->ndescs=1]
[  769.066618] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066619] [vq=00000000465a3846][vq->ndescs=0]
[  769.066622] [vq=00000000465a3846][start=14][count=1]
[  769.066765] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  769.066769] [vq=00000000201c16ab][start=68][count=1]
[  769.066828] [vq=00000000465a3846][vq->ndescs=1]
[  769.066829] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066831] [vq=00000000465a3846][vq->ndescs=0]
[  769.066835] [vq=00000000465a3846][start=15][count=1]
[  769.066967] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  769.066968] [vq=00000000201c16ab][start=69][count=1]
[  769.067024] [vq=00000000465a3846][vq->ndescs=1]
[  769.067025] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067026] [vq=00000000465a3846][vq->ndescs=0]
[  769.067029] [vq=00000000465a3846][start=16][count=1]
[  769.067170] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  769.067172] [vq=00000000201c16ab][start=70][count=1]
[  769.067224] [vq=00000000465a3846][vq->ndescs=1]
[  769.067225] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067226] [vq=00000000465a3846][vq->ndescs=0]
[  769.067229] [vq=00000000465a3846][start=17][count=1]
[  769.067365] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  769.067366] [vq=00000000201c16ab][start=71][count=1]
[  769.067416] [vq=00000000465a3846][vq->ndescs=1]
[  769.067417] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067418] [vq=00000000465a3846][vq->ndescs=0]
[  769.067421] [vq=00000000465a3846][start=18][count=1]
[  769.067557] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  769.067558] [vq=00000000201c16ab][start=72][count=1]
[  769.067605] [vq=00000000465a3846][vq->ndescs=1]
[  769.067606] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067607] [vq=00000000465a3846][vq->ndescs=0]
[  769.067610] [vq=00000000465a3846][start=19][count=1]
[  769.067758] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  769.067759] [vq=00000000201c16ab][start=73][count=1]
[  769.067805] [vq=00000000465a3846][vq->ndescs=1]
[  769.067806] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067807] [vq=00000000465a3846][vq->ndescs=0]
[  769.067811] [vq=00000000465a3846][start=20][count=1]
[  769.067959] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  769.067961] [vq=00000000201c16ab][start=74][count=1]
[  769.068045] [vq=00000000465a3846][vq->ndescs=1]
[  769.068047] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068049] [vq=00000000465a3846][vq->ndescs=0]
[  769.068052] [vq=00000000465a3846][start=21][count=1]
[  769.068181] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  769.068186] [vq=00000000201c16ab][start=75][count=1]
[  769.068241] [vq=00000000465a3846][vq->ndescs=1]
[  769.068242] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068244] [vq=00000000465a3846][vq->ndescs=0]
[  769.068250] [vq=00000000465a3846][start=22][count=1]
[  769.068379] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  769.068381] [vq=00000000201c16ab][start=76][count=1]
[  769.068425] [vq=00000000465a3846][vq->ndescs=1]
[  769.068426] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068428] [vq=00000000465a3846][vq->ndescs=0]
[  769.068432] [vq=00000000465a3846][start=23][count=1]
[  769.068555] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  769.068557] [vq=00000000201c16ab][start=77][count=1]
[  769.068599] [vq=00000000465a3846][vq->ndescs=1]
[  769.068600] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068605] [vq=00000000465a3846][vq->ndescs=0]
[  769.068608] [vq=00000000465a3846][start=24][count=1]
[  769.068737] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  769.068739] [vq=00000000201c16ab][start=78][count=1]
[  769.068781] [vq=00000000465a3846][vq->ndescs=1]
[  769.068782] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068784] [vq=00000000465a3846][vq->ndescs=0]
[  769.068787] [vq=00000000465a3846][start=25][count=1]
[  769.068914] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  769.068916] [vq=00000000201c16ab][start=79][count=1]
[  769.068959] [vq=00000000465a3846][vq->ndescs=1]
[  769.068960] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068962] [vq=00000000465a3846][vq->ndescs=0]
[  769.068965] [vq=00000000465a3846][start=26][count=1]
[  769.069094] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  769.069097] [vq=00000000201c16ab][start=80][count=1]
[  769.069139] [vq=00000000465a3846][vq->ndescs=1]
[  769.069140] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069143] [vq=00000000465a3846][vq->ndescs=0]
[  769.069145] [vq=00000000465a3846][start=27][count=1]
[  769.069271] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  769.069274] [vq=00000000201c16ab][start=81][count=1]
[  769.069316] [vq=00000000465a3846][vq->ndescs=1]
[  769.069317] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069319] [vq=00000000465a3846][vq->ndescs=0]
[  769.069322] [vq=00000000465a3846][start=28][count=1]
[  769.069452] [vq=00000000201c16ab][vq->ndescs=65]
[  769.069453] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  769.069455] [vq=00000000201c16ab][start=82][count=1]
[  769.069499] [vq=00000000465a3846][vq->ndescs=1]
[  769.069500] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069501] [vq=00000000465a3846][vq->ndescs=0]
[  769.069504] [vq=00000000465a3846][start=29][count=1]
[  769.069650] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  769.069654] [vq=00000000201c16ab][start=83][count=1]
[  769.069711] [vq=00000000465a3846][vq->ndescs=1]
[  769.069712] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069714] [vq=00000000465a3846][vq->ndescs=0]
[  769.069718] [vq=00000000465a3846][start=30][count=1]
[  769.069851] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  769.069852] [vq=00000000201c16ab][start=84][count=1]
[  769.069905] [vq=00000000465a3846][vq->ndescs=1]
[  769.069906] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069908] [vq=00000000465a3846][vq->ndescs=0]
[  769.069910] [vq=00000000465a3846][start=31][count=1]
[  769.070045] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  769.070046] [vq=00000000201c16ab][start=85][count=1]
[  769.070115] [vq=00000000465a3846][vq->ndescs=1]
[  769.070116] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.070117] [vq=00000000465a3846][vq->ndescs=0]
[  769.070120] [vq=00000000465a3846][start=32][count=1]
[  769.070253] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  769.070254] [vq=00000000201c16ab][start=86][count=1]
[  769.093873] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  769.093878] [vq=00000000201c16ab][start=87][count=1]
[  769.107789] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  769.107791] [vq=00000000201c16ab][start=88][count=1]
[  769.130423] [vq=00000000465a3846][vq->ndescs=1]
[  769.130424] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.130426] [vq=00000000465a3846][vq->ndescs=0]
[  769.130437] [vq=00000000465a3846][start=33][count=1]
[  769.133437] [vq=00000000465a3846][vq->ndescs=1]
[  769.133438] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.133440] [vq=00000000465a3846][vq->ndescs=0]
[  769.133443] [vq=00000000465a3846][start=34][count=1]
[  769.170381] [vq=00000000465a3846][vq->ndescs=1]
[  769.170383] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.170384] [vq=00000000465a3846][vq->ndescs=0]
[  769.170388] [vq=00000000465a3846][start=35][count=1]
[  769.370375] [vq=00000000465a3846][vq->ndescs=1]
[  769.370377] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.370380] [vq=00000000465a3846][vq->ndescs=0]
[  769.370386] [vq=00000000465a3846][start=36][count=1]
[  769.432770] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  769.432773] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  769.432775] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  769.432776] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  769.432778] [vq=00000000201c16ab][start=89][count=4]
[  769.432999] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  769.433001] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  769.433003] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  769.433005] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  769.433006] [vq=00000000201c16ab][start=93][count=4]
[  769.500358] [vq=00000000465a3846][vq->ndescs=1]
[  769.500361] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.500363] [vq=00000000465a3846][vq->ndescs=0]
[  769.500368] [vq=00000000465a3846][start=37][count=1]
[  769.855119] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  769.855123] [vq=00000000201c16ab][start=97][count=1]
[  769.855886] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  769.855888] [vq=00000000201c16ab][start=98][count=1]
[  770.136903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  770.136906] [vq=00000000201c16ab][start=99][count=1]
[  770.147613] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  770.147616] [vq=00000000201c16ab][start=100][count=1]
[  770.433405] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  770.433408] [vq=00000000201c16ab][start=101][count=1]
[  770.433716] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  770.433718] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  770.433720] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  770.433721] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  770.433723] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  770.433724] [vq=00000000201c16ab][start=102][count=5]
[  770.433903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  770.433904] [vq=00000000201c16ab][start=107][count=1]
[  770.830869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  770.830874] [vq=00000000201c16ab][start=108][count=1]
[  770.863937] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  770.863939] [vq=00000000201c16ab][start=109][count=1]
[  770.921316] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  770.921319] [vq=00000000201c16ab][start=110][count=1]
[  771.176836] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  771.176839] [vq=00000000201c16ab][start=111][count=1]
[  771.434412] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  771.434415] [vq=00000000201c16ab][start=112][count=1]
[  771.434701] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  771.434704] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  771.434705] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  771.434707] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  771.434708] [vq=00000000201c16ab][start=113][count=4]
[  771.434885] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  771.434887] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  771.434888] [vq=00000000201c16ab][start=117][count=2]
[  771.904063] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  771.904066] [vq=00000000201c16ab][start=119][count=1]
[  771.961232] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  771.961234] [vq=00000000201c16ab][start=120][count=1]
[  772.435587] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  772.435590] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  772.435592] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  772.435593] [vq=00000000201c16ab][start=121][count=3]
[  772.435862] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  772.435864] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  772.435865] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  772.435867] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  772.435868] [vq=00000000201c16ab][start=124][count=4]
[  775.057134] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  775.057139] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  775.057141] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  775.057143] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  775.057144] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  775.057146] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  775.057148] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  775.057149] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  775.057151] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  775.057153] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  775.057155] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  775.057156] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  775.057158] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  775.057160] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  775.057162] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  775.057163] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  775.057165] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  775.057166] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  775.057168] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  775.057170] [vq=00000000201c16ab][vq->ndescs=0]
[  775.057172] [vq=00000000201c16ab][start=0][count=19]
[  775.057175] [vq=00000000465a3846][vq->ndescs=0]
[  775.746716] [vq=00000000201c16ab][vq->ndescs=65]
[  775.746717] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  775.746719] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  775.746721] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  775.746722] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  775.746723] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  775.746725] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  775.746726] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  775.746727] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  775.746729] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  775.746730] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  775.746731] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  775.746732] [vq=00000000201c16ab][start=19][count=11]
[  775.779189] [vq=00000000465a3846][vq->ndescs=1]
[  775.779191] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.779193] [vq=00000000465a3846][vq->ndescs=0]
[  775.779201] [vq=00000000465a3846][start=0][count=1]
[  775.794833] [vq=00000000465a3846][vq->ndescs=1]
[  775.794835] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.794836] [vq=00000000465a3846][vq->ndescs=0]
[  775.794842] [vq=00000000465a3846][start=1][count=1]
[  775.803247] [vq=00000000465a3846][vq->ndescs=1]
[  775.803248] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.803249] [vq=00000000465a3846][vq->ndescs=0]
[  775.803253] [vq=00000000465a3846][start=2][count=1]
[  775.803932] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  775.803938] [vq=00000000201c16ab][start=30][count=1]
[  775.803977] [vq=00000000465a3846][vq->ndescs=2]
[  775.803998] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  775.804000] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  775.804002] [vq=00000000465a3846][vq->ndescs=0]
[  775.804010] [vq=00000000465a3846][start=3][count=2]
[  775.804194] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  775.804196] [vq=00000000201c16ab][start=31][count=1]
[  775.804205] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  775.804207] [vq=00000000201c16ab][start=32][count=1]
[  775.913969] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  775.913974] [vq=00000000201c16ab][start=33][count=1]
[  775.997471] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  775.997474] [vq=00000000201c16ab][start=34][count=1]
[  776.006405] [vq=00000000465a3846][vq->ndescs=2]
[  776.006407] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.006409] [vq=00000000465a3846][vq->ndescs=0]
[  776.006416] [vq=00000000465a3846][start=5][count=1]
[  776.036919] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  776.036924] [vq=00000000201c16ab][start=35][count=1]
[  776.046852] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  776.046856] [vq=00000000201c16ab][start=36][count=1]
[  776.061540] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  776.061542] [vq=00000000201c16ab][start=37][count=1]
[  776.111415] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  776.111419] [vq=00000000201c16ab][start=38][count=1]
[  776.154841] [vq=00000000465a3846][vq->ndescs=1]
[  776.154843] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.154845] [vq=00000000465a3846][vq->ndescs=0]
[  776.154850] [vq=00000000465a3846][start=6][count=1]
[  776.209078] [vq=00000000465a3846][vq->ndescs=2]
[  776.209079] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.209080] [vq=00000000465a3846][vq->ndescs=0]
[  776.209083] [vq=00000000465a3846][start=7][count=1]
[  776.234725] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  776.234727] [vq=00000000201c16ab][start=39][count=1]
[  776.306637] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  776.306639] [vq=00000000201c16ab][start=40][count=1]
[  776.394831] [vq=00000000465a3846][vq->ndescs=1]
[  776.394832] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.394833] [vq=00000000465a3846][vq->ndescs=0]
[  776.394837] [vq=00000000465a3846][start=8][count=1]
[  776.411908] [vq=00000000465a3846][vq->ndescs=2]
[  776.411910] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.411911] [vq=00000000465a3846][vq->ndescs=0]
[  776.411914] [vq=00000000465a3846][start=9][count=1]
[  776.439659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  776.439660] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  776.439661] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  776.439663] [vq=00000000201c16ab][start=41][count=3]
[  776.439929] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  776.439931] [vq=00000000201c16ab][start=44][count=1]
[  776.448982] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  776.448984] [vq=00000000201c16ab][start=45][count=1]
[  776.504845] [vq=00000000465a3846][vq->ndescs=1]
[  776.504846] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.504848] [vq=00000000465a3846][vq->ndescs=0]
[  776.504851] [vq=00000000465a3846][start=10][count=1]
[  776.615546] [vq=00000000465a3846][vq->ndescs=2]
[  776.615548] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.615549] [vq=00000000465a3846][vq->ndescs=0]
[  776.615551] [vq=00000000465a3846][start=11][count=1]
[  776.650494] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  776.650495] [vq=00000000201c16ab][start=46][count=1]
[  777.099061] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  777.099068] [vq=00000000201c16ab][start=47][count=1]
[  777.151245] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  777.151247] [vq=00000000201c16ab][start=48][count=1]
[  777.283424] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  777.283427] [vq=00000000201c16ab][start=49][count=1]
[  777.337441] [vq=00000000465a3846][vq->ndescs=1]
[  777.337442] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.337443] [vq=00000000465a3846][vq->ndescs=0]
[  777.337447] [vq=00000000465a3846][start=12][count=1]
[  777.337944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  777.337948] [vq=00000000201c16ab][start=50][count=1]
[  777.337968] [vq=00000000465a3846][vq->ndescs=1]
[  777.337969] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.337971] [vq=00000000465a3846][vq->ndescs=0]
[  777.337975] [vq=00000000465a3846][start=13][count=1]
[  777.338197] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  777.338198] [vq=00000000201c16ab][start=51][count=1]
[  777.338238] [vq=00000000465a3846][vq->ndescs=1]
[  777.338239] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338241] [vq=00000000465a3846][vq->ndescs=0]
[  777.338244] [vq=00000000465a3846][start=14][count=1]
[  777.338382] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  777.338383] [vq=00000000201c16ab][start=52][count=1]
[  777.338415] [vq=00000000465a3846][vq->ndescs=1]
[  777.338416] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338417] [vq=00000000465a3846][vq->ndescs=0]
[  777.338419] [vq=00000000465a3846][start=15][count=1]
[  777.338556] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  777.338557] [vq=00000000201c16ab][start=53][count=1]
[  777.338588] [vq=00000000465a3846][vq->ndescs=1]
[  777.338589] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338600] [vq=00000000465a3846][vq->ndescs=0]
[  777.338603] [vq=00000000465a3846][start=16][count=1]
[  777.338760] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  777.338762] [vq=00000000201c16ab][start=54][count=1]
[  777.338795] [vq=00000000465a3846][vq->ndescs=1]
[  777.338796] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338797] [vq=00000000465a3846][vq->ndescs=0]
[  777.338799] [vq=00000000465a3846][start=17][count=1]
[  777.338939] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  777.338940] [vq=00000000201c16ab][start=55][count=1]
[  777.338975] [vq=00000000465a3846][vq->ndescs=1]
[  777.338977] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338979] [vq=00000000465a3846][vq->ndescs=0]
[  777.338983] [vq=00000000465a3846][start=18][count=1]
[  777.339124] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  777.339126] [vq=00000000201c16ab][start=56][count=1]
[  777.339158] [vq=00000000465a3846][vq->ndescs=1]
[  777.339159] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339160] [vq=00000000465a3846][vq->ndescs=0]
[  777.339163] [vq=00000000465a3846][start=19][count=1]
[  777.339314] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  777.339316] [vq=00000000201c16ab][start=57][count=1]
[  777.339345] [vq=00000000465a3846][vq->ndescs=1]
[  777.339346] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339348] [vq=00000000465a3846][vq->ndescs=0]
[  777.339350] [vq=00000000465a3846][start=20][count=1]
[  777.339489] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  777.339490] [vq=00000000201c16ab][start=58][count=1]
[  777.339521] [vq=00000000465a3846][vq->ndescs=1]
[  777.339522] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339524] [vq=00000000465a3846][vq->ndescs=0]
[  777.339526] [vq=00000000465a3846][start=21][count=1]
[  777.339662] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  777.339664] [vq=00000000201c16ab][start=59][count=1]
[  777.339694] [vq=00000000465a3846][vq->ndescs=1]
[  777.339695] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339696] [vq=00000000465a3846][vq->ndescs=0]
[  777.339699] [vq=00000000465a3846][start=22][count=1]
[  777.339842] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  777.339843] [vq=00000000201c16ab][start=60][count=1]
[  777.339874] [vq=00000000465a3846][vq->ndescs=1]
[  777.339876] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339877] [vq=00000000465a3846][vq->ndescs=0]
[  777.339879] [vq=00000000465a3846][start=23][count=1]
[  777.340015] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  777.340016] [vq=00000000201c16ab][start=61][count=1]
[  777.340045] [vq=00000000465a3846][vq->ndescs=1]
[  777.340046] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340048] [vq=00000000465a3846][vq->ndescs=0]
[  777.340050] [vq=00000000465a3846][start=24][count=1]
[  777.340187] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  777.340188] [vq=00000000201c16ab][start=62][count=1]
[  777.340217] [vq=00000000465a3846][vq->ndescs=1]
[  777.340218] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340220] [vq=00000000465a3846][vq->ndescs=0]
[  777.340222] [vq=00000000465a3846][start=25][count=1]
[  777.340358] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  777.340360] [vq=00000000201c16ab][start=63][count=1]
[  777.340389] [vq=00000000465a3846][vq->ndescs=1]
[  777.340390] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340391] [vq=00000000465a3846][vq->ndescs=0]
[  777.340393] [vq=00000000465a3846][start=26][count=1]
[  777.340528] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  777.340529] [vq=00000000201c16ab][start=64][count=1]
[  777.340559] [vq=00000000465a3846][vq->ndescs=1]
[  777.340560] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340561] [vq=00000000465a3846][vq->ndescs=0]
[  777.340564] [vq=00000000465a3846][start=27][count=1]
[  777.340697] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  777.340698] [vq=00000000201c16ab][start=65][count=1]
[  777.340730] [vq=00000000465a3846][vq->ndescs=1]
[  777.340731] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340732] [vq=00000000465a3846][vq->ndescs=0]
[  777.340734] [vq=00000000465a3846][start=28][count=1]
[  777.340870] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  777.340871] [vq=00000000201c16ab][start=66][count=1]
[  777.340900] [vq=00000000465a3846][vq->ndescs=1]
[  777.340901] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340902] [vq=00000000465a3846][vq->ndescs=0]
[  777.340904] [vq=00000000465a3846][start=29][count=1]
[  777.341038] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  777.341039] [vq=00000000201c16ab][start=67][count=1]
[  777.341068] [vq=00000000465a3846][vq->ndescs=1]
[  777.341069] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341070] [vq=00000000465a3846][vq->ndescs=0]
[  777.341073] [vq=00000000465a3846][start=30][count=1]
[  777.341209] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  777.341210] [vq=00000000201c16ab][start=68][count=1]
[  777.341239] [vq=00000000465a3846][vq->ndescs=1]
[  777.341240] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341242] [vq=00000000465a3846][vq->ndescs=0]
[  777.341244] [vq=00000000465a3846][start=31][count=1]
[  777.341379] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  777.341381] [vq=00000000201c16ab][start=69][count=1]
[  777.341410] [vq=00000000465a3846][vq->ndescs=1]
[  777.341411] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341412] [vq=00000000465a3846][vq->ndescs=0]
[  777.341415] [vq=00000000465a3846][start=32][count=1]
[  777.341550] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  777.341551] [vq=00000000201c16ab][start=70][count=1]
[  777.347869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  777.347874] [vq=00000000201c16ab][start=71][count=1]
[  777.404861] [vq=00000000465a3846][vq->ndescs=1]
[  777.404863] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.404864] [vq=00000000465a3846][vq->ndescs=0]
[  777.404872] [vq=00000000465a3846][start=33][count=1]
[  777.434829] [vq=00000000465a3846][vq->ndescs=1]
[  777.434831] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.434832] [vq=00000000465a3846][vq->ndescs=0]
[  777.434836] [vq=00000000465a3846][start=34][count=1]
[  777.440676] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  777.440678] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  777.440679] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  777.440681] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  777.440682] [vq=00000000201c16ab][start=72][count=4]
[  777.440939] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  777.440940] [vq=00000000201c16ab][start=76][count=1]
[  777.844890] [vq=00000000465a3846][vq->ndescs=1]
[  777.844893] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.844896] [vq=00000000465a3846][vq->ndescs=0]
[  777.844905] [vq=00000000465a3846][start=35][count=1]
[  777.861124] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  777.861127] [vq=00000000201c16ab][start=77][count=1]
[  777.892947] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  777.892950] [vq=00000000201c16ab][start=78][count=1]
[  778.136716] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  778.136718] [vq=00000000201c16ab][start=79][count=1]
[  778.201376] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  778.201378] [vq=00000000201c16ab][start=80][count=1]
[  778.204884] [vq=00000000465a3846][vq->ndescs=1]
[  778.204885] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  778.204887] [vq=00000000465a3846][vq->ndescs=0]
[  778.204892] [vq=00000000465a3846][start=36][count=1]
[  778.335800] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  778.335803] [vq=00000000201c16ab][start=81][count=1]
[  778.387704] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  778.387706] [vq=00000000201c16ab][start=82][count=1]
[  778.441538] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  778.441543] [vq=00000000201c16ab][vq->ndescs=65]
[  778.441545] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  778.441546] [vq=00000000201c16ab][start=83][count=2]
[  778.441804] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  778.441805] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  778.441807] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  778.441808] [vq=00000000201c16ab][start=85][count=3]
[  778.808385] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  778.808390] [vq=00000000201c16ab][start=88][count=1]
[  778.863990] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  778.863992] [vq=00000000201c16ab][start=89][count=1]
[  778.907345] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  778.907348] [vq=00000000201c16ab][start=90][count=1]
[  779.176815] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  779.176819] [vq=00000000201c16ab][start=91][count=1]
[  779.358511] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  779.358513] [vq=00000000201c16ab][start=92][count=1]
[  779.437933] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  779.437935] [vq=00000000201c16ab][start=93][count=1]
[  779.442422] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  779.442424] [vq=00000000201c16ab][start=94][count=1]
[  779.442665] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  779.442667] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  779.442668] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  779.442669] [vq=00000000201c16ab][start=95][count=3]
[  779.904029] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  779.904032] [vq=00000000201c16ab][start=98][count=1]
[  780.443688] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  780.443691] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  780.443692] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  780.443694] [vq=00000000201c16ab][start=99][count=3]
[  780.443985] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  780.443987] [vq=00000000201c16ab][start=102][count=1]
[  780.467631] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  780.467633] [vq=00000000201c16ab][start=103][count=1]
[  780.651623] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  780.651626] [vq=00000000201c16ab][start=104][count=1]
[  783.326504] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  783.326510] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  783.326512] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  783.326514] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  783.326515] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  783.326517] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  783.326518] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  783.326520] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  783.326522] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  783.326523] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  783.326524] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  783.326526] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  783.326528] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  783.326530] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  783.326532] [vq=00000000201c16ab][start=0][count=14]
[  783.326535] [vq=00000000465a3846][vq->ndescs=0]
[  783.446605] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  783.446608] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  783.446610] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  783.446611] [vq=00000000201c16ab][start=14][count=3]
[  783.903893] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  783.903896] [vq=00000000201c16ab][start=17][count=1]
[  784.022786] [vq=00000000465a3846][vq->ndescs=1]
[  784.022788] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.022790] [vq=00000000465a3846][vq->ndescs=0]
[  784.022797] [vq=00000000465a3846][start=0][count=1]
[  784.039300] [vq=00000000465a3846][vq->ndescs=1]
[  784.039301] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.039303] [vq=00000000465a3846][vq->ndescs=0]
[  784.039306] [vq=00000000465a3846][start=1][count=1]
[  784.039827] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  784.039829] [vq=00000000201c16ab][start=18][count=1]
[  784.044823] [vq=00000000465a3846][vq->ndescs=1]
[  784.044824] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.044826] [vq=00000000465a3846][vq->ndescs=0]
[  784.044831] [vq=00000000465a3846][start=2][count=1]
[  784.167484] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  784.167486] [vq=00000000201c16ab][start=19][count=1]
[  784.329413] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  784.329416] [vq=00000000201c16ab][start=20][count=1]
[  784.414819] [vq=00000000465a3846][vq->ndescs=1]
[  784.414820] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.414822] [vq=00000000465a3846][vq->ndescs=0]
[  784.414826] [vq=00000000465a3846][start=3][count=1]
[  784.447615] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  784.447618] [vq=00000000201c16ab][start=21][count=1]
[  784.447900] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  784.447902] [vq=00000000201c16ab][start=22][count=1]
[  784.934838] [vq=00000000465a3846][vq->ndescs=1]
[  784.934840] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.934841] [vq=00000000465a3846][vq->ndescs=0]
[  784.934845] [vq=00000000465a3846][start=4][count=1]
[  785.100943] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  785.100946] [vq=00000000201c16ab][start=23][count=1]
[  785.104788] [vq=00000000465a3846][vq->ndescs=1]
[  785.104789] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.104791] [vq=00000000465a3846][vq->ndescs=0]
[  785.104797] [vq=00000000465a3846][start=5][count=1]
[  785.105315] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  785.105316] [vq=00000000201c16ab][start=24][count=1]
[  785.264810] [vq=00000000465a3846][vq->ndescs=1]
[  785.264812] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.264813] [vq=00000000465a3846][vq->ndescs=0]
[  785.264817] [vq=00000000465a3846][start=6][count=1]
[  785.448591] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  785.448593] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  785.448595] [vq=00000000201c16ab][start=25][count=2]
[  785.812091] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  785.812093] [vq=00000000201c16ab][start=27][count=1]
[  785.865622] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  785.865624] [vq=00000000201c16ab][start=28][count=1]
[  785.956012] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  785.956014] [vq=00000000201c16ab][start=29][count=1]
[  785.984841] [vq=00000000465a3846][vq->ndescs=1]
[  785.984843] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.984844] [vq=00000000465a3846][vq->ndescs=0]
[  785.984849] [vq=00000000465a3846][start=7][count=1]
[  785.985751] [vq=00000000465a3846][vq->ndescs=1]
[  785.985752] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.985753] [vq=00000000465a3846][vq->ndescs=0]
[  785.985756] [vq=00000000465a3846][start=8][count=1]
[  786.022926] [vq=00000000465a3846][vq->ndescs=1]
[  786.022927] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.022928] [vq=00000000465a3846][vq->ndescs=0]
[  786.022931] [vq=00000000465a3846][start=9][count=1]
[  786.024797] [vq=00000000465a3846][vq->ndescs=1]
[  786.024798] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.024799] [vq=00000000465a3846][vq->ndescs=0]
[  786.024802] [vq=00000000465a3846][start=10][count=1]
[  786.054795] [vq=00000000465a3846][vq->ndescs=1]
[  786.054796] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.054798] [vq=00000000465a3846][vq->ndescs=0]
[  786.054802] [vq=00000000465a3846][start=11][count=1]
[  786.136759] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  786.136761] [vq=00000000201c16ab][start=30][count=1]
[  786.144783] [vq=00000000465a3846][vq->ndescs=1]
[  786.144784] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.144786] [vq=00000000465a3846][vq->ndescs=0]
[  786.144790] [vq=00000000465a3846][start=12][count=1]
[  786.145388] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  786.145389] [vq=00000000201c16ab][start=31][count=1]
[  786.166419] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  786.166421] [vq=00000000201c16ab][start=32][count=1]
[  786.449605] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  786.449608] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  786.449610] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  786.449611] [vq=00000000201c16ab][start=33][count=3]
[  786.831294] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  786.831296] [vq=00000000201c16ab][start=36][count=1]
[  786.943969] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  786.943971] [vq=00000000201c16ab][start=37][count=1]
[  786.944849] [vq=00000000465a3846][vq->ndescs=1]
[  786.944850] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.944853] [vq=00000000465a3846][vq->ndescs=0]
[  786.944858] [vq=00000000465a3846][start=13][count=1]
[  786.980009] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  786.980011] [vq=00000000201c16ab][start=38][count=1]
[  787.155382] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  787.155386] [vq=00000000201c16ab][start=39][count=1]
[  787.176736] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  787.176738] [vq=00000000201c16ab][start=40][count=1]
[  787.185039] [vq=00000000465a3846][vq->ndescs=1]
[  787.185040] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  787.185042] [vq=00000000465a3846][vq->ndescs=0]
[  787.185050] [vq=00000000465a3846][start=14][count=1]
[  787.185639] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  787.185641] [vq=00000000201c16ab][start=41][count=1]
[  787.450691] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  787.450693] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  787.450695] [vq=00000000201c16ab][start=42][count=2]
[  787.485512] Guest moved vq 00000000201c16ab used index from 0 to 289
[  787.485513] [vq=00000000201c16ab][vq->ndescs=0]
[  788.023034] [vq=00000000465a3846][vq->ndescs=1]
[  788.023036] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  788.023037] [vq=00000000465a3846][vq->ndescs=0]
[  788.023042] [vq=00000000465a3846][start=15][count=1]
[  788.224798] [vq=00000000465a3846][vq->ndescs=1]
[  788.224800] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  788.224802] [vq=00000000465a3846][vq->ndescs=0]
[  788.224806] [vq=00000000465a3846][start=16][count=1]
[  789.264807] [vq=00000000465a3846][vq->ndescs=1]
[  789.264809] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  789.264811] [vq=00000000465a3846][vq->ndescs=0]
[  789.264816] [vq=00000000465a3846][start=17][count=1]
[  789.885117] [vq=00000000465a3846][vq->ndescs=1]
[  789.885118] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  789.885120] [vq=00000000465a3846][vq->ndescs=0]
[  789.885124] [vq=00000000465a3846][start=18][count=1]
[  790.304957] [vq=00000000465a3846][vq->ndescs=1]
[  790.304959] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  790.304961] [vq=00000000465a3846][vq->ndescs=0]
[  790.304967] [vq=00000000465a3846][start=19][count=1]
[  791.344839] [vq=00000000465a3846][vq->ndescs=1]
[  791.344842] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  791.344843] [vq=00000000465a3846][vq->ndescs=0]
[  791.344850] [vq=00000000465a3846][start=20][count=1]
[  792.384804] [vq=00000000465a3846][vq->ndescs=1]
[  792.384807] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  792.384808] [vq=00000000465a3846][vq->ndescs=0]
[  792.384814] [vq=00000000465a3846][start=21][count=1]
[  793.424946] [vq=00000000465a3846][vq->ndescs=1]
[  793.424948] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  793.424950] [vq=00000000465a3846][vq->ndescs=0]
[  793.424955] [vq=00000000465a3846][start=22][count=1]
[  793.886441] [vq=00000000465a3846][vq->ndescs=1]
[  793.886442] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  793.886444] [vq=00000000465a3846][vq->ndescs=0]
[  793.886449] [vq=00000000465a3846][start=23][count=1]
[  794.464802] [vq=00000000465a3846][vq->ndescs=1]
[  794.464804] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  794.464806] [vq=00000000465a3846][vq->ndescs=0]
[  794.464811] [vq=00000000465a3846][start=24][count=1]
[  795.504831] [vq=00000000465a3846][vq->ndescs=1]
[  795.504835] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  795.504838] [vq=00000000465a3846][vq->ndescs=0]

Christian Borntraeger

2020-Feb-11 13:13 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 11.02.20 14:04, Eugenio P?rez wrote:> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>
>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>> Hi Christian.
>>>
>>> I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>> Your configuration (libvirt or qemu line), and host's dmesg
output if any?
>>>
>>> Thanks!
>>
>> If it was not obvious, this is on s390x, a big endian system.
>>
> 
> Hi Christian. Thank you very much for your fast responses.
> 
> Could you try this patch on top of
eccb852f1fe6bede630e2e4f1a121a81e34354ab?
I still get 
[   43.665145] Guest moved used index from 0 to 289
after some reboots.

> 
> Thanks!
> 
> From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> eperezma at redhat.com>
> Date: Tue, 11 Feb 2020 13:19:10 +0100
> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> 
> Before of the batch change, it was the chain's head. Need to keep that
> way or we will not be able to free a chain of descriptors.
> 
> Fixes: eccb852f1fe6 ("vhost: batching fetches")
> ---
>  drivers/vhost/vhost.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..fc422c3e5c08 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  			*out_num += ret;
>  		}
>  
> -		ret = desc->id;
> -
>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>  			break;
>  	}
>  
> +	ret = vq->descs[vq->first_desc].id;
>  	vq->first_desc = i + 1;
>  
>  	return ret;
>

Michael S. Tsirkin

2020-Feb-11 13:57 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On Tue, Feb 11, 2020 at 02:04:54PM +0100, Eugenio P??rez
wrote:> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > 
> > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > Hi Christian.
> > > 
> > > I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > > Your configuration (libvirt or qemu line), and host's dmesg
output if any?
> > > 
> > > Thanks!
> > 
> > If it was not obvious, this is on s390x, a big endian system.
> > 
> 
> Hi Christian. Thank you very much for your fast responses.
> 
> Could you try this patch on top of
eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> 
> Thanks!
> 
> >From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> eperezma at redhat.com>
> Date: Tue, 11 Feb 2020 13:19:10 +0100
> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> 
> Before of the batch change, it was the chain's head. Need to keep that
> way or we will not be able to free a chain of descriptors.
I think it's cleaner to have all descriptors in the chain
have the same id.
> Fixes: eccb852f1fe6 ("vhost: batching fetches")
> ---
>  drivers/vhost/vhost.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..fc422c3e5c08 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  			*out_num += ret;
>  		}
>  
> -		ret = desc->id;
> -
>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>  			break;
>  	}
>  
> +	ret = vq->descs[vq->first_desc].id;
>  	vq->first_desc = i + 1;
>  
>  	return ret;
> -- 
> 2.18.1

Christian Borntraeger

2020-Feb-13 09:30 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 12.02.20 17:34, Eugenio P?rez wrote:> On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
>>
>> On 11.02.20 14:04, Eugenio P?rez wrote:
>>> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>>>> Hi Christian.
>>>>>
>>>>> I'm not able to reproduce the failure with
eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more
>>>>> data?
>>>>> Your configuration (libvirt or qemu line), and host's
dmesg output if any?
>>>>>
>>>>> Thanks!
>>>>
>>>> If it was not obvious, this is on s390x, a big endian system.
>>>>
>>>
>>> Hi Christian. Thank you very much for your fast responses.
>>>
>>> Could you try this patch on top of
eccb852f1fe6bede630e2e4f1a121a81e34354ab?
>>
>> I still get 
>> [   43.665145] Guest moved used index from 0 to 289
>> after some reboots.
>>
>>
>>> Thanks!
>>>
>>> From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00
2001
>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
>>> eperezma at redhat.com>
>>> Date: Tue, 11 Feb 2020 13:19:10 +0100
>>> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
>>>
>>> Before of the batch change, it was the chain's head. Need to
keep that
>>> way or we will not be able to free a chain of descriptors.
>>>
>>> Fixes: eccb852f1fe6 ("vhost: batching fetches")
>>> ---
>>>  drivers/vhost/vhost.c | 3 +--
>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>> index b5a51b1f2e79..fc422c3e5c08 100644
>>> --- a/drivers/vhost/vhost.c
>>> +++ b/drivers/vhost/vhost.c
>>> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct
vhost_virtqueue *vq,
>>>  			*out_num += ret;
>>>  		}
>>>  
>>> -		ret = desc->id;
>>> -
>>>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>>>  			break;
>>>  	}
>>>  
>>> +	ret = vq->descs[vq->first_desc].id;
>>>  	vq->first_desc = i + 1;
>>>  
>>>  	return ret;
>>>
> 
> Sorry, still not able to reproduce the issue.
> 
> Could we try to disable all the vhost features?
> 
> Thanks!
> 
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..08f6d2ccb697 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool
enabled);
>         } while (0)
>  
>  enum {
> -       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> -                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> -                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> -                        (1ULL << VHOST_F_LOG_ALL) |
> -                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
> +       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> +                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
*/
> +                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> +                        /* (1ULL << VHOST_F_LOG_ALL) | */
> +                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>                          (1ULL << VIRTIO_F_VERSION_1)
>  };
> 

I still get  guest crashes with this on top of eccb852f1fe6. (The patch did not
apply, I had to manually comment out these things)

Christian Borntraeger

2020-Feb-13 13:09 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 13.02.20 11:47, Eugenio P?rez wrote:> 
> Can we try tracing last_avail_idx with the attached patch? Can you enable
also line and thread id (dyndbg='+plt')?





-------------- next part --------------
[  326.584446] [2127] 1648: VHOST_SET_VRING_BASE
[vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0]
[  326.584457] [2127] 1648: VHOST_SET_VRING_BASE
[vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=0]
[  326.584474] [2124] 2441: DISCARD
[vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[  327.281901] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  327.281902] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  327.281903] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  327.281904] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  327.281905] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  327.281906] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  327.281907] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  327.281908] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  327.281909] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  327.281910] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  327.281911] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  327.281911] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  327.281912] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  327.281913] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  327.281914] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  327.281915] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  327.281916] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  327.281916] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  327.281917] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  327.281918] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  327.281919] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  327.281920] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  327.281921] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  327.281921] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  327.281922] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  327.281923] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  327.281924] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  327.281925] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  327.281926] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  327.281926] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  327.281927] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  327.281928] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  327.281929] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  327.281930] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  327.281931] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  327.281932] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  327.281933] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  327.281934] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  327.281934] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  327.281935] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  327.281936] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  327.281937] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  327.281938] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  327.281939] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  327.281939] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  327.281940] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  327.281941] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  327.281942] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  327.281943] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  327.281944] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  327.281945] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  327.281946] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  327.281947] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  327.281948] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  327.281948] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  327.281949] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=55][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  327.281950] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=56][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  327.281951] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=57][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  327.281952] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=58][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  327.281953] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=59][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  327.281954] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=60][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  327.281955] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=61][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  327.281955] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=62][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  327.281956] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=63][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  327.281958] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=64][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  327.295289] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  327.313733] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  327.328259] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=2][vq->avail_idx=3][vq->ndescs=1][vq->first_desc=0]
[  327.328816] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=3][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  327.328818] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=2][vq->first_desc=0]
[  327.553778] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  327.663779] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1][vq->first_desc=0]
[  328.433781] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  328.522897] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2][vq->first_desc=0]
[  328.593777] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1][vq->first_desc=0]
[  328.723911] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2][vq->first_desc=0]
[  328.926482] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=2][vq->first_desc=0]
[  329.129059] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=2][vq->first_desc=0]
[  329.295360] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1][vq->first_desc=0]
[  329.633801] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1][vq->first_desc=0]
[  329.663784] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1][vq->first_desc=0]
[  330.053043] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=1][vq->first_desc=0]
[  330.111522] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=1][vq->first_desc=0]
[  330.112176] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=1][vq->first_desc=0]
[  330.112449] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=1][vq->first_desc=0]
[  330.112619] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1][vq->first_desc=0]
[  330.112793] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=1][vq->first_desc=0]
[  330.113028] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1][vq->first_desc=0]
[  330.113203] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1][vq->first_desc=0]
[  330.113366] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1][vq->first_desc=0]
[  330.113527] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1][vq->first_desc=0]
[  330.113686] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1][vq->first_desc=0]
[  330.113895] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1][vq->first_desc=0]
[  330.114064] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1][vq->first_desc=0]
[  330.114227] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1][vq->first_desc=0]
[  330.114389] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1][vq->first_desc=0]
[  330.114549] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1][vq->first_desc=0]
[  330.114708] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1][vq->first_desc=0]
[  330.114869] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1][vq->first_desc=0]
[  330.115033] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1][vq->first_desc=0]
[  330.115194] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1][vq->first_desc=0]
[  330.115355] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1][vq->first_desc=0]
[  330.115515] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1][vq->first_desc=0]
[  330.433757] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1][vq->first_desc=0]
[  330.593759] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1][vq->first_desc=0]
[  332.511365] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1][vq->first_desc=0]
[  333.245392] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=65][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  333.245393] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=66][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  333.245394] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=67][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  333.245395] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=68][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  333.245396] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=69][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  333.245397] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=70][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  333.245398] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=71][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  333.245398] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=72][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  333.245399] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=73][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  333.245400] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=74][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  333.245401] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=75][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  333.245402] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=76][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  333.245403] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=77][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  333.245404] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=78][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  333.245405] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=79][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  333.245406] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=80][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  333.245407] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=81][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  333.245408] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=82][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  333.245408] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=83][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  333.245409] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=84][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  333.245410] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=85][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  333.245411] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=86][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  333.245412] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=87][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  333.245413] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=88][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  333.245414] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=89][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  333.245414] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=90][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  333.245415] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=91][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  333.245416] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=92][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  333.245417] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=93][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  333.245418] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=94][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  333.245419] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=95][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  333.245420] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=96][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  333.245421] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=97][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  333.245422] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=98][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  333.245422] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=99][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  333.245423] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=100][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  333.245424] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=101][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  333.245425] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=102][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  333.245426] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=103][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  333.245427] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=104][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  333.245427] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=105][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  333.245428] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=106][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  333.245429] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=107][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  333.245430] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=108][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  333.245431] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=109][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  333.245431] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=110][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  333.245432] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=111][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  333.245433] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=112][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  333.245434] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=113][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  333.245435] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=114][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  333.245436] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=115][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  333.245437] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=116][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  333.245438] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=117][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  333.245438] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=118][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  333.245439] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=119][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  333.245440] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=120][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  333.245441] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=121][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  333.245442] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=122][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  333.245443] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=123][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  333.245443] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=124][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  333.245444] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=125][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  333.245445] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=126][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  333.245446] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=127][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  333.245447] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=128][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  333.245448] [2124] 2322:
[vq=0000000036fdfcb7][vq->last_avail_idx=129][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  336.121638] [2126] 1648: VHOST_SET_VRING_BASE
[vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0]
[  336.121654] [2126] 1648: VHOST_SET_VRING_BASE
[vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=0]
[  336.943877] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  336.957322] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  336.973438] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=2][vq->avail_idx=3][vq->ndescs=1][vq->first_desc=0]
[  337.223453] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=1][vq->first_desc=0]
[  337.383453] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  337.473429] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  337.993434] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1][vq->first_desc=0]
[  338.393446] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  338.423446] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=1][vq->first_desc=0]
[  338.693357] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1][vq->first_desc=0]
[  338.843477] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=1][vq->first_desc=0]
[  338.943915] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=1][vq->first_desc=0]
[  339.033436] [2124] 2322:
[vq=0000000060d7302b][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=1][vq->first_desc=0]

Christian Borntraeger

2020-Feb-13 16:32 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 13.02.20 17:29, Eugenio P?rez wrote:> Can we try with this traces?
Does not apply on eccb852f1fe6bede630e2e4f1a121a81e34354ab, can you double
check?
> 
> From b793b4106085ab1970bdedb340e49f37843ed585 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma at redhat.com>
> Date: Thu, 13 Feb 2020 17:27:05 +0100
> Subject: [PATCH] vhost: Add debug in ioctl calls
> 
> ---
>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>  drivers/vhost/vhost.c | 16 ++++++++++++++--
>  2 files changed, 31 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..e4d5f843f9c0 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d",
index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index,
fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d
!vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err
r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock,
oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r,
vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd,
r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r,
vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned
int ioctl,
>  
>  	switch (ioctl) {
>  	case VHOST_NET_SET_BACKEND:
> +		pr_debug("VHOST_NET_SET_BACKEND");
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 021d70bed015..7f4848f9cec3 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1642,18 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d,
unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		if (vq->last_avail_idx || vq->avail_idx) {
> +			pr_debug(
> +				"strange VHOST_SET_VRING_BASE
[vq=%p][s.index=%u][s.num=%u]",
> +				vq, s.index, s.num);
> +			dump_stack();
> +			r = 0;
> +			break;
> +		}
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
>  		pr_debug(
> -			"VHOST_SET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
> -			vq, vq->last_avail_idx, vq->avail_idx);
> +			"VHOST_SET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
>

Christian Borntraeger

2020-Feb-14 07:33 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

I did 
ping -c 20 -f ... ; reboot 
twice

The ping after the first reboot showed .......E 

this was on the host console

[   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
[   55.951892] Call Trace:
[   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
[   55.951951] Call Trace:
[   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
[   56.609831] unexpected descriptor format for RX: out 0, in 0
[   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
[   86.540466] Call Trace:
[   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
[   86.540526] Call Trace:
[   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   86.540570] unexpected descriptor format for RX: out 0, in 0
[   86.540577] Unexpected header len for TX: 0 expected 0


On 14.02.20 08:06, Eugenio P?rez wrote:> Hi Christian.
> 
> Sorry, that was meant to be applied over previous debug patch.
> 
> Here I inline the one meant to be applied over
eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> 
> Thanks!
> 
> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma at redhat.com>
> Date: Fri, 14 Feb 2020 08:02:26 +0100
> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>  ioctl calls
> 
> ---
>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>  drivers/vhost/vhost.h | 10 +++++-----
>  3 files changed, 45 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..e4d5f843f9c0 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d",
index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index,
fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d
!vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err
r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock,
oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r,
vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd,
r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r,
vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net
*n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned
int ioctl,
>  
>  	switch (ioctl) {
>  	case VHOST_NET_SET_BACKEND:
> +		pr_debug("VHOST_NET_SET_BACKEND");
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..ec25ba32fe81 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d,
unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		if (vq->last_avail_idx || vq->avail_idx) {
> +			pr_debug(
> +				"strange VHOST_SET_VRING_BASE
[vq=%p][s.index=%u][s.num=%u]",
> +				vq, s.index, s.num);
> +			dump_stack();
> +			r = 0;
> +			break;
> +		}
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
> +		pr_debug(
> +			"VHOST_SET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>  
>  		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num))
{
> -			vq_err(vq, "Guest moved used index from %u to %u",
> -				last_avail_idx, vq->avail_idx);
> +			vq_err(vq, "Guest moved vq %p used index from %u to %u",
> +				vq, last_avail_idx, vq->avail_idx);
>  			return -EFAULT;
>  		}
>  
> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>  
>  	/* On success, increment avail index. */
> +	pr_debug(
> +	
"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs,
vq->first_desc);
>  	vq->last_avail_idx++;
>  
>  	return 0;
> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>  {
> +	pr_debug(
> +		"DISCARD
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, n);
>  	vq->last_avail_idx -= n;
>  }
>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..08f6d2ccb697 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool
enabled);
>  	} while (0)
>  
>  enum {
> -	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> -			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> -			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> -			 (1ULL << VHOST_F_LOG_ALL) |
> -			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
> +	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> +			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> +			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> +			 /* (1ULL << VHOST_F_LOG_ALL) | */
> +			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>  			 (1ULL << VIRTIO_F_VERSION_1)
>  };
>  
>

Christian Borntraeger

2020-Feb-14 07:43 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 14.02.20 08:40, Eugenio Perez Martin wrote:> Hi.
> 
> Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I
miss
> all the others regular debug traces on that one.
I did 

 echo -n 'file drivers/vhost/vhost.c +plt' > control
and
echo -n 'file drivers/vhost/net.c +plt'  > control

but apparently it did not work...me hates dynamic debug.
> 
> Thanks!
> 
> On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
> <borntraeger at de.ibm.com> wrote:
>>
>> I did
>> ping -c 20 -f ... ; reboot
>> twice
>>
>> The ping after the first reboot showed .......E
>>
>> this was on the host console
>>
>> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   55.951892] Call Trace:
>> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   55.951915]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
>> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   55.951951] Call Trace:
>> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   55.951956]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
>> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
>> [   56.609831] unexpected descriptor format for RX: out 0, in 0
>> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   86.540466] Call Trace:
>> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   86.540486]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
>> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   86.540526] Call Trace:
>> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   86.540531]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
>> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   86.540570] unexpected descriptor format for RX: out 0, in 0
>> [   86.540577] Unexpected header len for TX: 0 expected 0
>>
>>
>> On 14.02.20 08:06, Eugenio P?rez wrote:
>>> Hi Christian.
>>>
>>> Sorry, that was meant to be applied over previous debug patch.
>>>
>>> Here I inline the one meant to be applied over
eccb852f1fe6bede630e2e4f1a121a81e34354ab.
>>>
>>> Thanks!
>>>
>>> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00
2001
>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma at
redhat.com>
>>> Date: Fri, 14 Feb 2020 08:02:26 +0100
>>> Subject: [PATCH] vhost: disable all features and trace
last_avail_idx and
>>>  ioctl calls
>>>
>>> ---
>>>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>>>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>>>  drivers/vhost/vhost.h | 10 +++++-----
>>>  3 files changed, 45 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
>>> index e158159671fa..e4d5f843f9c0 100644
>>> --- a/drivers/vhost/net.c
>>> +++ b/drivers/vhost/net.c
>>> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct
vhost_net *n, unsigned index, int fd)
>>>
>>>       mutex_lock(&n->dev.mutex);
>>>       r = vhost_dev_check_owner(&n->dev);
>>> -     if (r)
>>> +     if (r) {
>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d
rc r=%d", index, fd, r);
>>>               goto err;
>>> +     }
>>>
>>>       if (index >= VHOST_NET_VQ_MAX) {
>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d
MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>>>               r = -ENOBUFS;
>>>               goto err;
>>>       }
>>> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct
vhost_net *n, unsigned index, int fd)
>>>
>>>       /* Verify that ring has been setup correctly. */
>>>       if (!vhost_vq_access_ok(vq)) {
>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d
!vhost_vq_access_ok", index, fd);
>>>               r = -EFAULT;
>>>               goto err_vq;
>>>       }
>>>       sock = get_socket(fd);
>>>       if (IS_ERR(sock)) {
>>>               r = PTR_ERR(sock);
>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d
get_socket err r=%d", index, fd, r);
>>>               goto err_vq;
>>>       }
>>>
>>>       /* start polling new socket */
>>>       oldsock = vq->private_data;
>>>       if (sock != oldsock) {
>>> +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d
vq=%p", sock, oldsock, index, fd, vq);
>>>               ubufs = vhost_net_ubuf_alloc(vq,
>>>                                            sock &&
vhost_sock_zcopy(sock));
>>>               if (IS_ERR(ubufs)) {
>>>                       r = PTR_ERR(ubufs);
>>> +                     pr_debug("ubufs index=%u fd=%d err r=%d
vq=%p", index, fd, r, vq);
>>>                       goto err_ubufs;
>>>               }
>>>
>>> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct
vhost_net *n, unsigned index, int fd)
>>>               vq->private_data = sock;
>>>               vhost_net_buf_unproduce(nvq);
>>>               r = vhost_vq_init_access(vq);
>>> -             if (r)
>>> +             if (r) {
>>> +                     pr_debug("init_access index=%u fd=%d
r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_used;
>>> +             }
>>>               r = vhost_net_enable_vq(n, vq);
>>> -             if (r)
>>> +             if (r) {
>>> +                     pr_debug("enable_vq index=%u fd=%d r=%d
vq=%p", index, fd, r, vq);
>>>                       goto err_used;
>>> +             }
>>>               if (index == VHOST_NET_VQ_RX)
>>>                       nvq->rx_ring = get_tap_ptr_ring(fd);
>>>
>>> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct
vhost_net *n, unsigned index, int fd)
>>>
>>>       mutex_unlock(&vq->mutex);
>>>
>>> +     pr_debug("sock=%p", sock);
>>> +
>>>       if (oldubufs) {
>>>               vhost_net_ubuf_put_wait_and_free(oldubufs);
>>>               mutex_lock(&vq->mutex);
>>> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f,
unsigned int ioctl,
>>>
>>>       switch (ioctl) {
>>>       case VHOST_NET_SET_BACKEND:
>>> +             pr_debug("VHOST_NET_SET_BACKEND");
>>>               if (copy_from_user(&backend, argp, sizeof
backend))
>>>                       return -EFAULT;
>>>               return vhost_net_set_backend(n, backend.index,
backend.fd);
>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>> index b5a51b1f2e79..ec25ba32fe81 100644
>>> --- a/drivers/vhost/vhost.c
>>> +++ b/drivers/vhost/vhost.c
>>> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d,
unsigned int ioctl, void __user *arg
>>>                       r = -EINVAL;
>>>                       break;
>>>               }
>>> +
>>> +             if (vq->last_avail_idx || vq->avail_idx) {
>>> +                     pr_debug(
>>> +                             "strange VHOST_SET_VRING_BASE
[vq=%p][s.index=%u][s.num=%u]",
>>> +                             vq, s.index, s.num);
>>> +                     dump_stack();
>>> +                     r = 0;
>>> +                     break;
>>> +             }
>>>               vq->last_avail_idx = s.num;
>>>               /* Forget the cached index value. */
>>>               vq->avail_idx = vq->last_avail_idx;
>>> +             pr_debug(
>>> +                     "VHOST_SET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>> +                     vq, vq->last_avail_idx, vq->avail_idx,
s.index, s.num);
>>>               break;
>>>       case VHOST_GET_VRING_BASE:
>>>               s.index = idx;
>>>               s.num = vq->last_avail_idx;
>>>               if (copy_to_user(argp, &s, sizeof s))
>>>                       r = -EFAULT;
>>> +             pr_debug(
>>> +                     "VHOST_GET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>> +                     vq, vq->last_avail_idx, vq->avail_idx,
s.index, s.num);
>>>               break;
>>>       case VHOST_SET_VRING_KICK:
>>>               if (copy_from_user(&f, argp, sizeof f)) {
>>> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue
*vq)
>>>               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>
>>>               if (unlikely((u16)(vq->avail_idx - last_avail_idx)
> vq->num)) {
>>> -                     vq_err(vq, "Guest moved used index from
%u to %u",
>>> -                             last_avail_idx, vq->avail_idx);
>>> +                     vq_err(vq, "Guest moved vq %p used index
from %u to %u",
>>> +                             vq, last_avail_idx,
vq->avail_idx);
>>>                       return -EFAULT;
>>>               }
>>>
>>> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue
*vq)
>>>       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>>>
>>>       /* On success, increment avail index. */
>>> +     pr_debug(
>>> +            
"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
>>> +             vq, vq->last_avail_idx, vq->avail_idx,
vq->ndescs, vq->first_desc);
>>>       vq->last_avail_idx++;
>>>
>>>       return 0;
>>> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>>>  /* Reverse the effect of vhost_get_vq_desc. Useful for error
handling. */
>>>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>>>  {
>>> +     pr_debug(
>>> +             "DISCARD
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
>>> +             vq, vq->last_avail_idx, vq->avail_idx, n);
>>>       vq->last_avail_idx -= n;
>>>  }
>>>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
>>> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
>>> index 661088ae6dc7..08f6d2ccb697 100644
>>> --- a/drivers/vhost/vhost.h
>>> +++ b/drivers/vhost/vhost.h
>>> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev
*d, bool enabled);
>>>       } while (0)
>>>
>>>  enum {
>>> -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
>>> -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC)
|
>>> -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
>>> -                      (1ULL << VHOST_F_LOG_ALL) |
>>> -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
>>> +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY)
| */
>>> +                      /* (1ULL <<
VIRTIO_RING_F_INDIRECT_DESC) | */
>>> +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) |
*/
>>> +                      /* (1ULL << VHOST_F_LOG_ALL) | */
>>> +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>>>                        (1ULL << VIRTIO_F_VERSION_1)
>>>  };
>>>
>>>
>>
>

Christian Borntraeger

2020-Feb-14 07:47 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

repro


On 14.02.20 08:43, Christian Borntraeger wrote:> 
> 
> On 14.02.20 08:40, Eugenio Perez Martin wrote:
>> Hi.
>>
>> Were the vhost and vhost_net modules loaded with dyndbg='+plt'?
I miss
>> all the others regular debug traces on that one.
> 
> I did 
> 
>  echo -n 'file drivers/vhost/vhost.c +plt' > control
> and
> echo -n 'file drivers/vhost/net.c +plt'  > control
> 
> but apparently it did not work...me hates dynamic debug.
> 
>>
>> Thanks!
>>
>> On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
>> <borntraeger at de.ibm.com> wrote:
>>>
>>> I did
>>> ping -c 20 -f ... ; reboot
>>> twice
>>>
>>> The ping after the first reboot showed .......E
>>>
>>> this was on the host console
>>>
>>> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+
#21
>>> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   55.951892] Call Trace:
>>> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   55.951915]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   55.951919]  [<000003ff8042a608>]
vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   55.951927]  [<0000001ede3c51ba>]
__s390x_sys_ioctl+0x2a/0x38
>>> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+
#21
>>> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   55.951951] Call Trace:
>>> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   55.951956]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   55.951958]  [<000003ff8042a608>]
vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   55.951962]  [<0000001ede3c51ba>]
__s390x_sys_ioctl+0x2a/0x38
>>> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44
to 0
>>> [   56.609831] unexpected descriptor format for RX: out 0, in 0
>>> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+
#21
>>> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   86.540466] Call Trace:
>>> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   86.540486]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   86.540490]  [<000003ff8042a608>]
vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   86.540498]  [<0000001ede3c51ba>]
__s390x_sys_ioctl+0x2a/0x38
>>> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+
#21
>>> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   86.540526] Call Trace:
>>> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   86.540531]  [<000003ff803736a6>]
vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   86.540532]  [<000003ff8042a608>]
vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   86.540537]  [<0000001ede3c51ba>]
__s390x_sys_ioctl+0x2a/0x38
>>> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   86.540570] unexpected descriptor format for RX: out 0, in 0
>>> [   86.540577] Unexpected header len for TX: 0 expected 0
>>>
>>>
>>> On 14.02.20 08:06, Eugenio P?rez wrote:
>>>> Hi Christian.
>>>>
>>>> Sorry, that was meant to be applied over previous debug patch.
>>>>
>>>> Here I inline the one meant to be applied over
eccb852f1fe6bede630e2e4f1a121a81e34354ab.
>>>>
>>>> Thanks!
>>>>
>>>> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17
00:00:00 2001
>>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma at
redhat.com>
>>>> Date: Fri, 14 Feb 2020 08:02:26 +0100
>>>> Subject: [PATCH] vhost: disable all features and trace
last_avail_idx and
>>>>  ioctl calls
>>>>
>>>> ---
>>>>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>>>>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>>>>  drivers/vhost/vhost.h | 10 +++++-----
>>>>  3 files changed, 45 insertions(+), 10 deletions(-)
>>>>
>>>> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
>>>> index e158159671fa..e4d5f843f9c0 100644
>>>> --- a/drivers/vhost/net.c
>>>> +++ b/drivers/vhost/net.c
>>>> @@ -1505,10 +1505,13 @@ static long
vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       mutex_lock(&n->dev.mutex);
>>>>       r = vhost_dev_check_owner(&n->dev);
>>>> -     if (r)
>>>> +     if (r) {
>>>> +             pr_debug("vhost_dev_check_owner index=%u
fd=%d rc r=%d", index, fd, r);
>>>>               goto err;
>>>> +     }
>>>>
>>>>       if (index >= VHOST_NET_VQ_MAX) {
>>>> +             pr_debug("vhost_dev_check_owner index=%u
fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>>>>               r = -ENOBUFS;
>>>>               goto err;
>>>>       }
>>>> @@ -1518,22 +1521,26 @@ static long
vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       /* Verify that ring has been setup correctly. */
>>>>       if (!vhost_vq_access_ok(vq)) {
>>>> +             pr_debug("vhost_net_set_backend index=%u
fd=%d !vhost_vq_access_ok", index, fd);
>>>>               r = -EFAULT;
>>>>               goto err_vq;
>>>>       }
>>>>       sock = get_socket(fd);
>>>>       if (IS_ERR(sock)) {
>>>>               r = PTR_ERR(sock);
>>>> +             pr_debug("vhost_net_set_backend index=%u
fd=%d get_socket err r=%d", index, fd, r);
>>>>               goto err_vq;
>>>>       }
>>>>
>>>>       /* start polling new socket */
>>>>       oldsock = vq->private_data;
>>>>       if (sock != oldsock) {
>>>> +             pr_debug("sock=%p != oldsock=%p index=%u
fd=%d vq=%p", sock, oldsock, index, fd, vq);
>>>>               ubufs = vhost_net_ubuf_alloc(vq,
>>>>                                            sock &&
vhost_sock_zcopy(sock));
>>>>               if (IS_ERR(ubufs)) {
>>>>                       r = PTR_ERR(ubufs);
>>>> +                     pr_debug("ubufs index=%u fd=%d err
r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_ubufs;
>>>>               }
>>>>
>>>> @@ -1541,11 +1548,15 @@ static long
vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>               vq->private_data = sock;
>>>>               vhost_net_buf_unproduce(nvq);
>>>>               r = vhost_vq_init_access(vq);
>>>> -             if (r)
>>>> +             if (r) {
>>>> +                     pr_debug("init_access index=%u fd=%d
r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_used;
>>>> +             }
>>>>               r = vhost_net_enable_vq(n, vq);
>>>> -             if (r)
>>>> +             if (r) {
>>>> +                     pr_debug("enable_vq index=%u fd=%d
r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_used;
>>>> +             }
>>>>               if (index == VHOST_NET_VQ_RX)
>>>>                       nvq->rx_ring = get_tap_ptr_ring(fd);
>>>>
>>>> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct
vhost_net *n, unsigned index, int fd)
>>>>
>>>>       mutex_unlock(&vq->mutex);
>>>>
>>>> +     pr_debug("sock=%p", sock);
>>>> +
>>>>       if (oldubufs) {
>>>>               vhost_net_ubuf_put_wait_and_free(oldubufs);
>>>>               mutex_lock(&vq->mutex);
>>>> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file
*f, unsigned int ioctl,
>>>>
>>>>       switch (ioctl) {
>>>>       case VHOST_NET_SET_BACKEND:
>>>> +             pr_debug("VHOST_NET_SET_BACKEND");
>>>>               if (copy_from_user(&backend, argp, sizeof
backend))
>>>>                       return -EFAULT;
>>>>               return vhost_net_set_backend(n, backend.index,
backend.fd);
>>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>>> index b5a51b1f2e79..ec25ba32fe81 100644
>>>> --- a/drivers/vhost/vhost.c
>>>> +++ b/drivers/vhost/vhost.c
>>>> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev
*d, unsigned int ioctl, void __user *arg
>>>>                       r = -EINVAL;
>>>>                       break;
>>>>               }
>>>> +
>>>> +             if (vq->last_avail_idx || vq->avail_idx) {
>>>> +                     pr_debug(
>>>> +                             "strange
VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
>>>> +                             vq, s.index, s.num);
>>>> +                     dump_stack();
>>>> +                     r = 0;
>>>> +                     break;
>>>> +             }
>>>>               vq->last_avail_idx = s.num;
>>>>               /* Forget the cached index value. */
>>>>               vq->avail_idx = vq->last_avail_idx;
>>>> +             pr_debug(
>>>> +                     "VHOST_SET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>>> +                     vq, vq->last_avail_idx,
vq->avail_idx, s.index, s.num);
>>>>               break;
>>>>       case VHOST_GET_VRING_BASE:
>>>>               s.index = idx;
>>>>               s.num = vq->last_avail_idx;
>>>>               if (copy_to_user(argp, &s, sizeof s))
>>>>                       r = -EFAULT;
>>>> +             pr_debug(
>>>> +                     "VHOST_GET_VRING_BASE
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>>> +                     vq, vq->last_avail_idx,
vq->avail_idx, s.index, s.num);
>>>>               break;
>>>>       case VHOST_SET_VRING_KICK:
>>>>               if (copy_from_user(&f, argp, sizeof f)) {
>>>> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct
vhost_virtqueue *vq)
>>>>               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>>
>>>>               if (unlikely((u16)(vq->avail_idx -
last_avail_idx) > vq->num)) {
>>>> -                     vq_err(vq, "Guest moved used index
from %u to %u",
>>>> -                             last_avail_idx,
vq->avail_idx);
>>>> +                     vq_err(vq, "Guest moved vq %p used
index from %u to %u",
>>>> +                             vq, last_avail_idx,
vq->avail_idx);
>>>>                       return -EFAULT;
>>>>               }
>>>>
>>>> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct
vhost_virtqueue *vq)
>>>>       BUG_ON(!(vq->used_flags &
VRING_USED_F_NO_NOTIFY));
>>>>
>>>>       /* On success, increment avail index. */
>>>> +     pr_debug(
>>>> +            
"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
>>>> +             vq, vq->last_avail_idx, vq->avail_idx,
vq->ndescs, vq->first_desc);
>>>>       vq->last_avail_idx++;
>>>>
>>>>       return 0;
>>>> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>>>>  /* Reverse the effect of vhost_get_vq_desc. Useful for error
handling. */
>>>>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>>>>  {
>>>> +     pr_debug(
>>>> +             "DISCARD
[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
>>>> +             vq, vq->last_avail_idx, vq->avail_idx, n);
>>>>       vq->last_avail_idx -= n;
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
>>>> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
>>>> index 661088ae6dc7..08f6d2ccb697 100644
>>>> --- a/drivers/vhost/vhost.h
>>>> +++ b/drivers/vhost/vhost.h
>>>> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct
vhost_dev *d, bool enabled);
>>>>       } while (0)
>>>>
>>>>  enum {
>>>> -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY)
|
>>>> -                      (1ULL <<
VIRTIO_RING_F_INDIRECT_DESC) |
>>>> -                      (1ULL << VIRTIO_RING_F_EVENT_IDX)
|
>>>> -                      (1ULL << VHOST_F_LOG_ALL) |
>>>> -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
>>>> +     VHOST_FEATURES = /* (1ULL <<
VIRTIO_F_NOTIFY_ON_EMPTY) | */
>>>> +                      /* (1ULL <<
VIRTIO_RING_F_INDIRECT_DESC) | */
>>>> +                      /* (1ULL <<
VIRTIO_RING_F_EVENT_IDX) | */
>>>> +                      /* (1ULL << VHOST_F_LOG_ALL) | */
>>>> +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) |
*/
>>>>                        (1ULL << VIRTIO_F_VERSION_1)
>>>>  };
>>>>
>>>>
>>>
>>-------------- next part --------------
[  924.337135] [1901] 1666: VHOST_GET_VRING_BASE
[vq=0000000063d896c6][vq->last_avail_idx=109][vq->avail_idx=0][s.index=1][s.num=109]
[  937.802750] [2237] 1657: VHOST_SET_VRING_BASE
[vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=0][s.index=0][s.num=0]
[  937.802757] [2237] 1657: VHOST_SET_VRING_BASE
[vq=000000007d36f807][vq->last_avail_idx=0][vq->avail_idx=0][s.index=1][s.num=0]
[  937.802763] [2237] 1726: VHOST_NET_SET_BACKEND
[  937.802766] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88
index=0 fd=39 vq=00000000742665e9
[  937.802770] [2237] 1573: sock=00000000de4d87a3
[  937.802771] [2237] 1726: VHOST_NET_SET_BACKEND
[  937.802772] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88
index=1 fd=39 vq=000000007d36f807
[  937.802773] [2237] 1573: sock=00000000de4d87a3
[  937.802775] [2235] 2453: DISCARD
[vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[  938.475541] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  938.475543] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  938.475544] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  938.475545] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  938.475546] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  938.475547] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  938.475548] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  938.475549] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  938.475550] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  938.475551] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  938.475552] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  938.475553] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  938.475554] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  938.475555] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  938.475556] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  938.475557] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  938.475558] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  938.475559] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  938.475560] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  938.475561] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  938.475562] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  938.475563] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  938.475564] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  938.475564] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  938.475565] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  938.475566] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  938.475567] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  938.475568] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  938.475569] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  938.475570] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  938.475571] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  938.475572] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  938.475573] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  938.475574] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  938.475575] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  938.475576] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  938.475577] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  938.475577] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  938.475578] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  938.475579] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  938.475580] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  938.475581] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  938.475582] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  938.475583] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  938.475584] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  938.475585] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  938.475586] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  938.475587] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  938.475588] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  938.475589] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  938.475589] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  938.475590] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  938.475591] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  938.475592] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  938.475593] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  938.475594] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=55][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  938.475595] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=56][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  938.475596] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=57][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  938.475597] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=58][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  938.475598] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=59][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  938.475599] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=60][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  938.475600] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=61][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  938.475601] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=62][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  938.475601] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=63][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  938.475603] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=64][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  938.591320] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  938.607139] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  938.607538] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=2][vq->avail_idx=4][vq->ndescs=1][vq->first_desc=0]
[  938.607540] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=2][vq->first_desc=0]
[  938.616010] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  938.796034] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  938.811243] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=2][vq->first_desc=0]
[  938.905985] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  939.011405] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2][vq->first_desc=0]
[  939.212298] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=2][vq->first_desc=0]
[  939.413649] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2][vq->first_desc=0]
[  939.596036] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=1][vq->first_desc=0]
[  939.996031] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=1][vq->first_desc=0]
[  940.591371] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1][vq->first_desc=0]
[  940.636040] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1][vq->first_desc=0]
[  940.665984] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1][vq->first_desc=0]
[  940.865048] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=2][vq->first_desc=0]
[  941.067594] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=2][vq->first_desc=0]
[  941.126013] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=1][vq->first_desc=0]
[  941.273297] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=2][vq->first_desc=0]
[  941.294416] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1][vq->first_desc=0]
[  941.476486] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=2][vq->first_desc=0]
[  941.516034] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1][vq->first_desc=0]
[  941.651361] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1][vq->first_desc=0]
[  941.651942] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1][vq->first_desc=0]
[  941.652190] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1][vq->first_desc=0]
[  941.652363] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1][vq->first_desc=0]
[  941.652542] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1][vq->first_desc=0]
[  941.652715] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1][vq->first_desc=0]
[  941.652882] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1][vq->first_desc=0]
[  941.653050] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1][vq->first_desc=0]
[  941.653218] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1][vq->first_desc=0]
[  941.653388] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1][vq->first_desc=0]
[  941.653558] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1][vq->first_desc=0]
[  941.653725] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1][vq->first_desc=0]
[  941.653896] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1][vq->first_desc=0]
[  941.654063] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1][vq->first_desc=0]
[  941.654229] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1][vq->first_desc=0]
[  941.654398] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1][vq->first_desc=0]
[  941.654567] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1][vq->first_desc=0]
[  941.654738] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1][vq->first_desc=0]
[  941.654906] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=41][vq->avail_idx=42][vq->ndescs=1][vq->first_desc=0]
[  941.655070] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=42][vq->avail_idx=43][vq->ndescs=1][vq->first_desc=0]
[  941.655240] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=43][vq->avail_idx=44][vq->ndescs=1][vq->first_desc=0]
[  943.627121] [2235] 2334:
[vq=000000007d36f807][vq->last_avail_idx=44][vq->avail_idx=45][vq->ndescs=1][vq->first_desc=0]
[  945.158600] [2228] 1726: VHOST_NET_SET_BACKEND
[  945.158603] [2228] 1538: sock=0000000012e3bc88 != oldsock=00000000de4d87a3
index=0 fd=-1 vq=00000000742665e9
[  945.158604] [2228] 1573: sock=0000000012e3bc88
[  945.158622] [2228] 1726: VHOST_NET_SET_BACKEND
[  945.158623] [2228] 1538: sock=0000000012e3bc88 != oldsock=00000000de4d87a3
index=1 fd=-1 vq=000000007d36f807
[  945.158624] [2228] 1573: sock=0000000012e3bc88
[  945.158646] [2228] 1666: VHOST_GET_VRING_BASE
[vq=00000000742665e9][vq->last_avail_idx=65][vq->avail_idx=256][s.index=0][s.num=65]
[  945.158648] [2228] 1666: VHOST_GET_VRING_BASE
[vq=000000007d36f807][vq->last_avail_idx=45][vq->avail_idx=45][s.index=1][s.num=45]
[  947.678597] [2237] 1647: strange VHOST_SET_VRING_BASE
[vq=00000000742665e9][s.index=0][s.num=0]
[  947.678600] CPU: 22 PID: 2237 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[  947.678604] Hardware name: IBM 3906 M04 704 (LPAR)
[  947.678606] Call Trace:
[  947.678612]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[  947.678615]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[  947.678625]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[  947.678628]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[  947.678632]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[  947.678634]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[  947.678635]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[  947.678638]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[  947.678660] [2237] 1647: strange VHOST_SET_VRING_BASE
[vq=000000007d36f807][s.index=1][s.num=0]
[  947.678661] CPU: 22 PID: 2237 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[  947.678662] Hardware name: IBM 3906 M04 704 (LPAR)
[  947.678662] Call Trace:
[  947.678664]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[  947.678665]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[  947.678667]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[  947.678669]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[  947.678671]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[  947.678672]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[  947.678673]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[  947.678675]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[  947.678694] [2237] 1726: VHOST_NET_SET_BACKEND
[  947.678697] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88
index=0 fd=39 vq=00000000742665e9
[  947.678702] [2237] 1573: sock=00000000de4d87a3
[  947.678703] [2237] 1726: VHOST_NET_SET_BACKEND
[  947.678704] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88
index=1 fd=39 vq=000000007d36f807
[  947.678705] [2237] 1573: sock=00000000de4d87a3
[  947.678717] Guest moved vq 000000007d36f807 used index from 45 to 0
[  950.562666] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=65][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  950.562671] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=66][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  950.562672] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=67][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  950.562673] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=68][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  950.562674] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=69][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  950.562675] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=70][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  950.562676] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=71][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  950.562677] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=72][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  950.562678] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=73][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  950.562679] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=74][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  950.562679] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=75][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  950.562680] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=76][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  950.562681] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=77][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  950.562682] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=78][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  950.562683] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=79][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  950.562684] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=80][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  950.562685] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=81][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  950.562686] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=82][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  950.562687] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=83][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  950.562688] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=84][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  950.562689] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=85][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  950.562690] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=86][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  950.562690] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=87][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  950.562691] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=88][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  950.562692] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=89][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  950.562693] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=90][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  950.562694] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=91][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  950.562695] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=92][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  950.562696] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=93][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  950.562697] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=94][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  950.562698] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=95][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  950.562699] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=96][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  950.562700] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=97][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  950.562700] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=98][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  950.562701] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=99][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  950.562702] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=100][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  950.562703] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=101][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  950.562704] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=102][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  950.562705] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=103][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  950.562706] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=104][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  950.562707] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=105][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  950.562707] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=106][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  950.562708] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=107][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  950.562709] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=108][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  950.562710] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=109][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  950.562711] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=110][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  950.562712] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=111][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  950.562713] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=112][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  950.562714] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=113][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  950.562715] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=114][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  950.562715] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=115][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  950.562716] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=116][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  950.562717] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=117][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  950.562718] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=118][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  950.562719] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=119][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  950.562720] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=120][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  950.562721] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=121][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  950.562722] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=122][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  950.562723] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=123][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  950.562724] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=124][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  950.562725] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=125][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  950.562726] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=126][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  950.562727] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=127][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  950.562728] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=128][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  950.562730] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=129][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  968.267262] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=130][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  968.267264] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=131][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  968.267265] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=132][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  968.267266] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=133][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  968.267267] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=134][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  968.267268] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=135][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  968.267269] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=136][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  968.267270] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=137][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  968.267271] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=138][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  968.267271] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=139][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  968.267272] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=140][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  968.267273] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=141][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  968.267274] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=142][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  968.267275] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=143][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  968.267276] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=144][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  968.267277] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=145][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  968.267278] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=146][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  968.267278] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=147][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  968.267279] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=148][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  968.267280] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=149][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  968.267281] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=150][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  968.267282] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=151][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  968.267283] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=152][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  968.267284] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=153][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  968.267284] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=154][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  968.267285] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=155][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  968.267286] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=156][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  968.267287] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=157][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  968.267288] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=158][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  968.267289] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=159][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  968.267290] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=160][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  968.267291] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=161][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  968.267292] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=162][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  968.267292] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=163][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  968.267293] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=164][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  968.267294] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=165][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  968.267295] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=166][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  968.267296] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=167][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  968.267297] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=168][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  968.267298] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=169][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  968.267298] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=170][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  968.267299] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=171][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  968.267300] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=172][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  968.267301] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=173][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  968.267302] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=174][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  968.267303] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=175][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  968.267304] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=176][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  968.267304] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=177][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  968.267305] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=178][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  968.267306] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=179][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  968.267307] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=180][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  968.267308] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=181][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  968.267309] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=182][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  968.267310] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=183][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  968.267310] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=184][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  968.267311] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=185][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  968.267312] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=186][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  968.267313] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=187][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  968.267314] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=188][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  968.267315] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=189][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  968.267316] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=190][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  968.267317] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=191][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  968.267318] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=192][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  968.267319] [2235] 2334:
[vq=00000000742665e9][vq->last_avail_idx=193][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]

Christian Borntraeger

2020-Feb-14 12:22 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 14.02.20 13:17, Eugenio P?rez wrote:> Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched
version by default")? My intention is to check if
> "strange VHOST_SET_VRING_BASE" line appears. In previous tests,
it appears very fast, but maybe it takes some time for
> it to appear, or it does not appear anymore.
  LD [M]  drivers/vhost/vhost_vsock.o
  CC [M]  drivers/vhost/vhost.o
In file included from ./include/linux/printk.h:331,
                 from ./include/linux/kernel.h:15,
                 from ./include/linux/list.h:9,
                 from ./include/linux/wait.h:7,
                 from ./include/linux/eventfd.h:13,
                 from drivers/vhost/vhost.c:13:
drivers/vhost/vhost.c: In function ?fetch_descs?:
drivers/vhost/vhost.c:2330:56: error: ?struct vhost_virtqueue? has no member
named ?first_desc?
 2330 |   vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs,
vq->first_desc);
      |                                                        ^~
./include/linux/dynamic_debug.h:125:15: note: in definition of macro
?__dynamic_func_call?
  125 |   func(&id, ##__VA_ARGS__);  \
      |               ^~~~~~~~~~~
./include/linux/dynamic_debug.h:153:2: note: in expansion of macro
?_dynamic_func_call?
  153 |  _dynamic_func_call(fmt, __dynamic_pr_debug,  \
      |  ^~~~~~~~~~~~~~~~~~
./include/linux/printk.h:335:2: note: in expansion of macro ?dynamic_pr_debug?
  335 |  dynamic_pr_debug(fmt, ##__VA_ARGS__)
      |  ^~~~~~~~~~~~~~~~
drivers/vhost/vhost.c:2328:2: note: in expansion of macro ?pr_debug?
 2328 |  pr_debug(
      |  ^~~~~~~~
make[2]: *** [scripts/Makefile.build:266: drivers/vhost/vhost.o] Error 1
make[1]: *** [scripts/Makefile.build:503: drivers/vhost] Error 2

Christian Borntraeger

2020-Feb-14 12:34 UTC

head link

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

On 14.02.20 13:26, Eugenio P?rez wrote:> On Fri, 2020-02-14 at 13:22 +0100, Christian Borntraeger wrote:
>>
>> On 14.02.20 13:17, Eugenio P?rez wrote:
>>> Can you try the inlined patch over 52c36ce7f334 ("vhost: use
batched version by default")? My intention is to check
>>> if
>>> "strange VHOST_SET_VRING_BASE" line appears. In previous
tests, it appears very fast, but maybe it takes some time
>>> for
>>> it to appear, or it does not appear anymore.

yep it does:

[   67.801012] [1917] vhost:vhost_vring_ioctl:1655: VHOST_SET_VRING_BASE
[vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=0][s.index=0][s.num=0]
[   67.801018] [1917] vhost:vhost_vring_ioctl:1655: VHOST_SET_VRING_BASE
[vq=00000000175f11ec][vq->last_avail_idx=0][vq->avail_idx=0][s.index=1][s.num=0]
[   67.801026] [1917] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   67.801028] [1917] vhost_net:vhost_net_set_backend:1538:
sock=0000000082d8d291 != oldsock=000000001ae027fd index=0 fd=39
vq=0000000088199421
[   67.801032] [1917] vhost_net:vhost_net_set_backend:1573:
sock=0000000082d8d291
[   67.801033] [1917] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   67.801034] [1917] vhost_net:vhost_net_set_backend:1538:
sock=0000000082d8d291 != oldsock=000000001ae027fd index=1 fd=39
vq=00000000175f11ec
[   67.801035] [1917] vhost_net:vhost_net_set_backend:1573:
sock=0000000082d8d291
[   67.801037] [1915] vhost:vhost_discard_vq_desc:2424: DISCARD
[vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[   68.648803] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1]
[   68.648810] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=1]
[   68.648812] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=1]
[   68.648815] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=1]
[   68.648817] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=1]
[   68.648818] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=1]
[   68.648820] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=1]
[   68.648822] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=1]
[   68.648824] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=1]
[   68.648826] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=1]
[   68.648828] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=1]
[   68.648829] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=1]
[   68.648831] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=1]
[   68.648832] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=1]
[   68.648833] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=1]
[   68.670292] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1]
[   68.687187] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1]
[   68.687623] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=1]
[   68.687641] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=2][vq->avail_idx=4][vq->ndescs=1]
[   68.687642] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=1]
[   68.690274] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=1]
[   68.690539] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1]
[   68.715379] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=1]
[   68.800525] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1]
[   68.890537] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1]
[   68.900587] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1]
[   68.916837] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2]
[   68.928828] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=1]
[   69.090540] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1]
[   69.119651] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2]
[   69.132325] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=1]
[   69.323473] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=2]
[   69.354557] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=1]
[   69.442550] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=1]
[   69.523593] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=2]
[   69.557360] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=1]
[   69.980634] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1]
[   69.981364] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1]
[   70.010545] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1]
[   70.161316] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=1]
[   70.177640] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=1]
[   70.280564] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=1]
[   70.670327] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=1]
[   70.932887] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=2]
[   70.940587] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=1]
[   70.947598] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=1]
[   71.070388] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1]
[   71.070770] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=1]
[   71.070805] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=1]
[   71.070977] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=1]
[   71.071049] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1]
[   71.071195] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=1]
[   71.071243] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1]
[   71.071386] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=1]
[   71.071433] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1]
[   71.071575] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=1]
[   71.071611] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1]
[   71.071747] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=1]
[   71.071789] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1]
[   71.071923] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=1]
[   71.071960] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1]
[   71.072096] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=1]
[   71.072128] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1]
[   71.072267] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=1]
[   71.072300] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1]
[   71.072432] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=1]
[   71.072463] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1]
[   71.072596] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=1]
[   71.072630] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1]
[   71.072759] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=1]
[   71.072791] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1]
[   71.072933] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=1]
[   71.073054] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1]
[   71.073193] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=1]
[   71.073247] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1]
[   71.073383] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=1]
[   71.073434] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1]
[   71.073571] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=1]
[   71.073627] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1]
[   71.073762] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=1]
[   71.073813] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1]
[   71.073948] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=1]
[   71.073998] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1]
[   71.074136] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=1]
[   71.074186] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1]
[   71.074320] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=1]
[   71.074370] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1]
[   71.074503] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=1]
[   72.344493] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=1]
[   72.553413] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=1]
[   73.522174] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=1]
[   73.705202] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=1]
[   73.705239] [1915] vhost:fetch_descs:2328:
[vq=00000000175f11ec][vq->last_avail_idx=41][vq->avail_idx=42][vq->ndescs=1]
[   73.994388] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=1]
[   74.208443] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=1]
[   74.433345] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=1]
[   74.594756] [1910] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   74.594761] [1910] vhost_net:vhost_net_set_backend:1538:
sock=000000001ae027fd != oldsock=0000000082d8d291 index=0 fd=-1
vq=0000000088199421
[   74.594762] [1910] vhost_net:vhost_net_set_backend:1573:
sock=000000001ae027fd
[   74.594803] [1910] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   74.594804] [1910] vhost_net:vhost_net_set_backend:1538:
sock=000000001ae027fd != oldsock=0000000082d8d291 index=1 fd=-1
vq=00000000175f11ec
[   74.594805] [1910] vhost_net:vhost_net_set_backend:1573:
sock=000000001ae027fd
[   74.594847] [1910] vhost:vhost_vring_ioctl:1664: VHOST_GET_VRING_BASE
[vq=0000000088199421][vq->last_avail_idx=54][vq->avail_idx=256][s.index=0][s.num=54]
[   74.594850] [1910] vhost:vhost_vring_ioctl:1664: VHOST_GET_VRING_BASE
[vq=00000000175f11ec][vq->last_avail_idx=42][vq->avail_idx=42][s.index=1][s.num=42]
[   77.003191] [1918] vhost:vhost_vring_ioctl:1645: strange VHOST_SET_VRING_BASE
[vq=0000000088199421][s.index=0][s.num=0]
[   77.003194] CPU: 62 PID: 1918 Comm: CPU 1/KVM Not tainted 5.5.0+ #22
[   77.003197] Hardware name: IBM 3906 M04 704 (LPAR)
[   77.003198] Call Trace:
[   77.003207]  [<0000000b8d93c132>] show_stack+0x8a/0xd0 
[   77.003211]  [<0000000b8e32e72a>] dump_stack+0x8a/0xb8 
[   77.003224]  [<000003ff803567ae>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[   77.003228]  [<000003ff8036c670>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[   77.003234]  [<0000000b8dbecdd8>] do_vfs_ioctl+0x430/0x6f8 
[   77.003235]  [<0000000b8dbed124>] ksys_ioctl+0x84/0xb0 
[   77.003237]  [<0000000b8dbed1ba>] __s390x_sys_ioctl+0x2a/0x38 
[   77.003240]  [<0000000b8e34ff72>] system_call+0x2a6/0x2c8 
[   77.003261] [1918] vhost:vhost_vring_ioctl:1645: strange VHOST_SET_VRING_BASE
[vq=00000000175f11ec][s.index=1][s.num=0]
[   77.003262] CPU: 62 PID: 1918 Comm: CPU 1/KVM Not tainted 5.5.0+ #22
[   77.003263] Hardware name: IBM 3906 M04 704 (LPAR)
[   77.003264] Call Trace:
[   77.003266]  [<0000000b8d93c132>] show_stack+0x8a/0xd0 
[   77.003267]  [<0000000b8e32e72a>] dump_stack+0x8a/0xb8 
[   77.003270]  [<000003ff803567ae>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
[   77.003271]  [<000003ff8036c670>] vhost_net_ioctl+0x510/0x570
[vhost_net]
[   77.003273]  [<0000000b8dbecdd8>] do_vfs_ioctl+0x430/0x6f8 
[   77.003274]  [<0000000b8dbed124>] ksys_ioctl+0x84/0xb0 
[   77.003276]  [<0000000b8dbed1ba>] __s390x_sys_ioctl+0x2a/0x38 
[   77.003277]  [<0000000b8e34ff72>] system_call+0x2a6/0x2c8 
[   77.003297] [1918] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   77.003300] [1918] vhost_net:vhost_net_set_backend:1538:
sock=0000000082d8d291 != oldsock=000000001ae027fd index=0 fd=39
vq=0000000088199421
[   77.003304] [1918] vhost_net:vhost_net_set_backend:1573:
sock=0000000082d8d291
[   77.003305] [1918] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   77.003306] [1918] vhost_net:vhost_net_set_backend:1538:
sock=0000000082d8d291 != oldsock=000000001ae027fd index=1 fd=39
vq=00000000175f11ec
[   77.003308] [1915] vhost:fetch_descs:2328:
[vq=0000000088199421][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=1]
[   77.003308] [1918] vhost_net:vhost_net_set_backend:1573:
sock=0000000082d8d291
[   77.003310] [1915] vhost_net:get_rx_bufs:1061: unexpected descriptor format
for RX: out 0, in 0
[   77.003312] [1915] vhost:vhost_discard_vq_desc:2424: DISCARD
[vq=0000000088199421][vq->last_avail_idx=55][vq->avail_idx=256][n=0]

Reasonably Related Threads

Search for more possibly parallel threads

Linux Virtualization - Feb 2020 - vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

Reasonably Related Threads