Libguestfs - Jul 2022 - nbdkit test-checkwrite.sh failing occasionally on ppc64le

On Wed, Jul 13, 2022 at 03:10:37PM +0200, Laszlo Ersek
wrote:
> On 07/13/22 13:27, Richard W.M. Jones wrote:
> > 
> > It's not 100% reproducible, but it seems to happen reasonably
often.
> > Unfortunately it only happens on ppc64le (so far).  The error message
> > lacks sufficient detail to tell what's going on.
> > 
> > nbdkit: backend.c:624: backend_pread: Assertion `backend_valid_range
(c, offset, count)' failed.
> > nbdkit: backend.c:624: backend_pread: Assertion `backend_valid_range
(c, offset, count)' failed.
> > 
> > Full log attached.
> 
> "tests/disk" has size 105923072.
> 
> Considering all the count=... offset=... lines in the last section of
> the log, there isn't one line where backend_valid_range():
> 
>   return count > 0 && offset <= c->exportsize &&
>     offset + count <= c->exportsize;
> 
> would return false.
> 
> I've printed a table, with offset, count, and offset+count colums, from
> the log entries, then sorted the table three times separately, each time
> by a different column. I got the following min and max elements:
> 
> offset    count     offset+count
> --------- --------- -----------
>         0 105923072   105923072 (min by offset)
> 105906176     16896   105923072 (max by offset)
>         0      4096        4096 (min by count)
>         0 105923072   105923072 (max by count)
>         0      4096        4096 (min by end)
> 105906176     16896   105923072 (max by end)
> 
> All of these look valid... I think you'll have to look at the coredump
> with gdb.
Indeed - unfortunately the coredump is on a Fedora build system :-(

I'll reserve a POWER machine later (unless someone gets there first).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On Wed, Jul 13, 2022 at 02:13:14PM +0100, Richard W.M. Jones
wrote:
> On Wed, Jul 13, 2022 at 03:10:37PM +0200, Laszlo Ersek wrote:
> > On 07/13/22 13:27, Richard W.M. Jones wrote:
> > > 
> > > It's not 100% reproducible, but it seems to happen reasonably
often.
> > > Unfortunately it only happens on ppc64le (so far).  The error
message
> > > lacks sufficient detail to tell what's going on.
> > > 
> > > nbdkit: backend.c:624: backend_pread: Assertion
`backend_valid_range (c, offset, count)' failed.
> > > nbdkit: backend.c:624: backend_pread: Assertion
`backend_valid_range (c, offset, count)' failed.
> > > 
> > > Full log attached.
> > 
> > "tests/disk" has size 105923072.
> > 
> > Considering all the count=... offset=... lines in the last section of
> > the log, there isn't one line where backend_valid_range():
> > 
> >   return count > 0 && offset <= c->exportsize
&&
> >     offset + count <= c->exportsize;
> > 
> > would return false.
> > 
> > I've printed a table, with offset, count, and offset+count colums,
from
> > the log entries, then sorted the table three times separately, each
time
> > by a different column. I got the following min and max elements:
> > 
> > offset    count     offset+count
> > --------- --------- -----------
> >         0 105923072   105923072 (min by offset)
> > 105906176     16896   105923072 (max by offset)
> >         0      4096        4096 (min by count)
> >         0 105923072   105923072 (max by count)
> >         0      4096        4096 (min by end)
> > 105906176     16896   105923072 (max by end)
> > 
> > All of these look valid... I think you'll have to look at the
coredump
> > with gdb.
> 
> Indeed - unfortunately the coredump is on a Fedora build system :-(
> 
> I'll reserve a POWER machine later (unless someone gets there first).
I can't reproduce this after a few hours.  I'll leave it running
overnight, but the bug seems peculiarly rare on this reserved system,
versus Fedora Koji where I observed it quite frequently.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Arrrgh!

  https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
  FAIL: test-checkwrite.sh

Last week I reserved a ppc64le machine which had almost exactly the
same spec as one of the Koji builders and ran both the full test suite
and this single test thousands of times over a few days and it didn't
fail even once.  Ideas gratefully accepted :-/

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top