thr3ads.net - Libguestfs - [Libguestfs] libnbd golang failure on RISC-V [Jun 2022]

If this information is useful, please help other people find it:
Share via:

Richard W.M. Jones

2022-Jun-09 14:08 UTC

[Libguestfs] libnbd golang failure on RISC-V

On Thu, Jun 09, 2022 at 02:14:07PM +0100, Daniel P. Berrang?
wrote:> Setting GODEBUG=cgocheck=1   or GODEBUG=cgocheck=2  can sometimes
> get more info.
Took me a while to work out that we are actually setting that already
(in ./run) which explains a couple of things: why I couldn't reproduce
the error using go build directly, and why things like
"GODEBUG=cgocheck=1 ../run go build" make no difference.
> If you can get the test to core dump, then plain old GDB core
> dump could also be useful - might identify which specific API
> is being called, which can help restrict the size of the haystack
> for code review.
Eventually careful reading of this page revealed how to do this:
https://pkg.go.dev/runtime

The results unfortunately don't seem very useful.  The stack trace
appears to point to where the error was detected rather than where it
was caused, but I've copied it below anyway.

Also attached is the failure in the test suite if you turn *off*
cgocheck, maybe that is useful.

Rich.

$ GODEBUG=cgocheck=2,invalidptr=1 GOTRACEBACK=crash go test -count=1 -v
write of Go pointer 0x3f94026000 to non-Go memory 0x3fbfd6fb20
fatal error: Go pointer stored into non-Go memory

runtime stack:
runtime.dopanic_m
	../../../libgo/go/runtime/panic.go:1211
runtime.fatalthrow
	../../../libgo/go/runtime/panic.go:1071
runtime.throw
	../../../libgo/go/runtime/panic.go:1042
runtime.cgoCheckWriteBarrier..func1
	../../../libgo/go/runtime/cgocheck.go:55
runtime.systemstack..func1
	../../../libgo/go/runtime/stubs.go:63
runtime_mstart
	../../../libgo/runtime/proc.c:593

goroutine 1 [running, locked to thread]:
runtime.mcall
	../../../libgo/runtime/proc.c:343
runtime.systemstack
	../../../libgo/go/runtime/stubs.go:66
runtime.cgoCheckWriteBarrier
	../../../libgo/go/runtime/cgocheck.go:53
runtime.wbBufFlush
	../../../libgo/go/runtime/mwbbuf.go:196
runtime.gcWriteBarrier
	../../../libgo/go/runtime/mgc_gccgo.go:168
runtime.main
	../../../libgo/go/runtime/proc.go:209
Aborted (core dumped)

$ coredumpctl gdb
           PID: 191012 (go)
           UID: 1001 (rjones)
           GID: 1001 (rjones)
        Signal: 6 (ABRT)
     Timestamp: Thu 2022-06-09 10:04:10 EDT (1min 24s ago)
  Command Line: go test -count=1 -v
    Executable: /usr/bin/go.gcc
 Control Group: /user.slice/user-1001.slice/session-11.scope
          Unit: session-11.scope
         Slice: user-1001.slice
       Session: 11
     Owner UID: 1001 (rjones)
       Boot ID: c2a10a42ad324704a32e04b47621d3a9
    Machine ID: 1640aa86edd8444fb3261e75d4390014
      Hostname: nufive.home.annexia.org
       Storage:
/var/lib/systemd/coredump/core.go.1001.c2a10a42ad324704a32e04b47621d3a9.191012.1654783450000000.zst
       Message: Process 191012 (go) of user 1001 dumped core.
                
                Stack trace of thread 191012:
                #0  0x0000003fbe65b084 raise (libc.so.6 + 0x33084)
                #1  0x0000003fbf524cca runtime.dieFromSignal (libgo.so.16 +
0xd49cca)
                #2  0x0000003fbf542bdc runtime.sigfwdgo (libgo.so.16 + 0xd67bdc)
                #3  0x0000003fbfdbb800 n/a (linux-vdso.so.1 + 0x800)

GNU gdb (GDB) Fedora 10.2-4.0.riscv64.fc33
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "riscv64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to
"word"...
Reading symbols from /usr/bin/go.gcc...
[New LWP 191012]
[New LWP 191028]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/lp64d/libthread_db.so.1".
Core was generated by `go test -count=1 -v'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000003fbe65b084 in raise () from /lib64/lp64d/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install
gcc-go-10.3.1-1.fc33.riscv64
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (Thread 0x3fbe5f5630 (LWP 191012))]

(gdb) t a a bt

Thread 2 (Thread 0x3f99563150 (LWP 191028)):
#0  0x0000003fbe6cda60 in syscall () from /lib64/lp64d/libc.so.6
#1  0x0000003fbf246f88 in __go_syscall6 (flag=<optimized out>,
a1=<optimized out>, a2=<optimized out>, a3=<optimized out>,
a4=<optimized out>, a5=<optimized out>, a6=<optimized out>) at
../../../libgo/runtime/go-varargs.c:109
#2  0x0000003fbf51f05c in runtime.futex (val3=<optimized out>,
addr2=<optimized out>, ts=<optimized out>, val=<optimized
out>, op=<optimized out>, addr=<optimized out>) at
../../../libgo/go/runtime/os_linux.go:17
#3  runtime.futexsleep (addr=0x3fbfd6e878 <runtime.sched+240>,
val=<optimized out>, ns=<optimized out>) at
../../../libgo/go/runtime/os_linux.go:59
#4  0x0000003fbf52d70c in runtime.notetsleep_internal (n=n at entry=0x3fbfd6e878
<runtime.sched+240>, ns=ns at entry=60000000000) at
../../../libgo/go/runtime/lock_futex.go:204
#5  0x0000003fbf52d8ce in runtime.notetsleep (n=n at entry=0x3fbfd6e878
<runtime.sched+240>, ns=60000000000) at
../../../libgo/go/runtime/lock_futex.go:227
#6  0x0000003fbf54de9c in runtime.sysmon () at
../../../libgo/go/runtime/proc.go:4210
#7  0x0000003fbf5501a6 in runtime.mstart1 () at
../../../libgo/go/runtime/proc.go:1074
#8  0x0000003fbf24779e in runtime_mstart (arg=<optimized out>) at
../../../libgo/runtime/proc.c:611
#9  0x0000003fbe613606 in start_thread () from /lib64/lp64d/libpthread.so.0
#10 0x0000003fbe6d028e in __thread_start () from /lib64/lp64d/libc.so.6

Thread 1 (Thread 0x3fbe5f5630 (LWP 191012)):
#0  0x0000003fbe65b084 in raise () from /lib64/lp64d/libc.so.6
#1  0x0000003fbf524cca in runtime.dieFromSignal (sig=sig at entry=6) at
../../../libgo/go/runtime/signal_unix.go:668
#2  0x0000003fbf542bdc in runtime.sigfwdgo (ctx=0x3f99973c40, info=0x3f99973bc0,
sig=<optimized out>) at ../../../libgo/go/runtime/signal_unix.go:883
#3  runtime.sigtrampgo (sig=<optimized out>, info=0x3f99973bc0,
ctx=0x3f99973c40) at ../../../libgo/go/runtime/signal_unix.go:392
#4  <signal handler called>
#5  0x0000003fbe65b084 in raise () from /lib64/lp64d/libc.so.6
#6  0x0000003fbf524cca in runtime.dieFromSignal (sig=sig at entry=6) at
../../../libgo/go/runtime/signal_unix.go:668
#7  0x0000003fbf524d42 in runtime.crash () at
../../../libgo/go/runtime/signal_unix.go:763
#8  0x0000003fbf529132 in runtime.fatalthrow () at
../../../libgo/go/runtime/panic.go:1072
#9  runtime.throw (s=...) at ../../../libgo/go/runtime/panic.go:1042
#10 0x0000003fbf5291c4 in runtime.cgoCheckWriteBarrier..func1 () at
../../../libgo/go/runtime/cgocheck.go:55
#11 0x0000003fbf51ff3e in runtime.systemstack..func1 (origg=0x3f94000a80) at
../../../libgo/go/runtime/stubs.go:63
#12 0x0000003fbf24776e in runtime_mstart (arg=<optimized out>) at
../../../libgo/runtime/proc.c:593
#13 0x0000000000035abe in main (argc=<optimized out>, argv=<optimized
out>) at ../../../libgo/runtime/go-main.c:58
#14 0x0000003fbe64b956 in __libc_start_main () from /lib64/lp64d/libc.so.6
#15 0x0000000000035afc in _start ()



-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
-------------- next part --------------
+ requires nbdkit --version
+ go test -count=1 -v
=== RUN   Test010Load
--- PASS: Test010Load (0.00s)
=== RUN   TestAioBuffer
--- PASS: TestAioBuffer (0.00s)
=== RUN   TestAioBufferFree
--- PASS: TestAioBufferFree (0.00s)
=== RUN   TestAioBufferBytesAfterFree
SIGABRT: abort
PC=0x3fdf6f9bac m=0 sigcode=18446744073709551610

goroutine 9 [running]:
gsignal
	:0
abort
	:0
panic
	../../../libgo/go/runtime/panic.go:750
_wordcopy_fwd_aligned
	:0
__GI_memmove
	:0

goroutine 1 [chan receive]:
testing.T.Run
	../../../libgo/go/testing/testing.go:1127
testing.runTests..func1
	../../../libgo/go/testing/testing.go:1370
testing.tRunner
	../../../libgo/go/testing/testing.go:1075
testing.runTests
	../../../libgo/go/testing/testing.go:1368
testing.M.Run
	../../../libgo/go/testing/testing.go:1285
main.main
	/tmp/go-build/b001/_testmain.go:108

exit status 2
FAIL	libguestfs.org/libnbd	1.691s
++ _run_cleanup_hooks
++ local _status=1 _i
++ set +e
++ trap '' INT QUIT TERM EXIT ERR
++ echo /home/rjones/d/libnbd/golang/run-tests.sh: run cleanup hooks: exit code
1
/home/rjones/d/libnbd/golang/run-tests.sh: run cleanup hooks: exit code 1
++ (( _i = 0 ))
++ (( _i < 0 ))
++ exit 1
FAIL run-tests.sh (exit status: 1)

Daniel P. Berrangé

2022-Jun-09 14:20 UTC

head link

[Libguestfs] libnbd golang failure on RISC-V

On Thu, Jun 09, 2022 at 03:08:51PM +0100, Richard W.M. Jones
wrote:> On Thu, Jun 09, 2022 at 02:14:07PM +0100, Daniel P. Berrang? wrote:
> > If you can get the test to core dump, then plain old GDB core
> > dump could also be useful - might identify which specific API
> > is being called, which can help restrict the size of the haystack
> > for code review.
> 
> Eventually careful reading of this page revealed how to do this:
> https://pkg.go.dev/runtime
> 
> The results unfortunately don't seem very useful.  The stack trace
> appears to point to where the error was detected rather than where it
> was caused, but I've copied it below anyway.
> 
> Also attached is the failure in the test suite if you turn *off*
> cgocheck, maybe that is useful.
> -- 
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> nbdkit - Flexible, fast NBD server with plugins
> https://gitlab.com/nbdkit/nbdkit
> + requires nbdkit --version
> + go test -count=1 -v
> === RUN   Test010Load
> --- PASS: Test010Load (0.00s)
> === RUN   TestAioBuffer
> --- PASS: TestAioBuffer (0.00s)
> === RUN   TestAioBufferFree
> --- PASS: TestAioBufferFree (0.00s)
> === RUN   TestAioBufferBytesAfterFree
> SIGABRT: abort
> PC=0x3fdf6f9bac m=0 sigcode=18446744073709551610
So suggesting TestAioBufferBytesAfterFree is as fault, but quite
odd as that test case is trivial and whle it allocates some
native memory it doesn't seem to write to it. Unless the problem
happened in an earlier test case and we have delayed detection ?

I guess I'd try throwing darts at the wall by chopping out bits
of test code to see what makes it disappear.

Perhaps also try swapping MakeAioBuffer with MakeAioBufferZero
in case pre-existing data into the C.malloc()d block is confusing
Go ? 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Libguestfs - Jun 2022 - libnbd golang failure on RISC-V

[Libguestfs] libnbd golang failure on RISC-V

[Libguestfs] libnbd golang failure on RISC-V