Hi, I''m working on some kind of microOS for Xen on ARM a bit like miniOS but with less features. I''m having totally random crashes after make use of variadic arguments ( using newlib ). To debug this I''d really need a way to force a register dump from the guest, is there a good way to do this? I don''t think any of the available hypercalls has that ability? Regards, Sander
On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote:> Hi, > > I''m working on some kind of microOS for Xen on ARM a bit like miniOS > but with less features. I''m having totally random crashes after make > use of variadic arguments ( using newlib ). > > To debug this I''d really need a way to force a register dump from the > guest, is there a good way to do this? I don''t think any of the > available hypercalls has that ability?Sounds like you want "xenctx" which is in tools/xentrace/ Ian
On 5 June 2013 17:11, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote: >> Hi, >> >> I''m working on some kind of microOS for Xen on ARM a bit like miniOS >> but with less features. I''m having totally random crashes after make >> use of variadic arguments ( using newlib ). >> >> To debug this I''d really need a way to force a register dump from the >> guest, is there a good way to do this? I don''t think any of the >> available hypercalls has that ability? > > Sounds like you want "xenctx" which is in tools/xentrace/I doesn''t show any output when invoked. Is it supposed to work for Xen on ARM? It''s also not really what I need, some hypercall or other break that crashes the domain and does a show_execution_state() is more like it. That way I can test how far in my code I get before the crash and see the state that leads to it. It might be easiest to trap something in Xen ( or use the SMC trap ) and run a custom Xen build then? My application crashes whenever I use variadic arguments, the crashes are totally random and sometimes non-existent, if this would raise any red flags in your brain I''d be very happy to hear some suggestions too :p But my main concern is some way to debug as described. Thanks!
On Wed, 2013-06-05 at 20:18 +0200, Sander Bogaert wrote:> On 5 June 2013 17:11, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote: > >> Hi, > >> > >> I''m working on some kind of microOS for Xen on ARM a bit like miniOS > >> but with less features. I''m having totally random crashes after make > >> use of variadic arguments ( using newlib ). > >> > >> To debug this I''d really need a way to force a register dump from the > >> guest, is there a good way to do this? I don''t think any of the > >> available hypercalls has that ability? > > > > Sounds like you want "xenctx" which is in tools/xentrace/ > > I doesn''t show any output when invoked. Is it supposed to work for Xen > on ARM?Yes, although you will need a new enough tree, where new enough means at least 66425df7ce61, from mid-April> It''s also not really what I need, some hypercall or other > break that crashes the domain and does a show_execution_state() is > more like it.Well, such a thing would be trivial to write, but checkout do_debug_trap in xen/arch/arm/traps.c, you probably want "hvc #0xffff" in your code.> My application crashes whenever I use variadic arguments,crashes how, segfault or something else? Where did your stdargs.h come from? I''d have thought the compiler provided stuff ought to be reasonably robust assuming your application isn''t buggy. Ian.
On 5 June 2013 20:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Wed, 2013-06-05 at 20:18 +0200, Sander Bogaert wrote: >> On 5 June 2013 17:11, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> > On Wed, 2013-06-05 at 16:36 +0200, Sander Bogaert wrote: >> >> Hi, >> >> >> >> I''m working on some kind of microOS for Xen on ARM a bit like miniOS >> >> but with less features. I''m having totally random crashes after make >> >> use of variadic arguments ( using newlib ). >> >> >> >> To debug this I''d really need a way to force a register dump from the >> >> guest, is there a good way to do this? I don''t think any of the >> >> available hypercalls has that ability? >> > >> > Sounds like you want "xenctx" which is in tools/xentrace/ >> >> I doesn''t show any output when invoked. Is it supposed to work for Xen >> on ARM? > > Yes, although you will need a new enough tree, where new enough means > at least 66425df7ce61, from mid-AprilAs you may have read I''m having an issue with the latest Xen / linux kernel at the moment. I haven''t been able to find a working combination of commits in both trees yet to get this functionallity in and I was smart enough not to write down the hashes of the working setup I do have :-)>> It''s also not really what I need, some hypercall or other >> break that crashes the domain and does a show_execution_state() is >> more like it. > > Well, such a thing would be trivial to write, but checkout do_debug_trap > in xen/arch/arm/traps.c, you probably want "hvc #0xffff" in your code.This seems perfect! I''l try using these traps.>> My application crashes whenever I use variadic arguments, > > crashes how, segfault or something else?It''s not as much an application as my "micro" OS :) The crashes are totally random but it''s always the domain that crashes. Sometimes a translation fault, sometimes some "unexpected hypervisor trap" ending up in SVC or even IRQ mode ( which shouldn''t even be enabled on domain boot?! ). Maybe it''s related to the segfaulting Julien mailed about because it always seems to be on CPU 1 according to Xen''s crash dump and maybe when it just works and doesn''t crash it''s scheduled on CPU 0? I haven''t done enough testing to be sure this behavior is consistent.> Where did your stdargs.h come from? I''d have thought the compiler > provided stuff ought to be reasonably robust assuming your application > isn''t buggy.I''m using newlib for now.
At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote:> My application crashes whenever I use variadic arguments, the crashes > are totally random and sometimes non-existent, if this would raise any > red flags in your brain I''d be very happy to hear some suggestions tooIn the early stages of the Xen ARM port we had intermittent problems with varadics when the stack pointer got misaligned -- the toolchain''s code for laying out stack arguments assumes that the stack pointer is properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do some confusing things if not. Cheers, Tim.
On 6 June 2013 11:33, Tim Deegan <tim@xen.org> wrote:> At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote: >> My application crashes whenever I use variadic arguments, the crashes >> are totally random and sometimes non-existent, if this would raise any >> red flags in your brain I''d be very happy to hear some suggestions too > > In the early stages of the Xen ARM port we had intermittent problems > with varadics when the stack pointer got misaligned -- the toolchain''s > code for laying out stack arguments assumes that the stack pointer is > properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do > some confusing things if not.Thank you for the suggestion. I did however verify the stack is properly aligned in the linkerscript. I tried a bunch of things and I''m getting nowhere so even your craziest ideas (if you have any) are more than welcome! Thanks
At 16:48 +0200 on 07 Jun (1370623706), Sander Bogaert wrote:> On 6 June 2013 11:33, Tim Deegan <tim@xen.org> wrote: > > At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote: > >> My application crashes whenever I use variadic arguments, the crashes > >> are totally random and sometimes non-existent, if this would raise any > >> red flags in your brain I''d be very happy to hear some suggestions too > > > > In the early stages of the Xen ARM port we had intermittent problems > > with varadics when the stack pointer got misaligned -- the toolchain''s > > code for laying out stack arguments assumes that the stack pointer is > > properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do > > some confusing things if not. > > Thank you for the suggestion. I did however verify the stack is > properly aligned in the linkerscript.It needs a bit more than that -- the stack pointer must be 8-byte aligned on all (visible) function calls. In our case, although the stack area was properly aligned, there were some places (in asm code) where we called C functions with the stack only 4-byte aligned. Anyway, if the application runs fine on linux without Xen, I guess we should be looking at the Xen code rather than yours. :) If it''s really non-deterministic, even with the same call stacks, then maybe something is messed up with interrupt delivery. It might be that Xen''s return-from-interrupt code doesn''t unwind non-aligned stacks properly. Tim.
At 16:41 +0100 on 07 Jun (1370623276), Tim Deegan wrote:> At 16:48 +0200 on 07 Jun (1370623706), Sander Bogaert wrote: > > On 6 June 2013 11:33, Tim Deegan <tim@xen.org> wrote: > > > At 20:18 +0200 on 05 Jun (1370463493), Sander Bogaert wrote: > > >> My application crashes whenever I use variadic arguments, the crashes > > >> are totally random and sometimes non-existent, if this would raise any > > >> red flags in your brain I''d be very happy to hear some suggestions too > > > > > > In the early stages of the Xen ARM port we had intermittent problems > > > with varadics when the stack pointer got misaligned -- the toolchain''s > > > code for laying out stack arguments assumes that the stack pointer is > > > properly (i.e. 8-byte, on 32-bit ARM!) aligned to start with, and can do > > > some confusing things if not. > > > > Thank you for the suggestion. I did however verify the stack is > > properly aligned in the linkerscript. > > It needs a bit more than that -- the stack pointer must be 8-byte aligned > on all (visible) function calls. In our case, although the stack area > was properly aligned, there were some places (in asm code) where we > called C functions with the stack only 4-byte aligned. > > Anyway, if the application runs fine on linux without XenLooking back at the thread, I realise this makes no sense. Sorry. :) Tim.> If it''s really > non-deterministic, even with the same call stacks, then maybe something > is messed up with interrupt delivery. It might be that Xen''s > return-from-interrupt code doesn''t unwind non-aligned stacks properly. > > Tim. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel