Xen devel - Feb 2004 - megaraid problem on hp netserver lh6000 - ignorebiostables o.k.

testresults from XEN1.0 and XEN1.2 democd shows following:

XEN1.0 democd default boot
   o.k. ... but no smp

XEN1.2 democd default boot 2.4.25
   stop at ... megaraid scanning raidchannel with error:

      scsi0: scanning virtual channel0 for scsi devices.
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      (... endless)

XEN1.2 democd safemode
   o.k. ... but no smp
XEN1.2 democd default boot 2.4.25
   o.k. with xen-kernel parameter ignorebiostables ... but no smp

XEN1.2 democd ... linux 2.4.22
   o.k. (with smp)

XEN1.2 democd ... linux 2.4.24
XEN1.2 democd ... linux 2.4.25
   stop at ... VFS: Cannot open root device "ram0" or 01:00


i prepared two LH6000/4CPU/4GB for XEN-production :o(
so what next?

   Manni

> i prepared two LH6000/4CPU/4GB for XEN-production :o(
> so what next?
> XEN1.2 democd ... linux 2.4.24
Please boot linux and send us the output of dmesg, "lspci -v" and
"cat /proc/cpuinfo"

It sounds like the BIOS is passing in some weird stuff in the
tables. dmesg output may help us pin this down. It might be worth
doing a BIOS upgrade if one is available.

We run Xen on HP DL360''s but have never seen a netserver lh6000.


Ian


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> 
> > i prepared two LH6000/4CPU/4GB for XEN-production :o(
> > so what next?
> 
> > XEN1.2 democd ... linux 2.4.24
> 
> Please boot linux and send us the output of dmesg, "lspci -v" and
> "cat /proc/cpuinfo"
> 
> It sounds like the BIOS is passing in some weird stuff in the
> tables. dmesg output may help us pin this down. It might be worth
> doing a BIOS upgrade if one is available.
the latest BIOS/Firmware-upgrades are done
   BIOS       4.06.45S
   MngmtCtrl. E.10.44
   Netraid    B.02.04
              E.01.10
> 
> We run Xen on HP DL360''s but have never seen a netserver lh6000.
> 
> 
> Ian
one more debug info:
   same error for Adaptec7880 and megaraid

> i prepared two LH6000/4CPU/4GB for XEN-production :o(
> so what next?
Well, we have never booted that chipset, we have never booted with
that many I/O devices configured, we have never booted that many CPUs,
and we have never booted with that amount of memory. So it''s fair to
say that you''re outside the tested operational envelope. :-)

I''d also guess that this may be non-trivial to debug.

 -- Keir


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > i prepared two LH6000/4CPU/4GB for XEN-production :o(
> > so what next?
> 
> Well, we have never booted that chipset, we have never booted with
> that many I/O devices configured, we have never booted that many CPUs,
> and we have never booted with that amount of memory. So it''s fair
to
> say that you''re outside the tested operational envelope. :-)
> 
uuupppsss :-)
> I''d also guess that this may be non-trivial to debug.
> 
>  -- Keir
So what would you suggest?

1. Sell this servers and get smaller an newer systems.
or
2. This systems are very good for quality assurance. You can test
   many cpu´s, nic´s and other i/o.

If "2." then:
I own 3 LH6000 systems. Server1 with 10HDs, Server2 with 6HDs and
Server3 with 2HDs.
Server1 is for production.
Server2 for a while we can run reliability and load tests.
Server3 is free for testing new configs/changes.

best regards
   Manni

> > > i prepared two LH6000/4CPU/4GB for XEN-production :o(
> > > so what next?
Judging from the Linux boot output, this looks like a 5 (!) CPU
system. Is this correct? 

You''re definitely outside the tested envelope with this system...

A few things to try:

1) Remove half the RAM (2GB). We have some 4GB systems that work
fine, but its possible there''s a clash with IO memory space.

2) Remove all but one of the Ethernet cards. Multiple Ethernet
cards shouldn''t cause problems, but we''ve never used multiple
of
these particular cards. (You can''t currently use more than one
card anyhow, but this will be fixed relatively soon)

3) Boot a 2.4.16 version of Linux (e.g. from the 1.0
XenDemoCD). If that fails, step forward versions until you find
the one where it starts working. (It''s probably a BIOS table or
weird APIC issue)

4) Boot Xen with a serial line connected so that you can capture
the serial output to send to us.


Ian


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> So what would you suggest?
> 
> 1. Sell this servers and get smaller an newer systems.
> or
> 2. This systems are very good for quality assurance. You can test
>    many cpu´s, nic´s and other i/o.
> 
> If "2." then:
> I own 3 LH6000 systems. Server1 with 10HDs, Server2 with 6HDs and
> Server3 with 2HDs.
> Server1 is for production.
> Server2 for a while we can run reliability and load tests.
> Server3 is free for testing new configs/changes.
I think Ian is going to mail you some suggestions of his own.

I would suggest pulling some of the hardware out of one of teh boxes
and see if a more ''minimal'' config gets you anywhere:
 e.g., one cpu, one network card, one disc, 1GB memory.

If we can get you to a working configuration it makes debugging much
easier! 

 -- Keir


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> 
> > > > i prepared two LH6000/4CPU/4GB for XEN-production :o(
> > > > so what next?
> 
> Judging from the Linux boot output, this looks like a 5 (!) CPU
> system. Is this correct? 
> 
yes ... LH6000 is up to 6 CPU, I thought it´s good for QOS resource
        sharing if I can pin domains to a "free" cpu.
> You''re definitely outside the tested envelope with this system...
> 
not so good for production use :o)
>
> A few things to try:
> 
> 1) Remove half the RAM (2GB). We have some 4GB systems that work
> fine, but its possible there''s a clash with IO memory space.
> 
ack
>
> 2) Remove all but one of the Ethernet cards. Multiple Ethernet
> cards shouldn''t cause problems, but we''ve never used
multiple of
> these particular cards. (You can''t currently use more than one
> card anyhow, but this will be fixed relatively soon)
> 
ack
> 3) Boot a 2.4.16 version of Linux (e.g. from the 1.0
> XenDemoCD). If that fails, step forward versions until you find
> the one where it starts working. (It''s probably a BIOS table or
> weird APIC issue)
> 
? sorry ... what to do in detail ?
On my XenDemoCD 1.0 download/22.01.2004 is only 2.4.22 as a boot
config.
This XenoLinux config booting is o.k.!
In dmesg only one cpu.
With mii-tools no network interface is accessible.
> 4) Boot Xen with a serial line connected so that you can capture
> the serial output to send to us.
>
ack ... need equipment for it ... next week
> 
> Ian
   Manni

> I think Ian is going to mail you some suggestions of his own.
> 
> I would suggest pulling some of the hardware out of one of teh boxes
> and see if a more ''minimal'' config gets you anywhere:
>  e.g., one cpu, one network card, one disc, 1GB memory.
> 
> If we can get you to a working configuration it makes debugging much
> easier! 
> 
ack ...

   Manni

> > 3) Boot a 2.4.16 version of Linux (e.g. from the 1.0
> > XenDemoCD). If that fails, step forward versions until you find
> > the one where it starts working. (It''s probably a BIOS table
or
> > weird APIC issue)
> > 
> ? sorry ... what to do in detail ?
> On my XenDemoCD 1.0 download/22.01.2004 is only 2.4.22 as a boot
> config.
> This XenoLinux config booting is o.k.!
> In dmesg only one cpu.
> With mii-tools no network interface is accessible.
Hang-on, you mean the Linux 2.4.22 kernel is only finding one
CPU? I though an early message said it was OK SMP i.e. ''top''
shows 5 CPUs etc.

If 2.4.22 does work, please can you try compiling up a 2.4.20 to
see if that breaks. (there''s a 2.4.16 on the *1.0* version of the
demoCD.)

If we can find the most recent version that fails to work, that
will give us a clue as to what we''re missing.

Having the Xen boot output will helpful too. Has this machine got
builtin-in remote console facilities that you could cut and paste
from instead of setting up a serial line?

Cheers,
Ian


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2004-02-27 at 09:46, Manfred.Herrmann@zipptec.de
wrote:
> XEN1.2 democd default boot 2.4.25
>    stop at ... megaraid scanning raidchannel with error:
> 
>       scsi0: scanning virtual channel0 for scsi devices.
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       (... endless)

I can also confirm this behavior with a Dell PowerEdge, dual Xeon, 2GB
system with the MPT Fusion (LSI1030) SCSI controller. 

This happens from ''time to time'' with the driver I ported way
back in
November. I brushed it off as a driver problem as its awful code. Of
course, we do have reports of the thing working quite well with a domain
configured sufficiently.

My machine is well within the specs too ;) 

This machine has 3 SCSI controllers (aic79xx, sym83cxx_2 and the
mptfusion) and I think I only see the above when there is a drive
plugged in to the mptfusion controller. 

I do have a spark of a memory creating a patch that timed out in
scsi.c:scsi_wait_req because this appeared with _NO_ drives attached and
the driver enabled.

Actually, the more I think about it the more I remeber that when this
happened something strange was going on with the DMA for the
request/reply frames.. and a quick check of some debug output confirms
that.

When this happens with the MPT Fusion controller you see this in the
initialization and subsequent commands:

http://www.clarkson.edu/~evanchsa/xen/debug/dma.txt

and then later that day I made changes to get this:

http://www.clarkson.edu/~evanchsa/xen/debug/dma.111003.txt

I now recall that I got rid of some of their PCI handling code as it was
no longer needed in the newer kernels. I think that was the change I
mentioned above. 



I could be completely wrong of course ;) I''m not a pro at this by any
means.



The machine is currently being used so I''ll have to schedule some time
to check in to this more.



Stephen



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

XenDemoCD 1.2 - The problem exist with more than 1 CPU.

  Following configurations are o.k. with 1 CPU:
   4GByte RAM o.k.
   Adaptec AIC7880 o.k.
   Netraid o.k.
   5 NICs (4 3C905 + 1 EEPRO100) o.k.
  Multiple domains are o.k. (ssh login possible)

XenDemoCD 1.0 - No boot problem, dom0 o.k.

  4 CPUs

[root@xendemo0 root]# xenctl domain list
id: 0 (Domain-0)
  processor: 0
  has cpu: true
  state: 0 running
  mcu advance: 10
  total pages: 25000
id: 1 (XenoLinux)
  processor: 1
  has cpu: false         <--- what is "false" ?
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 2 (XenoLinux)
  processor: 2
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 3 (XenoLinux)
  processor: 3
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 4 (XenoLinux)
  processor: 0
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 5 (XenoLinux)
  processor: 1
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 6 (XenoLinux)
  processor: 2
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 7 (XenoLinux)
  processor: 3
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
id: 8 (XenoLinux)
  processor: 0
  has cpu: false
  state: 8 suspended
  mcu advance: 10
  total pages: 24576
[root@xendemo0 root]#

> 
> > > 3) Boot a 2.4.16 version of Linux (e.g. from the 1.0
> > > XenDemoCD). If that fails, step forward versions until you find
> > > the one where it starts working. (It''s probably a BIOS
table or
> > > weird APIC issue)
> > > 
> > ? sorry ... what to do in detail ?
> > On my XenDemoCD 1.0 download/22.01.2004 is only 2.4.22 as a boot
> > config.
> > This XenoLinux config booting is o.k.!
> > In dmesg only one cpu.
> > With mii-tools no network interface is accessible.
> 
> Hang-on, you mean the Linux 2.4.22 kernel is only finding one
> CPU? I though an early message said it was OK SMP i.e.
''top''
> shows 5 CPUs etc.
> 
> If 2.4.22 does work, please can you try compiling up a 2.4.20 to
> see if that breaks. (there''s a 2.4.16 on the *1.0* version of the
> demoCD.)
> 
Native Linux 2.4.22 from XenDemoCD 1.2 is o.k..
Do you mean a standard Linux as debian 2.4.18?

Give me a detailed instruction what and how ... and I will try it.
> If we can find the most recent version that fails to work, that
> will give us a clue as to what we''re missing.
> 
> Having the Xen boot output will helpful too. Has this machine got
> builtin-in remote console facilities that you could cut and paste
> from instead of setting up a serial line?
> 
sorry, I don´t know about the remote console features
> Cheers,
> Ian

> XenDemoCD 1.2 - The problem exist with more than 1 CPU.
> 
>   Following configurations are o.k. with 1 CPU:
>    4GByte RAM o.k.
>    Adaptec AIC7880 o.k.
>    Netraid o.k.
>    5 NICs (4 3C905 + 1 EEPRO100) o.k.
>   Multiple domains are o.k. (ssh login possible)
> 
> XenDemoCD 1.0 - No boot problem, dom0 o.k.
> 
>   4 CPUs
So, just to make sure I understand:

  XenDemoCD 1.0 works fine with 4 CPUs installed. What about 5?

  XenDemoCD 2.0 works OK with one CPU, but fails with 4 and
5. What''s the output when it fails? What about 2 CPUs?
> [root@xendemo0 root]# xenctl domain list
> id: 1 (XenoLinux)
>   processor: 1
>   has cpu: false         <--- what is "false" ?
It just means that the domain happens not to be using the CPU at
this instant (i.e. it''s idle). Totally normal.


Ian


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > XenDemoCD 1.2 - The problem exist with more than 1 CPU.
> > 
> >   Following configurations are o.k. with 1 CPU:
> >    4GByte RAM o.k.
> >    Adaptec AIC7880 o.k.
> >    Netraid o.k.
> >    5 NICs (4 3C905 + 1 EEPRO100) o.k.
> >   Multiple domains are o.k. (ssh login possible)
>
I will refer to this TEST1
> > 
> > XenDemoCD 1.0 - No boot problem, dom0 o.k.
> > 
> >   4 CPUs
I will refer to this TEST2
> 
> So, just to make sure I understand:
> 
>   XenDemoCD 1.0 works fine with 4 CPUs installed. What about 5?
> 
Yes XenDemoCD 1.0 (TEST2) boot is o.k. no more tests done.
If it make sense I will test domain starting, ssh ...

More Info about 5 CPUs:
1. Only "server1" has 5 CPUs (TEST1 1 CPU o.k., 2CPUs fail, 5CPUs
fail)
2. TEST2 was on "server2" (4 CPUs)
>   XenDemoCD 2.0 works OK with one CPU, but fails with 4 and
> 5. What''s the output when it fails? What about 2 CPUs?
> 
With 2 CPUs fail.

Output when it fails:
    megaraid scanning raidchannel with error:
      scsi0: scanning virtual channel0 for scsi devices.
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      (... endless)

Helpful info from a previous TESTx:
    BIOS AIC-7880 enabled + Netraid (megaraid) enabled
    Bootsequence 1. AIC-7880, 2. Netraid

    Output when it fails:
      ... "adaptec scanning" ... (this text is not copy/paste)
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      scsi_wait_req: still waiting...!
      (... endless)
> > [root@xendemo0 root]# xenctl domain list
> > id: 1 (XenoLinux)
> >   processor: 1
> >   has cpu: false         <--- what is "false" ?
> 
> It just means that the domain happens not to be using the CPU at
> this instant (i.e. it''s idle). Totally normal.
> 
very good :-)

> More Info about 5 CPUs:
> 1. Only "server1" has 5 CPUs (TEST1 1 CPU o.k., 2CPUs fail, 5CPUs
fail)
OK, so whenever you get a fail its a problem with the megaraid
driver looping? 

Can you bring the machine up using either NFS root or a spare IDE
driver?  I''d like to know whether interrupt routeing is totally
screwed with >1 CPU, or whether its just a problem with the
megaraid driver and the network drivers are OK.

Judging from Stephen''s comment, it sounds like there''s a known
problem with the megaraid driver. 

A sensible first move would be to upgrade the driver from the
current version 1.18d to the latest 1.18k from the 2.4.25 tree.

Ian




-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > More Info about 5 CPUs:
> > 1. Only "server1" has 5 CPUs (TEST1 1 CPU o.k., 2CPUs fail,
5CPUs
fail)
> 
> OK, so whenever you get a fail its a problem with the megaraid
> driver looping? 
> 
Not only megaraid, with builtin aic7880 enabled the same behaviour.

More testresults with XenDemoCD 1.0:
All "pass"
- CPU o.k. (dom0=CPU0, dom1=CPU1, dom2=CPU2)
- networking o.k. (scp from dom2 to dom0 /dev/sdb1)
- megaraid drive access o.k.
> Can you bring the machine up using either NFS root or a spare IDE
> driver?  I''d like to know whether interrupt routeing is totally
> screwed with >1 CPU, or whether its just a problem with the
> megaraid driver and the network drivers are OK.
> 
I will try it.
>
> Judging from Stephen''s comment, it sounds like there''s a
known
> problem with the megaraid driver. 
> 
> A sensible first move would be to upgrade the driver from the
> current version 1.18d to the latest 1.18k from the 2.4.25 tree.
> 
I will try it.

Thanks a lot!

   Manni

> >   XenDemoCD 2.0 works OK with one CPU, but fails with 4 and
> > 5. What''s the output when it fails? What about 2 CPUs?
> > 
> With 2 CPUs fail.
> 
> Output when it fails:
>     megaraid scanning raidchannel with error:
>       scsi0: scanning virtual channel0 for scsi devices.
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       (... endless)
> 
> Helpful info from a previous TESTx:
>     BIOS AIC-7880 enabled + Netraid (megaraid) enabled
>     Bootsequence 1. AIC-7880, 2. Netraid
> 
>     Output when it fails:
>       ... "adaptec scanning" ... (this text is not copy/paste)
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       scsi_wait_req: still waiting...!
>       (... endless)

Can you try appling the following minor patch (against 1.2) and 
retrying? It should essentially just terminate the endless printks
and allow us to see when (if?) it falls over later on. 

cheers, 

S.

-------------------------------------------------------------------------
# This is a BitKeeper generated patch for the following project:
# Project Name: Xen Virtual Machine Monitor source code
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
#                  ChangeSet    1.742   -> 1.743  
#       xen/drivers/scsi/scsi.c 1.10    -> 1.11   
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 04/03/04      smh22@tempest.cl.cam.ac.uk      1.743
# debugging tweaks
# --------------------------------------------
#
diff -Nru a/xen/drivers/scsi/scsi.c b/xen/drivers/scsi/scsi.c
--- a/xen/drivers/scsi/scsi.c   Thu Mar  4 14:57:22 2004
+++ b/xen/drivers/scsi/scsi.c   Thu Mar  4 14:57:22 2004
@@ -244,8 +244,8 @@
     }
 #else 
     /* XXX SMH: just use a flag to signal completion; caller spins */
-    if (*(int *)(req->waiting) != 0) {
-//        printk("scsi_wait_done: flipping wait status on req %p\n",
req);
+    if (*(volatile int *)(req->waiting) != 0) {
+        printk("scsi_wait_done: flipping wait status on req %p\n",
req);
         *(int *)(req->waiting) = 0; 
     }
 #endif
@@ -812,6 +812,7 @@
 #else 
     int wait = 1; 
     int usecs = 0;
+    int nsecs = 0; 
 #endif
 
 
@@ -845,8 +846,16 @@
         usecs += 500; 
         if(usecs > 1000000) {
             printk("scsi_wait_req: still waiting...!\n"); 
+            nsecs = nsecs++; 
             usecs = 0; 
         } 
+
+        if(nsecs > 10) {
+            printk("scsi_wait_req: bored of waiting (wait is %d, &wait
%p)\n",
+                   wait, &wait); 
+            wait = 0; 
+        }
+        
     }
 #endif




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Can you try appling the following minor patch (against 1.2) and 
> retrying? It should essentially just terminate the endless printks
> and allow us to see when (if?) it falls over later on. 
Oh, and please fix the typo ("nsecs = nsecs++;" should of course be
"nsecs = nsecs + 1;" :-) 

S.