Xen devel - Feb 2011 - re:RE: blue screen in windows balloon driver

Thanks James.
 
I think it is GPLPV. The driver is from
http://xenbits.xen.org/ext/win-pvdrivers.hg
But, I have done some other things
 
1) Add pod support 
2) enable a meminfo thread, periodically write VM meminfo into xenstore 
    We use info of Current Memory, Free memory, and Committed memory, retrived
through NativeAPI
3) our code is based from changeset 823, attached is the diff of my current code
with changset of 853.
 
Maybe I need add my code to 853, and test again.
Thanks.
 
>From: James Harper [mailto:james.harper@bendigoit.com.au] 
>To: xiaoyun.maoxy; xen-devel@lists.xensource.com
>cc: george.dunlap@eu.citrix.com
>Subject: RE: blue screen in windows balloon driver
>
>> Hi:
>>      We''ve confronted the windows 2003 Blue screen when using
balloon
>> driver.
>
>>      The blue screen has the information of
"NO_PAGES_AVAILABLE", and
>>      Technical information:
>>      ***STOP: 0x0000004D (0x0002A8FB,0x0002A8FB,0x00000000,0x00000000)
>
>>      In fact we are doing the stress test on 24 windows2003 HVM on a
>>      24G, 16core host.
>>      In dom0, there is a daemon process (balloond) will give memory
>>      back  to the VM who acclaim memory.
>>      The balloond will ensure every VM at least has 512M memory.
>>      Each VM is started with memory=512 maxmem=2048
>>      Inside each of the VM, there is two processes eat memory, each
        of which will eat 1G memory.
>>      And the page file size configured on C:\ is init size 1536M, max
>>      size 3072M, which I think even
>>      If balloond process not give back the memory back to VM, the VM
>       can has 512M+3072M = 3.5G virtual
>>      memory.
>> 
>>      Am I right?, if so , since our eat memory process only consume
>>      2G  memory, how could "NO_PAGES_AVAILABLE" happen?
>>      Also, since we enable VM writes its memory status into xenstore,
>>      when blue screen, it has only has 972KB memory.
>>      Any comments? Many thanks.
>> 
> 
>
>You''ve put myself (I wrote GPLPV) and George Dunlap (from Citrix)
in the
>CC list but you don''t say if you are using GPLPV drivers or Citrix
PV
>drivers. If you are using GPLPV drivers let me know and I''ll get
some
>more information about the problem from you. If you aren''t using
GPLPV
>drivers then I don''t think I can be of much assistance...
>
>Thanks
>
>James
> 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Thanks James.
> 
> I think it is GPLPV. The driver is from
http://xenbits.xen.org/ext/win-
> pvdrivers.hg
> But, I have done some other things
> 
> 1) Add pod support
> 2) enable a meminfo thread, periodically write VM meminfo into
xenstore
>     We use info of Current Memory, Free memory, and Committed memory,
retrived
> through NativeAPI
> 3) our code is based from changeset 823, attached is the diff of my
current
> code with changset of 853.
> 
> Maybe I need add my code to 853, and test again.
> Thanks.
> 
As per other post, I have just committed some patches and PoD should now
be working properly. I can start a DomU with 4GB of maxmem but only
128MB of populated memory without any problems. This now works because I
do the initial balloon down in DriverEntry, way before xenpci does
anything else. Before it would blow up in DriverEntry. I think I
determine the amount to initially balloon down a little differently from
you too.

It takes a while to balloon down the memory though... I think Windows
tends to delay large allocations or something, because ballooning up
again is pretty much instant.

How often are you writing meminfo stuff into xenstore? Could you do that
in userspace (the interface to xenstore exists and seems to work well
although it''s a little tedious)? You would then be able to just run it
as a service and not need to patch GPLPV.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for fixing the POD. It''s is better make it earlier to avoid
crash.
 
The meminfo is written every 1 seconds into xenstore dir
/local/domain/did/memory/meminfo.
And to avoid to many writes, only the memory changes large than 5M, the thread
will do the write.
 
As for userspace daemon, it is our first choice, but we found it xenstore daemon
in dom0 comsume
many CPU(we test in linux only), so we decide to move it into driver.
 
I''ve done merge my code with latest changeset 866, and do the stree
test later.
 
many thanks.
 
> Subject: RE: RE: blue screen in windows balloon driver
> Date: Sun, 27 Feb 2011 22:25:28 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> CC: george.dunlap@eu.citrix.com
> 
> > Thanks James.
> > 
> > I think it is GPLPV. The driver is from
> http://xenbits.xen.org/ext/win-
> > pvdrivers.hg
> > But, I have done some other things
> > 
> > 1) Add pod support
> > 2) enable a meminfo thread, periodically write VM meminfo into
> xenstore
> > We use info of Current Memory, Free memory, and Committed memory,
> retrived
> > through NativeAPI
> > 3) our code is based from changeset 823, attached is the diff of my
> current
> > code with changset of 853.
> > 
> > Maybe I need add my code to 853, and test again.
> > Thanks.
> > 
> 
> As per other post, I have just committed some patches and PoD should now
> be working properly. I can start a DomU with 4GB of maxmem but only
> 128MB of populated memory without any problems. This now works because I
> do the initial balloon down in DriverEntry, way before xenpci does
> anything else. Before it would blow up in DriverEntry. I think I
> determine the amount to initially balloon down a little differently from
> you too.
> 
> It takes a while to balloon down the memory though... I think Windows
> tends to delay large allocations or something, because ballooning up
> again is pretty much instant.
> 
> How often are you writing meminfo stuff into xenstore? Could you do that
> in userspace (the interface to xenstore exists and seems to work well
> although it''s a little tedious)? You would then be able to just
run it
> as a service and not need to patch GPLPV.
> 
> James
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi James:
 
     Unfortunately, We still hit the blue screen on the stress test.
     (Start total 24 HVMS on a single 16core, 24G host,
      each HVM owns 2G Memory, start with memory=512M, 
      and inside two eating memory  processes, each of which will each 1G
memory)
 
     As I go though the code, I noticed that all memory allocation relates to 
     "ExAllocatePoolWithTag(NonPagedPool,...)", which is from
NonePaged Pool,
      As I know, the NonePagePool memory is the memory could not be paged out, 
      and that is limited, and for the blue screen VMS, I also found the free
memory
      is quite low, only about hundreds KB left. 
 
     So, when memory overcommit, some of the VM will not got enough memory, 
and if  most of its Memory is occupied by eating memory process, then
ExAllocatePoolWithTag
will fail, thus caused "NO_PAGES_AVALIABLE" blue screen. Is this
possible?
 
      Meanwhile, I will have your PVdriver tested to see if blue exists, thanks.
      
>From: tinnycloud@hotmail.com
>To: tinnycloud@hotmail.com
>Subject: FW: blue screen in windows balloon driver
>Date: Mon, 28 Feb 2011 16:16:59 +0800
>
>
>Thanks for fixing the POD. It''s is better make it earlier to avoid
crash.
> 
>The meminfo is written every 1 seconds into xenstore dir
/local/domain/did/memory/meminfo.
>And to avoid to many writes, only the memory changes large than 5M, the
thread will do the write.
> 
>As for userspace daemon, it is our first choice, but we found it xenstore
daemon in dom0 comsume
>many CPU(we test in linux only), so we decide to move it into driver.
> 
>I''ve done merge my code with latest changeset 866, and do the stree
test later.
> 
>many thanks.
> 
>> Subject: RE: RE: blue screen in windows balloon driver
>> Date: Sun, 27 Feb 2011 22:25:28 +1100
>> From: james.harper@bendigoit.com.au
>> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
>> CC: george.dunlap@eu.citrix.com
>> 
>> > Thanks James.
>> > 
>> > I think it is GPLPV. The driver is from
>> http://xenbits.xen.org/ext/win-
>> > pvdrivers.hg
>> > But, I have done some other things
>> > 
>> > 1) Add pod support
>> > 2) enable a meminfo thread, periodically write VM meminfo into
>> xenstore
>> > We use info of Current Memory, Free memory, and Committed memory,
>> retrived
>> > through NativeAPI
>> > 3) our code is based from changeset 823, attached is the diff of
my
>> current
>> > code with changset of 853.
>> > 
>> > Maybe I need add my code to 853, and test again.
>> > Thanks.
>> > 
>> 
>> As per other post, I have just committed some patches and PoD should
now
>> be working properly. I can start a DomU with 4GB of maxmem but only
>> 128MB of populated memory without any problems. This now works because
I
>> do the initial balloon down in DriverEntry, way before xenpci does
>> anything else. Before it would blow up in DriverEntry. I think I
>> determine the amount to initially balloon down a little differently
from
>> you too.
>> 
>> It takes a while to balloon down the memory though... I think Windows
>> tends to delay large allocations or something, because ballooning up
>> again is pretty much instant.
>> 
>> How often are you writing meminfo stuff into xenstore? Could you do
that
>> in userspace (the interface to xenstore exists and seems to work well
>> although it''s a little tedious)? You would then be able to
just run it
>> as a service and not need to patch GPLPV.
>> 
>> James
> 		 	   		  
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> Hi James:
> 
> Unfortunately, We still hit the blue screen on the stress test.
> (Start total 24 HVMS on a single 16core, 24G host,
> each HVM owns 2G Memory, start with memory=512M,
> and inside two eating memory  processes, each of which will each 1G
> memory)
> 
> As I go though the code, I noticed that all memory allocation relates
to
> "ExAllocatePoolWithTag(NonPagedPool,...)", which is from
NonePaged
Pool,
> As I know, the NonePagePool memory is the memory could not be paged
out,
> and that is limited, and for the blue screen VMS, I also found the
free
> memory is quite low, only about hundreds KB left.
> 
> So, when memory overcommit, some of the VM will not got enough memory,
> and if  most of its Memory is occupied by eating memory process, then
> ExAllocatePoolWithTag
> will fail, thus caused "NO_PAGES_AVALIABLE" blue screen. Is this
possible?
> 
> Meanwhile, I will have your PVdriver tested to see if blue exists,
Yes you are correct - NonPagedPool memory that will always be available.
Most of my driver code is required at some point to run at
DISPATCH_LEVEL (an IRQ priority level - not sure if you are familiar
with the concept) and at DISPATCH_LEVEL any attempt to access memory
that is paged out to disk will result in a blue screen.

There are overheads with adding more memory to the system. Windows has
to keep track of every page of memory so the more memory you have the
more memory windows has to keep track of, and this in turn uses up more
memory. If you had a system with maxmem=32768 and memory=256 or
something in that order of numbers then maybe Windows needs most of that
256MB to keep track of the 32GB of memory and doesn''t have enough left
for itself.

In your original email you said you were using maxmem=2048 and
memory=512 so maybe that isn''t the problem.

Looking at the bug check, parameter 1 and 2 are both 0x0002A8FB (about
714MB). This means that there is 714MB of ''dirty'' pages to be
written to
the pagefile. I think this might happen if you tried to allocate too
much memory too fast and Windows couldn''t write it out to disk fast
enough (it could also happen if xenvbd had hung or something). Maybe a
delay is needed when allocating memory, or else some sort of feedback
into the ballooning down code.

I''ll post on ntdev for advice.

James




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I have just pushed a change to check the
"\KernelObjects\LowMemoryCondition" event before allocating memory for
ballooning, and waiting if the event is set. This may resolve the
problems you are seeing.

What I have seen is that initially the event gets set, but then as
Windows pages some active memory out the event gets cleared again and
further ballooning down is possible. It may prevent you ballooning down
quite as low as you could before, but if it stops windows crashing then
I think it is good.

James
> -----Original Message-----
> From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> Sent: Monday, 28 February 2011 19:30
> To: xen devel
> Cc: James Harper
> Subject: RE: blue screen in windows balloon driver
> 
> Hi James:
> 
>      Unfortunately, We still hit the blue screen on the stress test.
>      (Start total 24 HVMS on a single 16core, 24G host,
>       each HVM owns 2G Memory, start with memory=512M,
>       and inside two eating memory  processes, each of which will each
1G
> memory)
> 
>      As I go though the code, I noticed that all memory allocation
relates to
>      "ExAllocatePoolWithTag(NonPagedPool,...)", which is from
NonePaged Pool,
>       As I know, the NonePagePool memory is the memory could not be
paged out,
>       and that is limited, and for the blue screen VMS, I also found
the free
> memory
>       is quite low, only about hundreds KB left.
> 
>      So, when memory overcommit, some of the VM will not got enough
memory,
> and if  most of its Memory is occupied by eating memory process, then
> ExAllocatePoolWithTag
> will fail, thus caused "NO_PAGES_AVALIABLE" blue screen. Is this
possible?
> 
>       Meanwhile, I will have your PVdriver tested to see if blue
exists,
> thanks.
> 
> 
> >From: tinnycloud@hotmail.com
> >To: tinnycloud@hotmail.com
> >Subject: FW: blue screen in windows balloon driver
> >Date: Mon, 28 Feb 2011 16:16:59 +0800
> >
> >
> >Thanks for fixing the POD. It''s is better make it earlier to
avoid
crash.
> >
> >The meminfo is written every 1 seconds into xenstore dir
> /local/domain/did/memory/meminfo.
> >And to avoid to many writes, only the memory changes large than 5M,
the
> thread will do the write.
> >
> >As for userspace daemon, it is our first choice, but we found it
xenstore
> daemon in dom0 comsume
> >many CPU(we test in linux only), so we decide to move it into driver.
> >
> >I''ve done merge my code with latest changeset 866, and do the
stree
test
> later.
> >
> >many thanks.
> >
> >> Subject: RE: RE: blue screen in windows balloon driver
> >> Date: Sun, 27 Feb 2011 22:25:28 +1100
> >> From: james.harper@bendigoit.com.au
> >> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> >> CC: george.dunlap@eu.citrix.com
> >>
> >> > Thanks James.
> >> >
> >> > I think it is GPLPV. The driver is from
> >> http://xenbits.xen.org/ext/win-
> >> > pvdrivers.hg
> >> > But, I have done some other things
> >> >
> >> > 1) Add pod support
> >> > 2) enable a meminfo thread, periodically write VM meminfo
into
> >> xenstore
> >> > We use info of Current Memory, Free memory, and Committed
memory,
> >> retrived
> >> > through NativeAPI
> >> > 3) our code is based from changeset 823, attached is the diff
of
my
> >> current
> >> > code with changset of 853.
> >> >
> >> > Maybe I need add my code to 853, and test again.
> >> > Thanks.
> >> >
> >>
> >> As per other post, I have just committed some patches and PoD
should now
> >> be working properly. I can start a DomU with 4GB of maxmem but
only
> >> 128MB of populated memory without any problems. This now works
because I
> >> do the initial balloon down in DriverEntry, way before xenpci does
> >> anything else. Before it would blow up in DriverEntry. I think I
> >> determine the amount to initially balloon down a little
differently
from
> >> you too.
> >>
> >> It takes a while to balloon down the memory though... I think
Windows
> >> tends to delay large allocations or something, because ballooning
up
> >> again is pretty much instant.
> >>
> >> How often are you writing meminfo stuff into xenstore? Could you
do
that
> >> in userspace (the interface to xenstore exists and seems to work
well
> >> although it''s a little tedious)? You would then be able
to just run
it
> >> as a service and not need to patch GPLPV.
> >>
> >> James
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks James.
 
Well, what if the memory is balloon dow already?
In my test, the eat memory process(named memhog) is started after the server
starts,
(that is all VMs have already ballooned down to 512M)
It looks like the "balloon down threads " is not working at that time.
 
One more question is, if memhog eat process at very fast speed, will it  consume
the
NopagePool memory? (I am not whether NopagePool and Page Pool is seperate pool).
If so, if the memory is exhausted, some other places
like"ExAllocatePoolWithTag(NonPagedPool,...)",
will gets no memory, and could it cause bluescreen?
 
I will have the latest driver tested, thanks.
 
 
> Subject: RE: blue screen in windows balloon driver
> Date: Tue, 1 Mar 2011 10:45:52 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> 
> I have just pushed a change to check the
> "\KernelObjects\LowMemoryCondition" event before allocating
memory for
> ballooning, and waiting if the event is set. This may resolve the
> problems you are seeing.
> 
> What I have seen is that initially the event gets set, but then as
> Windows pages some active memory out the event gets cleared again and
> further ballooning down is possible. It may prevent you ballooning down
> quite as low as you could before, but if it stops windows crashing then
> I think it is good.
> 
> James
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Monday, 28 February 2011 19:30
> > To: xen devel
> > Cc: James Harper
> > Subject: RE: blue screen in windows balloon driver
> > 
> > Hi James:
> > 
> > Unfortunately, We still hit the blue screen on the stress test.
> > (Start total 24 HVMS on a single 16core, 24G host,
> > each HVM owns 2G Memory, start with memory=512M,
> > and inside two eating memory processes, each of which will each
> 1G
> > memory)
> > 
> > As I go though the code, I noticed that all memory allocation
> relates to
> > "ExAllocatePoolWithTag(NonPagedPool,...)", which is from
> NonePaged Pool,
> > As I know, the NonePagePool memory is the memory could not be
> paged out,
> > and that is limited, and for the blue screen VMS, I also found
> the free
> > memory
> > is quite low, only about hundreds KB left.
> > 
> > So, when memory overcommit, some of the VM will not got enough
> memory,
> > and if most of its Memory is occupied by eating memory process, then
> > ExAllocatePoolWithTag
> > will fail, thus caused "NO_PAGES_AVALIABLE" blue screen. Is
this
> possible?
> > 
> > Meanwhile, I will have your PVdriver tested to see if blue
> exists,
> > thanks.
> > 
> > 
> > >From: tinnycloud@hotmail.com
> > >To: tinnycloud@hotmail.com
> > >Subject: FW: blue screen in windows balloon driver
> > >Date: Mon, 28 Feb 2011 16:16:59 +0800
> > >
> > >
> > >Thanks for fixing the POD. It''s is better make it earlier
to avoid
> crash.
> > >
> > >The meminfo is written every 1 seconds into xenstore dir
> > /local/domain/did/memory/meminfo.
> > >And to avoid to many writes, only the memory changes large than
5M,
> the
> > thread will do the write.
> > >
> > >As for userspace daemon, it is our first choice, but we found it
> xenstore
> > daemon in dom0 comsume
> > >many CPU(we test in linux only), so we decide to move it into
driver.
> > >
> > >I''ve done merge my code with latest changeset 866, and do
the stree
> test
> > later.
> > >
> > >many thanks.
> > >
> > >> Subject: RE: RE: blue screen in windows balloon driver
> > >> Date: Sun, 27 Feb 2011 22:25:28 +1100
> > >> From: james.harper@bendigoit.com.au
> > >> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > >> CC: george.dunlap@eu.citrix.com
> > >>
> > >> > Thanks James.
> > >> >
> > >> > I think it is GPLPV. The driver is from
> > >> http://xenbits.xen.org/ext/win-
> > >> > pvdrivers.hg
> > >> > But, I have done some other things
> > >> >
> > >> > 1) Add pod support
> > >> > 2) enable a meminfo thread, periodically write VM
meminfo into
> > >> xenstore
> > >> > We use info of Current Memory, Free memory, and
Committed memory,
> > >> retrived
> > >> > through NativeAPI
> > >> > 3) our code is based from changeset 823, attached is the
diff of
> my
> > >> current
> > >> > code with changset of 853.
> > >> >
> > >> > Maybe I need add my code to 853, and test again.
> > >> > Thanks.
> > >> >
> > >>
> > >> As per other post, I have just committed some patches and PoD
> should now
> > >> be working properly. I can start a DomU with 4GB of maxmem
but only
> > >> 128MB of populated memory without any problems. This now
works
> because I
> > >> do the initial balloon down in DriverEntry, way before xenpci
does
> > >> anything else. Before it would blow up in DriverEntry. I
think I
> > >> determine the amount to initially balloon down a little
differently
> from
> > >> you too.
> > >>
> > >> It takes a while to balloon down the memory though... I think
> Windows
> > >> tends to delay large allocations or something, because
ballooning
> up
> > >> again is pretty much instant.
> > >>
> > >> How often are you writing meminfo stuff into xenstore? Could
you do
> that
> > >> in userspace (the interface to xenstore exists and seems to
work
> well
> > >> although it''s a little tedious)? You would then be
able to just run
> it
> > >> as a service and not need to patch GPLPV.
> > >>
> > >> James
> > >
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Please send logs and bug check codes for any future crashes

Can you also send me your memhog program?

Sent from my iPhone

On 01/03/2011, at 13:37, "MaoXiaoyun" <tinnycloud@hotmail.com>
wrote:
> Thanks James.
>  
> Well, what if the memory is balloon dow already?
> In my test, the eat memory process(named memhog) is started after the
server starts,
> (that is all VMs have already ballooned down to 512M)
> It looks like the "balloon down threads " is not working at that
time.
>  
> One more question is, if memhog eat process at very fast speed, will it 
consume the
> NopagePool memory? (I am not whether NopagePool and Page Pool is seperate
pool).
> If so, if the memory is exhausted, some other places
like"ExAllocatePoolWithTag(NonPagedPool,...)",
> will gets no memory, and could it cause bluescreen?
>  
> I will have the latest driver tested, thanks.
>  
>  
> > Subject: RE: blue screen in windows balloon driver
> > Date: Tue, 1 Mar 2011 10:45:52 +1100
> > From: james.harper@bendigoit.com.au
> > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > 
> > I have just pushed a change to check the
> > "\KernelObjects\LowMemoryCondition" event before allocating
memory for
> > ballooning, and waiting if the event is set. This may resolve the
> > problems you are seeing.
> > 
> > What I have seen is that initially the event gets set, but then as
> > Windows pages some active memory out the event gets cleared again and
> > further ballooning down is possible. It may prevent you ballooning
down
> > quite as low as you could before, but if it stops windows crashing
then
> > I think it is good.
> > 
> > James
> > 
> > > -----Original Message-----
> > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > Sent: Monday, 28 February 2011 19:30
> > > To: xen devel
> > > Cc: James Harper
> > > Subject: RE: blue screen in windows balloon driver
> > > 
> > > Hi James:
> > > 
> > > Unfortunately, We still hit the blue screen on the stress test.
> > > (Start total 24 HVMS on a single 16core, 24G host,
> > > each HVM owns 2G Memory, start with memory=512M,
> > > and inside two eating memory processes, each of which will each
> > 1G
> > > memory)
> > > 
> > > As I go though the code, I noticed that all memory allocation
> > relates to
> > > "ExAllocatePoolWithTag(NonPagedPool,...)", which is
from
> > NonePaged Pool,
> > > As I know, the NonePagePool memory is the memory could not be
> > paged out,
> > > and that is limited, and for the blue screen VMS, I also found
> > the free
> > > memory
> > > is quite low, only about hundreds KB left.
> > > 
> > > So, when memory overcommit, some of the VM will not got enough
> > memory,
> > > and if most of its Memory is occupied by eating memory process,
then
> > > ExAllocatePoolWithTag
> > > will fail, thus caused "NO_PAGES_AVALIABLE" blue
screen. Is this
> > possible?
> > > 
> > > Meanwhile, I will have your PVdriver tested to see if blue
> > exists,
> > > thanks.
> > > 
> > > 
> > > >From: tinnycloud@hotmail.com
> > > >To: tinnycloud@hotmail.com
> > > >Subject: FW: blue screen in windows balloon driver
> > > >Date: Mon, 28 Feb 2011 16:16:59 +0800
> > > >
> > > >
> > > >Thanks for fixing the POD. It''s is better make it
earlier to avoid
> > crash.
> > > >
> > > >The meminfo is written every 1 seconds into xenstore dir
> > > /local/domain/did/memory/meminfo.
> > > >And to avoid to many writes, only the memory changes large
than 5M,
> > the
> > > thread will do the write.
> > > >
> > > >As for userspace daemon, it is our first choice, but we found
it
> > xenstore
> > > daemon in dom0 comsume
> > > >many CPU(we test in linux only), so we decide to move it into
driver.
> > > >
> > > >I''ve done merge my code with latest changeset 866,
and do the stree
> > test
> > > later.
> > > >
> > > >many thanks.
> > > >
> > > >> Subject: RE: RE: blue screen in windows balloon driver
> > > >> Date: Sun, 27 Feb 2011 22:25:28 +1100
> > > >> From: james.harper@bendigoit.com.au
> > > >> To: tinnycloud@hotmail.com;
xen-devel@lists.xensource.com
> > > >> CC: george.dunlap@eu.citrix.com
> > > >>
> > > >> > Thanks James.
> > > >> >
> > > >> > I think it is GPLPV. The driver is from
> > > >> http://xenbits.xen.org/ext/win-
> > > >> > pvdrivers.hg
> > > >> > But, I have done some other things
> > > >> >
> > > >> > 1) Add pod support
> > > >> > 2) enable a meminfo thread, periodically write VM
meminfo into
> > > >> xenstore
> > > >> > We use info of Current Memory, Free memory, and
Committed memory,
> > > >> retrived
> > > >> > through NativeAPI
> > > >> > 3) our code is based from changeset 823, attached
is the diff of
> > my
> > > >> current
> > > >> > code with changset of 853.
> > > >> >
> > > >> > Maybe I need add my code to 853, and test again.
> > > >> > Thanks.
> > > >> >
> > > >>
> > > >> As per other post, I have just committed some patches
and PoD
> > should now
> > > >> be working properly. I can start a DomU with 4GB of
maxmem but only
> > > >> 128MB of populated memory without any problems. This now
works
> > because I
> > > >> do the initial balloon down in DriverEntry, way before
xenpci does
> > > >> anything else. Before it would blow up in DriverEntry. I
think I
> > > >> determine the amount to initially balloon down a little
differently
> > from
> > > >> you too.
> > > >>
> > > >> It takes a while to balloon down the memory though... I
think
> > Windows
> > > >> tends to delay large allocations or something, because
ballooning
> > up
> > > >> again is pretty much instant.
> > > >>
> > > >> How often are you writing meminfo stuff into xenstore?
Could you do
> > that
> > > >> in userspace (the interface to xenstore exists and seems
to work
> > well
> > > >> although it''s a little tedious)? You would then
be able to just run
> > it
> > > >> as a service and not need to patch GPLPV.
> > > >>
> > > >> James
> > > >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Attached is the memhog source code.
 
In test, I run two memhogs at same time with options " -t 2 -M 512 -R 30 -s
2 -K 100"
which means, 2threads, total alloc 512M memory, run 30times, sleep every 2
seconds between
rounds, and every malloc, malloc 100KB.
 
Also, in currently test, after VM starts with 512M memory, no memory is given
back to VM.
 
Actually, we have three physical host to test the blue screen, quite strange is
only on machine
hit the blue screen quite easily(your driver with/without low memory event hit
the screen).
But on other two machine, everything goes fine. Even I recompile the host
system, different result
still exists.
 
 
I also noticed the your post on ntdev, is it possible
"ExAllocatePoolWithTag" failed, but in code
the return NULL is not fully checked?
 
More, what kind of logs your need, and does "check code" means 
something like "
***STOP: 0x0000004D (0x00015742,0x00015742,0x00000000,0x00000000)
***STOP: 0x0000004D (0x0001583B,0x0001583B,0x00000000,0x00000000)
this the latest two crashes.
 
> Subject: Re: blue screen in windows balloon driver
> From: james.harper@bendigoit.com.au
> Date: Tue, 1 Mar 2011 16:01:46 +1100
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com
> 
> Please send logs and bug check codes for any future crashes
> 
> Can you also send me your memhog program?
> 
> Sent from my iPhone
> 
> On 01/03/2011, at 13:37, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> 
> > Thanks James.
> > 
> > Well, what if the memory is balloon dow already?
> > In my test, the eat memory process(named memhog) is started after the
server starts,
> > (that is all VMs have already ballooned down to 512M)
> > It looks like the "balloon down threads " is not working at
that time.
> > 
> > One more question is, if memhog eat process at very fast speed, will
it consume the
> > NopagePool memory? (I am not whether NopagePool and Page Pool is
seperate pool).
> > If so, if the memory is exhausted, some other places
like"ExAllocatePoolWithTag(NonPagedPool,...)",
> > will gets no memory, and could it cause bluescreen?
> > 
> > I will have the latest driver tested, thanks.
> > 
> > 
> > > Subject: RE: blue screen in windows balloon driver
> > > Date: Tue, 1 Mar 2011 10:45:52 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > > 
> > > I have just pushed a change to check the
> > > "\KernelObjects\LowMemoryCondition" event before
allocating memory for
> > > ballooning, and waiting if the event is set. This may resolve the
> > > problems you are seeing.
> > > 
> > > What I have seen is that initially the event gets set, but then
as
> > > Windows pages some active memory out the event gets cleared again
and
> > > further ballooning down is possible. It may prevent you
ballooning down
> > > quite as low as you could before, but if it stops windows
crashing then
> > > I think it is good.
> > > 
> > > James
> > > 
> > > > -----Original Message-----
> > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > Sent: Monday, 28 February 2011 19:30
> > > > To: xen devel
> > > > Cc: James Harper
> > > > Subject: RE: blue screen in windows balloon driver
> > > > 
> > > > Hi James:
> > > > 
> > > > Unfortunately, We still hit the blue screen on the stress
test.
> > > > (Start total 24 HVMS on a single 16core, 24G host,
> > > > each HVM owns 2G Memory, start with memory=512M,
> > > > and inside two eating memory processes, each of which will
each
> > > 1G
> > > > memory)
> > > > 
> > > > As I go though the code, I noticed that all memory
allocation
> > > relates to
> > > > "ExAllocatePoolWithTag(NonPagedPool,...)", which
is from
> > > NonePaged Pool,
> > > > As I know, the NonePagePool memory is the memory could not
be
> > > paged out,
> > > > and that is limited, and for the blue screen VMS, I also
found
> > > the free
> > > > memory
> > > > is quite low, only about hundreds KB left.
> > > > 
> > > > So, when memory overcommit, some of the VM will not got
enough
> > > memory,
> > > > and if most of its Memory is occupied by eating memory
process, then
> > > > ExAllocatePoolWithTag
> > > > will fail, thus caused "NO_PAGES_AVALIABLE" blue
screen. Is this
> > > possible?
> > > > 
> > > > Meanwhile, I will have your PVdriver tested to see if blue
> > > exists,
> > > > thanks.
> > > > 
> > > > 
> > > > >From: tinnycloud@hotmail.com
> > > > >To: tinnycloud@hotmail.com
> > > > >Subject: FW: blue screen in windows balloon driver
> > > > >Date: Mon, 28 Feb 2011 16:16:59 +0800
> > > > >
> > > > >
> > > > >Thanks for fixing the POD. It''s is better make
it earlier to avoid
> > > crash.
> > > > >
> > > > >The meminfo is written every 1 seconds into xenstore dir
> > > > /local/domain/did/memory/meminfo.
> > > > >And to avoid to many writes, only the memory changes
large than 5M,
> > > the
> > > > thread will do the write.
> > > > >
> > > > >As for userspace daemon, it is our first choice, but we
found it
> > > xenstore
> > > > daemon in dom0 comsume
> > > > >many CPU(we test in linux only), so we decide to move it
into driver.
> > > > >
> > > > >I''ve done merge my code with latest changeset
866, and do the stree
> > > test
> > > > later.
> > > > >
> > > > >many thanks.
> > > > >
> > > > >> Subject: RE: RE: blue screen in windows balloon
driver
> > > > >> Date: Sun, 27 Feb 2011 22:25:28 +1100
> > > > >> From: james.harper@bendigoit.com.au
> > > > >> To: tinnycloud@hotmail.com;
xen-devel@lists.xensource.com
> > > > >> CC: george.dunlap@eu.citrix.com
> > > > >>
> > > > >> > Thanks James.
> > > > >> >
> > > > >> > I think it is GPLPV. The driver is from
> > > > >> http://xenbits.xen.org/ext/win-
> > > > >> > pvdrivers.hg
> > > > >> > But, I have done some other things
> > > > >> >
> > > > >> > 1) Add pod support
> > > > >> > 2) enable a meminfo thread, periodically write
VM meminfo into
> > > > >> xenstore
> > > > >> > We use info of Current Memory, Free memory,
and Committed memory,
> > > > >> retrived
> > > > >> > through NativeAPI
> > > > >> > 3) our code is based from changeset 823,
attached is the diff of
> > > my
> > > > >> current
> > > > >> > code with changset of 853.
> > > > >> >
> > > > >> > Maybe I need add my code to 853, and test
again.
> > > > >> > Thanks.
> > > > >> >
> > > > >>
> > > > >> As per other post, I have just committed some
patches and PoD
> > > should now
> > > > >> be working properly. I can start a DomU with 4GB of
maxmem but only
> > > > >> 128MB of populated memory without any problems.
This now works
> > > because I
> > > > >> do the initial balloon down in DriverEntry, way
before xenpci does
> > > > >> anything else. Before it would blow up in
DriverEntry. I think I
> > > > >> determine the amount to initially balloon down a
little differently
> > > from
> > > > >> you too.
> > > > >>
> > > > >> It takes a while to balloon down the memory
though... I think
> > > Windows
> > > > >> tends to delay large allocations or something,
because ballooning
> > > up
> > > > >> again is pretty much instant.
> > > > >>
> > > > >> How often are you writing meminfo stuff into
xenstore? Could you do
> > > that
> > > > >> in userspace (the interface to xenstore exists and
seems to work
> > > well
> > > > >> although it''s a little tedious)? You would
then be able to just run
> > > it
> > > > >> as a service and not need to patch GPLPV.
> > > > >>
> > > > >> James
> > > > >
 		 	   		  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi James:
 
Attached are three logs. (I started test PV in debug mode)
 
qemu-dm-w3.MR_cp7.vhd.log.normal:  
is the VM not crash
 
 
qemu-dm-w3.MR_cp23.vhd.log.crash:
 is the vm crashed, but the log show a Assertion failed.
*** Assertion failed: srb != NULL
***   Source File: e:\download\win-pvdrivers.hg\xenvbd\xenvbd_scsiport.c, line
988
Blue screen on "NO_PAGES_AVAILABLE"
***STOP: 0x0000004D (0x0001566c,0x0001566c,0x00000000,0x00000000)
 
 
qemu-dm-w3.MR_cp6.vhd.log.crash: is the vm crashed, bug no special error in log
Blue screen on "NO_PAGES_AVAILABLE"
***STOP: 0x0000004D (0x0001590f,0x0001590f,0x00000000,0x00000000)
 
thanks.
 
> Subject: Re: blue screen in windows balloon driver
> From: james.harper@bendigoit.com.au
> Date: Tue, 1 Mar 2011 16:01:46 +1100
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com
> 
> Please send logs and bug check codes for any future crashes
> 
> Can you also send me your memhog program?
> 
> Sent from my iPhone
> 
> On 01/03/2011, at 13:37, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> 
> > Thanks James.
> > 
> > Well, what if the memory is balloon dow already?
> > In my test, the eat memory process(named memhog) is started after the
server starts,
> > (that is all VMs have already ballooned down to 512M)
> > It looks like the "balloon down threads " is not working at
that time.
> > 
> > One more question is, if memhog eat process at very fast speed, will
it consume the
> > NopagePool memory? (I am not whether NopagePool and Page Pool is
seperate pool).
> > If so, if the memory is exhausted, some other places
like"ExAllocatePoolWithTag(NonPagedPool,...)",
> > will gets no memory, and could it cause bluescreen?
> > 
> > I will have the latest driver tested, thanks.
> > 
> > 
> > > Subject: RE: blue screen in windows balloon driver
> > > Date: Tue, 1 Mar 2011 10:45:52 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > > 
> > > I have just pushed a change to check the
> > > "\KernelObjects\LowMemoryCondition" event before
allocating memory for
> > > ballooning, and waiting if the event is set. This may resolve the
> > > problems you are seeing.
> > > 
> > > What I have seen is that initially the event gets set, but then
as
> > > Windows pages some active memory out the event gets cleared again
and
> > > further ballooning down is possible. It may prevent you
ballooning down
> > > quite as low as you could before, but if it stops windows
crashing then
> > > I think it is good.
> > > 
> > > James
> > > 
> > > > -----Original Message-----
> > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > Sent: Monday, 28 February 2011 19:30
> > > > To: xen devel
> > > > Cc: James Harper
> > > > Subject: RE: blue screen in windows balloon driver
> > > > 
> > > > Hi James:
> > > > 
> > > > Unfortunately, We still hit the blue screen on the stress
test.
> > > > (Start total 24 HVMS on a single 16core, 24G host,
> > > > each HVM owns 2G Memory, start with memory=512M,
> > > > and inside two eating memory processes, each of which will
each
> > > 1G
> > > > memory)
> > > > 
> > > > As I go though the code, I noticed that all memory
allocation
> > > relates to
> > > > "ExAllocatePoolWithTag(NonPagedPool,...)", which
is from
> > > NonePaged Pool,
> > > > As I know, the NonePagePool memory is the memory could not
be
> > > paged out,
> > > > and that is limited, and for the blue screen VMS, I also
found
> > > the free
> > > > memory
> > > > is quite low, only about hundreds KB left.
> > > > 
> > > > So, when memory overcommit, some of the VM will not got
enough
> > > memory,
> > > > and if most of its Memory is occupied by eating memory
process, then
> > > > ExAllocatePoolWithTag
> > > > will fail, thus caused "NO_PAGES_AVALIABLE" blue
screen. Is this
> > > possible?
> > > > 
> > > > Meanwhile, I will have your PVdriver tested to see if blue
> > > exists,
> > > > thanks.
> > > > 
> > > > 
> > > > >From: tinnycloud@hotmail.com
> > > > >To: tinnycloud@hotmail.com
> > > > >Subject: FW: blue screen in windows balloon driver
> > > > >Date: Mon, 28 Feb 2011 16:16:59 +0800
> > > > >
> > > > >
> > > > >Thanks for fixing the POD. It''s is better make
it earlier to avoid
> > > crash.
> > > > >
> > > > >The meminfo is written every 1 seconds into xenstore dir
> > > > /local/domain/did/memory/meminfo.
> > > > >And to avoid to many writes, only the memory changes
large than 5M,
> > > the
> > > > thread will do the write.
> > > > >
> > > > >As for userspace daemon, it is our first choice, but we
found it
> > > xenstore
> > > > daemon in dom0 comsume
> > > > >many CPU(we test in linux only), so we decide to move it
into driver.
> > > > >
> > > > >I''ve done merge my code with latest changeset
866, and do the stree
> > > test
> > > > later.
> > > > >
> > > > >many thanks.
> > > > >
> > > > >> Subject: RE: RE: blue screen in windows balloon
driver
> > > > >> Date: Sun, 27 Feb 2011 22:25:28 +1100
> > > > >> From: james.harper@bendigoit.com.au
> > > > >> To: tinnycloud@hotmail.com;
xen-devel@lists.xensource.com
> > > > >> CC: george.dunlap@eu.citrix.com
> > > > >>
> > > > >> > Thanks James.
> > > > >> >
> > > > >> > I think it is GPLPV. The driver is from
> > > > >> http://xenbits.xen.org/ext/win-
> > > > >> > pvdrivers.hg
> > > > >> > But, I have done some other things
> > > > >> >
> > > > >> > 1) Add pod support
> > > > >> > 2) enable a meminfo thread, periodically write
VM meminfo into
> > > > >> xenstore
> > > > >> > We use info of Current Memory, Free memory,
and Committed memory,
> > > > >> retrived
> > > > >> > through NativeAPI
> > > > >> > 3) our code is based from changeset 823,
attached is the diff of
> > > my
> > > > >> current
> > > > >> > code with changset of 853.
> > > > >> >
> > > > >> > Maybe I need add my code to 853, and test
again.
> > > > >> > Thanks.
> > > > >> >
> > > > >>
> > > > >> As per other post, I have just committed some
patches and PoD
> > > should now
> > > > >> be working properly. I can start a DomU with 4GB of
maxmem but only
> > > > >> 128MB of populated memory without any problems.
This now works
> > > because I
> > > > >> do the initial balloon down in DriverEntry, way
before xenpci does
> > > > >> anything else. Before it would blow up in
DriverEntry. I think I
> > > > >> determine the amount to initially balloon down a
little differently
> > > from
> > > > >> you too.
> > > > >>
> > > > >> It takes a while to balloon down the memory
though... I think
> > > Windows
> > > > >> tends to delay large allocations or something,
because ballooning
> > > up
> > > > >> again is pretty much instant.
> > > > >>
> > > > >> How often are you writing meminfo stuff into
xenstore? Could you do
> > > that
> > > > >> in userspace (the interface to xenstore exists and
seems to work
> > > well
> > > > >> although it''s a little tedious)? You would
then be able to just run
> > > it
> > > > >> as a service and not need to patch GPLPV.
> > > > >>
> > > > >> James
> > > > >
 		 	   		  




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Attached are three logs. (I started test PV in debug mode)
> 
> qemu-dm-w3.MR_cp7.vhd.log.normal:
> is the VM not crash
> 
> 
> qemu-dm-w3.MR_cp23.vhd.log.crash:
>  is the vm crashed, but the log show a Assertion failed.
> *** Assertion failed: srb != NULL
> ***   Source File:
e:\download\win-pvdrivers.hg\xenvbd\xenvbd_scsiport.c,
line
> 988
> Blue screen on "NO_PAGES_AVAILABLE"
> ***STOP: 0x0000004D (0x0001566c,0x0001566c,0x00000000,0x00000000)
> 
> 
> qemu-dm-w3.MR_cp6.vhd.log.crash: is the vm crashed, bug no special
error in
> log
> Blue screen on "NO_PAGES_AVAILABLE"
> ***STOP: 0x0000004D (0x0001590f,0x0001590f,0x00000000,0x00000000)
> 
I have checked both your bug checks. Both of them have some scsi resets
just before the crash, eg:

XenVbd --> XenVbd_HwScsiResetBus
XenVbd     IRQL = 9
XenVbd <-- XenVbd_HwScsiResetBus
XenVbd --> XenVbd_HwScsiResetBus
XenVbd     IRQL = 9
XenVbd <-- XenVbd_HwScsiResetBus

So I''m wondering if xenvbd is stuck. If that happened, Windows would be
able to write out to the pagefile, which is more in line with the
documentation of Bug Check 0x4D.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I am testing windows 2003, and the pvdriver msi is build in XP env.
 
Well, I check all other not crashed VMS, all of them has the 
XenVbd_HwScsiResetBus.
What does this mean? Is it reasonable.
 
I will run the debug mode pv on the other two physical to see if the log exists.
(Since not blue srceen ever happen on those two hosts).
 
Also, how to check whether xenvbd is stuck?
 
many thanks.
 
> Subject: RE: blue screen in windows balloon driver
> Date: Tue, 1 Mar 2011 20:41:18 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> 
> Are you testing under Windows 2008?
> 
> When you build the drivers under the windows 2008 build environment, you
> should get a storport xenvbd driver, not a scsiport xenvbd driver, but
> in your logs I see scsiport.
> 
> This shouldn''t affect the crash we are seeing though.
> 
> James
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Tuesday, 1 March 2011 18:14
> > To: xen devel
> > Cc: James Harper
> > Subject: RE: blue screen in windows balloon driver
> > 
> > Hi James:
> > 
> > Attached are three logs. (I started test PV in debug mode)
> > 
> > qemu-dm-w3.MR_cp7.vhd.log.normal:
> > is the VM not crash
> > 
> > 
> > qemu-dm-w3.MR_cp23.vhd.log.crash:
> > is the vm crashed, but the log show a Assertion failed.
> > *** Assertion failed: srb != NULL
> > *** Source File:
> e:\download\win-pvdrivers.hg\xenvbd\xenvbd_scsiport.c, line
> > 988
> > Blue screen on "NO_PAGES_AVAILABLE"
> > ***STOP: 0x0000004D (0x0001566c,0x0001566c,0x00000000,0x00000000)
> > 
> > 
> > qemu-dm-w3.MR_cp6.vhd.log.crash: is the vm crashed, bug no special
> error in
> > log
> > Blue screen on "NO_PAGES_AVAILABLE"
> > ***STOP: 0x0000004D (0x0001590f,0x0001590f,0x00000000,0x00000000)
> > 
> > thanks.
> > 
> > > Subject: Re: blue screen in windows balloon driver
> > > From: james.harper@bendigoit.com.au
> > > Date: Tue, 1 Mar 2011 16:01:46 +1100
> > > To: tinnycloud@hotmail.com
> > > CC: xen-devel@lists.xensource.com
> > >
> > > Please send logs and bug check codes for any future crashes
> > >
> > > Can you also send me your memhog program?
> > >
> > > Sent from my iPhone
> > >
> > > On 01/03/2011, at 13:37, "MaoXiaoyun"
<tinnycloud@hotmail.com>
> wrote:
> > >
> > > > Thanks James.
> > > >
> > > > Well, what if the memory is balloon dow already?
> > > > In my test, the eat memory process(named memhog) is started
after
> the
> > server starts,
> > > > (that is all VMs have already ballooned down to 512M)
> > > > It looks like the "balloon down threads " is not
working at that
> time.
> > > >
> > > > One more question is, if memhog eat process at very fast
speed,
> will it
> > consume the
> > > > NopagePool memory? (I am not whether NopagePool and Page
Pool is
> seperate
> > pool).
> > > > If so, if the memory is exhausted, some other places
> > like"ExAllocatePoolWithTag(NonPagedPool,...)",
> > > > will gets no memory, and could it cause bluescreen?
> > > >
> > > > I will have the latest driver tested, thanks.
> > > >
> > > >
> > > > > Subject: RE: blue screen in windows balloon driver
> > > > > Date: Tue, 1 Mar 2011 10:45:52 +1100
> > > > > From: james.harper@bendigoit.com.au
> > > > > To: tinnycloud@hotmail.com;
xen-devel@lists.xensource.com
> > > > >
> > > > > I have just pushed a change to check the
> > > > > "\KernelObjects\LowMemoryCondition" event
before allocating
> memory for
> > > > > ballooning, and waiting if the event is set. This may
resolve
> the
> > > > > problems you are seeing.
> > > > >
> > > > > What I have seen is that initially the event gets set,
but then
> as
> > > > > Windows pages some active memory out the event gets
cleared
> again and
> > > > > further ballooning down is possible. It may prevent you
> ballooning down
> > > > > quite as low as you could before, but if it stops
windows
> crashing then
> > > > > I think it is good.
> > > > >
> > > > > James
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > > > Sent: Monday, 28 February 2011 19:30
> > > > > > To: xen devel
> > > > > > Cc: James Harper
> > > > > > Subject: RE: blue screen in windows balloon driver
> > > > > >
> > > > > > Hi James:
> > > > > >
> > > > > > Unfortunately, We still hit the blue screen on the
stress
> test.
> > > > > > (Start total 24 HVMS on a single 16core, 24G host,
> > > > > > each HVM owns 2G Memory, start with memory=512M,
> > > > > > and inside two eating memory processes, each of
which will
> each
> > > > > 1G
> > > > > > memory)
> > > > > >
> > > > > > As I go though the code, I noticed that all memory
allocation
> > > > > relates to
> > > > > >
"ExAllocatePoolWithTag(NonPagedPool,...)", which is from
> > > > > NonePaged Pool,
> > > > > > As I know, the NonePagePool memory is the memory
could not be
> > > > > paged out,
> > > > > > and that is limited, and for the blue screen VMS,
I also found
> > > > > the free
> > > > > > memory
> > > > > > is quite low, only about hundreds KB left.
> > > > > >
> > > > > > So, when memory overcommit, some of the VM will
not got enough
> > > > > memory,
> > > > > > and if most of its Memory is occupied by eating
memory
> process, then
> > > > > > ExAllocatePoolWithTag
> > > > > > will fail, thus caused
"NO_PAGES_AVALIABLE" blue screen. Is
> this
> > > > > possible?
> > > > > >
> > > > > > Meanwhile, I will have your PVdriver tested to see
if blue
> > > > > exists,
> > > > > > thanks.
> > > > > >
> > > > > >
> > > > > > >From: tinnycloud@hotmail.com
> > > > > > >To: tinnycloud@hotmail.com
> > > > > > >Subject: FW: blue screen in windows balloon
driver
> > > > > > >Date: Mon, 28 Feb 2011 16:16:59 +0800
> > > > > > >
> > > > > > >
> > > > > > >Thanks for fixing the POD. It''s is
better make it earlier to
> avoid
> > > > > crash.
> > > > > > >
> > > > > > >The meminfo is written every 1 seconds into
xenstore dir
> > > > > > /local/domain/did/memory/meminfo.
> > > > > > >And to avoid to many writes, only the memory
changes large
> than 5M,
> > > > > the
> > > > > > thread will do the write.
> > > > > > >
> > > > > > >As for userspace daemon, it is our first
choice, but we found
> it
> > > > > xenstore
> > > > > > daemon in dom0 comsume
> > > > > > >many CPU(we test in linux only), so we decide
to move it into
> driver.
> > > > > > >
> > > > > > >I''ve done merge my code with latest
changeset 866, and do the
> stree
> > > > > test
> > > > > > later.
> > > > > > >
> > > > > > >many thanks.
> > > > > > >
> > > > > > >> Subject: RE: RE: blue screen in windows
balloon driver
> > > > > > >> Date: Sun, 27 Feb 2011 22:25:28 +1100
> > > > > > >> From: james.harper@bendigoit.com.au
> > > > > > >> To: tinnycloud@hotmail.com;
xen-devel@lists.xensource.com
> > > > > > >> CC: george.dunlap@eu.citrix.com
> > > > > > >>
> > > > > > >> > Thanks James.
> > > > > > >> >
> > > > > > >> > I think it is GPLPV. The driver is
from
> > > > > > >> http://xenbits.xen.org/ext/win-
> > > > > > >> > pvdrivers.hg
> > > > > > >> > But, I have done some other things
> > > > > > >> >
> > > > > > >> > 1) Add pod support
> > > > > > >> > 2) enable a meminfo thread,
periodically write VM meminfo
> into
> > > > > > >> xenstore
> > > > > > >> > We use info of Current Memory, Free
memory, and Committed
> memory,
> > > > > > >> retrived
> > > > > > >> > through NativeAPI
> > > > > > >> > 3) our code is based from changeset
823, attached is the
> diff of
> > > > > my
> > > > > > >> current
> > > > > > >> > code with changset of 853.
> > > > > > >> >
> > > > > > >> > Maybe I need add my code to 853, and
test again.
> > > > > > >> > Thanks.
> > > > > > >> >
> > > > > > >>
> > > > > > >> As per other post, I have just committed
some patches and
> PoD
> > > > > should now
> > > > > > >> be working properly. I can start a DomU
with 4GB of maxmem
> but only
> > > > > > >> 128MB of populated memory without any
problems. This now
> works
> > > > > because I
> > > > > > >> do the initial balloon down in
DriverEntry, way before
> xenpci does
> > > > > > >> anything else. Before it would blow up in
DriverEntry. I
> think I
> > > > > > >> determine the amount to initially balloon
down a little
> differently
> > > > > from
> > > > > > >> you too.
> > > > > > >>
> > > > > > >> It takes a while to balloon down the
memory though... I
> think
> > > > > Windows
> > > > > > >> tends to delay large allocations or
something, because
> ballooning
> > > > > up
> > > > > > >> again is pretty much instant.
> > > > > > >>
> > > > > > >> How often are you writing meminfo stuff
into xenstore?
> Could you do
> > > > > that
> > > > > > >> in userspace (the interface to xenstore
exists and seems to
> work
> > > > > well
> > > > > > >> although it''s a little tedious)?
You would then be able to
> just run
> > > > > it
> > > > > > >> as a service and not need to patch GPLPV.
> > > > > > >>
> > > > > > >> James
> > > > > > >
> 
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> I am testing windows 2003, and the pvdriver msi is build in XP env.
> 
> Well, I check all other not crashed VMS, all of them has the
> XenVbd_HwScsiResetBus.
> What does this mean? Is it reasonable.
> 
> I will run the debug mode pv on the other two physical to see if the
log
> exists.
> (Since not blue srceen ever happen on those two hosts).
> 
> Also, how to check whether xenvbd is stuck?
> 
I''m just making some changes now to test this. I will make
HwScsiResetBus print out all the outstanding requests.

HwScsiResetBus is called when Windows hasn''t received a response to a
scsi request (eg a read or a write). Either the request is stuck in my
driver or it is stuck somewhere else.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I will have new driver tested. 
Attached is the xentop snapshot.
 
thanks.
 
> Subject: RE: blue screen in windows balloon driver
> Date: Tue, 1 Mar 2011 23:11:14 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> 
> > 
> > exe attached, thanks.
> > 
> > I have three machines, on each sum the *XenVbd_HwScsiResetBus* event.
> > 24 VMS, so
> > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp* | wc -l
> > 
> > machine 25: VM easily got crash, the sum is 200
> > machine 23: VM never got crash, the sum is 10
> > machine 212: VM never got crash, the sum is 16
> > 
> > it seems that machine 25 has much more XenVbd_HwScsiResetBus event
> > than other two machines.
> > 
> > BTW, when start 24VM concurrently, the starting process is quite slow,
> takes
> > about 20 minutes more to whole started.
> > 
> > I commented line 505 in xenpci_pdo.c to avoid timed out.
> > 
> > 505 //remaining -= thiswait;
> > 
> 
> It sounds like you are overloading your disk IO bandwidth. With many
> DomU''s swapping heavily, Dom0 may simply not be able to keep up
with the
> IO throughput required resulting in windows thinking that the scsi
> device isn''t responding. Can you check xentop and see what sort of
IO
> operations per second you are getting?
> 
> I have just pushed a change to dump out the in-flight scsi requests
> (srb) when HwScsiResetBus is called. Please apply the patch and send me
> the next crash.
> 
> Thanks
> 
> James
 		 	   		  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hold off on testing. I''m fixing up the reset code so that it does what
Windows wants. I''ll post something soon if it doesn''t take too
long.

James
> -----Original Message-----
> From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> Sent: Tuesday, 1 March 2011 23:34
> To: James Harper
> Cc: xen devel
> Subject: RE: blue screen in windows balloon driver
> 
> I will have new driver tested.
> Attached is the xentop snapshot.
> 
> thanks.
> 
> > Subject: RE: blue screen in windows balloon driver
> > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > From: james.harper@bendigoit.com.au
> > To: tinnycloud@hotmail.com
> >
> > >
> > > exe attached, thanks.
> > >
> > > I have three machines, on each sum the *XenVbd_HwScsiResetBus*
event.
> > > 24 VMS, so
> > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp* | wc -l
> > >
> > > machine 25: VM easily got crash, the sum is 200
> > > machine 23: VM never got crash, the sum is 10
> > > machine 212: VM never got crash, the sum is 16
> > >
> > > it seems that machine 25 has much more XenVbd_HwScsiResetBus
event
> > > than other two machines.
> > >
> > > BTW, when start 24VM concurrently, the starting process is quite
slow,
> > takes
> > > about 20 minutes more to whole started.
> > >
> > > I commented line 505 in xenpci_pdo.c to avoid timed out.
> > >
> > > 505 //remaining -= thiswait;
> > >
> >
> > It sounds like you are overloading your disk IO bandwidth. With many
> > DomU''s swapping heavily, Dom0 may simply not be able to keep
up with
the
> > IO throughput required resulting in windows thinking that the scsi
> > device isn''t responding. Can you check xentop and see what
sort of
IO
> > operations per second you are getting?
> >
> > I have just pushed a change to dump out the in-flight scsi requests
> > (srb) when HwScsiResetBus is called. Please apply the patch and send
me
> > the next crash.
> >
> > Thanks
> >
> > James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''ve pushed a possible fix for the reset code for Windows 2000, XP and
2003. I haven''t fixed the Vista/2008/7/2008R2 storport driver yet.

I''ll see what I can do tomorrow to actually test a scsi reset but I
can''t reproduce the problem you are seeing on my system.
You''ll still
see the reset messages in the logs which I think simply indicates that
your system is too loaded to complete the requests in time and Windows
thinks the scsi bus is hung, but this way it might pick itself up again
afterwards. On the other hand it may be that too many timeouts and
resets will cause windows to throw its hands in the air and give up and
declare the scsi device offline, in which case there might not be much
we can do.

James
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of James Harper
> Sent: Tuesday, 1 March 2011 23:36
> To: MaoXiaoyun
> Cc: xen devel
> Subject: [Xen-devel] RE: blue screen in windows balloon driver
> 
> Hold off on testing. I''m fixing up the reset code so that it does
what
> Windows wants. I''ll post something soon if it doesn''t
take too long.
> 
> James
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Tuesday, 1 March 2011 23:34
> > To: James Harper
> > Cc: xen devel
> > Subject: RE: blue screen in windows balloon driver
> >
> > I will have new driver tested.
> > Attached is the xentop snapshot.
> >
> > thanks.
> >
> > > Subject: RE: blue screen in windows balloon driver
> > > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com
> > >
> > > >
> > > > exe attached, thanks.
> > > >
> > > > I have three machines, on each sum the
*XenVbd_HwScsiResetBus*
> event.
> > > > 24 VMS, so
> > > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp* | wc -l
> > > >
> > > > machine 25: VM easily got crash, the sum is 200
> > > > machine 23: VM never got crash, the sum is 10
> > > > machine 212: VM never got crash, the sum is 16
> > > >
> > > > it seems that machine 25 has much more XenVbd_HwScsiResetBus
event
> > > > than other two machines.
> > > >
> > > > BTW, when start 24VM concurrently, the starting process is
quite
> slow,
> > > takes
> > > > about 20 minutes more to whole started.
> > > >
> > > > I commented line 505 in xenpci_pdo.c to avoid timed out.
> > > >
> > > > 505 //remaining -= thiswait;
> > > >
> > >
> > > It sounds like you are overloading your disk IO bandwidth. With
many
> > > DomU''s swapping heavily, Dom0 may simply not be able to
keep up
with
> the
> > > IO throughput required resulting in windows thinking that the
scsi
> > > device isn''t responding. Can you check xentop and see
what sort of
> IO
> > > operations per second you are getting?
> > >
> > > I have just pushed a change to dump out the in-flight scsi
requests
> > > (srb) when HwScsiResetBus is called. Please apply the patch and
send
> me
> > > the next crash.
> > >
> > > Thanks
> > >
> > > James
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Attached is the three logs for crash.
cp17 & 21 crash on 
Assertion failed: srb != NULL
 
thanks.
 
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> Date: Tue, 1 Mar 2011 23:48:04 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com
> 
> I''ve pushed a possible fix for the reset code for Windows 2000, XP
and
> 2003. I haven''t fixed the Vista/2008/7/2008R2 storport driver yet.
> 
> I''ll see what I can do tomorrow to actually test a scsi reset but
I
> can''t reproduce the problem you are seeing on my system.
You''ll still
> see the reset messages in the logs which I think simply indicates that
> your system is too loaded to complete the requests in time and Windows
> thinks the scsi bus is hung, but this way it might pick itself up again
> afterwards. On the other hand it may be that too many timeouts and
> resets will cause windows to throw its hands in the air and give up and
> declare the scsi device offline, in which case there might not be much
> we can do.
> 
> James
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > bounces@lists.xensource.com] On Behalf Of James Harper
> > Sent: Tuesday, 1 March 2011 23:36
> > To: MaoXiaoyun
> > Cc: xen devel
> > Subject: [Xen-devel] RE: blue screen in windows balloon driver
> > 
> > Hold off on testing. I''m fixing up the reset code so that it
does what
> > Windows wants. I''ll post something soon if it
doesn''t take too long.
> > 
> > James
> > 
> > > -----Original Message-----
> > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > Sent: Tuesday, 1 March 2011 23:34
> > > To: James Harper
> > > Cc: xen devel
> > > Subject: RE: blue screen in windows balloon driver
> > >
> > > I will have new driver tested.
> > > Attached is the xentop snapshot.
> > >
> > > thanks.
> > >
> > > > Subject: RE: blue screen in windows balloon driver
> > > > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > > > From: james.harper@bendigoit.com.au
> > > > To: tinnycloud@hotmail.com
> > > >
> > > > >
> > > > > exe attached, thanks.
> > > > >
> > > > > I have three machines, on each sum the
*XenVbd_HwScsiResetBus*
> > event.
> > > > > 24 VMS, so
> > > > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp* | wc -l
> > > > >
> > > > > machine 25: VM easily got crash, the sum is 200
> > > > > machine 23: VM never got crash, the sum is 10
> > > > > machine 212: VM never got crash, the sum is 16
> > > > >
> > > > > it seems that machine 25 has much more
XenVbd_HwScsiResetBus
> event
> > > > > than other two machines.
> > > > >
> > > > > BTW, when start 24VM concurrently, the starting process
is quite
> > slow,
> > > > takes
> > > > > about 20 minutes more to whole started.
> > > > >
> > > > > I commented line 505 in xenpci_pdo.c to avoid timed
out.
> > > > >
> > > > > 505 //remaining -= thiswait;
> > > > >
> > > >
> > > > It sounds like you are overloading your disk IO bandwidth.
With
> many
> > > > DomU''s swapping heavily, Dom0 may simply not be
able to keep up
> with
> > the
> > > > IO throughput required resulting in windows thinking that
the scsi
> > > > device isn''t responding. Can you check xentop and
see what sort of
> > IO
> > > > operations per second you are getting?
> > > >
> > > > I have just pushed a change to dump out the in-flight scsi
> requests
> > > > (srb) when HwScsiResetBus is called. Please apply the patch
and
> send
> > me
> > > > the next crash.
> > > >
> > > > Thanks
> > > >
> > > > James
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
 		 	   		  




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

That assertion is a bit misleading as it occurs during dump mode when
the crash has actually already occurred. It still shouldn''t occur but
it''s not the problem we are looking for.

Does this problem occur when not using GPLPV?

When you are running GPLPV, can you do a tail -f on the logfile and see
how quickly the log messages are coming out? If they are printing out
slowly then I think your physical machine is just overloaded with IO.

James


> -----Original Message-----
> From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> Sent: Wednesday, 2 March 2011 14:02
> To: James Harper
> Cc: xen devel
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> 
> 
> Attached is the three logs for crash.
> cp17 & 21 crash on
> Assertion failed: srb != NULL
> 
> thanks.
> 
> > Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> > Date: Tue, 1 Mar 2011 23:48:04 +1100
> > From: james.harper@bendigoit.com.au
> > To: tinnycloud@hotmail.com
> > CC: xen-devel@lists.xensource.com
> >
> > I''ve pushed a possible fix for the reset code for Windows
2000, XP
and
> > 2003. I haven''t fixed the Vista/2008/7/2008R2 storport driver
yet.
> >
> > I''ll see what I can do tomorrow to actually test a scsi reset
but I
> > can''t reproduce the problem you are seeing on my system.
You''ll
still
> > see the reset messages in the logs which I think simply indicates
that
> > your system is too loaded to complete the requests in time and
Windows
> > thinks the scsi bus is hung, but this way it might pick itself up
again
> > afterwards. On the other hand it may be that too many timeouts and
> > resets will cause windows to throw its hands in the air and give up
and
> > declare the scsi device offline, in which case there might not be
much
> > we can do.
> >
> > James
> >
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > > bounces@lists.xensource.com] On Behalf Of James Harper
> > > Sent: Tuesday, 1 March 2011 23:36
> > > To: MaoXiaoyun
> > > Cc: xen devel
> > > Subject: [Xen-devel] RE: blue screen in windows balloon driver
> > >
> > > Hold off on testing. I''m fixing up the reset code so
that it does
what
> > > Windows wants. I''ll post something soon if it
doesn''t take too
long.
> > >
> > > James
> > >
> > > > -----Original Message-----
> > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > Sent: Tuesday, 1 March 2011 23:34
> > > > To: James Harper
> > > > Cc: xen devel
> > > > Subject: RE: blue screen in windows balloon driver
> > > >
> > > > I will have new driver tested.
> > > > Attached is the xentop snapshot.
> > > >
> > > > thanks.
> > > >
> > > > > Subject: RE: blue screen in windows balloon driver
> > > > > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > > > > From: james.harper@bendigoit.com.au
> > > > > To: tinnycloud@hotmail.com
> > > > >
> > > > > >
> > > > > > exe attached, thanks.
> > > > > >
> > > > > > I have three machines, on each sum the
*XenVbd_HwScsiResetBus*
> > > event.
> > > > > > 24 VMS, so
> > > > > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp* | wc
-l
> > > > > >
> > > > > > machine 25: VM easily got crash, the sum is 200
> > > > > > machine 23: VM never got crash, the sum is 10
> > > > > > machine 212: VM never got crash, the sum is 16
> > > > > >
> > > > > > it seems that machine 25 has much more
XenVbd_HwScsiResetBus
> > event
> > > > > > than other two machines.
> > > > > >
> > > > > > BTW, when start 24VM concurrently, the starting
process is
quite
> > > slow,
> > > > > takes
> > > > > > about 20 minutes more to whole started.
> > > > > >
> > > > > > I commented line 505 in xenpci_pdo.c to avoid
timed out.
> > > > > >
> > > > > > 505 //remaining -= thiswait;
> > > > > >
> > > > >
> > > > > It sounds like you are overloading your disk IO
bandwidth.
With
> > many
> > > > > DomU''s swapping heavily, Dom0 may simply not
be able to keep
up
> > with
> > > the
> > > > > IO throughput required resulting in windows thinking
that the
scsi
> > > > > device isn''t responding. Can you check xentop
and see what
sort of
> > > IO
> > > > > operations per second you are getting?
> > > > >
> > > > > I have just pushed a change to dump out the in-flight
scsi
> > requests
> > > > > (srb) when HwScsiResetBus is called. Please apply the
patch
and
> > send
> > > me
> > > > > the next crash.
> > > > >
> > > > > Thanks
> > > > >
> > > > > James
> > >
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Can you please try again with the following patch attached:

diff -r 34e72b071e51 xenpci/xenpci_dbgprint.c
--- a/xenpci/xenpci_dbgprint.c	Tue Mar 01 23:47:47 2011 +1100
+++ b/xenpci/xenpci_dbgprint.c	Wed Mar 02 17:27:31 2011 +1100
@@ -69,10 +69,23 @@
 static void XenDbgPrint(PCHAR string, ULONG length)
 {
   ULONG i;
+  ULONGLONG j;
+  LARGE_INTEGER current_time;
   //KIRQL old_irql = 0;
 
   while(InterlockedCompareExchange(&debug_print_lock, 1, 0) == 1)
     KeStallExecutionProcessor(1);
+  
+  KeQuerySystemTime(&current_time);
+  current_time.QuadPart /= 10000; /* convert to ms */
+  for (j = 1000000000000000000L; j >= 1; j /= 10)
+    if (current_time.QuadPart / j)
+      break;
+  for (; j >= 1; j /= 10)
+    WRITE_PORT_UCHAR(XEN_IOPORT_LOG, ''0'' +
((current_time.QuadPart / j)
% 10));
+  WRITE_PORT_UCHAR(XEN_IOPORT_LOG, '':'');
+  WRITE_PORT_UCHAR(XEN_IOPORT_LOG, '' '');
+      
   for (i = 0; i < length; i++)
     WRITE_PORT_UCHAR(XEN_IOPORT_LOG, string[i]);
   /* release the lock */

That will put a timestamp on each debug message which will help a lot in
diagnosing the problem.

James
> -----Original Message-----
> From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> Sent: Wednesday, 2 March 2011 14:02
> To: James Harper
> Cc: xen devel
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> 
> 
> Attached is the three logs for crash.
> cp17 & 21 crash on
> Assertion failed: srb != NULL
> 
> thanks.
> 
> > Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> > Date: Tue, 1 Mar 2011 23:48:04 +1100
> > From: james.harper@bendigoit.com.au
> > To: tinnycloud@hotmail.com
> > CC: xen-devel@lists.xensource.com
> >
> > I''ve pushed a possible fix for the reset code for Windows
2000, XP
and
> > 2003. I haven''t fixed the Vista/2008/7/2008R2 storport driver
yet.
> >
> > I''ll see what I can do tomorrow to actually test a scsi reset
but I
> > can''t reproduce the problem you are seeing on my system.
You''ll
still
> > see the reset messages in the logs which I think simply indicates
that
> > your system is too loaded to complete the requests in time and
Windows
> > thinks the scsi bus is hung, but this way it might pick itself up
again
> > afterwards. On the other hand it may be that too many timeouts and
> > resets will cause windows to throw its hands in the air and give up
and
> > declare the scsi device offline, in which case there might not be
much
> > we can do.
> >
> > James
> >
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > > bounces@lists.xensource.com] On Behalf Of James Harper
> > > Sent: Tuesday, 1 March 2011 23:36
> > > To: MaoXiaoyun
> > > Cc: xen devel
> > > Subject: [Xen-devel] RE: blue screen in windows balloon driver
> > >
> > > Hold off on testing. I''m fixing up the reset code so
that it does
what
> > > Windows wants. I''ll post something soon if it
doesn''t take too
long.
> > >
> > > James
> > >
> > > > -----Original Message-----
> > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > Sent: Tuesday, 1 March 2011 23:34
> > > > To: James Harper
> > > > Cc: xen devel
> > > > Subject: RE: blue screen in windows balloon driver
> > > >
> > > > I will have new driver tested.
> > > > Attached is the xentop snapshot.
> > > >
> > > > thanks.
> > > >
> > > > > Subject: RE: blue screen in windows balloon driver
> > > > > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > > > > From: james.harper@bendigoit.com.au
> > > > > To: tinnycloud@hotmail.com
> > > > >
> > > > > >
> > > > > > exe attached, thanks.
> > > > > >
> > > > > > I have three machines, on each sum the
*XenVbd_HwScsiResetBus*
> > > event.
> > > > > > 24 VMS, so
> > > > > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp* | wc
-l
> > > > > >
> > > > > > machine 25: VM easily got crash, the sum is 200
> > > > > > machine 23: VM never got crash, the sum is 10
> > > > > > machine 212: VM never got crash, the sum is 16
> > > > > >
> > > > > > it seems that machine 25 has much more
XenVbd_HwScsiResetBus
> > event
> > > > > > than other two machines.
> > > > > >
> > > > > > BTW, when start 24VM concurrently, the starting
process is
quite
> > > slow,
> > > > > takes
> > > > > > about 20 minutes more to whole started.
> > > > > >
> > > > > > I commented line 505 in xenpci_pdo.c to avoid
timed out.
> > > > > >
> > > > > > 505 //remaining -= thiswait;
> > > > > >
> > > > >
> > > > > It sounds like you are overloading your disk IO
bandwidth.
With
> > many
> > > > > DomU''s swapping heavily, Dom0 may simply not
be able to keep
up
> > with
> > > the
> > > > > IO throughput required resulting in windows thinking
that the
scsi
> > > > > device isn''t responding. Can you check xentop
and see what
sort of
> > > IO
> > > > > operations per second you are getting?
> > > > >
> > > > > I have just pushed a change to dump out the in-flight
scsi
> > requests
> > > > > (srb) when HwScsiResetBus is called. Please apply the
patch
and
> > send
> > > me
> > > > > the next crash.
> > > > >
> > > > > Thanks
> > > > >
> > > > > James
> > >
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Currently I test GPLPV only.
I agree on the problem is caused by the heavy IO.
Since machine 212, 23 works fine, never hit the crash, and less reset event.
 

The frequency is not high, attched is a VM running 4hours, but *not* crashed,
on see 9  XenVbd <-- XenVbd_HwScsiResetBus.
 
It looks like our test is too stressed to machine 25, which cause the reset
event
and thus produce more reset event, and make the VM crashed.
 
Well, is it difficult for  XenVbd_HwScsiResetBus to handle this properly?
 
many thanks.
 
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> Date: Wed, 2 Mar 2011 17:07:03 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com
> 
> That assertion is a bit misleading as it occurs during dump mode when
> the crash has actually already occurred. It still shouldn''t occur
but
> it''s not the problem we are looking for.
> 
> Does this problem occur when not using GPLPV?
> 
> When you are running GPLPV, can you do a tail -f on the logfile and see
> how quickly the log messages are coming out? If they are printing out
> slowly then I think your physical machine is just overloaded with IO.
> 
> James
> 
> 
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Wednesday, 2 March 2011 14:02
> > To: James Harper
> > Cc: xen devel
> > Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> > 
> > 
> > Attached is the three logs for crash.
> > cp17 & 21 crash on
> > Assertion failed: srb != NULL
> > 
> > thanks.
> > 
> > > Subject: RE: [Xen-devel] RE: blue screen in windows balloon
driver
> > > Date: Tue, 1 Mar 2011 23:48:04 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com
> > > CC: xen-devel@lists.xensource.com
> > >
> > > I''ve pushed a possible fix for the reset code for
Windows 2000, XP
> and
> > > 2003. I haven''t fixed the Vista/2008/7/2008R2 storport
driver yet.
> > >
> > > I''ll see what I can do tomorrow to actually test a scsi
reset but I
> > > can''t reproduce the problem you are seeing on my system.
You''ll
> still
> > > see the reset messages in the logs which I think simply indicates
> that
> > > your system is too loaded to complete the requests in time and
> Windows
> > > thinks the scsi bus is hung, but this way it might pick itself up
> again
> > > afterwards. On the other hand it may be that too many timeouts
and
> > > resets will cause windows to throw its hands in the air and give
up
> and
> > > declare the scsi device offline, in which case there might not be
> much
> > > we can do.
> > >
> > > James
> > >
> > > > -----Original Message-----
> > > > From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-
> > > > bounces@lists.xensource.com] On Behalf Of James Harper
> > > > Sent: Tuesday, 1 March 2011 23:36
> > > > To: MaoXiaoyun
> > > > Cc: xen devel
> > > > Subject: [Xen-devel] RE: blue screen in windows balloon
driver
> > > >
> > > > Hold off on testing. I''m fixing up the reset code
so that it does
> what
> > > > Windows wants. I''ll post something soon if it
doesn''t take too
> long.
> > > >
> > > > James
> > > >
> > > > > -----Original Message-----
> > > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > > Sent: Tuesday, 1 March 2011 23:34
> > > > > To: James Harper
> > > > > Cc: xen devel
> > > > > Subject: RE: blue screen in windows balloon driver
> > > > >
> > > > > I will have new driver tested.
> > > > > Attached is the xentop snapshot.
> > > > >
> > > > > thanks.
> > > > >
> > > > > > Subject: RE: blue screen in windows balloon driver
> > > > > > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > > > > > From: james.harper@bendigoit.com.au
> > > > > > To: tinnycloud@hotmail.com
> > > > > >
> > > > > > >
> > > > > > > exe attached, thanks.
> > > > > > >
> > > > > > > I have three machines, on each sum the
> *XenVbd_HwScsiResetBus*
> > > > event.
> > > > > > > 24 VMS, so
> > > > > > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp*
| wc -l
> > > > > > >
> > > > > > > machine 25: VM easily got crash, the sum is
200
> > > > > > > machine 23: VM never got crash, the sum is 10
> > > > > > > machine 212: VM never got crash, the sum is
16
> > > > > > >
> > > > > > > it seems that machine 25 has much more
XenVbd_HwScsiResetBus
> > > event
> > > > > > > than other two machines.
> > > > > > >
> > > > > > > BTW, when start 24VM concurrently, the
starting process is
> quite
> > > > slow,
> > > > > > takes
> > > > > > > about 20 minutes more to whole started.
> > > > > > >
> > > > > > > I commented line 505 in xenpci_pdo.c to avoid
timed out.
> > > > > > >
> > > > > > > 505 //remaining -= thiswait;
> > > > > > >
> > > > > >
> > > > > > It sounds like you are overloading your disk IO
bandwidth.
> With
> > > many
> > > > > > DomU''s swapping heavily, Dom0 may simply
not be able to keep
> up
> > > with
> > > > the
> > > > > > IO throughput required resulting in windows
thinking that the
> scsi
> > > > > > device isn''t responding. Can you check
xentop and see what
> sort of
> > > > IO
> > > > > > operations per second you are getting?
> > > > > >
> > > > > > I have just pushed a change to dump out the
in-flight scsi
> > > requests
> > > > > > (srb) when HwScsiResetBus is called. Please apply
the patch
> and
> > > send
> > > > me
> > > > > > the next crash.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > James
> > > >
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> 
 		 	   		  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Two crash logs attached.
 
The last two XenVbd_HwScsiResetBus happened almost at the same time.
May it be the problem?
 
Thanks.
 
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> Date: Wed, 2 Mar 2011 17:28:48 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com
> 
> Can you please try again with the following patch attached:
> 
> diff -r 34e72b071e51 xenpci/xenpci_dbgprint.c
> --- a/xenpci/xenpci_dbgprint.c Tue Mar 01 23:47:47 2011 +1100
> +++ b/xenpci/xenpci_dbgprint.c Wed Mar 02 17:27:31 2011 +1100
> @@ -69,10 +69,23 @@
> static void XenDbgPrint(PCHAR string, ULONG length)
> {
> ULONG i;
> + ULONGLONG j;
> + LARGE_INTEGER current_time;
> //KIRQL old_irql = 0;
> 
> while(InterlockedCompareExchange(&debug_print_lock, 1, 0) == 1)
> KeStallExecutionProcessor(1);
> + 
> + KeQuerySystemTime(&current_time);
> + current_time.QuadPart /= 10000; /* convert to ms */
> + for (j = 1000000000000000000L; j >= 1; j /= 10)
> + if (current_time.QuadPart / j)
> + break;
> + for (; j >= 1; j /= 10)
> + WRITE_PORT_UCHAR(XEN_IOPORT_LOG, ''0'' +
((current_time.QuadPart / j)
> % 10));
> + WRITE_PORT_UCHAR(XEN_IOPORT_LOG, '':'');
> + WRITE_PORT_UCHAR(XEN_IOPORT_LOG, '' '');
> + 
> for (i = 0; i < length; i++)
> WRITE_PORT_UCHAR(XEN_IOPORT_LOG, string[i]);
> /* release the lock */
> 
> That will put a timestamp on each debug message which will help a lot in
> diagnosing the problem.
> 
> James
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Wednesday, 2 March 2011 14:02
> > To: James Harper
> > Cc: xen devel
> > Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> > 
> > 
> > Attached is the three logs for crash.
> > cp17 & 21 crash on
> > Assertion failed: srb != NULL
> > 
> > thanks.
> > 
> > > Subject: RE: [Xen-devel] RE: blue screen in windows balloon
driver
> > > Date: Tue, 1 Mar 2011 23:48:04 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com
> > > CC: xen-devel@lists.xensource.com
> > >
> > > I''ve pushed a possible fix for the reset code for
Windows 2000, XP
> and
> > > 2003. I haven''t fixed the Vista/2008/7/2008R2 storport
driver yet.
> > >
> > > I''ll see what I can do tomorrow to actually test a scsi
reset but I
> > > can''t reproduce the problem you are seeing on my system.
You''ll
> still
> > > see the reset messages in the logs which I think simply indicates
> that
> > > your system is too loaded to complete the requests in time and
> Windows
> > > thinks the scsi bus is hung, but this way it might pick itself up
> again
> > > afterwards. On the other hand it may be that too many timeouts
and
> > > resets will cause windows to throw its hands in the air and give
up
> and
> > > declare the scsi device offline, in which case there might not be
> much
> > > we can do.
> > >
> > > James
> > >
> > > > -----Original Message-----
> > > > From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-
> > > > bounces@lists.xensource.com] On Behalf Of James Harper
> > > > Sent: Tuesday, 1 March 2011 23:36
> > > > To: MaoXiaoyun
> > > > Cc: xen devel
> > > > Subject: [Xen-devel] RE: blue screen in windows balloon
driver
> > > >
> > > > Hold off on testing. I''m fixing up the reset code
so that it does
> what
> > > > Windows wants. I''ll post something soon if it
doesn''t take too
> long.
> > > >
> > > > James
> > > >
> > > > > -----Original Message-----
> > > > > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > > > > Sent: Tuesday, 1 March 2011 23:34
> > > > > To: James Harper
> > > > > Cc: xen devel
> > > > > Subject: RE: blue screen in windows balloon driver
> > > > >
> > > > > I will have new driver tested.
> > > > > Attached is the xentop snapshot.
> > > > >
> > > > > thanks.
> > > > >
> > > > > > Subject: RE: blue screen in windows balloon driver
> > > > > > Date: Tue, 1 Mar 2011 23:11:14 +1100
> > > > > > From: james.harper@bendigoit.com.au
> > > > > > To: tinnycloud@hotmail.com
> > > > > >
> > > > > > >
> > > > > > > exe attached, thanks.
> > > > > > >
> > > > > > > I have three machines, on each sum the
> *XenVbd_HwScsiResetBus*
> > > > event.
> > > > > > > 24 VMS, so
> > > > > > > grep XenVbd_HwScsiResetBus qemu-dm-w3.MR_cp*
| wc -l
> > > > > > >
> > > > > > > machine 25: VM easily got crash, the sum is
200
> > > > > > > machine 23: VM never got crash, the sum is 10
> > > > > > > machine 212: VM never got crash, the sum is
16
> > > > > > >
> > > > > > > it seems that machine 25 has much more
XenVbd_HwScsiResetBus
> > > event
> > > > > > > than other two machines.
> > > > > > >
> > > > > > > BTW, when start 24VM concurrently, the
starting process is
> quite
> > > > slow,
> > > > > > takes
> > > > > > > about 20 minutes more to whole started.
> > > > > > >
> > > > > > > I commented line 505 in xenpci_pdo.c to avoid
timed out.
> > > > > > >
> > > > > > > 505 //remaining -= thiswait;
> > > > > > >
> > > > > >
> > > > > > It sounds like you are overloading your disk IO
bandwidth.
> With
> > > many
> > > > > > DomU''s swapping heavily, Dom0 may simply
not be able to keep
> up
> > > with
> > > > the
> > > > > > IO throughput required resulting in windows
thinking that the
> scsi
> > > > > > device isn''t responding. Can you check
xentop and see what
> sort of
> > > > IO
> > > > > > operations per second you are getting?
> > > > > >
> > > > > > I have just pushed a change to dump out the
in-flight scsi
> > > requests
> > > > > > (srb) when HwScsiResetBus is called. Please apply
the patch
> and
> > > send
> > > > me
> > > > > > the next crash.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > James
> > > >
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> 
 		 	   		  



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> Two crash logs attached.
> 
> The last two XenVbd_HwScsiResetBus happened almost at the same time.
> May it be the problem?
> 
Timestamps are in milliseconds.

HwScsiResetBus completes the queued requests not yet sent to Dom0 with a
status of SRB_STATUS_BUS_RESET. It can''t touch the requests on the ring
so what it does then is set their status to SRB_STATUS_BUS_RESET and
then checks for that when Dom0 signals that they are complete.

The first reset at 12943529855375 is completed in under 1 second. All
the requests on the ring are marked with SRB_STATUS_BUS_RESET and then
they complete shortly after and the system seems to recover.

The second reset happens at 12943530977390. All the requests on the ring
are marked but it is 12 seconds before Dom0 completes one of them. Two
more are completed immediately and then 10 seconds later still no more
requests are completed by Dom0 and Windows issues another reset. This
time after 5 seconds all the requests are finished by Dom0.

The last reset happens at 12943531085390. At 12943531148593 (over 60
seconds later) with no response from Dom0, Windows finally gives up and
crashes.

I don''t think the problem really lies with GPLPV and I don''t
think there
is anything I can do about it. Maybe you could look at the IO scheduling
policy. It seems that maybe Dom0 is not servicing IO requests fairly
from your DomU''s. Either that or there is a bug in Dom0 and your disk
IO
requests are getting stuck under high IO load.

I could be wrong and it is a bug in GPLPV but I can''t see any evidence
of it.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi James:
 
      I am still working on the blue screen issue.
 
     This time I found 6 more machines to test. Two machines confronts  a new
blue screen in
2 days long test. 
 
KERNEL_STACK_INPAGE_ERROR
stop:0x00000077(0xc0000185,0xc0000185,0x00000000,0x009d9000)
 
 
      Attached is the log, cp1, 9, 15, 16, 17, 20, are broken, and has the log
like:
-----------------------------
12944494073046: XenVbd --> XenVbd_HwScsiResetBus
12944494073046: XenVbd     IRQL = 31
12944494073046: XenVbd     Completing in-flight srb 892C7058 with status
SRB_STATUS_BUS_RESET
12944494073046: XenVbd <-- XenVbd_HwScsiResetBus
12944494073046: XenPCI     Grant Entry 16382 for ISCS still in use by ISCS
12944494073046:
*** Assertion failed: !xpdd->gnttbl_table[ref].flags
***   Source File: e:\download\win-pvdrivers.hg\xenpci\gnttbl.c, line 120
 ---------------------------
 
Thanks.
 
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> Date: Thu, 3 Mar 2011 20:21:40 +0800
> 
> I''ve just pushed a fix that would cause a loop then a crash in the
event
> of a concurrent write.
> 
> James
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Thursday, 3 March 2011 23:18
> > To: James Harper
> > Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> > 
> > Well, quite strange is no blue screen show up now.
> > Should have some VMS got crash according to previous test result.
> > 
> > I will restart the test again.
> > 
> > > Subject: RE: [Xen-devel] RE: blue screen in windows balloon
driver
> > > Date: Thu, 3 Mar 2011 22:44:12 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com
> > >
> > > >
> > > > Two VM restart during the test.
> > > > From the log, I guess it is your fix, right?
> > > >
> > > > Log attatched.
> > > >
> > >
> > > It looks like at least one device failed to start up, probably
due
> to
> > > delays in the backend. I don''t know why it would shut
down itself
> > > though.
> > >
> > > James
> 
 		 	   		  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Forget to address the serial output:
 
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch input to Xen)
(XEN) Freed 156kB init memory.
(XEN) ioapic_guest_write: apic=0, pin=0, irq=0
(XEN) ioapic_guest_write: new_entry=000100f0
(XEN) ioapic_guest_write: old_entry=00010000 pirq=0
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) ioapic_guest_write: apic=0, pin=2, irq=0
(XEN) ioapic_guest_write: new_entry=000100f0
(XEN) ioapic_guest_write: old_entry=000000f0 pirq=0
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) irq.c:1445: dom0: pirq 0 or irq 3 already mapped
(XEN) ioapic_guest_write: apic=0, pin=4, irq=4
(XEN) ioapic_guest_write: new_entry=000100f1
(XEN) ioapic_guest_write: old_entry=000000f1 pirq=0
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) irq.c:1445: dom0: pirq 0 or irq 5 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 6 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 7 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 8 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 9 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 10 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 11 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 12 already mapped
(XEN) irq.c:1445: dom0: pirq 0 or irq 13 already mapped
(XEN) ioapic_guest_write: apic=0, pin=0, irq=0
(XEN) ioapic_guest_write: new_entry=000000f0
(XEN) ioapic_guest_write: old_entry=00010000 pirq=0
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) ioapic_guest_write: apic=0, pin=23, irq=23
(XEN) ioapic_guest_write: new_entry=0001a0d0
(XEN) ioapic_guest_write: old_entry=0000a0d0 pirq=23
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) ioapic_guest_write: apic=0, pin=18, irq=18
(XEN) ioapic_guest_write: new_entry=0001a0a8
(XEN) ioapic_guest_write: old_entry=0000a0a8 pirq=18
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) ioapic_guest_write: apic=0, pin=18, irq=18
(XEN) ioapic_guest_write: new_entry=0001a0a8
(XEN) ioapic_guest_write: old_entry=0000a0a8 pirq=18
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!

 


From: tinnycloud@hotmail.com
To: james.harper@bendigoit.com.au
CC: xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
Date: Mon, 14 Mar 2011 10:24:14 +0800




Hi James:
 
      I am still working on the blue screen issue.
 
     This time I found 6 more machines to test. Two machines confronts  a new
blue screen in
2 days long test. 
 
KERNEL_STACK_INPAGE_ERROR
stop:0x00000077(0xc0000185,0xc0000185,0x00000000,0x009d9000)
 
 
      Attached is the log, cp1, 9, 15, 16, 17, 20, are broken, and has the log
like:
-----------------------------
12944494073046: XenVbd --> XenVbd_HwScsiResetBus
12944494073046: XenVbd     IRQL = 31
12944494073046: XenVbd     Completing in-flight srb 892C7058 with status
SRB_STATUS_BUS_RESET
12944494073046: XenVbd <-- XenVbd_HwScsiResetBus
12944494073046: XenPCI     Grant Entry 16382 for ISCS still in use by ISCS
12944494073046:
*** Assertion failed: !xpdd->gnttbl_table[ref].flags
***   Source File: e:\download\win-pvdrivers.hg\xenpci\gnttbl.c, line 120
 ---------------------------
 
Thanks.
 
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> Date: Thu, 3 Mar 2011 20:21:40 +0800
> 
> I''ve just pushed a fix that would cause a loop then a crash in the
event
> of a concurrent write.
> 
> James
> 
> > -----Original Message-----
> > From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
> > Sent: Thursday, 3 March 2011 23:18
> > To: James Harper
> > Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> > 
> > Well, quite strange is no blue screen show up now.
> > Should have some VMS got crash according to previous test result.
> > 
> > I will restart the test again.
> > 
> > > Subject: RE: [Xen-devel] RE: blue screen in windows balloon
driver
> > > Date: Thu, 3 Mar 2011 22:44:12 +1100
> > > From: james.harper@bendigoit.com.au
> > > To: tinnycloud@hotmail.com
> > >
> > > >
> > > > Two VM restart during the test.
> > > > From the log, I guess it is your fix, right?
> > > >
> > > > Log attatched.
> > > >
> > >
> > > It looks like at least one device failed to start up, probably
due
> to
> > > delays in the backend. I don''t know why it would shut
down itself
> > > though.
> > >
> > > James
> 
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Hi James:
> 
>       I am still working on the blue screen issue.
> 
>      This time I found 6 more machines to test. Two machines confronts
a new
> blue screen in
> 2 days long test.
> 
> KERNEL_STACK_INPAGE_ERROR
> stop:0x00000077(0xc0000185,0xc0000185,0x00000000,0x009d9000)
> 
> 
>       Attached is the log, cp1, 9, 15, 16, 17, 20, are broken, and has
the log
> like:
> -----------------------------
> 12944494073046: XenVbd --> XenVbd_HwScsiResetBus
> 12944494073046: XenVbd     IRQL = 31
> 12944494073046: XenVbd     Completing in-flight srb 892C7058 with
status
> SRB_STATUS_BUS_RESET
> 12944494073046: XenVbd <-- XenVbd_HwScsiResetBus
> 12944494073046: XenPCI     Grant Entry 16382 for ISCS still in use by
ISCS
> 12944494073046:
> *** Assertion failed: !xpdd->gnttbl_table[ref].flags
> ***   Source File: e:\download\win-pvdrivers.hg\xenpci\gnttbl.c, line
120
>  ---------------------------
> 
That Assertion only happens in dump mode though right? Due to the delays
in requests getting completed, dump mode is still seeing request
completions from non-dump mode. I should have a fix for that in a
minute.

Bug check 0x77 means that windows couldn''t read the page into memory
from the page file. This could still be caused by timeouts. The first
and second parameter 0xc0000185 means ''STATUS_IO_DEVICE_ERROR''
which I
guess would be consistent with windows repeatedly trying to read the
pagefile and then giving up after multiple timeout/reset cycles.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 12944494073046: XenVbd --> XenVbd_HwScsiResetBus
> 12944494073046: XenVbd     IRQL = 31
> 12944494073046: XenVbd     Completing in-flight srb 892C7058 with
status
> SRB_STATUS_BUS_RESET
> 12944494073046: XenVbd <-- XenVbd_HwScsiResetBus
> 12944494073046: XenPCI     Grant Entry 16382 for ISCS still in use by
ISCS
> 12944494073046:
> *** Assertion failed: !xpdd->gnttbl_table[ref].flags
> ***   Source File: e:\download\win-pvdrivers.hg\xenpci\gnttbl.c, line
120
>  ---------------------------
I''ve just pushed a fix for that assertion, but can''t really
test it. It
isn''t going to fix your original problem, but the dump should now work.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks James.
I''ve just noticed that all 3 machine with *blue screen issue* have io
scheduler: "CFQ"
while other 4 none issue machines have io schedule DEADLINE.
 
I think this might be the difference and cause the problem.
I am re-running the test now.
 
> Subject: RE: [Xen-devel] RE: blue screen in windows balloon driver
> Date: Mon, 14 Mar 2011 16:08:15 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com
> 
> > 12944494073046: XenVbd --> XenVbd_HwScsiResetBus
> > 12944494073046: XenVbd IRQL = 31
> > 12944494073046: XenVbd Completing in-flight srb 892C7058 with
> status
> > SRB_STATUS_BUS_RESET
> > 12944494073046: XenVbd <-- XenVbd_HwScsiResetBus
> > 12944494073046: XenPCI Grant Entry 16382 for ISCS still in use by
> ISCS
> > 12944494073046:
> > *** Assertion failed: !xpdd->gnttbl_table[ref].flags
> > *** Source File: e:\download\win-pvdrivers.hg\xenpci\gnttbl.c, line
> 120
> > ---------------------------
> 
> I''ve just pushed a fix for that assertion, but can''t
really test it. It
> isn''t going to fix your original problem, but the dump should now
work.
> 
> James
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel