Xen devel - Jan 2011 - [PATCH] mem_sharing: fix race condition of nominate and unshare

Hi, this patch does the following
(1) When updating/checking p2m type for mem_sharing, we must hold shr_lock
(2) For nominate operation, if the page is already nominated, return the
handle from page_info->shr_handle
(3) For unshare operation, it is possible that multiple users unshare a page
via hvm_hap_nested_page_fault() at the same time. If the page is already
un-shared by someone else, simply return success.

NOTE: we assume that nobody holds page_alloc_lock/p2m_lock before calling
nominate/share/unshare.

Signed-off-by: Jui-Hao Chiang <juihaochiang@gmail.com>
Signed-off-by: Han-Lin Li <Han-Lin.Li@itri.org.tw>

Bests,
Jui-Hao



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 16:11 +0000 on 06 Jan (1294330319), Jui-Hao Chiang
wrote:
> Hi, this patch does the following
> (1) When updating/checking p2m type for mem_sharing, we must hold shr_lock
> (2) For nominate operation, if the page is already nominated, return the
handle from page_info->shr_handle
> (3) For unshare operation, it is possible that multiple users unshare a
page via hvm_hap_nested_page_fault() at the same time. If the page is already
un-shared by someone else, simply return success.
I''m going to apply this, since it looks like an improvement, but
I''m not
convinced it properly solves the problem.
> NOTE: we assume that nobody holds page_alloc_lock/p2m_lock before calling
nominate/share/unshare.
As I told you earlier, that''s not the case.  p2m_teardown() can call
mem_sharing_unshare_page() with the p2m lock held.

Cheers,

Tim.
> Signed-off-by: Jui-Hao Chiang
<juihaochiang@gmail.com<mailto:juihaochiang@gmail.com>>
> Signed-off-by: Han-Lin Li
<Han-Lin.Li@itri.org.tw<mailto:Li@itri.org.tw>>
> 
> Bests,
> Jui-Hao
> diff -r 7b4c82f07281 xen/arch/x86/mm/mem_sharing.c
> --- a/xen/arch/x86/mm/mem_sharing.c	Wed Jan 05 23:54:15 2011 +0000
> +++ b/xen/arch/x86/mm/mem_sharing.c	Thu Jan 06 23:46:28 2011 +0800
> @@ -502,6 +502,7 @@ int mem_sharing_nominate_page(struct p2m
>  
>      *phandle = 0UL;
>  
> +    shr_lock(); 
>      mfn = gfn_to_mfn(p2m, gfn, &p2mt);
>  
>      /* Check if mfn is valid */
> @@ -509,29 +510,33 @@ int mem_sharing_nominate_page(struct p2m
>      if (!mfn_valid(mfn))
>          goto out;
>  
> +    /* Return the handle if the page is already shared */
> +    page = mfn_to_page(mfn);
> +    if (p2m_is_shared(p2mt)) {
> +        *phandle = page->shr_handle;
> +        ret = 0;
> +        goto out;
> +    }
> +
>      /* Check p2m type */
>      if (!p2m_is_sharable(p2mt))
>          goto out;
>  
>      /* Try to convert the mfn to the sharable type */
> -    page = mfn_to_page(mfn);
>      ret = page_make_sharable(d, page, expected_refcnt); 
>      if(ret) 
>          goto out;
>  
>      /* Create the handle */
>      ret = -ENOMEM;
> -    shr_lock(); 
>      handle = next_handle++;  
>      if((hash_entry = mem_sharing_hash_insert(handle, mfn)) == NULL)
>      {
> -        shr_unlock();
>          goto out;
>      }
>      if((gfn_info = mem_sharing_gfn_alloc()) == NULL)
>      {
>          mem_sharing_hash_destroy(hash_entry);
> -        shr_unlock();
>          goto out;
>      }
>  
> @@ -545,7 +550,6 @@ int mem_sharing_nominate_page(struct p2m
>          BUG_ON(page_make_private(d, page) != 0);
>          mem_sharing_hash_destroy(hash_entry);
>          mem_sharing_gfn_destroy(gfn_info, 0);
> -        shr_unlock();
>          goto out;
>      }
>  
> @@ -559,11 +563,11 @@ int mem_sharing_nominate_page(struct p2m
>      gfn_info->domain = d->domain_id;
>      page->shr_handle = handle;
>      *phandle = handle;
> -    shr_unlock();
>  
>      ret = 0;
>  
>  out:
> +    shr_unlock();
>      return ret;
>  }
>  
> @@ -633,14 +637,21 @@ int mem_sharing_unshare_page(struct p2m_
>      struct list_head *le;
>      struct domain *d = p2m->domain;
>  
> +    mem_sharing_audit();
> +    /* Remove the gfn_info from the list */
> +    shr_lock();
> +    
>      mfn = gfn_to_mfn(p2m, gfn, &p2mt);
> +    
> +    /* Has someone already unshared it? */
> +    if (!p2m_is_shared(p2mt)) {
> +        shr_unlock();
> +        return 0;
> +    }
>  
>      page = mfn_to_page(mfn);
>      handle = page->shr_handle;
>   
> -    mem_sharing_audit();
> -    /* Remove the gfn_info from the list */
> -    shr_lock();
>      hash_entry = mem_sharing_hash_lookup(handle); 
>      list_for_each(le, &hash_entry->gfns)
>      {
> @@ -707,7 +718,6 @@ private_page_found:
>          mem_sharing_hash_delete(handle);
>      else
>          atomic_dec(&nr_saved_mfns);
> -    shr_unlock();
>  
>      if(p2m_change_type(p2m, gfn, p2m_ram_shared, p2m_ram_rw) != 
>                                                  p2m_ram_shared) 
> @@ -718,6 +728,7 @@ private_page_found:
>      /* Update m2p entry */
>      set_gpfn_from_mfn(mfn_x(page_to_mfn(page)), gfn);
>  
> +    shr_unlock();
>      return 0;
>  }
>  

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim and Hao:

 

         The patch failed to fix the bug.

 

         I applied patch, and add some more log info.

 

in  mem_sharing_unshare_page

664     shr_lock();

665     mfn = gfn_to_mfn(d, gfn, &p2mt);

666     /* Has someone already unshared it? */

667     if (!p2m_is_shared(p2mt)) {

668         printk("===someone unshare mfn %lx\n", mfn);


669         shr_unlock();

670         return 0;

671     }

 

In  mem_sharing_nominate_page

508     shr_lock();

509     mfn = gfn_to_mfn(d, gfn, &p2mt);

510 

511     /* Check if mfn is valid */

512     ret = -EINVAL;

513     if (!mfn_valid(mfn))

514         goto out;

515 

516     if (p2m_is_shared(p2mt)) {

517         page = mfn_to_page(mfn);

518         printk("===page h %lu, mfx %lx is already shared\n",
page->shr_handle, mfn);


519     }

520     /* Check p2m type */

521     if (!p2m_is_sharable(p2mt))

522         goto out;

523  

 

 

Also in mem_sharing_share_pages, I print some free info

 

584     shr_lock();

585 

586     ret = XEN_DOMCTL_MEM_SHARING_S_HANDLE_INVALID;

587     se = mem_sharing_hash_lookup(sh);

588     if(se == NULL) goto err_out;

589     ret = XEN_DOMCTL_MEM_SHARING_C_HANDLE_INVALID;

590     ce = mem_sharing_hash_lookup(ch);

591     if(ce == NULL) goto err_out;

592     spage = mfn_to_page(se->mfn);

593     cpage = mfn_to_page(ce->mfn);

594     printk("===will free cpage_mfn %lx spage_mfn %lx \n",
ce->mfn,
se->mfn);

 

 

 

634     ASSERT(list_empty(&ce->gfns));

635     mem_sharing_hash_delete(ch);

636     atomic_inc(&nr_saved_mfns);

637     /* Free the client page */

638     if(test_and_clear_bit(_PGC_allocated, &cpage->count_info)){

639         put_page(cpage);

640         printk("===free cpage_mfn %lx spage_mfn %lx \n",
ce->mfn,
se->mfn);

641     }

642     ret = 0;

643     

644 err_out:

645     shr_unlock();

646 

647     return ret;

 

 

Below is the serial output.

We can see neither line 668 and nor line 518 is print out.

 

blktap_sysfs_create: adding attributes for dev ffff88012148ee00

(XEN) ===will free cpage_mfn 1406fd spage_mfn 2df6a6 

(XEN) ===free cpage_mfn 0 spage_mfn 2df6a6 

(XEN) ===will free cpage_mfn 14083c spage_mfn 2df6a8 

(XEN) ===free cpage_mfn 0 spage_mfn 2df6a8 

(XEN) ===will free cpage_mfn 1409e5 spage_mfn 2df6a9 

(XEN) ===free cpage_mfn 0 spage_mfn 2df6a9 

(XEN) ===will free cpage_mfn 142eb4 spage_mfn 141c4b 

(XEN) ===free cpage_mfn 0 spage_mfn 141c4b 

(XEN) printk: 32 messages suppressed.

(XEN) mm.c:859:d0 Error getting mfn 2df6a8 (pfn fffffffffffffffe) from L1
entry 80000002df6a8627 for l1e_owner=0, pg_owner=2

(XEN) mm.c:859:d0 Error getting mfn 2df6a9 (pfn fffffffffffffffe) from L1
entry 80000002df6a9627 for l1e_owner=0, pg_owner=2

(XEN) ===will free cpage_mfn 219a1c spage_mfn 1421b1 

(XEN) ===free cpage_mfn 0 spage_mfn 1421b1 

(XEN) ===will free cpage_mfn 2dea05 spage_mfn 142124 

(XEN) ===free cpage_mfn 0 spage_mfn 142124 

(XEN) ===will free cpage_mfn 147127 spage_mfn 146cbb 

(XEN) ===free cpage_mfn 0 spage_mfn 146cbb 

(XEN) ===will free cpage_mfn 146ecf spage_mfn 14127f 

(XEN) ===free cpage_mfn 0 spage_mfn 14127f

 

 

From: Jui-Hao Chiang [mailto:juihaochiang@gmail.com] 
Date:  2011.1.7. 0:12
TO: Tim Deegan
CC: tinnycloud; xen-devel@lists.xensource.com
Sub: [PATCH] mem_sharing: fix race condition of nominate and unshare

 

Hi, this patch does the following
(1) When updating/checking p2m type for mem_sharing, we must hold shr_lock
(2) For nominate operation, if the page is already nominated, return the
handle from page_info->shr_handle
(3) For unshare operation, it is possible that multiple users unshare a page
via hvm_hap_nested_page_fault() at the same time. If the page is already
un-shared by someone else, simply return success.

NOTE: we assume that nobody holds page_alloc_lock/p2m_lock before calling
nominate/share/unshare.

Signed-off-by: Jui-Hao Chiang <juihaochiang@gmail.com>
Signed-off-by: Han-Lin Li <Han-Lin.Li@itri.org.tw>

Bests,
Jui-Hao



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Tim

On Fri, Jan 7, 2011 at 12:54 AM, Tim Deegan <Tim.Deegan@citrix.com> wrote:
> At 16:11 +0000 on 06 Jan (1294330319), Jui-Hao Chiang wrote:
> > Hi, this patch does the following
> > (1) When updating/checking p2m type for mem_sharing, we must hold
> shr_lock
> > (2) For nominate operation, if the page is already nominated, return
the
> handle from page_info->shr_handle
> > (3) For unshare operation, it is possible that multiple users unshare
a
> page via hvm_hap_nested_page_fault() at the same time. If the page is
> already un-shared by someone else, simply return success.
>
> I''m going to apply this, since it looks like an improvement, but
I''m not
> convinced it properly solves the problem.
>
>
It seems tinnycloud''s case is when dom0 try to RW maps a shared page,
which
should unshare it properly, and change the type count.
But there is still a bug hidden in page_make_sharable() which fails to
recover type count when the call fails.
Now I trace it again, and found something different than what we have
discussed before.
I will find it and submit patch again.


>  > NOTE: we assume that nobody holds page_alloc_lock/p2m_lock before
> calling nominate/share/unshare.
>
> As I told you earlier, that''s not the case.  p2m_teardown() can
call
> mem_sharing_unshare_page() with the p2m lock held.
>
>
Oops, I forgot again.
After this change, unshare() has a potential problem of deadlock for
shr_lock and p2m_lock with different locking order.
Assume two CPUs do the following
CPU1: hvm_hap_nested_page_fault() => unshare() => p2m_change_type()
(locking
order: shr_lock, p2m_lock)
CPU2: p2m_teardown() => unshare() (locking order: p2m_lock, shr_lock)
When CPU1 grabs shr_lock and CPU2 grabs p2m_lock, they deadlock later.

So it seems better to fix the following rules
(1) Fix locking order: p2m_lock ---> shr_lock
(2) Any function in mem_sharing, if modifying/checking p2m entry is
necessary, it must hold p2m_lock and then shr_lock. Later on, when changing
p2m entries, don''t call any p2m function which locks p2m again

So for p2m functions, it seems better to provide some functions which
don''t
call p2m_lock again.
What do you think? If that''s ok, I will do it in this way.

Bests,
Jui-Hao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Oops, I forgot again.
> After this change, unshare() has a potential problem of deadlock for
> shr_lock and p2m_lock with different locking order.
> Assume two CPUs do the following
> CPU1: hvm_hap_nested_page_fault() => unshare() => p2m_change_type()
> (locking order: shr_lock, p2m_lock)
> CPU2: p2m_teardown() => unshare() (locking order: p2m_lock, shr_lock)
> When CPU1 grabs shr_lock and CPU2 grabs p2m_lock, they deadlock later.
>
> So it seems better to fix the following rules
> (1) Fix locking order: p2m_lock ---> shr_lock
> (2) Any function in mem_sharing, if modifying/checking p2m entry is
> necessary, it must hold p2m_lock and then shr_lock. Later on, when changing
> p2m entries, don''t call any p2m function which locks p2m again
>
> So for p2m functions, it seems better to provide some functions which
don''t
> call p2m_lock again.
> What do you think? If that''s ok, I will do it in this way.
>
>
Hmm,  after looking it deeper, I summarize as the following
(1) It seems all the users of shr_lock, nominate/share/unshare, will
check/modify p2m type.
- nominate: p2m_change_type()
- share: set_shared_p2m_entry()
- unshare: set_shared_p2m_entry() and p2m_change_type()
(2) The functions which call unshare()
- hvm_hap_nested_page_fault(): I don''t see any p2m_lock holded
- p2m_tear_down(): hold p2m_lock
- gfn_to_mfn_unshare(): I don''t see any p2m_lock holded

One of the solution is to
(a) Simply replace shr_lock with p2m_lock.
(b) In unshare(), apply the following: if (!p2m_locked_by_me(p2m)) call
p2m_lock, otherwise, don''t lock it.
(c) p2m_change_type() and set_shared_p2m_entry() are pretty similar, we can
merge the functionality into one function, which does NOT take p2m_lock, and
keep the original p2m_change_type() unchanged.

Correct me if wrong.

Bests,
Jui-Hao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, tinnycloud:

(XEN) mm.c:859:d0 Error getting mfn 2df6a8 (pfn fffffffffffffffe) from
L1
> entry 80000002df6a8627 for l1e_owner=0, pg_owner=2
>
> (XEN) mm.c:859:d0 Error getting mfn 2df6a9 (pfn fffffffffffffffe) from L1
> entry 80000002df6a9627 for l1e_owner=0, pg_owner=2
>
Could you use dump_execution_state() in mm.c:859?
And in the unshare() function, could you move the printk outside the
(!p2m_is_shared(p2mt))
checking?
If you put inside it, we never know if the unshare() is being done or not
(please also print out the mfn, p2mt, gfn, domain_id).

Just out of curiosity, are you running stubdom? your HVM guest id =2 is
pretty weird.

Thanks,
Jui-Hao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

HI Jui-Hao:

 

         I have no stub-dom fro HVM. The domain ID starts from 1, and grows
on every new domain is created.

         Below is for you, thanks. 

 

---------code-----

 

664     shr_lock();

665     mfn = gfn_to_mfn(d, gfn, &p2mt);

666     /* Has someone already unshared it? */

667     printk("===will unshare mfn %lx p2mt %x gfn %lu did %d\n",
mfn,
p2mt, gfn, d->domain_id);

668     if (!p2m_is_shared(p2mt)) {

669         printk("===someone unshare mfn %lx p2mt %x gfn %lu did
%d\n",
mfn, p2mt, gfn, d->domain_id);


670         shr_unlock();

671         return 0;

672     }

 

 

------output -------

 

(XEN) ===will unshare mfn 1728ae p2mt d gfn 512686 did 1

(XEN) ===will unshare mfn 1728ef p2mt d gfn 512751 did 1

(XEN) ===will unshare mfn 1729aa p2mt d gfn 512938 did 1

(XEN) ===will unshare mfn 1728f6 p2mt d gfn 512758 did 1

(XEN) ===will unshare mfn 2de94a p2mt d gfn 39754 did 1

(XEN) ===will unshare mfn 2de94b p2mt d gfn 39755 did 1

(XEN) ===will unshare mfn 2de94c p2mt d gfn 39756 did 1

(XEN) printk: 32 messages suppressed.

(XEN) mm.c:859:d0 Error getting mfn 2de94c (pfn fffffffffffffffe) from L1
entry 80000002de94c627 for l1e_owner=0, pg_owner=1

(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----

(XEN) CPU:    0

(XEN) RIP:    e008:[<ffff82c48015d1d1>] get_page_from_l1e+0x351/0x4d0

(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor

(XEN) rax: 007fffffffffffff   rbx: 0000000000000001   rcx: 0000000000000092

(XEN) rdx: 8000000000000002   rsi: 8000000000000003   rdi: ffff82f605bd2980

(XEN) rbp: 00000000002de94c   rsp: ffff82c48035fcd8   r8:  0000000000000001

(XEN) r9:  0000000000000000   r10: 00000000fffffffb   r11: 0000000000000002

(XEN) r12: 0000000000000000   r13: fffffffffffffffe   r14: ffff82f605bd2980

(XEN) r15: 80000002de94c627   cr0: 000000008005003b   cr4: 00000000000026f0

(XEN) cr3: 000000031b870000   cr2: 000000000098efa0

(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008

(XEN) Xen stack trace from rsp=ffff82c48035fcd8:

(XEN)    80000002de94c627 ffff830200000000 ffff82c400000001 ffff82c4801df3d9

(XEN)    ffff83033e944930 ffff8300bf554000 ffff83023ff40000 000000000000014c

(XEN)    ffffffffffffffff 0000000000800627 ffff8302dd6a0000 0000000000000001

(XEN)    ffff83031ccb26b8 000000000031ccb2 80000002de94c627 ffff82c48016288b

(XEN)    000082c480168170 0000000000000000 0000000000000004 ffff8300bf554000

(XEN)    0000000000000009 000000000031ccb2 ffff83023fe60000 80000002de94c627

(XEN)    000000000031ccb2 ffff82c48035fedc 0000000000000000 000000010000014c

(XEN)    ffff83031babc000 0000000000000000 0000000000000000 0000000000000001

(XEN)    ffff83031ccb26b8 000000000031ccb2 8000000009b4c627 ffff82c480163f36

(XEN)    0000000000000001 ffff82c480161a49 ffff82c48035fe88 ffff82c48035fe88

(XEN)    00007ff0fffffffe 0000000000000000 0000000100000000 ffff8300bf554000

(XEN)    00000001bf554000 0000000000000000 000000010000c178 ffff82f606399640

(XEN)    0000000000000006 ffff83023fe60000 ffff8302dd6a0000 ffff8300bf554000

(XEN)    0000000000000000 0000000000000000 0000000000000000 000000008035ff28

(XEN)    ffff8801208d5c18 0000000080251008 000000031ccb26b8 8000000009b4c627

(XEN)    ffff82c480251008 ffff82c480251000 0000000000000000 ffff82c480113d7e

(XEN)    0000000d00000000 0000000000000001 00000001ffffffff ffff8300bf554000

(XEN)    ffff8801208d5d68 0000000000000001 ffff880121dbd0a8 00007f4de18d7000

(XEN)    0000000000000001 ffff82c4801e3169 0000000000000001 00007f4de18d7000

(XEN)    ffff880121dbd0a8 0000000000000001 ffff8801208d5d68 0000000000000000

(XEN) Xen call trace:

(XEN)    [<ffff82c48015d1d1>] get_page_from_l1e+0x351/0x4d0

(XEN)    [<ffff82c4801df3d9>] ept_get_entry+0xa9/0x1c0

(XEN)    [<ffff82c48016288b>] mod_l1_entry+0x37b/0x9a0

(XEN)    [<ffff82c480163f36>] do_mmu_update+0x9f6/0x1a70

(XEN)    [<ffff82c480161a49>] do_mmuext_op+0x859/0x1320

(XEN)    [<ffff82c480113d7e>] do_multicall+0x14e/0x340

(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae

(XEN)    

 

From: Jui-Hao Chiang [mailto:juihaochiang@gmail.com] 
Date: 2011.1.1 14:45
To: tinnycloud
CC: Tim Deegan; xen-devel@lists.xensource.com
Sub: Re: [PATCH] mem_sharing: fix race condition of nominate and unshare

 

Hi, tinnycloud:

(XEN) mm.c:859:d0 Error getting mfn 2df6a8 (pfn fffffffffffffffe) from L1
entry 80000002df6a8627 for l1e_owner=0, pg_owner=2

(XEN) mm.c:859:d0 Error getting mfn 2df6a9 (pfn fffffffffffffffe) from L1
entry 80000002df6a9627 for l1e_owner=0, pg_owner=2


Could you use dump_execution_state() in mm.c:859?
And in the unshare() function, could you move the printk outside the
(!p2m_is_shared(p2mt)) checking?
If you put inside it, we never know if the unshare() is being done or not
(please also print out the mfn, p2mt, gfn, domain_id).

Just out of curiosity, are you running stubdom? your HVM guest id =2 is
pretty weird.

Thanks,
Jui-Hao



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 06:02 +0000 on 07 Jan (1294380120), Jui-Hao Chiang
wrote:
> One of the solution is to
> (a) Simply replace shr_lock with p2m_lock.
I think this is the best choice.   If we find that the p2m lock is a
bottleneck we can address it later. 

Cheers,

Tim
> (b) In unshare(), apply the following: if (!p2m_locked_by_me(p2m)) call
p2m_lock, otherwise, don''t lock it.
> (c) p2m_change_type() and set_shared_p2m_entry() are pretty similar, we can
merge the functionality into one function, which does NOT take p2m_lock, and
keep the original p2m_change_type() unchanged.
> 
> Correct me if wrong.
> 
> Bests,
> Jui-Hao
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Tim:

On Sat, Jan 8, 2011 at 12:09 AM, Tim Deegan <Tim.Deegan@citrix.com> wrote:
> At 06:02 +0000 on 07 Jan (1294380120), Jui-Hao Chiang wrote:
> > One of the solution is to
> > (a) Simply replace shr_lock with p2m_lock.
>
> I think this is the best choice.   If we find that the p2m lock is a
> bottleneck we can address it later.
>
>
Just to be skeptic.
Why doesn''t mfn_to_gfn() take p2m lock when querying the p2m type? Is
there
any quarantee that the resulting type is correct and trustful?
For example:
(1) User1 query the p2m type:
mfn_to_gfn(...&p2mt);
if (p2mt == p2m_ram_rw) /* do something based on the p2m type result? */

(2) User2 modify the p2m type
p2m_lock(p2m);
set_p2m_entry(..... p2m_ram_rw);
p2m_unlock(p2m);

Thanks,
Jui-Hao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sorry, typo

On Mon, Jan 10, 2011 at 12:57 PM, Jui-Hao Chiang
<juihaochiang@gmail.com>wrote:
> Hi, Tim:
>
> On Sat, Jan 8, 2011 at 12:09 AM, Tim Deegan <Tim.Deegan@citrix.com>
wrote:
>
>> At 06:02 +0000 on 07 Jan (1294380120), Jui-Hao Chiang wrote:
>> > One of the solution is to
>> > (a) Simply replace shr_lock with p2m_lock.
>>
>> I think this is the best choice.   If we find that the p2m lock is a
>> bottleneck we can address it later.
>>
>>
> Just to be skeptic.
> Why doesn''t mfn_to_gfn() take p2m lock when querying the p2m type?
Is there
> any quarantee that the resulting type is correct and
>
I mean gfn_to_mfn()

> trustful?
> For example:
> (1) User1 query the p2m type:
> mfn_to_gfn(...&p2mt);
> if (p2mt == p2m_ram_rw) /* do something based on the p2m type result? */
>
> (2) User2 modify the p2m type
> p2m_lock(p2m);
> set_p2m_entry(..... p2m_ram_rw);
> p2m_unlock(p2m);
>
> Thanks,
> Jui-Hao
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sent: Jui-Hao Chiang [mailto:juihaochiang@gmail.com] 
Date: 2011年1月7日 14:02
To: Tim Deegan
CC: tinnycloud; xen-devel@lists.xensource.com
Sub: Re: [PATCH] mem_sharing: fix race condition of nominate and unshare

 

 

Oops, I forgot again.
After this change, unshare() has a potential problem of deadlock for
shr_lock and p2m_lock with different locking order.
Assume two CPUs do the following
CPU1: hvm_hap_nested_page_fault() => unshare() => p2m_change_type()
(locking
order: shr_lock, p2m_lock)
CPU2: p2m_teardown() => unshare() (locking order: p2m_lock, shr_lock)
When CPU1 grabs shr_lock and CPU2 grabs p2m_lock, they deadlock later.

So it seems better to fix the following rules
(1) Fix locking order: p2m_lock ---> shr_lock
(2) Any function in mem_sharing, if modifying/checking p2m entry is
necessary, it must hold p2m_lock and then shr_lock. Later on, when changing
p2m entries, don''t call any p2m function which locks p2m again

So for p2m functions, it seems better to provide some functions which
don''t
call p2m_lock again.
What do you think? If that''s ok, I will do it in this way.


Hmm,  after looking it deeper, I summarize as the following
(1) It seems all the users of shr_lock, nominate/share/unshare, will
check/modify p2m type.
- nominate: p2m_change_type()
- share: set_shared_p2m_entry()
- unshare: set_shared_p2m_entry() and p2m_change_type()
(2) The functions which call unshare()
- hvm_hap_nested_page_fault(): I don''t see any p2m_lock holded
- p2m_tear_down(): hold p2m_lock
- gfn_to_mfn_unshare(): I don''t see any p2m_lock holded



 

Thank for sharing the lock info.

I’ve go through the code too. 

 

1.       mem_sharing_unshare_page() has the routine  called from
gfn_to_mfn_unshare, which is called by gnttab_transfer  

Since no bug report on grant_table right now, so I think this is safe for
now

Also  p2m_tear_down è mem_sharing_unshare_page() , its flag is
MEM_SHARING_DESTROY_GFN, and won’t has the chance to

call set_shared_p2m_entry()

 

2.       as for p2m_change_type(), I found in other place is it called lock
free, so it is safe too

3.       set_shared_p2m_entry() which call set_p2m_entry() is not in
p2m_lock, and I found in other code set_p2m_entry is called in p2m_lock,

so here I think it is a problem

 

So I think at least set_p2m_entry should be put into p2m_lock.

I’ll do more investigation base on this.

 

One of the solution is to 
(a) Simply replace shr_lock with p2m_lock.
(b) In unshare(), apply the following: if (!p2m_locked_by_me(p2m)) call
p2m_lock, otherwise, don''t lock it.
(c) p2m_change_type() and set_shared_p2m_entry() are pretty similar, we can
merge the functionality into one function, which does NOT take p2m_lock, and
keep the original p2m_change_type() unchanged.

Correct me if wrong.

Bests,
Jui-Hao



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, tinnycloud:

Thanks for your testing info.
I assume you have mem_sharing_unshare_page called with successful return
value, otherwise the mod_l1_entry won''t be called.
Could you call mem_sharing_debug_gfn() before unshare() return success?
In addition, are there multiple CPUs touching the same page? e.g. you can
print out the cpu id inside unshare() and the mm.c:859.

> After this change, unshare() has a potential problem of deadlock for
> shr_lock and p2m_lock with different locking order.
> Assume two CPUs do the following
> CPU1: hvm_hap_nested_page_fault() => unshare() => p2m_change_type()
> (locking order: shr_lock, p2m_lock)
> CPU2: p2m_teardown() => unshare() (locking order: p2m_lock, shr_lock)
> When CPU1 grabs shr_lock and CPU2 grabs p2m_lock, they deadlock later.
>
>  1.       mem_sharing_unshare_page() has the routine  called from
> gfn_to_mfn_unshare, which is called by gnttab_transfer
>
> Since no bug report on grant_table right now, so I think this is safe for
> now
>
> Also  p2m_tear_down è mem_sharing_unshare_page() , its flag is
> MEM_SHARING_DESTROY_GFN, and won’t has the chance to
>
> call set_shared_p2m_entry()
>
>
>
Of course, the p2m_teardown won''t call set_shared_p2m_entry. But this
does
not change my argument that p2m_teardown() hold p2m_lock to wait on
shr_lock. Actaully, after looking for a while, I rebut myself that the
scenario of deadlock won''t exist.
When p2m_teardown is called, the domain is dying in its last few steps
(device, irq are released), and there is no way for hvm_hap_nested_page_fault()
to happen on the memory of the dying domain. If this case is eliminated,
then my patch should not have deadlock problem. Any comments?

3.       set_shared_p2m_entry() which call set_p2m_entry() is not
in
> p2m_lock, and I found in other code set_p2m_entry is called in p2m_lock,
>
> so here I think it is a problem
>
>
>
> So I think at least set_p2m_entry should be put into p2m_lock.
>
> I’ll do more investigation base on this.
>
>
>
See this http://xenbits.xensource.com/xen-unstable.hg?rev/a8d69de8eb31

Thanks,
Jui-Hao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

Can you please (both of you) sort out yout mail clients to do proper
indenting of quoted text?  The plain-text versions don''t have any
quote prefix, which makes them confusing to read.

At 04:57 +0000 on 10 Jan (1294635461), Jui-Hao Chiang
wrote:
> Hi, Tim:
> 
> On Sat, Jan 8, 2011 at 12:09 AM, Tim Deegan
<Tim.Deegan@citrix.com<mailto:Tim.Deegan@citrix.com>> wrote:
> At 06:02 +0000 on 07 Jan (1294380120), Jui-Hao Chiang wrote:
> > One of the solution is to
> > (a) Simply replace shr_lock with p2m_lock.
> 
> I think this is the best choice.   If we find that the p2m lock is a
> bottleneck we can address it later.
> 
> 
> Just to be skeptic.
> Why doesn''t mfn_to_gfn() take p2m lock when querying the p2m type?
Because gfn->mfn lookups happen very frequently and requiring the lock
would be a performance bottleneck on multi-vcpu guests.
> Is there any quarantee that the resulting type is correct and trustful?
Yes.  It''s not perfect (and as I said I need to overhaul the locking
here) but if the p2m lookup only reads each level once and the p2m
updates are careful about the order they change things in, the worst
that can happen is another CPU sees a slightly out-of-date value.

There is at least one issue there (now that some p2m code frees old p2m
pages there''s a potential race against other readers that needs a
tlbflush-timestamp-style interlock), but TBH there are other things that
need fixing first.

Tim.
> For example:
> (1) User1 query the p2m type:
> mfn_to_gfn(...&p2mt);
> if (p2mt == p2m_ram_rw) /* do something based on the p2m type result? */
> 
> (2) User2 modify the p2m type
> p2m_lock(p2m);
> set_p2m_entry(..... p2m_ram_rw);
> p2m_unlock(p2m);
> 
> Thanks,
> Jui-Hao
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Jui-Hao:

 

         Under what situation hvm_hap_nested_page_fault will be called?

 

         Attach is the latest log with CPU id.

 

         Before each call of unshare() I print out the caller.

         

         From the log u will find the error mfn is MFN=2df895,  in line 488

       Line 37:(XEN) ===>mem_sharing_share_pages mfn 2df895 gfn 520798 p2md
d did 2

Is the log in mem_sharing_share_pages, from below line 632

 

618    /* gfn lists always have at least one entry => save to call
list_entry */

619     mem_sharing_gfn_account(gfn_get_info(&ce->gfns), 1);

620     mem_sharing_gfn_account(gfn_get_info(&se->gfns), 1);

621     list_for_each_safe(le, te, &ce->gfns)

622     {

623         gfn = list_entry(le, struct gfn_info, list);

624         /* Get the source page and type, this should never fail 

625          * because we are under shr lock, and got non-null se */

626         BUG_ON(!get_page_and_type(spage, dom_cow, PGT_shared_page));

627         /* Move the gfn_info from ce list to se list */

628         list_del(&gfn->list);

629         d = get_domain_by_id(gfn->domain);

630         BUG_ON(!d);

631         gfn_to_mfn(d, gfn->gfn, &p2mt);

632         printk("===>mem_sharing_share_pages mfn %lx gfn %lu p2md %x
did
%d\n", se->mfn, gfn->gfn, p2mt,d->domain_id);

633         BUG_ON(set_shared_p2m_entry(d, gfn->gfn, se->mfn) == 0);

634         put_domain(d);

635         list_add(&gfn->list, &se->gfns);

636         put_page_and_type(cpage);

637     }   

 

Sent: Jui-Hao Chiang [mailto:juihaochiang@gmail.com] 
Date: 2011年1月10日 16:10
To: tinnycloud
CC: Tim Deegan; xen-devel@lists.xensource.com
Sub: Re: [PATCH] mem_sharing: fix race condition of nominate and unshare

 

Hi, tinnycloud:

Thanks for your testing info.
I assume you have mem_sharing_unshare_page called with successful return
value, otherwise the mod_l1_entry won''t be called.
Could you call mem_sharing_debug_gfn() before unshare() return success?
In addition, are there multiple CPUs touching the same page? e.g. you can
print out the cpu id inside unshare() and the mm.c:859.


After this change, unshare() has a potential problem of deadlock for
shr_lock and p2m_lock with different locking order.
Assume two CPUs do the following
CPU1: hvm_hap_nested_page_fault() => unshare() => p2m_change_type()
(locking
order: shr_lock, p2m_lock)
CPU2: p2m_teardown() => unshare() (locking order: p2m_lock, shr_lock)
When CPU1 grabs shr_lock and CPU2 grabs p2m_lock, they deadlock later.

 1.       mem_sharing_unshare_page() has the routine  called from
gfn_to_mfn_unshare, which is called by gnttab_transfer  

Since no bug report on grant_table right now, so I think this is safe for
now

Also  p2m_tear_down è mem_sharing_unshare_page() , its flag is
MEM_SHARING_DESTROY_GFN, and won’t has the chance to

call set_shared_p2m_entry()

 

Of course, the p2m_teardown won''t call set_shared_p2m_entry. But this
does
not change my argument that p2m_teardown() hold p2m_lock to wait on
shr_lock. Actaully, after looking for a while, I rebut myself that the
scenario of deadlock won''t exist.
When p2m_teardown is called, the domain is dying in its last few steps
(device, irq are released), and there is no way for
hvm_hap_nested_page_fault() to happen on the memory of the dying domain. If
this case is eliminated, then my patch should not have deadlock problem. Any
comments?

3.       set_shared_p2m_entry() which call set_p2m_entry() is not in
p2m_lock, and I found in other code set_p2m_entry is called in p2m_lock,

so here I think it is a problem

 

So I think at least set_p2m_entry should be put into p2m_lock.

I’ll do more investigation base on this.

 


See this http://xenbits.xensource.com/xen-unstable.hg?rev/a8d69de8eb31

Thanks,
Jui-Hao




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim:
 
      Sorry for the inconvenience, I think it''s better now when I reply
directly from hotmail.
      
      It looks like when unshare(), page_set_owner is forgetten, right?
      Since after I add this code the Error log is disappeared.
 
 
     But unfortunately, I meet HVM blue screen(windows) with serial output
below.
     Full log attached.
 
(XEN) ==>3Debug for domain=1, gfn=7de15, Debug page: MFN=171c15 is
ci=8000000000000001, ti=0, owner_id=1
(XEN) ===hvm_hap_nested_page_fault mfn 171fd6 p2mt d gfn 515542 did 1
(XEN) ===mem_sharing_unshare_page cpu 12 mfn 171fd6 p2mt d gfn 515542 did 1
(XEN) ==>3Debug for domain=1, gfn=7ddd6, Debug page: MFN=171fd6 is
ci=8000000000000001, ti=0, owner_id=1
(XEN) ===hvm_hap_nested_page_fault mfn 171c57 p2mt d gfn 515671 did 1
(XEN) ===mem_sharing_unshare_page cpu 12 mfn 171c57 p2mt d gfn 515671 did 1
(XEN) ==>3Debug for domain=1, gfn=7de57, Debug page: MFN=171c57 is
ci=8000000000000001, ti=0, owner_id=1
(XEN) printk: 32 messages suppressed.
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)
(XEN) grant_table.c:555:d0 Iomem mapping not permitted ffffffffffffffff (domain
1)

 
> Date: Mon, 10 Jan 2011 10:30:41 +0000
> From: Tim.Deegan@citrix.com
> To: juihaochiang@gmail.com
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> Hi, 
> 
> Can you please (both of you) sort out yout mail clients to do proper
> indenting of quoted text? The plain-text versions don''t have any
> quote prefix, which makes them confusing to read.
> 
> At 04:57 +0000 on 10 Jan (1294635461), Jui-Hao Chiang wrote:
> > Hi, Tim:
> > 
> > On Sat, Jan 8, 2011 at 12:09 AM, Tim Deegan
<Tim.Deegan@citrix.com<mailto:Tim.Deegan@citrix.com>> wrote:
> > At 06:02 +0000 on 07 Jan (1294380120), Jui-Hao Chiang wrote:
> > > One of the solution is to
> > > (a) Simply replace shr_lock with p2m_lock.
> > 
> > I think this is the best choice. If we find that the p2m lock is a
> > bottleneck we can address it later.
> > 
> > 
> > Just to be skeptic.
> > Why doesn''t mfn_to_gfn() take p2m lock when querying the p2m
type?
> 
> Because gfn->mfn lookups happen very frequently and requiring the lock
> would be a performance bottleneck on multi-vcpu guests.
> 
> > Is there any quarantee that the resulting type is correct and
trustful?
> 
> Yes. It''s not perfect (and as I said I need to overhaul the
locking
> here) but if the p2m lookup only reads each level once and the p2m
> updates are careful about the order they change things in, the worst
> that can happen is another CPU sees a slightly out-of-date value.
> 
> There is at least one issue there (now that some p2m code frees old p2m
> pages there''s a potential race against other readers that needs a
> tlbflush-timestamp-style interlock), but TBH there are other things that
> need fixing first.
> 
> Tim.
> 
> > For example:
> > (1) User1 query the p2m type:
> > mfn_to_gfn(...&p2mt);
> > if (p2mt == p2m_ram_rw) /* do something based on the p2m type result?
*/
> > 
> > (2) User2 modify the p2m type
> > p2m_lock(p2m);
> > set_p2m_entry(..... p2m_ram_rw);
> > p2m_unlock(p2m);
> > 
> > Thanks,
> > Jui-Hao
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, all:

I disable Rich formatting in gmail, does the format looks good now?
To tinnycloud: hvm_hap_nested_page_fault() will be called when a guest
writes a shared page.

2011/1/11 MaoXiaoyun <tinnycloud@hotmail.com>
>
> Hi Tim:
>
>       Sorry for the inconvenience, I think it''s better now when I
reply directly from hotmail.
>
>       It looks like when unshare(), page_set_owner is forgetten, right?
Inside unshare(), it does the following
(1) page_make_private() --> page_set_owner(page, d) to update
page_info structure. So there is no need for you to add page_set_owner
explicitly.
(2) set_shared_p2m_entry() to update the page table
>From your log file, the page_make_private() is not called/executed
properly. Could you debug into it also?

Thanks,
Jui-Hao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Jui_hao:
 
    Page_make_private() --> page_set_owner(page, d)  sets page owner when
page_make_private success.
    But in fact, I remember, when the Error log shows up, page_make_private
failed, thus
    page_set_owner(page, d) is not called.    
 
> Date: Tue, 11 Jan 2011 14:32:51 +0800
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> From: juihaochiang@gmail.com
> To: tinnycloud@hotmail.com
> CC: tim.deegan@citrix.com; xen-devel@lists.xensource.com
> 
> Hi, all:
> 
> I disable Rich formatting in gmail, does the format looks good now?
> To tinnycloud: hvm_hap_nested_page_fault() will be called when a guest
> writes a shared page.
> 
> 2011/1/11 MaoXiaoyun <tinnycloud@hotmail.com>
> >
> > Hi Tim:
> >
> >       Sorry for the inconvenience, I think it''s better now
when I reply directly from hotmail.
> >
> >       It looks like when unshare(), page_set_owner is forgetten,
right?
> 
> Inside unshare(), it does the following
> (1) page_make_private() --> page_set_owner(page, d) to update
> page_info structure. So there is no need for you to add page_set_owner
> explicitly.
> (2) set_shared_p2m_entry() to update the page table
> From your log file, the page_make_private() is not called/executed
> properly. Could you debug into it also?
> 
> Thanks,
> Jui-Hao
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi:
 
      I''ve been trying to fix memory sharing on xen for a few days.
      Now it works on my box, I''d like to share the patches.
 
     Attached is the fix for code of tools in xen.(Sorry, I don''t know
how to make git patches.)
     Also you should apply the patch of
http://lists.xensource.com/archives/html/xen-devel/2010-12/msg00149.html
     
     Currently, I am able to start 24 HVMS(each of which memory=512,
maxmem=2048)
     After all VMS start, Xen info shows that without mem_sharing,  the free
memory is 7486
     and with mem_sharing, free is 9000, so 1.5G memory is saved.
 
    Remember, you should start the blktap/drivers/blktapctrl first, since it is
enable hash share between tapdisk process.
    also add memory_sharing into your HVM file to enable sharing.
 
    thanks. 
   
    
 
 
  		 	   		  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Tim:

On Mon, Jan 10, 2011 at 4:10 PM, Jui-Hao Chiang <juihaochiang@gmail.com>
wrote:
>>
>> After this change, unshare() has a potential problem of deadlock for
>> shr_lock and p2m_lock with different locking order.
>> Assume two CPUs do the following
>> CPU1: hvm_hap_nested_page_fault() => unshare() =>
p2m_change_type()
>> (locking order: shr_lock, p2m_lock)
>> CPU2: p2m_teardown() => unshare() (locking order: p2m_lock,
shr_lock)
>> When CPU1 grabs shr_lock and CPU2 grabs p2m_lock, they deadlock later.
>>
>>  1.       mem_sharing_unshare_page() has the routine  called from
>> gfn_to_mfn_unshare, which is called by gnttab_transfer
>>
>> Since no bug report on grant_table right now, so I think this is safe
for
>> now
>>
>> Also  p2m_tear_down è mem_sharing_unshare_page() , its flag is
>> MEM_SHARING_DESTROY_GFN, and won’t has the chance to
>>
>> call set_shared_p2m_entry()
>>
>>
>
> Of course, the p2m_teardown won''t call set_shared_p2m_entry. But
this does
> not change my argument that p2m_teardown() hold p2m_lock to wait on
> shr_lock. Actaully, after looking for a while, I rebut myself that the
> scenario of deadlock won''t exist.
> When p2m_teardown is called, the domain is dying in its last few steps
> (device, irq are released), and there is no way for
> hvm_hap_nested_page_fault() to happen on the memory of the dying domain. If
> this case is eliminated, then my patch should not have deadlock problem.
Any
> comments?
>
After a discussion with tinnycloud, his test is working after applying
the previous patch
http://lists.xensource.com/archives/html/xen-devel/2010-12/txteWc7Bs5Yap.txt
(set_shared_p2m_entry is not executed since it is in ASSERT).

And after a few code tracing and testing, my own worry about the
deadlock between p2m_lock and shr_lock actually disappears as the
above discussion. So here I re-attach the patch again which includes
another fix to recover type count when nominate fails on a page (from
our previous dicussions).

See if anything wrong.

Bests,
Jui-Hao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

At 10:03 +0000 on 12 Jan (1294826637), Jui-Hao Chiang
wrote:
> After a discussion with tinnycloud, his test is working after applying
> the previous patch
>
http://lists.xensource.com/archives/html/xen-devel/2010-12/txteWc7Bs5Yap.txt
> (set_shared_p2m_entry is not executed since it is in ASSERT).
Great!
> And after a few code tracing and testing, my own worry about the
> deadlock between p2m_lock and shr_lock actually disappears as the
> above discussion. So here I re-attach the patch again which includes
> another fix to recover type count when nominate fails on a page (from
> our previous dicussions).
The locking parts of this patch are already applied to the staging tree,
thanks.

The "recover type count" I still think is wrong.  The page-sharing
code
can''t rely on the type of a page with typecount 0.  Also in this patch
you''re changing the code of the core __put_page_type() function, rather
than the page-sharing code.

Can you try the attached patch instead?  If it fixes your problem I''ll
apply it. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jan 10, 2011 at 10:30 AM, Tim Deegan <Tim.Deegan@citrix.com>
wrote:
>> Just to be skeptic.
>> Why doesn''t mfn_to_gfn() take p2m lock when querying the p2m
type?
>
> Because gfn->mfn lookups happen very frequently and requiring the lock
> would be a performance bottleneck on multi-vcpu guests.
I think there''s also a deadlock issue.  At some point a few months ago
I made ept_get_entry() grab the p2m lock, and it deadlocked with the
paging lock.  IIRC, the problem was that
* Sometimes the paging lock is grabbed after the p2m lock is taken
* Sometimes gfn_to_mfn() is called when the paging lock is held

So adding the p2m lock to gfn_to_mfn() gave you a circular lock
dependency, classic condition for deadlock.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim:
 
      Seems not work. See log below.
      Since the patch is for xen_unstable, I need to merge the code manually, I
will check my code carefully later.
 
blktap_sysfs_create: adding attributes for dev ffff880102523c00
(XEN) Bad type in alloc_page_type 8000000000000000 t=8000000000000001
c=8000000000000005
(XEN) Xen BUG at mm.c:2094
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e008:[<ffff82c48015cd08>] __get_page_type+0x4d8/0x1410
(XEN) RFLAGS: 0000000000010286   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000092
(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021e8c4
(XEN) rbp: ffff82f603663e80   rsp: ffff83023ff27b08   r8:  0000000000000001
(XEN) r9:  0000000000000001   r10: 00000000ffffffed   r11: ffff82c4801318d0
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 8000000000000001
(XEN) r15: ffff83023ff27bc8   cr0: 0000000080050033   cr4: 00000000000026f0
(XEN) cr3: 0000000336800000   cr2: ffff88010545d008
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff83023ff27b08:
(XEN)    0000000000000000 00000000000000ec 000000110000000e ffff83023ff27c1c
(XEN)    0000000100000002 ffff83023ff27be8 0000000800000001 0000000000000001
(XEN)    00000000001b31f4 8000000000000000 0000008700000100 0000000000000086
(XEN)    00972cd100000000 ffff82c480376980 0000000000000000 ffff880002b63dc8
(XEN)    0000000000000000 4c00000000000002 000000000000ffff 80000003276b0127
(XEN)    0000000000000000 ffff88002c11a258 0000000000000000 ffff8801060b00a8
(XEN)    0000000000000000 0000000000000000 0000000000000009 ffff82f603663e80
(XEN)    ffff8301bd350000 ffff8301bd350014 0000000000000003 ffff82f603663e80
(XEN)    00000000000099f4 ffff82c48015dc5b ffff82f603663e80 ffff82c48015e98e
(XEN)    ffff8301bd350000 00000000001b31f4 ffff8301bd350000 ffff83023ff27cc0
(XEN)    0000000000000003 ffff82c4801c02aa 0000000000000000 0000000000000092
(XEN)    00000000bf568000 0000000000000096 000000013ff27f28 ffff83023ff27e38
(XEN)    ffff8301bd350000 ffff83023ff27e28 0000000000305000 0000000000000006
(XEN)    0000000000000006 ffff82c4801c06a3 0000000000000000 0000000000000000
(XEN)    00000000000099f4 00000000006ee000 00000000006ee000 ffff82c4801457fd
(XEN)    ffff82c4801447da 0000000000000080 ffff83023ff27f28 ffff82c48015a1d8
(XEN)    0000000000000000 00000000000000fc 0000000000000000 0000000000000001
(XEN)    ffff83023ff27e28 ffff82c48015a1d8 0000000000000000 ffff82f6066d0000
(XEN)    0000000000000000 0000000000000000 00000000003369d0 0000000000000000
(XEN)    0000000000000000 ffff82f6066d3a00 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82c48015cd08>] __get_page_type+0x4d8/0x1410
(XEN)    [<ffff82c48015dc5b>] get_page_type+0xb/0x20
(XEN)    [<ffff82c48015e98e>] page_make_sharable+0x4e/0x1a0
(XEN)    [<ffff82c4801c02aa>] mem_sharing_nominate_page+0x18a/0x380
(XEN)    [<ffff82c4801c06a3>] mem_sharing_domctl+0x73/0x130
(XEN)    [<ffff82c4801457fd>] arch_do_domctl+0xdad/0x1f90
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c48015a1d8>] get_page+0x28/0xf0
(XEN)    [<ffff82c48015a1d8>] get_page+0x28/0xf0
(XEN)    [<ffff82c48015dc5b>] get_page_type+0xb/0x20
(XEN)    [<ffff82c48015e1c3>] get_page_and_type_from_pagenr+0x93/0xf0
(XEN)    [<ffff82c480104373>] do_domctl+0x163/0x1000
(XEN)    [<ffff82c480162134>] do_mmuext_op+0xf34/0x1320
(XEN)    [<ffff82c48014717d>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 3:
(XEN) Xen BUG at mm.c:2094
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified)

 
> Date: Wed, 12 Jan 2011 10:54:05 +0000
> From: Tim.Deegan@citrix.com
> To: juihaochiang@gmail.com
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> Hi, 
> 
> At 10:03 +0000 on 12 Jan (1294826637), Jui-Hao Chiang wrote:
> > After a discussion with tinnycloud, his test is working after applying
> > the previous patch
> >
http://lists.xensource.com/archives/html/xen-devel/2010-12/txteWc7Bs5Yap.txt
> > (set_shared_p2m_entry is not executed since it is in ASSERT).
> 
> Great!
> 
> > And after a few code tracing and testing, my own worry about the
> > deadlock between p2m_lock and shr_lock actually disappears as the
> > above discussion. So here I re-attach the patch again which includes
> > another fix to recover type count when nominate fails on a page (from
> > our previous dicussions).
> 
> The locking parts of this patch are already applied to the staging tree,
> thanks.
> 
> The "recover type count" I still think is wrong. The page-sharing
code
> can''t rely on the type of a page with typecount 0. Also in this
patch
> you''re changing the code of the core __put_page_type() function,
rather
> than the page-sharing code.
> 
> Can you try the attached patch instead? If it fixes your problem
I''ll
> apply it. 
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 12:39 +0000 on 12 Jan (1294835994), MaoXiaoyun wrote:
> Hi Tim:
> 
>       Seems not work. See log below.
>       Since the patch is for xen_unstable, I need to merge the code
manually, I will check my code carefully later.
> 
I think you need this change as well:

diff -r d8eef6e395a8 xen/arch/x86/mm.c
--- a/xen/arch/x86/mm.c	Wed Jan 12 10:51:38 2011 +0000
+++ b/xen/arch/x86/mm.c	Wed Jan 12 14:01:11 2011 +0000
@@ -2367,7 +2367,7 @@ static int __get_page_type(struct page_i
 
                 /* No special validation needed for writable pages. */
                 /* Page tables and GDT/LDT need to be scanned for validity. */
-                if ( type == PGT_writable_page )
+                if ( type == PGT_writable_page || type == PGT_shared_page )
                     nx |= PGT_validated;
             }
         }


Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim:
 
        That''s it, I am running the test, so far so good, I''ll
test more, thanks.
 
      Currently from the code of tapdisk, it indicates only *read only* IO with
secs 8 has the
chance to be shared, so does it mean only the parent image can be shared, still
it needs to
be opened read only, right?
 
      So it looks like page sharing are refer to those pages contain disk data
been loaded
into Guest IO buffer, and this is the page sharing in Xen, right?      
             
 
> Date: Wed, 12 Jan 2011 14:02:23 +0000
> From: Tim.Deegan@citrix.com
> To: tinnycloud@hotmail.com
> CC: juihaochiang@gmail.com; xen-devel@lists.xensource.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> At 12:39 +0000 on 12 Jan (1294835994), MaoXiaoyun wrote:
> > Hi Tim:
> > 
> > Seems not work. See log below.
> > Since the patch is for xen_unstable, I need to merge the code
manually, I will check my code carefully later.
> > 
> 
> I think you need this change as well:
> 
> diff -r d8eef6e395a8 xen/arch/x86/mm.c
> --- a/xen/arch/x86/mm.c Wed Jan 12 10:51:38 2011 +0000
> +++ b/xen/arch/x86/mm.c Wed Jan 12 14:01:11 2011 +0000
> @@ -2367,7 +2367,7 @@ static int __get_page_type(struct page_i
> 
> /* No special validation needed for writable pages. */
> /* Page tables and GDT/LDT need to be scanned for validity. */
> - if ( type == PGT_writable_page )
> + if ( type == PGT_writable_page || type == PGT_shared_page )
> nx |= PGT_validated;
> }
> }
> 
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim:
 
      More strees test failed on lock issue. 
      thanks.
 
(XEN) Error: p2m lock held by p2m_change_type
(XEN) Xen BUG at p2m-ept.c:38
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    9
(XEN) RIP:    e008:[<ffff82c4801df45a>]
ept_pod_check_and_populate+0x13a/0x150
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff830402750000   rcx: 0000000000000092
(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021e8c4
(XEN) rbp: ffff83023fed7f28   rsp: ffff83023fed7c18   r8:  0000000000000001
(XEN) r9:  0000000000000001   r10: ffff830000000000   r11: ffff82c4801318d0
(XEN) r12: ffff83058dda63e0   r13: 0000000000000001   r14: 0000000000000009
(XEN) r15: 000000000000f9c7   cr0: 0000000080050033   cr4: 00000000000026f0
(XEN) cr3: 000000058ddfb000   cr2: 00000000e1258000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff83023fed7c18:
(XEN)    0000000000000002 0000000000000001 0000000000000009 ffff830402750000
(XEN)    ffff83023fed7c78 0000000000000001 ffff83023fed7c70 ffff82c4801df5cf
(XEN)    0000000000000000 ffff83023fed7cc4 000000000000f9c7 000000000000f9c7
(XEN)    ffff83058dda6000 ffff830402750000 ffff83023fed7f28 000000000000f9c7
(XEN)    0000000000000002 0000000000000001 0000000000000030 ffff82c4801bcba4
(XEN)    ffff83058dda6000 000000043fed7f28 ffff83023fed7f28 000000000000f9c7
(XEN)    000000000003f7c7 0000000000000030 ffff83023fed7f28 ffff82c48019baf1
(XEN)    0000000000000000 00000001a8e46000 0000000000000000 0000000000000182
(XEN)    ffff8300a8e46000 ffff82c4801b3864 ffff830402750470 07008300a8e46000
(XEN)    ffff83023fe810e0 0000000000000009 ffff83023fe814c0 0000000000000040
(XEN)    ffff82c4801447da 0000000000000080 ffff83023fe814c0 0000000000000000
(XEN)    000000000f9c7000 000000000000e000 ffff83023fed7dc8 0000000000000080
(XEN)    ffff82c480251dd0 000000000000f9c7 00ff8304027504b0 ffff82c480251dc0
(XEN)    ffff82c480251080 ffff82c480251dc0 0000000000000080 ffff82c48011b3cc
(XEN)    ffff82c480263240 0000000000000040 ffff82c4801447da 0000000000000080
(XEN)    ffff83023fed7f28 0000000000000092 00000c2d0a7ff1db 00000000000000fc
(XEN)    0000000000000092 0000000000000009 ffff8304027504b0 0000000000000009
(XEN)    ffff82c480263100 ffff82c480251100 ffff82c480251100 0000000000000292
(XEN)    ffff8300a8e477f0 0000069731554460 0000000000000292 ffff82c4801a93c3
(XEN)    00000000000000d1 ffff8300a8e46000 ffff8300a8e46000 ffff8300a8e477e8
(XEN) Xen call trace:
(XEN)    [<ffff82c4801df45a>] ept_pod_check_and_populate+0x13a/0x150
(XEN)    [<ffff82c4801df5cf>] ept_get_entry+0x15f/0x1c0
(XEN)    [<ffff82c4801bcba4>] p2m_change_type+0x144/0x1b0
(XEN)    [<ffff82c48019baf1>] hvm_hap_nested_page_fault+0x121/0x190
(XEN)    [<ffff82c4801b3864>] vmx_vmexit_handler+0x304/0x1a90
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c48011b3cc>] vcpu_runstate_get+0xec/0xf0
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c4801a93c3>] pt_update_irq+0x33/0x1e0
(XEN)    [<ffff82c4801a6082>] vlapic_has_pending_irq+0x42/0x70
(XEN)    [<ffff82c4801a0cc8>] hvm_vcpu_has_pending_irq+0x88/0xa0
(XEN)    [<ffff82c4801b267b>] vmx_vmenter_helper+0x5b/0x150
(XEN)    [<ffff82c4801adaa3>] vmx_asm_do_vmentry+0x0/0xdd
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 9:
 
> Date: Wed, 12 Jan 2011 14:02:23 +0000
> From: Tim.Deegan@citrix.com
> To: tinnycloud@hotmail.com
> CC: juihaochiang@gmail.com; xen-devel@lists.xensource.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> At 12:39 +0000 on 12 Jan (1294835994), MaoXiaoyun wrote:
> > Hi Tim:
> > 
> > Seems not work. See log below.
> > Since the patch is for xen_unstable, I need to merge the code
manually, I will check my code carefully later.
> > 
> 
> I think you need this change as well:
> 
> diff -r d8eef6e395a8 xen/arch/x86/mm.c
> --- a/xen/arch/x86/mm.c Wed Jan 12 10:51:38 2011 +0000
> +++ b/xen/arch/x86/mm.c Wed Jan 12 14:01:11 2011 +0000
> @@ -2367,7 +2367,7 @@ static int __get_page_type(struct page_i
> 
> /* No special validation needed for writable pages. */
> /* Page tables and GDT/LDT need to be scanned for validity. */
> - if ( type == PGT_writable_page )
> + if ( type == PGT_writable_page || type == PGT_shared_page )
> nx |= PGT_validated;
> }
> }
> 
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, all:

I think there is still a problem.
(1) I think using the get_page_and_type is definitely better since
it''s a function already implemented there
There seems a typo:
"if ( get_page_and_type(page, d, PGT_shared_page) )" should be changed
to "if ( !get_page_and_type(page, d, PGT_shared_page) )" because the
function return 1 on success.

(2) The major problem is the __put_page_type() never handle the
special case for shared pages.

If the (1) is changed as I said, the problem still exists as the following
/* Before nominating domain 1, gfn 0x63 */
(XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
ci=8000000000000002, ti=0, owner_id=1
/* After a failed nominate  [desired: ci=8000000000000002, ti=0]*/
(XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
ci=8000000000000002, ti=8400000000000000, owner_id=1


2011/1/12 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi Tim:
>
>         That''s it, I am running the test, so far so good,
I''ll test more,
> thanks.
>
>       Currently from the code of tapdisk, it indicates only *read only* IO
> with secs 8 has the
> chance to be shared, so does it mean only the parent image can be shared,
> still it needs to
> be opened read only, right?
>
>       So it looks like page sharing are refer to those pages contain disk
> data been loaded
> into Guest IO buffer, and this is the page sharing in Xen, right?
>
>
Bests,
Jui-Hao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Well, I think the discuss is around get_page/put_page,
get_page_type/put_page_type
 
Could someone help to explain their usage and difference?
 
thanks.
 
> Date: Thu, 13 Jan 2011 10:26:55 +0800
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> From: juihaochiang@gmail.com
> To: tinnycloud@hotmail.com
> CC: tim.deegan@citrix.com; xen-devel@lists.xensource.com
> 
> Hi, all:
> 
> I think there is still a problem.
> (1) I think using the get_page_and_type is definitely better since
> it''s a function already implemented there
> There seems a typo:
> "if ( get_page_and_type(page, d, PGT_shared_page) )" should be
changed
> to "if ( !get_page_and_type(page, d, PGT_shared_page) )" because
the
> function return 1 on success.
> 
> (2) The major problem is the __put_page_type() never handle the
> special case for shared pages.
> 
> If the (1) is changed as I said, the problem still exists as the following
> /* Before nominating domain 1, gfn 0x63 */
> (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
> ci=8000000000000002, ti=0, owner_id=1
> /* After a failed nominate [desired: ci=8000000000000002, ti=0]*/
> (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
> ci=8000000000000002, ti=8400000000000000, owner_id=1
> 
> 
> 2011/1/12 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi Tim:
> >
> >         That''s it, I am running the test, so far so good,
I''ll test more,
> > thanks.
> >
> >       Currently from the code of tapdisk, it indicates only *read
only* IO
> > with secs 8 has the
> > chance to be shared, so does it mean only the parent image can be
shared,
> > still it needs to
> > be opened read only, right?
> >
> >       So it looks like page sharing are refer to those pages contain
disk
> > data been loaded
> > into Guest IO buffer, and this is the page sharing in Xen, right?
> >
> >
> 
> Bests,
> Jui-Hao
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 02:26 +0000 on 13 Jan (1294885615), Jui-Hao Chiang
wrote:
> There seems a typo:
> "if ( get_page_and_type(page, d, PGT_shared_page) )" should be
changed
> to "if ( !get_page_and_type(page, d, PGT_shared_page) )"
Oops!  Yes, thanks for that. :)
> (2) The major problem is the __put_page_type() never handle the
> special case for shared pages.
> 
> If the (1) is changed as I said, the problem still exists as the following
> /* Before nominating domain 1, gfn 0x63 */
> (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
> ci=8000000000000002, ti=0, owner_id=1
> /* After a failed nominate  [desired: ci=8000000000000002, ti=0]*/
> (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
> ci=8000000000000002, ti=8400000000000000, owner_id=1
Is this causing a real problem other than this printout?

One of the reasons to use get_page_and_type/put_page_and_type was that
it gets rid of the code that requires pages to have PGT_none before
they''re shared. 

As I have been trying to explain, when a page has typecount 0 its type
is only relevant for the TLB flushing logic.  If there''s still a place
in the page-sharing code that relies on (type == PGT_none && count == 0)
then AFAICS that''s a bug. 

Cheers,

Tim.
> 2011/1/12 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi Tim:
> >
> >         That''s it, I am running the test, so far so good,
I''ll test more,
> > thanks.
> >
> >       Currently from the code of tapdisk, it indicates only *read
only* IO
> > with secs 8 has the
> > chance to be shared, so does it mean only the parent image can be
shared,
> > still it needs to
> > be opened read only, right?
> >
> >       So it looks like page sharing are refer to those pages contain
disk
> > data been loaded
> > into Guest IO buffer, and this is the page sharing in Xen, right?
> >
> >
> 
> Bests,
> Jui-Hao
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 04:42 +0000 on 13 Jan (1294893768), MaoXiaoyun wrote:
> Well, I think the discuss is around get_page/put_page,
get_page_type/put_page_type
> 
> Could someone help to explain their usage and difference?
The reference counting mechanism is described at the top of
xen/arch/x86/mm.c.  get_page() takes a "TOT_COUNT" reference;
get_page_type() takes a "TYPE_COUNT" reference; get_page_and_type()
takes both. 

Cheers,

Tim.
> > Date: Thu, 13 Jan 2011 10:26:55 +0800
> > Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> > From: juihaochiang@gmail.com
> > To: tinnycloud@hotmail.com
> > CC: tim.deegan@citrix.com; xen-devel@lists.xensource.com
> >
> > Hi, all:
> >
> > I think there is still a problem.
> > (1) I think using the get_page_and_type is definitely better since
> > it''s a function already implemented there
> > There seems a typo:
> > "if ( get_page_and_type(page, d, PGT_shared_page) )" should
be changed
> > to "if ( !get_page_and_type(page, d, PGT_shared_page) )"
because the
> > function return 1 on success.
> >
> > (2) The major problem is the __put_page_type() never handle the
> > special case for shared pages.
> >
> > If the (1) is changed as I said, the problem still exists as the
following
> > /* Before nominating domain 1, gfn 0x63 */
> > (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
> > ci=8000000000000002, ti=0, owner_id=1
> > /* After a failed nominate [desired: ci=8000000000000002, ti=0]*/
> > (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
> > ci=8000000000000002, ti=8400000000000000, owner_id=1
> >
> >
> > 2011/1/12 MaoXiaoyun <tinnycloud@hotmail.com>:
> > > Hi Tim:
> > >
> > >         That''s it, I am running the test, so far so
good, I''ll test more,
> > > thanks.
> > >
> > >       Currently from the code of tapdisk, it indicates only *read
only* IO
> > > with secs 8 has the
> > > chance to be shared, so does it mean only the parent image can be
shared,
> > > still it needs to
> > > be opened read only, right?
> > >
> > >       So it looks like page sharing are refer to those pages
contain disk
> > > data been loaded
> > > into Guest IO buffer, and this is the page sharing in Xen, right?
> > >
> > >
> >
> > Bests,
> > Jui-Hao
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 01:48 +0000 on 13 Jan (1294883302), MaoXiaoyun wrote:
> Hi Tim:
> 
>       More strees test failed on lock issue.
Here''s the fix for that.  It''s a surprisingly old bug in the
populate-on-demand code for EPT.  I''ll check it in as soon as
we''ve
managed to tag 4.1.0 RC1

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Tim:
>> (2) The major problem is the __put_page_type() never handle the
>> special case for shared pages.
>>
>> If the (1) is changed as I said, the problem still exists as the
following
>> /* Before nominating domain 1, gfn 0x63 */
>> (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
>> ci=8000000000000002, ti=0, owner_id=1
>> /* After a failed nominate  [desired: ci=8000000000000002, ti=0]*/
>> (XEN) Debug for domain=1, gfn=63, Debug page: MFN=4836c7 is
>> ci=8000000000000002, ti=8400000000000000, owner_id=1
>
> Is this causing a real problem other than this printout?
>
> One of the reasons to use get_page_and_type/put_page_and_type was that
> it gets rid of the code that requires pages to have PGT_none before
> they''re shared.
>
> As I have been trying to explain, when a page has typecount 0 its type
> is only relevant for the TLB flushing logic.  If there''s still a
place
> in the page-sharing code that relies on (type == PGT_none && count
== 0)
> then AFAICS that''s a bug.
>
Thanks for the clarification. As you said, the following is excerpted
from the mm.c
"* So, type_count is a count of the number of times a frame is being
 * referred to in its current incarnation. Therefore, a page can only
 * change its type when its type count is zero."

After testing the code with your patch, it''s ok for the mem_sharing.
And as the argument says, when (type_count & PGT_count_mask) is zero,
it''s ok for changing the page type. (even when there is a old value in
type_count & PGT_type_mask, e.g., ti=8400000000000000)

Thanks,
Jui-Hao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 15:24 +0000 on 13 Jan (1294932299), Jui-Hao Chiang
wrote:
> After testing the code with your patch, it''s ok for the
mem_sharing.
> And as the argument says, when (type_count & PGT_count_mask) is zero,
> it''s ok for changing the page type. (even when there is a old
value in
> type_count & PGT_type_mask, e.g., ti=8400000000000000)
Great, thanks.  I''ve applied that change as cset 22745:32b7a4f2d399
of xen-unstable and the EPT locking fix as 22744:b01ef59c8c80 .
They''re in the staging tree and will hit the public tree the next time
the automatic regression tests pass. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim:
 
     Thanks for the patch, xen panic on more stressed test. ( 12HVMS, each of
them reboot every 30minutes).
     Please refer to below log.
 
blktap_sysfs_create: adding attributes for dev ffff8801044bc400
blktap_sysfs_create: adding attributes for dev ffff8801044bc200
__ratelimit: 4 callbacks suppressed
(XEN) Xen BUG at mem_sharing.c:454
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000000000001   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: 000000000000005f   rdi: 000000000000005f
(XEN) rbp: ffff8305894f0fc0   rsp: ffff82c48035fc48   r8:  ffff82f600000000
(XEN) r9:  00007fffcdbd0fb8   r10: ffff82c4801f8c70   r11: 0000000000000282
(XEN) r12: ffff82c48035fe28   r13: ffff8303192a3bf0   r14: ffff82f60b966700
(XEN) r15: 0000000000000006   cr0: 0000000080050033   cr4: 00000000000026f0
(XEN) cr3: 000000032ea58000   cr2: ffff880119c2e668
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48035fc48:
(XEN)    00000000fffffff7 ffff82c4801bf8c0 0000000000553b86 ffff8305894f0fc0
(XEN)    ffff8302f4d12cf0 0000000000553b86 ffff82f603e28580 ffff82c48035fe38
(XEN)    ffff83023fe60000 ffff82c48035fe28 0000000000305000 0000000000000006
(XEN)    0000000000000006 ffff82c4801c0724 ffff82c4801447da 0000000000553b86
(XEN)    000000000001a938 00000000006ee000 00000000006ee000 ffff82c4801457fd
(XEN)    0000000000000096 0000000000000001 ffff82c48035fd30 0000000000000080
(XEN)    ffff82c480376980 ffff82c480251080 0000000000000292 ffff82c48011c519
(XEN)    ffff82c48035fe28 0000000000000080 0000000000000000 ffff8302ef312fa0
(XEN)    ffff8300b4aee000 ffff82c48025f080 ffff82c480251080 ffff82c480118351
(XEN)    0000000000000080 0000000000000000 ffff8300b4aef708 00000de9e9529c40
(XEN)    ffff8300b4aee000 0000000000000292 ffff8305cf9f09b8 0000000000000001
(XEN)    0000000000000001 0000000000000000 00000000002159e6 fffffffffffffff3
(XEN)    00000000006ee000 ffff82c48035fe28 0000000000305000 0000000000000006
(XEN)    0000000000000006 ffff82c480104373 ffff8305cf9f09c0 ffff82c4801a0b63
(XEN)    00000000159e6070 ffff8305cf9f0000 0000000000000007 ffff83023fe60180
(XEN)    0000000600000039 0000000000000000 00007fae14b30003 000000000054fdad
(XEN)    0000000000553b86 ffffffffff600429 000000004d2f26e8 0000000000088742
(XEN)    0000000000000000 00007fae14b30070 00007fae14b30000 00007fffcdbd0f50
(XEN)    00007fae14b30078 0000000000430e98 00007fffcdbd0fb8 0000000000cd39c8
(XEN)    0005aeb700000007 00007fae15bd2ab0 0000000000000000 0000000000000246
(XEN) Xen call trace:
(XEN)    [<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
(XEN)    [<ffff82c4801bf8c0>] mem_sharing_share_pages+0x170/0x310
(XEN)    [<ffff82c4801c0724>] mem_sharing_domctl+0xe4/0x130
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c4801457fd>] arch_do_domctl+0xdad/0x1f90
(XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
(XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
(XEN)    [<ffff82c480104373>] do_domctl+0x163/0x1000
(XEN)    [<ffff82c4801a0b63>] hvm_set_callback_irq_level+0xe3/0x110
(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at mem_sharing.c:454
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified)
 
 
 

 
> Date: Thu, 13 Jan 2011 15:53:44 +0000
> From: Tim.Deegan@citrix.com
> To: juihaochiang@gmail.com
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> At 15:24 +0000 on 13 Jan (1294932299), Jui-Hao Chiang wrote:
> > After testing the code with your patch, it''s ok for the
mem_sharing.
> > And as the argument says, when (type_count & PGT_count_mask) is
zero,
> > it''s ok for changing the page type. (even when there is a old
value in
> > type_count & PGT_type_mask, e.g., ti=8400000000000000)
> 
> Great, thanks. I''ve applied that change as cset 22745:32b7a4f2d399
> of xen-unstable and the EPT locking fix as 22744:b01ef59c8c80 .
> They''re in the staging tree and will hit the public tree the next
time
> the automatic regression tests pass. 
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim:
 
     Thanks for the patch, xen panic on more stressed test. ( 12HVMS, each of
them reboot every 30minutes).
     Please refer to below log.
 
blktap_sysfs_create: adding attributes for dev ffff8801044bc400
blktap_sysfs_create: adding attributes for dev ffff8801044bc200
__ratelimit: 4 callbacks suppressed
(XEN) Xen BUG at mem_sharing.c:454
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000000000001   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: 000000000000005f   rdi: 000000000000005f
(XEN) rbp: ffff8305894f0fc0   rsp: ffff82c48035fc48   r8:  ffff82f600000000
(XEN) r9:  00007fffcdbd0fb8   r10: ffff82c4801f8c70   r11: 0000000000000282
(XEN) r12: ffff82c48035fe28   r13: ffff8303192a3bf0   r14: ffff82f60b966700
(XEN) r15: 0000000000000006   cr0: 0000000080050033   cr4: 00000000000026f0
(XEN) cr3: 000000032ea58000   cr2: ffff880119c2e668
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48035fc48:
(XEN)    00000000fffffff7 ffff82c4801bf8c0 0000000000553b86 ffff8305894f0fc0
(XEN)    ffff8302f4d12cf0 0000000000553b86 ffff82f603e28580 ffff82c48035fe38
(XEN)    ffff83023fe60000 ffff82c48035fe28 0000000000305000 0000000000000006
(XEN)    0000000000000006 ffff82c4801c0724 ffff82c4801447da 0000000000553b86
(XEN)    000000000001a938 00000000006ee000 00000000006ee000 ffff82c4801457fd
(XEN)    0000000000000096 0000000000000001 ffff82c48035fd30 0000000000000080
(XEN)    ffff82c480376980 ffff82c480251080 0000000000000292 ffff82c48011c519
(XEN)    ffff82c48035fe28 0000000000000080 0000000000000000 ffff8302ef312fa0
(XEN)    ffff8300b4aee000 ffff82c48025f080 ffff82c480251080 ffff82c480118351
(XEN)    0000000000000080 0000000000000000 ffff8300b4aef708 00000de9e9529c40
(XEN)    ffff8300b4aee000 0000000000000292 ffff8305cf9f09b8 0000000000000001
(XEN)    0000000000000001 0000000000000000 00000000002159e6 fffffffffffffff3
(XEN)    00000000006ee000 ffff82c48035fe28 0000000000305000 0000000000000006
(XEN)    0000000000000006 ffff82c480104373 ffff8305cf9f09c0 ffff82c4801a0b63
(XEN)    00000000159e6070 ffff8305cf9f0000 0000000000000007 ffff83023fe60180
(XEN)    0000000600000039 0000000000000000 00007fae14b30003 000000000054fdad
(XEN)    0000000000553b86 ffffffffff600429 000000004d2f26e8 0000000000088742
(XEN)    0000000000000000 00007fae14b30070 00007fae14b30000 00007fffcdbd0f50
(XEN)    00007fae14b30078 0000000000430e98 00007fffcdbd0fb8 0000000000cd39c8
(XEN)    0005aeb700000007 00007fae15bd2ab0 0000000000000000 0000000000000246
(XEN) Xen call trace:
(XEN)    [<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
(XEN)    [<ffff82c4801bf8c0>] mem_sharing_share_pages+0x170/0x310
(XEN)    [<ffff82c4801c0724>] mem_sharing_domctl+0xe4/0x130
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c4801457fd>] arch_do_domctl+0xdad/0x1f90
(XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
(XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
(XEN)    [<ffff82c480104373>] do_domctl+0x163/0x1000
(XEN)    [<ffff82c4801a0b63>] hvm_set_callback_irq_level+0xe3/0x110
(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at mem_sharing.c:454
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified)
 
 
 

 
> Date: Thu, 13 Jan 2011 15:53:44 +0000
> From: Tim.Deegan@citrix.com
> To: juihaochiang@gmail.com
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> At 15:24 +0000 on 13 Jan (1294932299), Jui-Hao Chiang wrote:
> > After testing the code with your patch, it''s ok for the
mem_sharing.
> > And as the argument says, when (type_count & PGT_count_mask) is
zero,
> > it''s ok for changing the page type. (even when there is a old
value in
> > type_count & PGT_type_mask, e.g., ti=8400000000000000)
> 
> Great, thanks. I''ve applied that change as cset 22745:32b7a4f2d399
> of xen-unstable and the EPT locking fix as 22744:b01ef59c8c80 .
> They''re in the staging tree and will hit the public tree the next
time
> the automatic regression tests pass. 
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, all:

Is that possible that the domain is dying?
In mem_sharing_gfn_account(): could you try the following?

d = get_domain_by_id(gfn->domain);
if (!d) printk("null domain %x\n", gfn->domain); /* add this line
to
see which domain id you see */
BUG_ON(!d);

When this domain id printed out, could you check if the printed domain
id is dying?
If the domain is dying, then the question seems to be:
"Given a domain id from the gfn_info, how do we know the domain is
dying? or we have stored a wrong information inside the hash list?"

2011/1/14 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi Tim:
>
>      Thanks for the patch, xen panic on more stressed test. ( 12HVMS, each
> of them reboot every 30minutes).
>      Please refer to below log.
>
> blktap_sysfs_create: adding attributes for dev ffff8801044bc400
> blktap_sysfs_create: adding attributes for dev ffff8801044bc200
> __ratelimit: 4 callbacks suppressed
> (XEN) Xen BUG at mem_sharing.c:454
> (XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e008:[<ffff82c4801bf52c>]
mem_sharing_gfn_account+0x5c/0x70
> (XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
> (XEN) rax: 0000000000000000   rbx: 0000000000000001   rcx: 0000000000000000
> (XEN) rdx: 0000000000000000   rsi: 000000000000005f   rdi: 000000000000005f
> (XEN) rbp: ffff8305894f0fc0   rsp: ffff82c48035fc48   r8:  ffff82f600000000
> (XEN) r9:  00007fffcdbd0fb8   r10: ffff82c4801f8c70   r11: 0000000000000282
> (XEN) r12: ffff82c48035fe28   r13: ffff8303192a3bf0   r14: ffff82f60b966700
> (XEN) r15: 0000000000000006   cr0: 0000000080050033   cr4: 00000000000026f0
> (XEN) cr3: 000000032ea58000   cr2: ffff880119c2e668
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=ffff82c48035fc48:
> (XEN)    00000000fffffff7 ffff82c4801bf8c0 0000000000553b86
ffff8305894f0fc0
> (XEN)    ffff8302f4d12cf0 0000000000553b86 ffff82f603e28580
ffff82c48035fe38
> (XEN)    ffff83023fe60000 ffff82c48035fe28 0000000000305000
0000000000000006
> (XEN)    0000000000000006 ffff82c4801c0724 ffff82c4801447da
0000000000553b86
> (XEN)    000000000001a938 00000000006ee000 00000000006ee000
ffff82c4801457fd
> (XEN)    0000000000000096 0000000000000001 ffff82c48035fd30
0000000000000080
> (XEN)    ffff82c480376980 ffff82c480251080 0000000000000292
ffff82c48011c519
> (XEN)    ffff82c48035fe28 0000000000000080 0000000000000000
ffff8302ef312fa0
> (XEN)    ffff8300b4aee000 ffff82c48025f080 ffff82c480251080
ffff82c480118351
> (XEN)    0000000000000080 0000000000000000 ffff8300b4aef708
00000de9e9529c40
> (XEN)    ffff8300b4aee000 0000000000000292 ffff8305cf9f09b8
0000000000000001
> (XEN)    0000000000000001 0000000000000000 00000000002159e6
fffffffffffffff3
> (XEN)    00000000006ee000 ffff82c48035fe28 0000000000305000
0000000000000006
> (XEN)    0000000000000006 ffff82c480104373 ffff8305cf9f09c0
ffff82c4801a0b63
> (XEN)    00000000159e6070 ffff8305cf9f0000 0000000000000007
ffff83023fe60180
> (XEN)    0000000600000039 0000000000000000 00007fae14b30003
000000000054fdad
> (XEN)    0000000000553b86 ffffffffff600429 000000004d2f26e8
0000000000088742
> (XEN)    0000000000000000 00007fae14b30070 00007fae14b30000
00007fffcdbd0f50
> (XEN)    00007fae14b30078 0000000000430e98 00007fffcdbd0fb8
0000000000cd39c8
> (XEN)    0005aeb700000007 00007fae15bd2ab0 0000000000000000
0000000000000246
> (XEN) Xen call trace:
> (XEN)    [<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
> (XEN)    [<ffff82c4801bf8c0>] mem_sharing_share_pages+0x170/0x310
> (XEN)    [<ffff82c4801c0724>] mem_sharing_domctl+0xe4/0x130
> (XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> (XEN)    [<ffff82c4801457fd>] arch_do_domctl+0xdad/0x1f90
> (XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
> (XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
> (XEN)    [<ffff82c480104373>] do_domctl+0x163/0x1000
> (XEN)    [<ffff82c4801a0b63>] hvm_set_callback_irq_level+0xe3/0x110
> (XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Xen BUG at mem_sharing.c:454
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required (''noreboot'' specified)
>
Bests,
Jui-Hao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Jui-Hao:
 
    Domain ID is 4.
   
   Well,
domain_destroy()->complete_domain_destroy->arch_domain_destroy->
paging_final_teardown()->hap_final_teardown->p2m_teardown->mem_sharing_unshare_page
 
   so it looks like it is possible that domain is destroyed before the handle is
removed for hash table.
 
  Further more, I add below code.
637     if(mem_sharing_gfn_account(gfn_get_info(&ce->gfns), 1) == -1){
638         printk("=====client not found, server %d client %d\n",
gfn_get_info(&se->gfns)->domain,
gfn_get_info(&ce->gfns)->domain);
639         ret = XEN_DOMCTL_MEM_SHARING_C_HANDLE_INVALID;
640         goto err_out;
641     }
642 
643     if(mem_sharing_gfn_account(gfn_get_info(&se->gfns), 1)==-1){
644         printk("=====server not found, server %d client %d\n",
gfn_get_info(&se->gfns)->domain,
gfn_get_info(&ce->gfns)->domain);
645         ret = XEN_DOMCTL_MEM_SHARING_C_HANDLE_INVALID;
646         goto err_out;
647     }
 
     those logs are printed out in test, when all domains are destroyed, I print
out all hash entry, it is empty,
     so it is correct.
      
     what''s your opinion?
 
> Date: Sat, 15 Jan 2011 01:00:27 +0800
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> From: juihaochiang@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> 
> Hi, all:
> 
> Is that possible that the domain is dying?
> In mem_sharing_gfn_account(): could you try the following?
> 
> d = get_domain_by_id(gfn->domain);
> if (!d) printk("null domain %x\n", gfn->domain); /* add this
line to
> see which domain id you see */
> BUG_ON(!d);
> 
> When this domain id printed out, could you check if the printed domain
> id is dying?
> If the domain is dying, then the question seems to be:
> "Given a domain id from the gfn_info, how do we know the domain is
> dying? or we have stored a wrong information inside the hash list?"
> 
> 2011/1/14 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi Tim:
> >
> >      Thanks for the patch, xen panic on more stressed test. ( 12HVMS,
each
> > of them reboot every 30minutes).
> >      Please refer to below log.
> >
> > blktap_sysfs_create: adding attributes for dev ffff8801044bc400
> > blktap_sysfs_create: adding attributes for dev ffff8801044bc200
> > __ratelimit: 4 callbacks suppressed
> > (XEN) Xen BUG at mem_sharing.c:454
> > (XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
> > (XEN) CPU:    0
> > (XEN) RIP:    e008:[<ffff82c4801bf52c>]
mem_sharing_gfn_account+0x5c/0x70
> > (XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
> > (XEN) rax: 0000000000000000   rbx: 0000000000000001   rcx:
0000000000000000
> > (XEN) rdx: 0000000000000000   rsi: 000000000000005f   rdi:
000000000000005f
> > (XEN) rbp: ffff8305894f0fc0   rsp: ffff82c48035fc48   r8: 
ffff82f600000000
> > (XEN) r9:  00007fffcdbd0fb8   r10: ffff82c4801f8c70   r11:
0000000000000282
> > (XEN) r12: ffff82c48035fe28   r13: ffff8303192a3bf0   r14:
ffff82f60b966700
> > (XEN) r15: 0000000000000006   cr0: 0000000080050033   cr4:
00000000000026f0
> > (XEN) cr3: 000000032ea58000   cr2: ffff880119c2e668
> > (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> > (XEN) Xen stack trace from rsp=ffff82c48035fc48:
> > (XEN)    00000000fffffff7 ffff82c4801bf8c0 0000000000553b86
ffff8305894f0fc0
> > (XEN)    ffff8302f4d12cf0 0000000000553b86 ffff82f603e28580
ffff82c48035fe38
> > (XEN)    ffff83023fe60000 ffff82c48035fe28 0000000000305000
0000000000000006
> > (XEN)    0000000000000006 ffff82c4801c0724 ffff82c4801447da
0000000000553b86
> > (XEN)    000000000001a938 00000000006ee000 00000000006ee000
ffff82c4801457fd
> > (XEN)    0000000000000096 0000000000000001 ffff82c48035fd30
0000000000000080
> > (XEN)    ffff82c480376980 ffff82c480251080 0000000000000292
ffff82c48011c519
> > (XEN)    ffff82c48035fe28 0000000000000080 0000000000000000
ffff8302ef312fa0
> > (XEN)    ffff8300b4aee000 ffff82c48025f080 ffff82c480251080
ffff82c480118351
> > (XEN)    0000000000000080 0000000000000000 ffff8300b4aef708
00000de9e9529c40
> > (XEN)    ffff8300b4aee000 0000000000000292 ffff8305cf9f09b8
0000000000000001
> > (XEN)    0000000000000001 0000000000000000 00000000002159e6
fffffffffffffff3
> > (XEN)    00000000006ee000 ffff82c48035fe28 0000000000305000
0000000000000006
> > (XEN)    0000000000000006 ffff82c480104373 ffff8305cf9f09c0
ffff82c4801a0b63
> > (XEN)    00000000159e6070 ffff8305cf9f0000 0000000000000007
ffff83023fe60180
> > (XEN)    0000000600000039 0000000000000000 00007fae14b30003
000000000054fdad
> > (XEN)    0000000000553b86 ffffffffff600429 000000004d2f26e8
0000000000088742
> > (XEN)    0000000000000000 00007fae14b30070 00007fae14b30000
00007fffcdbd0f50
> > (XEN)    00007fae14b30078 0000000000430e98 00007fffcdbd0fb8
0000000000cd39c8
> > (XEN)    0005aeb700000007 00007fae15bd2ab0 0000000000000000
0000000000000246
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
> > (XEN)    [<ffff82c4801bf8c0>]
mem_sharing_share_pages+0x170/0x310
> > (XEN)    [<ffff82c4801c0724>] mem_sharing_domctl+0xe4/0x130
> > (XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> > (XEN)    [<ffff82c4801457fd>] arch_do_domctl+0xdad/0x1f90
> > (XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
> > (XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
> > (XEN)    [<ffff82c480104373>] do_domctl+0x163/0x1000
> > (XEN)    [<ffff82c4801a0b63>]
hvm_set_callback_irq_level+0xe3/0x110
> > (XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
> > (XEN)
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Xen BUG at mem_sharing.c:454
> > (XEN) ****************************************
> > (XEN)
> > (XEN) Manual reset required (''noreboot'' specified)
> >
> 
> Bests,
> Jui-Hao
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Another failure on BUG() in mem_sharing_alloc_page()
 
 memset(&req, 0, sizeof(req));
 if(must_succeed) 
    {
        /* We do not support ''must_succeed'' any more. External
operations such
         * as grant table mappings may fail with OOM condition! 
         */
        BUG();===================>bug here
    }
    else
    {
        /* All foreign attempts to unshare pages should be handled through
         * ''must_succeed'' case. */
        ASSERT(v->domain->domain_id == d->domain_id);
        vcpu_pause_nosync(v);
        req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
    }
        
log below:

(XEN) Xen BUG at mem_sharing.c:347
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c4801c0344>]
mem_sharing_unshare_page+0x5d4/0x6e0
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff83030913eb78   rcx: 0000000000000000
(XEN) rdx: ffff82f6094d78e0   rsi: 0000000000000001   rdi: ffff82c48035f650
(XEN) rbp: ffff83048d950000   rsp: ffff82c48035f5e8   r8:  0000000000000000
(XEN) r9:  ffffffffffffffff   r10: 0000000000000001   r11: 0000000000000002
(XEN) r12: 000000000000ecbf   r13: ffff82c48035f628   r14: ffff83033f4fd760
(XEN) r15: ffff82f6025540c0   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000339044000   cr2: ffff8801589fac78
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48035f5e8:
(XEN)    ffff880159dcf000 ffff83033f4fd770 0000000000000000 ffff83030913eb60
(XEN)    0000000000235012 ffff8300bf554000 0000000100000009 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000d000000bf ffff83059b097000 000000000000ecbf
(XEN)    000000000012aa06 ffff82c48035f724 ffff8304f20d2a00 ffff8800b742bbc0
(XEN)    ffff83048d950000 ffff82c48010bfa9 ffff8304fca169b0 000000000053ea75
(XEN)    0000000000000000 ffff8304fca169b0 0000000000000000 ffff82c48035f6f8
(XEN)    0000000000000001 ffff8304fca169b0 ffff83023fe60000 ffff8304fca169b0
(XEN)    000002fd8035ff28 ffff8800b742bbc0 ffff880159dcf000 00003d3600000002
(XEN)    ffffffffffff003c 000000048cb88000 ffff82f60a7d4ea0 0000000d8010a650
(XEN)    0000000000000100 ffff83023fe60000 ffff83057cdbd8c0 ffffffffffffffea
(XEN)    ffff8800b742bb70 0000000000000000 ffff8800b742bbf0 ffff8800b742bbc0
(XEN)    0000000000000001 ffff82c48010da9b 0000000000000202 ffff82c48035fec8
(XEN)    ffff82c48035f7c8 00000000801880af ffff83023fe60010 0000000000000000
(XEN)    ffff82c400000001 ffff82c48035ff28 0000000000000008 ffff8800b742bbc0
(XEN)    ffff880159dcf000 0000000000000000 0000000000000000 00020000000002fd
(XEN)    000000000053ea75 ffff83021bdc47e8 ffff830310a20000 00000003370af718
(XEN)    0000000000000000 0000000000000000 001a000000000096 000000000053ea75
(XEN)    ffff83023fe514b0 ffff830310a20000 ffff880159d51000 0000000000000000
(XEN)    0000000000000000 00020000000005b6 0000000000509bb5 ffff83031884fdb0
(XEN) Xen call trace:
(XEN)    [<ffff82c4801c0344>] mem_sharing_unshare_page+0x5d4/0x6e0
(XEN)    [<ffff82c48010bfa9>] gnttab_map_grant_ref+0xbf9/0xe30
(XEN)    [<ffff82c48010da9b>] do_grant_table_op+0x14b/0x1080
(XEN)    [<ffff82c48015dc6b>] get_page_type+0xb/0x20
(XEN)    [<ffff82c48015df6d>] get_page_from_l1e+0x2ed/0x4c0
(XEN)    [<ffff82c480159ce9>] is_iomem_page+0x9/0x70
(XEN)    [<ffff82c48015b999>] put_page_from_l1e+0x59/0x150
(XEN)    [<ffff82c48015f6c5>] ptwr_emulated_update+0x2e5/0x420
(XEN)    [<ffff82c48015f946>] ptwr_emulated_write+0x86/0x90
(XEN)    [<ffff82c480159cb5>] ptwr_emulated_read+0x15/0x40
(XEN)    [<ffff82c48017511d>] x86_emulate+0x7ad/0x115d0
(XEN)    [<ffff82c48010fb44>] do_xen_version+0xb4/0x480
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
(XEN)    [<ffff82c4801872fc>] reprogram_hpet_evt_channel+0x8c/0x110
(XEN)    [<ffff82c4801880af>] handle_hpet_broadcast+0x16f/0x1d0
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c480187622>] hpet_legacy_irq_tick+0x42/0x50
(XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
(XEN)    [<ffff82c4801478e2>] update_runstate_area+0x102/0x110
(XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c48015a1d8>] get_page+0x28/0xf0
(XEN)    [<ffff82c48015ed72>] do_update_descriptor+0x1d2/0x210
(XEN)    [<ffff82c480113d7e>] do_multicall+0x14e/0x340
(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at mem_sharing.c:347
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified)
 
> Date: Sat, 15 Jan 2011 01:00:27 +0800
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> From: juihaochiang@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> 
> Hi, all:
> 
> Is that possible that the domain is dying?
> In mem_sharing_gfn_account(): could you try the following?
> 
> d = get_domain_by_id(gfn->domain);
> if (!d) printk("null domain %x\n", gfn->domain); /* add this
line to
> see which domain id you see */
> BUG_ON(!d);
> 
> When this domain id printed out, could you check if the printed domain
> id is dying?
> If the domain is dying, then the question seems to be:
> "Given a domain id from the gfn_info, how do we know the domain is
> dying? or we have stored a wrong information inside the hash list?"
> 
> 2011/1/14 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi Tim:
> >
> >      Thanks for the patch, xen panic on more stressed test. ( 12HVMS,
each
> > of them reboot every 30minutes).
> >      Please refer to below log.
> >
> > blktap_sysfs_create: adding attributes for dev ffff8801044bc400
> > blktap_sysfs_create: adding attributes for dev ffff8801044bc200
> > __ratelimit: 4 callbacks suppressed
> > (XEN) Xen BUG at mem_sharing.c:454
> > (XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
> > (XEN) CPU:    0
> > (XEN) RIP:    e008:[<ffff82c4801bf52c>]
mem_sharing_gfn_account+0x5c/0x70
> > (XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
> > (XEN) rax: 0000000000000000   rbx: 0000000000000001   rcx:
0000000000000000
> > (XEN) rdx: 0000000000000000   rsi: 000000000000005f   rdi:
000000000000005f
> > (XEN) rbp: ffff8305894f0fc0   rsp: ffff82c48035fc48   r8: 
ffff82f600000000
> > (XEN) r9:  00007fffcdbd0fb8   r10: ffff82c4801f8c70   r11:
0000000000000282
> > (XEN) r12: ffff82c48035fe28   r13: ffff8303192a3bf0   r14:
ffff82f60b966700
> > (XEN) r15: 0000000000000006   cr0: 0000000080050033   cr4:
00000000000026f0
> > (XEN) cr3: 000000032ea58000   cr2: ffff880119c2e668
> > (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> > (XEN) Xen stack trace from rsp=ffff82c48035fc48:
> > (XEN)    00000000fffffff7 ffff82c4801bf8c0 0000000000553b86
ffff8305894f0fc0
> > (XEN)    ffff8302f4d12cf0 0000000000553b86 ffff82f603e28580
ffff82c48035fe38
> > (XEN)    ffff83023fe60000 ffff82c48035fe28 0000000000305000
0000000000000006
> > (XEN)    0000000000000006 ffff82c4801c0724 ffff82c4801447da
0000000000553b86
> > (XEN)    000000000001a938 00000000006ee000 00000000006ee000
ffff82c4801457fd
> > (XEN)    0000000000000096 0000000000000001 ffff82c48035fd30
0000000000000080
> > (XEN)    ffff82c480376980 ffff82c480251080 0000000000000292
ffff82c48011c519
> > (XEN)    ffff82c48035fe28 0000000000000080 0000000000000000
ffff8302ef312fa0
> > (XEN)    ffff8300b4aee000 ffff82c48025f080 ffff82c480251080
ffff82c480118351
> > (XEN)    0000000000000080 0000000000000000 ffff8300b4aef708
00000de9e9529c40
> > (XEN)    ffff8300b4aee000 0000000000000292 ffff8305cf9f09b8
0000000000000001
> > (XEN)    0000000000000001 0000000000000000 00000000002159e6
fffffffffffffff3
> > (XEN)    00000000006ee000 ffff82c48035fe28 0000000000305000
0000000000000006
> > (XEN)    0000000000000006 ffff82c480104373 ffff8305cf9f09c0
ffff82c4801a0b63
> > (XEN)    00000000159e6070 ffff8305cf9f0000 0000000000000007
ffff83023fe60180
> > (XEN)    0000000600000039 0000000000000000 00007fae14b30003
000000000054fdad
> > (XEN)    0000000000553b86 ffffffffff600429 000000004d2f26e8
0000000000088742
> > (XEN)    0000000000000000 00007fae14b30070 00007fae14b30000
00007fffcdbd0f50
> > (XEN)    00007fae14b30078 0000000000430e98 00007fffcdbd0fb8
0000000000cd39c8
> > (XEN)    0005aeb700000007 00007fae15bd2ab0 0000000000000000
0000000000000246
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82c4801bf52c>] mem_sharing_gfn_account+0x5c/0x70
> > (XEN)    [<ffff82c4801bf8c0>]
mem_sharing_share_pages+0x170/0x310
> > (XEN)    [<ffff82c4801c0724>] mem_sharing_domctl+0xe4/0x130
> > (XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> > (XEN)    [<ffff82c4801457fd>] arch_do_domctl+0xdad/0x1f90
> > (XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
> > (XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
> > (XEN)    [<ffff82c480104373>] do_domctl+0x163/0x1000
> > (XEN)    [<ffff82c4801a0b63>]
hvm_set_callback_irq_level+0xe3/0x110
> > (XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
> > (XEN)
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Xen BUG at mem_sharing.c:454
> > (XEN) ****************************************
> > (XEN)
> > (XEN) Manual reset required (''noreboot'' specified)
> >
> 
> Bests,
> Jui-Hao
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, tinnycloud:

Do you have xenpaging tools running properly?
I haven''t gone through that one, but it seems you have run out of
memory.
When this case happens, mem_sharing will request memory to the
xenpaging daemon, which tends to page out and free some memory.
Otherwise, the allocation would fail.
Is this your scenario?

Bests,
Jui-Hao

2011/1/17 MaoXiaoyun <tinnycloud@hotmail.com>:
> Another failure on BUG() in mem_sharing_alloc_page()
>
>  memset(&req, 0, sizeof(req));
>  if(must_succeed)
>     {
>         /* We do not support ''must_succeed'' any more.
External operations
> such
>          * as grant table mappings may fail with OOM condition!
>          */
>         BUG();===================>bug here
>     }
>     else
>     {
>         /* All foreign attempts to unshare pages should be handled through
>          * ''must_succeed'' case. */
>         ASSERT(v->domain->domain_id == d->domain_id);
>         vcpu_pause_nosync(v);
>         req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
>     }
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Juihao:
 
     I have no xenpaging run. 
     Is it only necessary for memory sharing?
           
     But I don''t think my box confront with OOM, since I''ve
been periodically
check memory through "xm info", it shows about 10G free memory. Also, 
this is the first time the test run into this bug.
 
     Could you let me know how you set up memory sharing scenario?

 
> Date: Mon, 17 Jan 2011 17:02:02 +0800
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> From: juihaochiang@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> 
> Hi, tinnycloud:
> 
> Do you have xenpaging tools running properly?
> I haven''t gone through that one, but it seems you have run out of
memory.
> When this case happens, mem_sharing will request memory to the
> xenpaging daemon, which tends to page out and free some memory.
> Otherwise, the allocation would fail.
> Is this your scenario?
> 
> Bests,
> Jui-Hao
> 
> 2011/1/17 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Another failure on BUG() in mem_sharing_alloc_page()
> >
> >  memset(&req, 0, sizeof(req));
> >  if(must_succeed)
> >     {
> >         /* We do not support ''must_succeed'' any
more. External operations
> > such
> >          * as grant table mappings may fail with OOM condition!
> >          */
> >         BUG();===================>bug here
> >     }
> >     else
> >     {
> >         /* All foreign attempts to unshare pages should be handled
through
> >          * ''must_succeed'' case. */
> >         ASSERT(v->domain->domain_id == d->domain_id);
> >         vcpu_pause_nosync(v);
> >         req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
> >     }
> >
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Tim & Jui-Hao:
 
      When I use Linux HVM instead of Windows HVM, more bug shows up.
 
      I only start on VM, and when I destroy it , xen crashed on
mem_sharing_unshare_page()
which in line709, hash_entry is NULL. Later I found the handle has been removed
in
mem_sharing_share_pages(), please refer logs below.
 
----mem_sharing_unshare_page
708     /* Remove the gfn_info from the list */
709     hash_entry = mem_sharing_hash_lookup(handle); 
710     list_for_each(le, &hash_entry->gfns)
711     {
712         gfn_info = list_entry(le, struct gfn_info, list);
713         if((gfn_info->gfn == gfn) && (gfn_info->domain ==
d->domain_id))
714             goto gfn_found;
715     }
716     printk("Could not find gfn_info for shared gfn: %lx\n", gfn);
717     BUG();
 
 
----mem_sharing_share_page
 
649     printk("del %lu\n", ch);
650     list_for_each_safe(le, te, &ce->gfns)
651     {
652         gfn = list_entry(le, struct gfn_info, list);
653         /* Get the source page and type, this should never fail 
654          * because we are under shr lock, and got non-null se */
655         BUG_ON(!get_page_and_type(spage, dom_cow, PGT_shared_page));
656         /* Move the gfn_info from ce list to se list */
657         list_del(&gfn->list);
658         d = get_domain_by_id(gfn->domain);
659         BUG_ON(!d);
660         BUG_ON(set_shared_p2m_entry(d, gfn->gfn, se->mfn) == 0);
661         put_domain(d);
662         list_add(&gfn->list, &se->gfns);
663         put_page_and_type(cpage);
664         mem_sharing_debug_gfn(d, gfn->gfn);
665     } 
666     ASSERT(list_empty(&ce->gfns));
667     mem_sharing_hash_delete(ch);
668     atomic_inc(&nr_saved_mfns);
669     /* Free the client page */
670     if(test_and_clear_bit(_PGC_allocated, &cpage->count_info))
671         put_page(cpage);
672     mem_sharing_debug_gfn(d, gfn->gfn);
673     ret = 0;
 
 
-------log------------
      
(XEN) del 31261
(XEN) Debug for domain=1, gfn=75fd5, Debug page: MFN=179fd5 is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fd5, Debug page: MFN=179fd5 is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31262
(XEN) Debug for domain=1, gfn=75fd6, Debug page: MFN=179fd6 is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fd6, Debug page: MFN=179fd6 is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31263
(XEN) Debug for domain=1, gfn=75fd7, Debug page: MFN=179fd7 is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fd7, Debug page: MFN=179fd7 is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31264
(XEN) Debug for domain=1, gfn=75fd8, Debug page: MFN=179fd8 is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fd8, Debug page: MFN=179fd8 is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31265
(XEN) Debug for domain=1, gfn=75fd9, Debug page: MFN=179fd9 is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fd9, Debug page: MFN=179fd9 is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31266
(XEN) Debug for domain=1, gfn=75fda, Debug page: MFN=179fda is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fda, Debug page: MFN=179fda is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31267
(XEN) Debug for domain=1, gfn=75fdb, Debug page: MFN=179fdb is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fdb, Debug page: MFN=179fdb is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31268
(XEN) Debug for domain=1, gfn=75fdc, Debug page: MFN=179fdc is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fdc, Debug page: MFN=179fdc is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31269
(XEN) Debug for domain=1, gfn=75fdd, Debug page: MFN=179fdd is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fdd, Debug page: MFN=179fdd is ci=4,
ti=8400000000000001, owner_id=32755
(XEN) del 31270
(XEN) Debug for domain=1, gfn=75fde, Debug page: MFN=179fde is
ci=8000000000000005, ti=8400000000000001, owner_id=32755
(XEN) Debug for domain=1, gfn=75fde, Debug page: MFN=179fde is ci=4,
ti=8400000000000001, owner_id=32755
blktap_sysfs_destroy
(XEN) handle 31261
(XEN) Debug for domain=1, gfn=75fd5, Debug page: MFN=179fd5 is ci=1,
ti=8400000000000001, owner_id=32755
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801bfeeb>]
mem_sharing_unshare_page+0x1ab/0x740
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000000179fd5   rcx: 0000000000000082
(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021e9c4
(XEN) rbp: ffff8302dd6a0000   rsp: ffff83023ff3fcd0   r8:  0000000000000001
(XEN) r9:  0000000000000000   r10: 00000000fffffff8   r11: 0000000000000005
(XEN) r12: 0000000000075fd5   r13: 0000000000000002   r14: 0000000000000000
(XEN) r15: ffff82f602f3faa0   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000031b83b000   cr2: 0000000000000018
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff83023ff3fcd0:
(XEN)    ffff83023fe60000 ffff830173da4410 0000000100000000 0000000000000000
(XEN)    0000000000007a1d 0000000000000001 0000000000000009 ffff8302dd6a0000
(XEN)    ffff83023ff3fd40 ffff82c4801df7e9 ffff83023ff3ff28 ffff83023ff3fd94
(XEN)    0000000000075fd5 0000000d000001d5 ffff83033e950000 0000000000075fd5
(XEN)    ffff83063fde8ef0 ffff8302dd6a0000 ffff83023ff3fd94 ffff82c48037a980
(XEN)    ffff82c480253080 ffff82c4801b8949 ffff8302dd6a0180 ffff82c48037a980
(XEN)    0000000d80253080 ffff8302dd6a0000 ffff8302dd6a0000 00000000ffffffff
(XEN)    ffff8302dd6a0000 ffff82c4801b681c 0000000000000000 ffff82c480149b74
(XEN)    ffff8300bf560000 ffff8300bf560000 fffffffffffffff8 00000000ffffffff
(XEN)    ffff8302dd6a0000 ffff82c4801061fc ffff8300bf552060 0000000000000000
(XEN)    ffff82c4802531a0 0000000000000001 ffff82c480376980 ffff82c48012218c
(XEN)    0000000000000001 fffffffffffffffd ffff83023ff3ff28 ffff82c48011c588
(XEN)    ffff83023ff3ff28 ffff83063fdeb170 ffff83063fdeb230 ffff8300bf552000
(XEN)    000001831ea27db3 ffff82c480189c6a 7fffffffffffffff ffff82c4801441b5
(XEN)    ffff82c48037b7b0 ffff82c48011e474 0000000000000001 ffffffffffffffff
(XEN)    0000000000000000 0000000000000000 0000000080376980 00001833000116f2
(XEN)    ffffffffffffffff ffff83023ff3ff28 ffff82c480251b00 ffff83023ff3fe28
(XEN)    ffff8300bf552000 000001831ea27db3 ffff82c480253080 ffff82c480149ad6
(XEN)    0000000000000000 0000000000002000 ffff8300bf2fc000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff88015f8f9f10
(XEN) Xen call trace:
(XEN)    [<ffff82c4801bfeeb>] mem_sharing_unshare_page+0x1ab/0x740
(XEN)    [<ffff82c4801df7e9>] ept_get_entry+0xa9/0x1c0
(XEN)    [<ffff82c4801b8949>] p2m_teardown+0x129/0x170
(XEN)    [<ffff82c4801b681c>] paging_final_teardown+0x2c/0x40
(XEN)    [<ffff82c480149b74>] arch_domain_destroy+0x44/0x170
(XEN)    [<ffff82c4801061fc>] complete_domain_destroy+0x6c/0x130
(XEN)    [<ffff82c48012218c>] rcu_process_callbacks+0xac/0x220
(XEN)    [<ffff82c48011c588>] __do_softirq+0x58/0x80
(XEN)    [<ffff82c480189c6a>] acpi_processor_idle+0x14a/0x740
(XEN)    [<ffff82c4801441b5>] reprogram_timer+0x55/0x90
(XEN)    [<ffff82c48011e474>] timer_softirq_action+0x1a4/0x360
(XEN)    [<ffff82c480149ad6>] idle_loop+0x26/0x80
(XEN)    
(XEN) Pagetable walk from 0000000000000018:
(XEN)  L4[0x000] = 00000001676bd067 0000000000156103
(XEN)  L3[0x000] = 000000031b947067 0000000000121c8d
(XEN)  L2[0x000] = 0000000000000000 ffffffffffffffff 
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000018
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi:
 
 It is later found that caused by below patch code and I am using the blktap2.
The handle retruned from here will later become ch in mem_sharing_share_pages,
and then
in mem_sharing_share_pages will have ch = sh, thus caused the problem.
 
+    /* Return the handle if the page is already shared */
+    page = mfn_to_page(mfn);
+    if (p2m_is_shared(p2mt)) {
+        *phandle = page->shr_handle;
+        ret = 0;
+        goto out;
+    }
+
 
But. after I  removed the code, tests still failed, and this handle''s
value is not make sence.
 
 
(XEN) ===>total handles 17834 total gfns 255853
(XEN) handle 13856642536914634
(XEN) Debug for domain=1, gfn=19fed, Debug page: MFN=349c0a is
ci=8000000000000008, ti=8400000000000007, owner_id=32755
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    15
(XEN) RIP:    e008:[<ffff82c4801bff4b>]
mem_sharing_unshare_page+0x19b/0x720
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff83063fc67f28   rcx: 0000000000000092
(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021e9c4
(XEN) rbp: ffff830440000000   rsp: ffff83063fc67c48   r8:  0000000000000001
(XEN) r9:  0000000000000000   r10: 00000000fffffff8   r11: 0000000000000005
(XEN) r12: 0000000000019fed   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: ffff82f606938140   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000055513c000   cr2: 0000000000000018
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff83063fc67c48:
(XEN)    02c5f6c8b70fed66 39ef64058b487674 ffff82c4801a6082 0000000000000000
(XEN)    00313a8b00313eca 0000000000000001 0000000000000009 ffff830440000000
(XEN)    ffff83063fc67cb8 ffff82c4801df6f9 0000000000000040 ffff83063fc67d04
(XEN)    0000000000019fed 0000000d000001ed ffff83055458d000 ffff83063fc67f28
(XEN)    0000000000019fed 0000000000349c0a 0000000000000030 ffff83063fc67f28
(XEN)    0000000000000030 ffff82c48019baa6 ffff82c4802519c0 0000000d8016838e
(XEN)    0000000000000000 00000000000001aa ffff8300bf554000 ffff82c4801b3864
(XEN)    ffff830440000348 ffff8300bf554000 ffff8300bf5557f0 ffff8300bf5557e8
(XEN)    00000032027b81f2 ffff82c48026f080 ffff82c4801a9337 ffff8300bf448000
(XEN)    ffff8300bf554000 ffff830000000000 0000000019fed000 ffff8300bf2f2000
(XEN)    ffff82c48019985d 0000000000000080 ffff8300bf554000 0000000000019fed
(XEN)    ffff82c4801b08ba 000000000001e000 ffff82c48014931f ffff8305570c6d50
(XEN)    ffff82c480251080 00000032027b81f2 ffff8305570c6d50 ffff83052f3e2200
(XEN)    0000000f027b7de0 ffff82c48011e07a 000000000000000f ffff82c48026f0a0
(XEN)    0000000000000082 0000000000000000 0000000000000000 0000000000009e44
(XEN)    ffff8300bf554000 ffff8300bf2f2000 ffff82c48011e07a 000000000000000f
(XEN)    ffff8300bf555760 0000000000000292 ffff82c48011afca 00000032028a8fc0
(XEN)    0000000000000292 ffff82c4801a93c3 00000000000000ef ffff8300bf554000
(XEN)    ffff8300bf554000 ffff8300bf5557e8 ffff82c4801a6082 ffff8300bf554000
(XEN)    0000000000000000 ffff82c4801a0cc8 ffff8300bf554000 ffff8300bf554000
(XEN) Xen call trace:
(XEN)    [<ffff82c4801bff4b>] mem_sharing_unshare_page+0x19b/0x720
(XEN)    [<ffff82c4801a6082>] vlapic_has_pending_irq+0x42/0x70
(XEN)    [<ffff82c4801df6f9>] ept_get_entry+0xa9/0x1c0
(XEN)    [<ffff82c48019baa6>] hvm_hap_nested_page_fault+0xd6/0x190
(XEN)    [<ffff82c4801b3864>] vmx_vmexit_handler+0x304/0x1a90
(XEN)    [<ffff82c4801a9337>] pt_restore_timer+0x57/0xb0
(XEN)    [<ffff82c48019985d>] hvm_do_resume+0x1d/0x130
(XEN)    [<ffff82c4801b08ba>] vmx_do_resume+0x11a/0x1c0
(XEN)    [<ffff82c48014931f>] context_switch+0x76f/0xf00
(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
(XEN)    [<ffff82c48011afca>] schedule+0x1ea/0x500
(XEN)    [<ffff82c4801a93c3>] pt_update_irq+0x33/0x1e0
(XEN)    [<ffff82c4801a6082>] vlapic_has_pending_irq+0x42/0x70
(XEN)    [<ffff82c4801a0cc8>] hvm_vcpu_has_pending_irq+0x88/0xa0
(XEN)    [<ffff82c4801b267b>] vmx_vmenter_helper+0x5b/0x150
(XEN)    [<ffff82c4801adaa3>] vmx_asm_do_vmentry+0x0/0xdd
(XEN)    
(XEN) Pagetable walk from 0000000000000018:
(XEN)  L4[0x000] = 0000000000000000 ffffffffffffffff
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 15:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000018
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified)

 
 

 ---------------------------------------------------------------------------------------------------
>From: tinnycloud@hotmail.com
>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
>CC: tim.deegan@citrix.com
>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate and unshare
>Date: Tue, 18 Jan 2011 17:42:32 +0800


>Hi Tim & Jui-Hao:
 
 >     When I use Linux HVM instead of Windows HVM, more bug shows up.
 
>      I only start on VM, and when I destroy it , xen crashed on
mem_sharing_unshare_page()
>which in line709, hash_entry is NULL. Later I found the handle has been
removed in
>mem_sharing_share_pages(), please refer logs below.
 
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi George:
 
       I am working on the xen mem_sharing,  I think the bug below is related to
POD.
(Test shows when POD is enable, it is easily hit the bug, when disabled, no bug
occurs).
 
As I know when domU starts will POD, it gets memory from POD cache, and in some
situation, POD cached will scan from Zero pages for reusing(link the page into
POD
cache page list), and from the page_info define, list and handle share same
posistion,
I think when reusing the page, POD doest''t check page type, and if it
is a shared page
, it still can be put into POD cache, and thus handle is been overwritten.
      
      So maybe we need to check the page type before putting into cache,
What''s your opinion?
      thanks.
 
>--------------------------------------------------------------------------------
>From: tinnycloud@hotmail.com
>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
>CC: tim.deegan@citrix.com
>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate and unshare
>Date: Tue, 18 Jan 2011 20:05:16 +0800
>
>Hi:
> 
> It is later found that caused by below patch code and I am using the
blktap2.
>The handle retruned from here will later become ch in
mem_sharing_share_pages, and then
>in mem_sharing_share_pages will have ch = sh, thus caused the problem.
> 
>+    /* Return the handle if the page is already shared */
>+    page = mfn_to_page(mfn);
>+    if (p2m_is_shared(p2mt)) {
>+        *phandle = page->shr_handle;
>+        ret = 0;
>+        goto out;
>+    }
>+
> 
>But. after I  removed the code, tests still failed, and this
handle''s value is not make sence.
> 
> 
>(XEN) ===>total handles 17834 total gfns 255853
>(XEN) handle 13856642536914634
>(XEN) Debug for domain=1, gfn=19fed, Debug page: MFN=349c0a is
ci=8000000000000008, ti=8400000000000007, owner_id=32755
>(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
>(XEN) CPU:    15
>(XEN) RIP:    e008:[<ffff82c4801bff4b>]
mem_sharing_unshare_page+0x19b/0x720
>(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
>(XEN) rax: 0000000000000000   rbx: ffff83063fc67f28   rcx: 0000000000000092
>(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021e9c4
>(XEN) rbp: ffff830440000000   rsp: ffff83063fc67c48   r8:  0000000000000001
>(XEN) r9:  0000000000000000   r10: 00000000fffffff8   r11: 0000000000000005
>(XEN) r12: 0000000000019fed   r13: 0000000000000000   r14: 0000000000000000
>(XEN) r15: ffff82f606938140   cr0: 000000008005003b   cr4: 00000000000026f0
>(XEN) cr3: 000000055513c000   cr2: 0000000000000018
>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>(XEN) Xen stack trace from rsp=ffff83063fc67c48:
>(XEN)    02c5f6c8b70fed66 39ef64058b487674 ffff82c4801a6082 0000000000000000
>(XEN)    00313a8b00313eca 0000000000000001 0000000000000009 ffff830440000000
>(XEN)    ffff83063fc67cb8 ffff82c4801df6f9 0000000000000040 ffff83063fc67d04
>(XEN)    0000000000019fed 0000000d000001ed ffff83055458d000 ffff83063fc67f28
>(XEN)    0000000000019fed 0000000000349c0a 0000000000000030 ffff83063fc67f28
>(XEN)    0000000000000030 ffff82c48019baa6 ffff82c4802519c0 0000000d8016838e
>(XEN)    0000000000000000 00000000000001aa ffff8300bf554000 ffff82c4801b3864
>(XEN)    ffff830440000348 ffff8300bf554000 ffff8300bf5557f0 ffff8300bf5557e8
>(XEN)    00000032027b81f2 ffff82c48026f080 ffff82c4801a9337 ffff8300bf448000
>(XEN)    ffff8300bf554000 ffff830000000000 0000000019fed000 ffff8300bf2f2000
>(XEN)    ffff82c48019985d 0000000000000080 ffff8300bf554000 0000000000019fed
>(XEN)    ffff82c4801b08ba 000000000001e000 ffff82c48014931f ffff8305570c6d50
>(XEN)    ffff82c480251080 00000032027b81f2 ffff8305570c6d50 ffff83052f3e2200
>(XEN)    0000000f027b7de0 ffff82c48011e07a 000000000000000f ffff82c48026f0a0
>(XEN)    0000000000000082 0000000000000000 0000000000000000 0000000000009e44
>(XEN)    ffff8300bf554000 ffff8300bf2f2000 ffff82c48011e07a 000000000000000f
>(XEN)    ffff8300bf555760 0000000000000292 ffff82c48011afca 00000032028a8fc0
>(XEN)    0000000000000292 ffff82c4801a93c3 00000000000000ef ffff8300bf554000
>(XEN)    ffff8300bf554000 ffff8300bf5557e8 ffff82c4801a6082 ffff8300bf554000
>(XEN)    0000000000000000 ffff82c4801a0cc8 ffff8300bf554000 ffff8300bf554000
>(XEN) Xen call trace:
>(XEN)    [<ffff82c4801bff4b>] mem_sharing_unshare_page+0x19b/0x720
>(XEN)    [<ffff82c4801a6082>] vlapic_has_pending_irq+0x42/0x70
>(XEN)    [<ffff82c4801df6f9>] ept_get_entry+0xa9/0x1c0
>(XEN)    [<ffff82c48019baa6>] hvm_hap_nested_page_fault+0xd6/0x190
>(XEN)    [<ffff82c4801b3864>] vmx_vmexit_handler+0x304/0x1a90
>(XEN)    [<ffff82c4801a9337>] pt_restore_timer+0x57/0xb0
>(XEN)    [<ffff82c48019985d>] hvm_do_resume+0x1d/0x130
>(XEN)    [<ffff82c4801b08ba>] vmx_do_resume+0x11a/0x1c0
>(XEN)    [<ffff82c48014931f>] context_switch+0x76f/0xf00
>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
>(XEN)    [<ffff82c48011afca>] schedule+0x1ea/0x500
>(XEN)    [<ffff82c4801a93c3>] pt_update_irq+0x33/0x1e0
>(XEN)    [<ffff82c4801a6082>] vlapic_has_pending_irq+0x42/0x70
>(XEN)    [<ffff82c4801a0cc8>] hvm_vcpu_has_pending_irq+0x88/0xa0
>(XEN)    [<ffff82c4801b267b>] vmx_vmenter_helper+0x5b/0x150
>(XEN)    [<ffff82c4801adaa3>] vmx_asm_do_vmentry+0x0/0xdd
>(XEN)    
>(XEN) Pagetable walk from 0000000000000018:
>(XEN)  L4[0x000] = 0000000000000000 ffffffffffffffff
>(XEN) 
>(XEN) ****************************************
>(XEN) Panic on CPU 15:
>(XEN) FATAL PAGE FAULT
>(XEN) [error_code=0000]
>(XEN) Faulting linear address: 0000000000000018
>(XEN) ****************************************
>(XEN) 
>(XEN) Manual reset required (''noreboot'' specified)
>
> 
> 
>
>
---------------------------------------------------------------------------------------------------
>>From: tinnycloud@hotmail.com
>>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
>>CC: tim.deegan@citrix.com
>>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate and
unshare
>>Date: Tue, 18 Jan 2011 17:42:32 +0800
>
>>Hi Tim & Jui-Hao:
> 
> >     When I use Linux HVM instead of Windows HVM, more bug shows up.
> 
>>      I only start on VM, and when I destroy it , xen crashed on
mem_sharing_unshare_page()
>>which in line709, hash_entry is NULL. Later I found the handle has been
removed in
>>mem_sharing_share_pages(), please refer logs below.
>  		 	   		  
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Very likely.  If you look in xen/arch/x86/mm/p2m.c, the two functions
which check a page to see if it can be reclaimed are
"p2m_pod_zero_check*()".  A little ways into each function
there''s a
giant "if()" which has all of the conditions for reclaiming a page,
starting with p2m_is_ram().  The easiest way to fix it is to add
p2m_is_shared() to that "if" statement.

 -George

2011/1/19 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi George:
>
>        I am working on the xen mem_sharing,  I think the bug below is
> related to POD.
> (Test shows when POD is enable, it is easily hit the bug, when disabled, no
> bug occurs).
>
> As I know when domU starts will POD, it gets memory from POD cache, and in
> some
> situation, POD cached will scan from Zero pages for reusing(link the page
> into POD
> cache page list), and from the page_info define, list and handle share same
> posistion,
> I think when reusing the page, POD doest''t check page type, and if
it is a
> shared page
> , it still can be put into POD cache, and thus handle is been overwritten.
>
>       So maybe we need to check the page type before putting into cache,
> What''s your opinion?
>       thanks.
>
>>--------------------------------------------------------------------------------
>>From: tinnycloud@hotmail.com
>>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
>>CC: tim.deegan@citrix.com
>>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate and
>> unshare
>>Date: Tue, 18 Jan 2011 20:05:16 +0800
>>
>>Hi:
>>
>> It is later found that caused by below patch code and I am using the
>> blktap2.
>>The handle retruned from here will later become ch in
>> mem_sharing_share_pages, and then
>>in mem_sharing_share_pages will have ch = sh, thus caused the problem.
>>
>>+    /* Return the handle if the page is already shared */
>>+    page = mfn_to_page(mfn);
>>+    if (p2m_is_shared(p2mt)) {
>>+        *phandle = page->shr_handle;
>>+        ret = 0;
>>+        goto out;
>>+    }
>>+
>>
>>But. after I  removed the code, tests still failed, and this
handle''s value
>> is not make sence.
>>
>>
>>(XEN) ===>total handles 17834 total gfns 255853
>>(XEN) handle 13856642536914634
>>(XEN) Debug for domain=1, gfn=19fed, Debug page: MFN=349c0a is
>> ci=8000000000000008, ti=8400000000000007, owner_id=32755
>>(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
>>(XEN) CPU:    15
>>(XEN) RIP:    e008:[<ffff82c4801bff4b>]
>> mem_sharing_unshare_page+0x19b/0x720
>>(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
>>(XEN) rax: 0000000000000000   rbx: ffff83063fc67f28&nbsp ;  rcx:
>> 0000000000000092
>>(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi:
ffff82c48021e9c4
>>(XEN) rbp: ffff830440000000   rsp: ffff83063fc67c48   r8: 
0000000000000001
>>(XEN) r9:  0000000000000000   r10: 00000000fffffff8   r11:
0000000000000005
>>(XEN) r12: 0000000000019fed   r13: 0000000000000000   r14:
0000000000000000
>>(XEN) r15: ffff82f606938140   cr0: 000000008005003b   cr4:
00000000000026f0
>>(XEN) cr3: 000000055513c000   cr2: 0000000000000018
>>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>>(XEN) Xen stack trace from rsp=ffff83063fc67c48:
>>(XEN)    02c5f6c8b70fed66 39ef64058b487674 ffff82c4801a6082
>> 0000000000000000
>>(XEN)    00313a8b00313eca 0000000000000001 0000000000000009 ff
>> ff830440000000
>>(XEN)    ffff83063fc67cb8 ffff82c4801df6f9 0000000000000040
>> ffff83063fc67d04
>>(XEN)    0000000000019fed 0000000d000001ed ffff83055458d000
>> ffff83063fc67f28
>>(XEN)    0000000000019fed 0000000000349c0a 0000000000000030
>> ffff83063fc67f28
>>(XEN)    0000000000000030 ffff82c48019baa6 ffff82c4802519c0
>> 0000000d8016838e
>>(XEN)    0000000000000000 00000000000001aa ffff8300bf554000
>> ffff82c4801b3864
>>(XEN)    ffff830440000348 ffff8300bf554000 ffff8300bf5557f0
>> ffff8300bf5557e8
>>(XEN)    00000032027b81f2 ffff82c48026f080 ffff82c4801a9337
>> ffff8300bf448000
>>(XEN)    ffff8300bf554000 ffff830000000000 0000000019fed000
>> ffff8300bf2f2000
>>(XEN)    ffff82c48019985d 0000000000000080 ffff8300bf554000
>> 0000000000019fed
>>(XEN)    ffff82c4801b08ba 000000000001e000 ffff82c48014931f ff
>> ff8305570c6d50
>>(XEN)    ffff82c480251080 00000032027b81f2 ffff8305570c6d50
>> ffff83052f3e2200
>>(XEN)    0000000f027b7de0 ffff82c48011e07a 000000000000000f
>> ffff82c48026f0a0
>>(XEN)    0000000000000082 0000000000000000 0000000000000000
>> 0000000000009e44
>>(XEN)    ffff8300bf554000 ffff8300bf2f2000 ffff82c48011e07a
>> 000000000000000f
>>(XEN)    ffff8300bf555760 0000000000000292 ffff82c48011afca
>> 00000032028a8fc0
>>(XEN)    0000000000000292 ffff82c4801a93c3 00000000000000ef
>> ffff8300bf554000
>>(XEN)    ffff8300bf554000 ffff8300bf5557e8 ffff82c4801a6082
>> ffff8300bf554000
>>(XEN)    0000000000000000 ffff82c4801a0cc8 ffff8300bf554000
>> ffff8300bf554000
>>(XEN) Xen call trace:
>>(XEN)    [<ffff82c4801bff4b>] mem_sharing_unshare_page+0x19b/0x720
>>(XEN)    [<ffff82c4801a6082>] v lapic_has_pending_irq+0x42/0x70
>>(XEN)    [<ffff82c4801df6f9>] ept_get_entry+0xa9/0x1c0
>>(XEN)    [<ffff82c48019baa6>] hvm_hap_nested_page_fault+0xd6/0x190
>>(XEN)    [<ffff82c4801b3864>] vmx_vmexit_handler+0x304/0x1a90
>>(XEN)    [<ffff82c4801a9337>] pt_restore_timer+0x57/0xb0
>>(XEN)    [<ffff82c48019985d>] hvm_do_resume+0x1d/0x130
>>(XEN)    [<ffff82c4801b08ba>] vmx_do_resume+0x11a/0x1c0
>>(XEN)    [<ffff82c48014931f>] context_switch+0x76f/0xf00
>>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
>>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
>>(XEN)    [<ffff82c48011afca>] schedule+0x1ea/0x500
>>(XEN)    [<ffff82c4801a93c3>] pt_update_irq+0x33/0x1e0
>>(XEN)    [&lt ;ffff82c4801a6082>]
vlapic_has_pending_irq+0x42/0x70
>>(XEN)    [<ffff82c4801a0cc8>] hvm_vcpu_has_pending_irq+0x88/0xa0
>>(XEN)    [<ffff82c4801b267b>] vmx_vmenter_helper+0x5b/0x150
>>(XEN)    [<ffff82c4801adaa3>] vmx_asm_do_vmentry+0x0/0xdd
>>(XEN)
>>(XEN) Pagetable walk from 0000000000000018:
>>(XEN)  L4[0x000] = 0000000000000000 ffffffffffffffff
>>(XEN)
>>(XEN) ****************************************
>>(XEN) Panic on CPU 15:
>>(XEN) FATAL PAGE FAULT
>>(XEN) [error_code=0000]
>>(XEN) Faulting linear address: 0000000000000018
>>(XEN) ****************************************
>>(XEN)
>>(XEN) Manual reset required (''noreboot'' specified)
>>
>>
>>
>>
>>
>>
---------------------------------------------------------------------------------------------------
>>>From: tinnycloud@hotmail.com
>>>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
>>>CC: tim.deegan@citrix.com
>>>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate and
>>> unshare
>>>Date: Tue, 18 Jan 2011 17:42:32 +0800
>>
>>>Hi Tim & Jui-Hao:
>>
>> >     When I use Linux HVM instead of Windows HVM, more bug shows
up.
>>
>>>      I only start on VM, and when I destroy it , xen crashed on
>>> mem_sharing_unshare_page()
>>>which in line709, hash_entry is NULL. Later I found the handle has
been
>>> removed in
>>>mem_sharing_share_pages(), please refer logs below.
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi:
 
       More, as I previous found that, some of the nominate pages are already
shared in linux HVM test.
       But it doesn''t make sense. 
 
       Since the in my setup, the page is allocated from  blkfront in HVM, its
gref go through blktap2
and tapdisk2, it is assumed to be new, not possible has the type of shared,
since if nominate failed,
the page will be filled with IO data from disk, that is to say, the page will be
written again, and its previous
data is overwritten. I tested with only one HVM, it shows it has double nominate
pages.
 
       I also test the windows HVM, which is no same page nominated.
       So i think it is a problem of blkfront.
 
       Am I right, could some one kindly offer some hints?
       many thanks.
 
>
>>--------------------------------------------------------------------------------
> >>From: tinnycloud@hotmail.com
> >>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
> >>CC: tim.deegan@citrix.com
> >>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate
and
> >> unshare
> >>Date: Tue, 18 Jan 2011 20:05:16 +0800
> >>
> >>Hi:
> >>
> >> It is later found that caused by below patch code and I am using
the
> >> blktap2.
> >>The handle retruned from here will later become ch in
> >> mem_sharing_share_pages, and then
> >>in mem_sharing_share_pages will have ch = sh, thus caused the
problem.
> >>
> >>+    /* Return the handle if the page is already shared */
> >>+    page = mfn_to_page(mfn);
> >>+    if (p2m_is_shared(p2mt)) {
> >>+        *phandle = page->shr_handle;
> >>+        ret = 0;
> >>+        goto out;
> >>+    }
> >>+
> >>
> >>But. after I  removed the code, tests still failed, and this
handle''s value
> >> is not make sence.
> >>
> >>
> >>(XEN) ===>total handles 17834 total gfns 255853
> >>(XEN) handle 13856642536914634
> >>(XEN) Debug for domain=1, gfn=19fed, Debug page: MFN=349c0a is
> >> ci=8000000000000008, ti=8400000000000007, owner_id=32755
> >>(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
> >>(XEN) CPU:    15
> >>(XEN) RIP:    e008:[<ffff82c4801bff4b>]
> >> mem_sharing_unshare_page+0x19b/0x720
> >>(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
> >>(XEN) rax: 0000000000000000   rbx: ffff83063fc67f28&nbsp ; 
rcx:
> >> 0000000000000092
> >>(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi:
ffff82c48021e9c4
> >>(XEN) rbp: ffff830440000000   rsp: ffff83063fc67c48   r8: 
0000000000000001
> >>(XEN) r9:  0000000000000000   r10: 00000000fffffff8   r11:
0000000000000005
> >>(XEN) r12: 0000000000019fed   r13: 0000000000000000   r14:
0000000000000000
> >>(XEN) r15: ffff82f606938140   cr0: 000000008005003b   cr4:
00000000000026f0
> >>(XEN) cr3: 000000055513c000   cr2: 0000000000000018
> >>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs:
e008
> >>(XEN) Xen stack trace from rsp=ffff83063fc67c48:
> >>(XEN)    02c5f6c8b70fed66 39ef64058b487674 ffff82c4801a6082
> >> 0000000000000000
> >>(XEN)    00313a8b00313eca 0000000000000001 0000000000000009 ff
> >> ff830440000000
> >>(XEN)    ffff83063fc67cb8 ffff82c4801df6f9 0000000000000040
> >> ffff83063fc67d04
> >>(XEN)    0000000000019fed 0000000d000001ed ffff83055458d000
> >> ffff83063fc67f28
> >>(XEN)    0000000000019fed 0000000000349c0a 0000000000000030
> >> ffff83063fc67f28
> >>(XEN)    0000000000000030 ffff82c48019baa6 ffff82c4802519c0
> >> 0000000d8016838e
> >>(XEN)    0000000000000000 00000000000001aa ffff8300bf554000
> >> ffff82c4801b3864
> >>(XEN)    ffff830440000348 ffff8300bf554000 ffff8300bf5557f0
> >> ffff8300bf5557e8
> >>(XEN)    00000032027b81f2 ffff82c48026f080 ffff82c4801a9337
> >> ffff8300bf448000
> >>(XEN)    ffff8300bf554000 ffff830000000000 0000000019fed000
> >> ffff8300bf2f2000
> >>(XEN)    ffff82c48019985d 0000000000000080 ffff8300bf554000
> >> 0000000000019fed
> >>(XEN)    ffff82c4801b08ba 000000000001e000 ffff82c48014931f ff
> >> ff8305570c6d50
> >>(XEN)    ffff82c480251080 00000032027b81f2 ffff8305570c6d50
> >> ffff83052f3e2200
> >>(XEN)    0000000f027b7de0 ffff82c48011e07a 000000000000000f
> >> ffff82c48026f0a0
> >>(XEN)    0000000000000082 0000000000000000 0000000000000000
> >> 0000000000009e44
> >>(XEN)    ffff8300bf554000 ffff8300bf2f2000 ffff82c48011e07a
> >> 000000000000000f
> >>(XEN)    ffff8300bf555760 0000000000000292 ffff82c48011afca
> >> 00000032028a8fc0
> >>(XEN)    0000000000000292 ffff82c4801a93c3 00000000000000ef
> >> ffff8300bf554000
> >>(XEN)    ffff8300bf554000 ffff8300bf5557e8 ffff82c4801a6082
> >> ffff8300bf554000
> >>(XEN)    0000000000000000 ffff82c4801a0cc8 ffff8300bf554000
> >> ffff8300bf554000
> >>(XEN) Xen call trace:
> >>(XEN)    [<ffff82c4801bff4b>]
mem_sharing_unshare_page+0x19b/0x720
> >>(XEN)    [<ffff82c4801a6082>] v
lapic_has_pending_irq+0x42/0x70
> >>(XEN)    [<ffff82c4801df6f9>] ept_get_entry+0xa9/0x1c0
> >>(XEN)    [<ffff82c48019baa6>]
hvm_hap_nested_page_fault+0xd6/0x190
> >>(XEN)    [<ffff82c4801b3864>] vmx_vmexit_handler+0x304/0x1a90
> >>(XEN)    [<ffff82c4801a9337>] pt_restore_timer+0x57/0xb0
> >>(XEN)    [<ffff82c48019985d>] hvm_do_resume+0x1d/0x130
> >>(XEN)    [<ffff82c4801b08ba>] vmx_do_resume+0x11a/0x1c0
> >>(XEN)    [<ffff82c48014931f>] context_switch+0x76f/0xf00
> >>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
> >>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
> >>(XEN)    [<ffff82c48011afca>] schedule+0x1ea/0x500
> >>(XEN)    [<ffff82c4801a93c3>] pt_update_irq+0x33/0x1e0
> >>(XEN)    [&lt ;ffff82c4801a6082>]
vlapic_has_pending_irq+0x42/0x70
> >>(XEN)    [<ffff82c4801a0cc8>]
hvm_vcpu_has_pending_irq+0x88/0xa0
> >>(XEN)    [<ffff82c4801b267b>] vmx_vmenter_helper+0x5b/0x150
> >>(XEN)    [<ffff82c4801adaa3>] vmx_asm_do_vmentry+0x0/0xdd
> >>(XEN)
> >>(XEN) Pagetable walk from 0000000000000018:
> >>(XEN)  L4[0x000] = 0000000000000000 ffffffffffffffff
> >>(XEN)
> >>(XEN) ****************************************
> >>(XEN) Panic on CPU 15:
> >>(XEN) FATAL PAGE FAULT
> >>(XEN) [error_code=0000]
> >>(XEN) Faulting linear address: 0000000000000018
> >>(XEN) ****************************************
> >>(XEN)
> >>(XEN) Manual reset required (''noreboot''
specified)
> >>
> >>
> >>
> >>
> >>
> >>
---------------------------------------------------------------------------------------------------
> >>>From: tinnycloud@hotmail.com
> >>>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
> >>>CC: tim.deegan@citrix.com
> >>>Subject: RE: [PATCH] mem_sharing: fix race condition of
nominate and
> >>> unshare
> >>>Date: Tue, 18 Jan 2011 17:42:32 +0800
> >>
> >>>Hi Tim & Jui-Hao:
> >>
> >> >     When I use Linux HVM instead of Windows HVM, more bug
shows up.
> >>
> >>>      I only start on VM, and when I destroy it , xen crashed
on
> >>> mem_sharing_unshare_page()
> >>>which in line709, hash_entry is NULL. Later I found the handle
has been
> >>> removed in
> >>>mem_sharing_share_pages(), please refer logs below.
> >>
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> >
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi:
 
            The latest BUG in mem_sharing_alloc_page from
mem_sharing_unshare_page.
            I printed heap info, which shows plenty memory left.
            Could domain be NULL during in unshare, or should it be locked by
rcu_lock_domain_by_id ?
 
-----------code------------
422 extern void pagealloc_info(unsigned char key);
 423 static struct page_info* mem_sharing_alloc_page(struct domain *d, 
 424                                                 unsigned long gfn,
 425                                                 int must_succeed)
 426 {
 427     struct page_info* page;
 428     struct vcpu *v = current;
 429     mem_event_request_t req;
 430 
 431     page = alloc_domheap_page(d, 0); 
 432     if(page != NULL) return page;
 433 
 434     memset(&req, 0, sizeof(req));
 435     if(must_succeed) 
 436     {
 437         /* We do not support ''must_succeed'' any more.
External operations such
 438          * as grant table mappings may fail with OOM condition! 
 439          */
 440         pagealloc_info(''m'');
 441         BUG();
 442     }
 
-------------serial output-------
(XEN) Physical memory information:
(XEN)     Xen heap: 0kB free
(XEN)     heap[14]: 64480kB free
(XEN)     heap[15]: 131072kB free
(XEN)     heap[16]: 262144kB free
(XEN)     heap[17]: 524288kB free
(XEN)     heap[18]: 1048576kB free
(XEN)     heap[19]: 1037128kB free
(XEN)     heap[20]: 3035744kB free
(XEN)     heap[21]: 2610292kB free
(XEN)     heap[22]: 2866212kB free
(XEN)     Dom heap: 11579936kB free
(XEN) Xen BUG at mem_sharing.c:441
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c4801c0531>]
mem_sharing_unshare_page+0x681/0x790
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff83040092d808   rcx: 0000000000000096
(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021eac4
(XEN) rbp: 0000000000000000   rsp: ffff82c48035f5e8   r8:  0000000000000001
(XEN) r9:  0000000000000001   r10: 00000000fffffff5   r11: 0000000000000008
(XEN) r12: ffff8305c61f3980   r13: ffff83040eff0000   r14: 000000000001610f
(XEN) r15: ffff82c48035f628   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000052bc4f000   cr2: ffff880120126e88
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48035f5e8:
(XEN)    ffff8305c61f3990 00018300bf2f0000 ffff82f604e6a4a0 000000002ab84078
(XEN)    ffff83040092d7f0 00000000001b9c9c ffff8300bf2f0000 000000010eff0000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000d0000010f ffff8305447ec000 000000000001610f
(XEN)    0000000000273525 ffff82c48035f724 ffff830502c705a0 ffff82f602c89a00
(XEN)    ffff83040eff0000 ffff82c48010bfa9 ffff830572c5dbf0 000000000029e07f
(XEN)    0000000000000000 ffff830572c5dbf0 000000008035fbe8 ffff82c48035f6f8
(XEN)    0000000100000002 ffff830572c5dbf0 ffff83063fc30000 ffff830572c5dbf0
(XEN)    0000035900000000 ffff88010d14bbe0 ffff880159e09000 00003f7e00000002
(XEN)    ffffffffffff0032 ffff88010d14bbb0 ffff830438dfa920 0000000d8010a650
(XEN)    0000000000000100 ffff83063fc30000 ffff8305f9203730 ffffffffffffffea
(XEN)    ffff88010d14bb70 0000000000000000 ffff88010d14bc10 ffff88010d14bbc0
(XEN)    0000000000000002 ffff82c48010da9b 0000000000000202 ffff82c48035fec8
(XEN)    ffff82c48035f7c8 00000000801880af ffff83063fc30010 0000000000000000
(XEN)    ffff82c400000008 ffff82c48035ff28 0000000000000000 ffff88010d14bbc0
(XEN)    ffff880159e08000 0000000000000000 0000000000000000 00020000000002d7
(XEN)    00000000003f2b38 ffff8305b1f4b6b8 ffff8305b30f0000 ffff880159e09000
(XEN)    0000000000000000 0000000000000000 000200000000008a 00000000003ed1f9
(XEN)    ffff83063fc26450 ffff8305b30f0000 ffff880159e0a000 0000000000000000
(XEN)    0000000000000000 00020000000001fa 000000000029e2ba ffff83063fc26fd0
(XEN) Xen call trace:
(XEN)    [<ffff82c4801c0531>] mem_sharing_unshare_page+0x681/0x790
(XEN)    [<ffff82c48010bfa9>] gnttab_map_grant_ref+0xbf9/0xe30
(XEN)    [<ffff82c48010da9b>] do_grant_table_op+0x14b/0x1080
(XEN)    [<ffff82c48010fb44>] do_xen_version+0xb4/0x480
(XEN)    [<ffff82c4801b8215>] set_p2m_entry+0x85/0xc0
(XEN)    [<ffff82c4801bc92e>] set_shared_p2m_entry+0x1be/0x2f0
(XEN)    [<ffff82c480121c4c>] xmem_pool_free+0x2c/0x310
(XEN)    [<ffff82c4801bfaf8>] mem_sharing_share_pages+0xd8/0x3d0
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
(XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
(XEN)    [<ffff82c48014717d>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c48015a1d8>] get_page+0x28/0xf0
(XEN)    [<ffff82c48015ed72>] do_update_descriptor+0x1d2/0x210
(XEN)    [<ffff82c480113d7e>] do_multicall+0x14e/0x340
(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at mem_sharing.c:441
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required (''noreboot'' specified)
 
> Date: Mon, 17 Jan 2011 17:02:02 +0800
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> From: juihaochiang@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> 
> Hi, tinnycloud:
> 
> Do you have xenpaging tools running properly?
> I haven''t gone through that one, but it seems you have run out of
memory.
> When this case happens, mem_sharing will request memory to the
> xenpaging daemon, which tends to page out and free some memory.
> Otherwise, the allocation would fail.
> Is this your scenario?
> 
> Bests,
> Jui-Hao
> 
> 2011/1/17 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Another failure on BUG() in mem_sharing_alloc_page()
> >
> >  memset(&req, 0, sizeof(req));
> >  if(must_succeed)
> >     {
> >         /* We do not support ''must_succeed'' any
more. External operations
> > such
> >          * as grant table mappings may fail with OOM condition!
> >          */
> >         BUG();===================>bug here
> >     }
> >     else
> >     {
> >         /* All foreign attempts to unshare pages should be handled
through
> >          * ''must_succeed'' case. */
> >         ASSERT(v->domain->domain_id == d->domain_id);
> >         vcpu_pause_nosync(v);
> >         req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
> >     }
> >
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 07:19 +0000 on 20 Jan (1295507976), MaoXiaoyun wrote:
> Hi:
> 
>             The latest BUG in mem_sharing_alloc_page from
mem_sharing_unshare_page.
>             I printed heap info, which shows plenty memory left.
>             Could domain be NULL during in unshare, or should it be locked
by rcu_lock_domain_by_id ?
> 
''d'' probably isn''t NULL; more likely is that the
domain is not allowed
to have any more memory.  You should look at the values of d->max_pages
and d->tot_pages when the failure happens.

Cheers.

Tim.
> -----------code------------
> 422 extern void pagealloc_info(unsigned char key);
>  423 static struct page_info* mem_sharing_alloc_page(struct domain *d,
>  424                                                 unsigned long gfn,
>  425                                                 int must_succeed)
>  426 {
>  427     struct page_info* page;
>  428     struct vcpu *v = current;
>  429     mem_event_request_t req;
>  430
>  431     page = alloc_domheap_page(d, 0);
>  432     if(page != NULL) return page;
>  433
>  434     memset(&req, 0, sizeof(req));
>  435     if(must_succeed)
>  436     {
>  437         /* We do not support ''must_succeed'' any
more. External operations such
>  438          * as grant table mappings may fail with OOM condition!
>  439          */
>  440         pagealloc_info(''m'');
>  441         BUG();
>  442     }
> 
> -------------serial output-------
> (XEN) Physical memory information:
> (XEN)     Xen heap: 0kB free
> (XEN)     heap[14]: 64480kB free
> (XEN)     heap[15]: 131072kB free
> (XEN)     heap[16]: 262144kB free
> (XEN)     heap[17]: 524288kB free
> (XEN)     heap[18]: 1048576kB free
> (XEN)     heap[19]: 1037128kB free
> (XEN)     heap[20]: 3035744kB free
> (XEN)     heap[21]: 2610292kB free
> (XEN)     heap[22]: 2866212kB free
> (XEN)     Dom heap: 11579936kB free
> (XEN) Xen BUG at mem_sharing.c:441
> (XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e008:[<ffff82c4801c0531>]
mem_sharing_unshare_page+0x681/0x790
> (XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
> (XEN) rax: 0000000000000000   rbx: ffff83040092d808   rcx: 0000000000000096
> (XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021eac4
> (XEN) rbp: 0000000000000000   rsp: ffff82c48035f5e8   r8:  0000000000000001
> (XEN) r9:  0000000000000001   r10: 00000000fffffff5   r11: 0000000000000008
> (XEN) r12: ffff8305c61f3980   r13: ffff83040eff0000   r14: 000000000001610f
> (XEN) r15: ffff82c48035f628   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 000000052bc4f000   cr2: ffff880120126e88
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=ffff82c48035f5e8:
> (XEN)    ffff8305c61f3990 00018300bf2f0000 ffff82f604e6a4a0
000000002ab84078
> (XEN)    ffff83040092d7f0 00000000001b9c9c ffff8300bf2f0000
000000010eff0000
> (XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
> (XEN)    0000000000000000 0000000d0000010f ffff8305447ec000
000000000001610f
> (XEN)    0000000000273525 ffff82c48035f724 ffff830502c705a0
ffff82f602c89a00
> (XEN)    ffff83040eff0000 ffff82c48010bfa9 ffff830572c5dbf0
000000000029e07f
> (XEN)    0000000000000000 ffff830572c5dbf0 000000008035fbe8
ffff82c48035f6f8
> (XEN)    0000000100000002 ffff830572c5dbf0 ffff83063fc30000
ffff830572c5dbf0
> (XEN)    0000035900000000 ffff88010d14bbe0 ffff880159e09000
00003f7e00000002
> (XEN)    ffffffffffff0032 ffff88010d14bbb0 ffff830438dfa920
0000000d8010a650
> (XEN)    0000000000000100 ffff83063fc30000 ffff8305f9203730
ffffffffffffffea
> (XEN)    ffff88010d14bb70 0000000000000000 ffff88010d14bc10
ffff88010d14bbc0
> (XEN)    0000000000000002 ffff82c48010da9b 0000000000000202
ffff82c48035fec8
> (XEN)    ffff82c48035f7c8 00000000801880af ffff83063fc30010
0000000000000000
> (XEN)    ffff82c400000008 ffff82c48035ff28 0000000000000000
ffff88010d14bbc0
> (XEN)    ffff880159e08000 0000000000000000 0000000000000000
00020000000002d7
> (XEN)    00000000003f2b38 ffff8305b1f4b6b8 ffff8305b30f0000
ffff880159e09000
> (XEN)    0000000000000000 0000000000000000 000200000000008a
00000000003ed1f9
> (XEN)    ffff83063fc26450 ffff8305b30f0000 ffff880159e0a000
0000000000000000
> (XEN)    0000000000000000 00020000000001fa 000000000029e2ba
ffff83063fc26fd0
> (XEN) Xen call trace:
> (XEN)    [<ffff82c4801c0531>] mem_sharing_unshare_page+0x681/0x790
> (XEN)    [<ffff82c48010bfa9>] gnttab_map_grant_ref+0xbf9/0xe30
> (XEN)    [<ffff82c48010da9b>] do_grant_table_op+0x14b/0x1080
> (XEN)    [<ffff82c48010fb44>] do_xen_version+0xb4/0x480
> (XEN)    [<ffff82c4801b8215>] set_p2m_entry+0x85/0xc0
> (XEN)    [<ffff82c4801bc92e>] set_shared_p2m_entry+0x1be/0x2f0
> (XEN)    [<ffff82c480121c4c>] xmem_pool_free+0x2c/0x310
> (XEN)    [<ffff82c4801bfaf8>] mem_sharing_share_pages+0xd8/0x3d0
> (XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> (XEN)    [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
> (XEN)    [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
> (XEN)    [<ffff82c48014717d>] vcpu_kick+0x1d/0x80
> (XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> (XEN)    [<ffff82c48015a1d8>] get_page+0x28/0xf0
> (XEN)    [<ffff82c48015ed72>] do_update_descriptor+0x1d2/0x210
> (XEN)    [<ffff82c480113d7e>] do_multicall+0x14e/0x340
> (XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Xen BUG at mem_sharing.c:441
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required (''noreboot'' specified)
> 
> > Date: Mon, 17 Jan 2011 17:02:02 +0800
> > Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> > From: juihaochiang@gmail.com
> > To: tinnycloud@hotmail.com
> > CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> >
> > Hi, tinnycloud:
> >
> > Do you have xenpaging tools running properly?
> > I haven''t gone through that one, but it seems you have run
out of memory.
> > When this case happens, mem_sharing will request memory to the
> > xenpaging daemon, which tends to page out and free some memory.
> > Otherwise, the allocation would fail.
> > Is this your scenario?
> >
> > Bests,
> > Jui-Hao
> >
> > 2011/1/17 MaoXiaoyun <tinnycloud@hotmail.com>:
> > > Another failure on BUG() in mem_sharing_alloc_page()
> > >
> > >  memset(&req, 0, sizeof(req));
> > >  if(must_succeed)
> > >     {
> > >         /* We do not support ''must_succeed'' any
more. External operations
> > > such
> > >          * as grant table mappings may fail with OOM condition!
> > >          */
> > >         BUG();===================>bug here
> > >     }
> > >     else
> > >     {
> > >         /* All foreign attempts to unshare pages should be
handled through
> > >          * ''must_succeed'' case. */
> > >         ASSERT(v->domain->domain_id == d->domain_id);
> > >         vcpu_pause_nosync(v);
> > >         req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
> > >     }
> > >
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''ll do the check. Thanks.
Well, during the test, I still have another two failures
 
1) when all domain are destroyed, the handle in hash table are not decrease to 0
sometimes.
I print the handle count, most of time it is 0 after all domain destroyed.
(XEN) ===>total handles 2 total gfns 2 next_handle: 713269
 
2)  set_shared_p2m_entry failed
 745     list_for_each_safe(le, te, &ce->gfns)
 746     {
 747         gfn = list_entry(le, struct gfn_info, list);
 748         /* Get the source page and type, this should never fail 
 749          * because we are under shr lock, and got non-null se */
 750         BUG_ON(!get_page_and_type(spage, dom_cow, PGT_shared_page));
 751         /* Move the gfn_info from ce list to se list */
 752         list_del(&gfn->list);
 753         d = get_domain_by_id(gfn->domain);
 754 //      mem_sharing_debug_gfn(d, gfn->gfn);
 755         BUG_ON(!d);
 756         BUG_ON(set_shared_p2m_entry(d, gfn->gfn, se->mfn) == 0);
 757         put_domain(d);
 758         list_add(&gfn->list, &se->gfns);
 759         put_page_and_type(cpage);
 760 //      mem_sharing_debug_gfn(d, gfn->gfn);   
 
 
(XEN) printk: 33 messages suppressed.
(XEN) p2m.c:2442:d0 set_mmio_p2m_entry: set_p2m_entry failed! mfn=0023dbb7
(XEN) Xen BUG at mem_sharing.c:756
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c4801bfd90>]
mem_sharing_share_pages+0x370/0x3d0
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff83040ed20000   rcx: 0000000000000092
(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi: ffff82c48021eac4
(XEN) rbp: ffff8305a4bbe1b0   rsp: ffff82c48035fc58   r8:  0000000000000001
(XEN) r9:  0000000000000000   r10: 00000000fffffffb   r11: ffff82c4801318d0
(XEN) r12: ffff8305a4bbe1a0   r13: ffff8305a61d42a0   r14: ffff82f6047b76e0
(XEN) r15: ffff8304e5e918c8   cr0: 0000000080050033   cr4: 00000000000026f0
(XEN) cr3: 00000005203fc000   cr2: 00000000027b8000


 
 
 
> Date: Thu, 20 Jan 2011 09:19:34 +0000
> From: Tim.Deegan@citrix.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; juihaochiang@gmail.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> At 07:19 +0000 on 20 Jan (1295507976), MaoXiaoyun wrote:
> > Hi:
> > 
> > The latest BUG in mem_sharing_alloc_page from
mem_sharing_unshare_page.
> > I printed heap info, which shows plenty memory left.
> > Could domain be NULL during in unshare, or should it be locked by
rcu_lock_domain_by_id ?
> > 
> 
> ''d'' probably isn''t NULL; more likely is that the
domain is not allowed
> to have any more memory. You should look at the values of d->max_pages
> and d->tot_pages when the failure happens.
> 
> Cheers.
> 
> Tim.
> 
> > -----------code------------
> > 422 extern void pagealloc_info(unsigned char key);
> > 423 static struct page_info* mem_sharing_alloc_page(struct domain *d,
> > 424 unsigned long gfn,
> > 425 int must_succeed)
> > 426 {
> > 427 struct page_info* page;
> > 428 struct vcpu *v = current;
> > 429 mem_event_request_t req;
> > 430
> > 431 page = alloc_domheap_page(d, 0);
> > 432 if(page != NULL) return page;
> > 433
> > 434 memset(&req, 0, sizeof(req));
> > 435 if(must_succeed)
> > 436 {
> > 437 /* We do not support ''must_succeed'' any more.
External operations such
> > 438 * as grant table mappings may fail with OOM condition!
> > 439 */
> > 440 pagealloc_info(''m'');
> > 441 BUG();
> > 442 }
> > 
> > -------------serial output-------
> > (XEN) Physical memory information:
> > (XEN) Xen heap: 0kB free
> > (XEN) heap[14]: 64480kB free
> > (XEN) heap[15]: 131072kB free
> > (XEN) heap[16]: 262144kB free
> > (XEN) heap[17]: 524288kB free
> > (XEN) heap[18]: 1048576kB free
> > (XEN) heap[19]: 1037128kB free
> > (XEN) heap[20]: 3035744kB free
> > (XEN) heap[21]: 2610292kB free
> > (XEN) heap[22]: 2866212kB free
> > (XEN) Dom heap: 11579936kB free
> > (XEN) Xen BUG at mem_sharing.c:441
> > (XEN) ----[ Xen-4.0.0 x86_64 debug=n Not tainted ]----
> > (XEN) CPU: 0
> > (XEN) RIP: e008:[<ffff82c4801c0531>]
mem_sharing_unshare_page+0x681/0x790
> > (XEN) RFLAGS: 0000000000010282 CONTEXT: hypervisor
> > (XEN) rax: 0000000000000000 rbx: ffff83040092d808 rcx:
0000000000000096
> > (XEN) rdx: 000000000000000a rsi: 000000000000000a rdi:
ffff82c48021eac4
> > (XEN) rbp: 0000000000000000 rsp: ffff82c48035f5e8 r8: 0000000000000001
> > (XEN) r9: 0000000000000001 r10: 00000000fffffff5 r11: 0000000000000008
> > (XEN) r12: ffff8305c61f3980 r13: ffff83040eff0000 r14:
000000000001610f
> > (XEN) r15: ffff82c48035f628 cr0: 000000008005003b cr4:
00000000000026f0
> > (XEN) cr3: 000000052bc4f000 cr2: ffff880120126e88
> > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> > (XEN) Xen stack trace from rsp=ffff82c48035f5e8:
> > (XEN) ffff8305c61f3990 00018300bf2f0000 ffff82f604e6a4a0
000000002ab84078
> > (XEN) ffff83040092d7f0 00000000001b9c9c ffff8300bf2f0000
000000010eff0000
> > (XEN) 0000000000000000 0000000000000000 0000000000000000
0000000000000000
> > (XEN) 0000000000000000 0000000d0000010f ffff8305447ec000
000000000001610f
> > (XEN) 0000000000273525 ffff82c48035f724 ffff830502c705a0
ffff82f602c89a00
> > (XEN) ffff83040eff0000 ffff82c48010bfa9 ffff830572c5dbf0
000000000029e07f
> > (XEN) 0000000000000000 ffff830572c5dbf0 000000008035fbe8
ffff82c48035f6f8
> > (XEN) 0000000100000002 ffff830572c5dbf0 ffff83063fc30000
ffff830572c5dbf0
> > (XEN) 0000035900000000 ffff88010d14bbe0 ffff880159e09000
00003f7e00000002
> > (XEN) ffffffffffff0032 ffff88010d14bbb0 ffff830438dfa920
0000000d8010a650
> > (XEN) 0000000000000100 ffff83063fc30000 ffff8305f9203730
ffffffffffffffea
> > (XEN) ffff88010d14bb70 0000000000000000 ffff88010d14bc10
ffff88010d14bbc0
> > (XEN) 0000000000000002 ffff82c48010da9b 0000000000000202
ffff82c48035fec8
> > (XEN) ffff82c48035f7c8 00000000801880af ffff83063fc30010
0000000000000000
> > (XEN) ffff82c400000008 ffff82c48035ff28 0000000000000000
ffff88010d14bbc0
> > (XEN) ffff880159e08000 0000000000000000 0000000000000000
00020000000002d7
> > (XEN) 00000000003f2b38 ffff8305b1f4b6b8 ffff8305b30f0000
ffff880159e09000
> > (XEN) 0000000000000000 0000000000000000 000200000000008a
00000000003ed1f9
> > (XEN) ffff83063fc26450 ffff8305b30f0000 ffff880159e0a000
0000000000000000
> > (XEN) 0000000000000000 00020000000001fa 000000000029e2ba
ffff83063fc26fd0
> > (XEN) Xen call trace:
> > (XEN) [<ffff82c4801c0531>] mem_sharing_unshare_page+0x681/0x790
> > (XEN) [<ffff82c48010bfa9>] gnttab_map_grant_ref+0xbf9/0xe30
> > (XEN) [<ffff82c48010da9b>] do_grant_table_op+0x14b/0x1080
> > (XEN) [<ffff82c48010fb44>] do_xen_version+0xb4/0x480
> > (XEN) [<ffff82c4801b8215>] set_p2m_entry+0x85/0xc0
> > (XEN) [<ffff82c4801bc92e>] set_shared_p2m_entry+0x1be/0x2f0
> > (XEN) [<ffff82c480121c4c>] xmem_pool_free+0x2c/0x310
> > (XEN) [<ffff82c4801bfaf8>] mem_sharing_share_pages+0xd8/0x3d0
> > (XEN) [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> > (XEN) [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
> > (XEN) [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
> > (XEN) [<ffff82c48014717d>] vcpu_kick+0x1d/0x80
> > (XEN) [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> > (XEN) [<ffff82c48015a1d8>] get_page+0x28/0xf0
> > (XEN) [<ffff82c48015ed72>] do_update_descriptor+0x1d2/0x210
> > (XEN) [<ffff82c480113d7e>] do_multicall+0x14e/0x340
> > (XEN) [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
> > (XEN)
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Xen BUG at mem_sharing.c:441
> > (XEN) ****************************************
> > (XEN)
> > (XEN) Manual reset required (''noreboot'' specified)
> > 
> > > Date: Mon, 17 Jan 2011 17:02:02 +0800
> > > Subject: Re: [PATCH] mem_sharing: fix race condition of nominate
and unshare
> > > From: juihaochiang@gmail.com
> > > To: tinnycloud@hotmail.com
> > > CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> > >
> > > Hi, tinnycloud:
> > >
> > > Do you have xenpaging tools running properly?
> > > I haven''t gone through that one, but it seems you have
run out of memory.
> > > When this case happens, mem_sharing will request memory to the
> > > xenpaging daemon, which tends to page out and free some memory.
> > > Otherwise, the allocation would fail.
> > > Is this your scenario?
> > >
> > > Bests,
> > > Jui-Hao
> > >
> > > 2011/1/17 MaoXiaoyun <tinnycloud@hotmail.com>:
> > > > Another failure on BUG() in mem_sharing_alloc_page()
> > > >
> > > > memset(&req, 0, sizeof(req));
> > > > if(must_succeed)
> > > > {
> > > > /* We do not support ''must_succeed'' any
more. External operations
> > > > such
> > > > * as grant table mappings may fail with OOM condition!
> > > > */
> > > > BUG();===================>bug here
> > > > }
> > > > else
> > > > {
> > > > /* All foreign attempts to unshare pages should be handled
through
> > > > * ''must_succeed'' case. */
> > > > ASSERT(v->domain->domain_id == d->domain_id);
> > > > vcpu_pause_nosync(v);
> > > > req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
> > > > }
> > > >
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

it is later found that domain is dying, so when dying alloc page is prohibitted
 
 (XEN) ---domain is 1, max_pages 132096, total_pages 29736
 
output log from line 914
 
 909     old_page = page;
 910     page = mem_sharing_alloc_page(d, gfn, flags &
MEM_SHARING_MUST_SUCCEED);
 911     if(!page)
 912     {
 913         mem_sharing_debug_gfn(d, gfn);
 914         printk("---domain is %d, max_pages %u, total_pages %u
\n", d->is_dying, d->max_pages, d->tot_pages);
 915         BUG_ON(!d);
 
--------------
Well the logic is a bit of complicate, my fix is to set gfn''s mfn to 
INVALID_MFN
 
 
 876     ret = page_make_private(d, page);
 877     /*last_gfn shoule able to be make_private*/
 878     BUG_ON(last_gfn & ret);
 879     if(ret == 0) goto private_page_found;
 880 
 881     ld = rcu_lock_domain_by_id(d->domain_id);
 882     BUG_ON(!ld);
 883     if(ld->is_dying )
 884     {
 885         if(!ld)
 886             printk("d is NULL %d\n", d->domain_id);
 887         else
 888             printk("d is dying %d %d\n", d->is_dying,
d->domain_id);
 889 
 890         /*decrease page type count and destory gfn*/
 891         put_page_and_type(page);
 892         mem_sharing_gfn_destroy(gfn_info, !last_gfn);
 893 
 894         if(last_gfn) 
 895             mem_sharing_hash_delete(handle);
 896         else 
 897             /* Even though we don''t allocate a private page, we
have to account
 898              * for the MFN that originally backed this PFN. */
 899             atomic_dec(&nr_saved_mfns);
 900 
 901         /*set mfn invalid*/
 902         BUG_ON(set_shared_p2m_entry_invalid(d, gfn)==0);
 903         if(ld)
 904           rcu_unlock_domain(ld);
 905         shr_unlock();
 906         return 0;
 907     }
 
Any other suggestions?
 
 

> Date: Thu, 20 Jan 2011 09:19:34 +0000
> From: Tim.Deegan@citrix.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; juihaochiang@gmail.com
> Subject: Re: [PATCH] mem_sharing: fix race condition of nominate and
unshare
> 
> At 07:19 +0000 on 20 Jan (1295507976), MaoXiaoyun wrote:
> > Hi:
> > 
> > The latest BUG in mem_sharing_alloc_page from
mem_sharing_unshare_page.
> > I printed heap info, which shows plenty memory left.
> > Could domain be NULL during in unshare, or should it be locked by
rcu_lock_domain_by_id ?
> > 
> 
> ''d'' probably isn''t NULL; more likely is that the
domain is not allowed
> to have any more memory. You should look at the values of d->max_pages
> and d->tot_pages when the failure happens.
> 
> Cheers.
> 
> Tim.
> 
> > -----------code------------
> > 422 extern void pagealloc_info(unsigned char key);
> > 423 static struct page_info* mem_sharing_alloc_page(struct domain *d,
> > 424 unsigned long gfn,
> > 425 int must_succeed)
> > 426 {
> > 427 struct page_info* page;
> > 428 struct vcpu *v = current;
> > 429 mem_event_request_t req;
> > 430
> > 431 page = alloc_domheap_page(d, 0);
> > 432 if(page != NULL) return page;
> > 433
> > 434 memset(&req, 0, sizeof(req));
> > 435 if(must_succeed)
> > 436 {
> > 437 /* We do not support ''must_succeed'' any more.
External operations such
> > 438 * as grant table mappings may fail with OOM condition!
> > 439 */
> > 440 pagealloc_info(''m'');
> > 441 BUG();
> > 442 }
> > 
> > -------------serial output-------
> > (XEN) Physical memory information:
> > (XEN) Xen heap: 0kB free
> > (XEN) heap[14]: 64480kB free
> > (XEN) heap[15]: 131072kB free
> > (XEN) heap[16]: 262144kB free
> > (XEN) heap[17]: 524288kB free
> > (XEN) heap[18]: 1048576kB free
> > (XEN) heap[19]: 1037128kB free
> > (XEN) heap[20]: 3035744kB free
> > (XEN) heap[21]: 2610292kB free
> > (XEN) heap[22]: 2866212kB free
> > (XEN) Dom heap: 11579936kB free
> > (XEN) Xen BUG at mem_sharing.c:441
> > (XEN) ----[ Xen-4.0.0 x86_64 debug=n Not tainted ]----
> > (XEN) CPU: 0
> > (XEN) RIP: e008:[<ffff82c4801c0531>]
mem_sharing_unshare_page+0x681/0x790
> > (XEN) RFLAGS: 0000000000010282 CONTEXT: hypervisor
> > (XEN) rax: 0000000000000000 rbx: ffff83040092d808 rcx:
0000000000000096
> > (XEN) rdx: 000000000000000a rsi: 000000000000000a rdi:
ffff82c48021eac4
> > (XEN) rbp: 0000000000000000 rsp: ffff82c48035f5e8 r8: 0000000000000001
> > (XEN) r9: 0000000000000001 r10: 00000000fffffff5 r11: 0000000000000008
> > (XEN) r12: ffff8305c61f3980 r13: ffff83040eff0000 r14:
000000000001610f
> > (XEN) r15: ffff82c48035f628 cr0: 000000008005003b cr4:
00000000000026f0
> > (XEN) cr3: 000000052bc4f000 cr2: ffff880120126e88
> > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> > (XEN) Xen stack trace from rsp=ffff82c48035f5e8:
> > (XEN) ffff8305c61f3990 00018300bf2f0000 ffff82f604e6a4a0
000000002ab84078
> > (XEN) ffff83040092d7f0 00000000001b9c9c ffff8300bf2f0000
000000010eff0000
> > (XEN) 0000000000000000 0000000000000000 0000000000000000
0000000000000000
> > (XEN) 0000000000000000 0000000d0000010f ffff8305447ec000
000000000001610f
> > (XEN) 0000000000273525 ffff82c48035f724 ffff830502c705a0
ffff82f602c89a00
> > (XEN) ffff83040eff0000 ffff82c48010bfa9 ffff830572c5dbf0
000000000029e07f
> > (XEN) 0000000000000000 ffff830572c5dbf0 000000008035fbe8
ffff82c48035f6f8
> > (XEN) 0000000100000002 ffff830572c5dbf0 ffff83063fc30000
ffff830572c5dbf0
> > (XEN) 0000035900000000 ffff88010d14bbe0 ffff880159e09000
00003f7e00000002
> > (XEN) ffffffffffff0032 ffff88010d14bbb0 ffff830438dfa920
0000000d8010a650
> > (XEN) 0000000000000100 ffff83063fc30000 ffff8305f9203730
ffffffffffffffea
> > (XEN) ffff88010d14bb70 0000000000000000 ffff88010d14bc10
ffff88010d14bbc0
> > (XEN) 0000000000000002 ffff82c48010da9b 0000000000000202
ffff82c48035fec8
> > (XEN) ffff82c48035f7c8 00000000801880af ffff83063fc30010
0000000000000000
> > (XEN) ffff82c400000008 ffff82c48035ff28 0000000000000000
ffff88010d14bbc0
> > (XEN) ffff880159e08000 0000000000000000 0000000000000000
00020000000002d7
> > (XEN) 00000000003f2b38 ffff8305b1f4b6b8 ffff8305b30f0000
ffff880159e09000
> > (XEN) 0000000000000000 0000000000000000 000200000000008a
00000000003ed1f9
> > (XEN) ffff83063fc26450 ffff8305b30f0000 ffff880159e0a000
0000000000000000
> > (XEN) 0000000000000000 00020000000001fa 000000000029e2ba
ffff83063fc26fd0
> > (XEN) Xen call trace:
> > (XEN) [<ffff82c4801c0531>] mem_sharing_unshare_page+0x681/0x790
> > (XEN) [<ffff82c48010bfa9>] gnttab_map_grant_ref+0xbf9/0xe30
> > (XEN) [<ffff82c48010da9b>] do_grant_table_op+0x14b/0x1080
> > (XEN) [<ffff82c48010fb44>] do_xen_version+0xb4/0x480
> > (XEN) [<ffff82c4801b8215>] set_p2m_entry+0x85/0xc0
> > (XEN) [<ffff82c4801bc92e>] set_shared_p2m_entry+0x1be/0x2f0
> > (XEN) [<ffff82c480121c4c>] xmem_pool_free+0x2c/0x310
> > (XEN) [<ffff82c4801bfaf8>] mem_sharing_share_pages+0xd8/0x3d0
> > (XEN) [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> > (XEN) [<ffff82c48011c519>] cpumask_raise_softirq+0x89/0xa0
> > (XEN) [<ffff82c480118351>] csched_vcpu_wake+0x101/0x1b0
> > (XEN) [<ffff82c48014717d>] vcpu_kick+0x1d/0x80
> > (XEN) [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
> > (XEN) [<ffff82c48015a1d8>] get_page+0x28/0xf0
> > (XEN) [<ffff82c48015ed72>] do_update_descriptor+0x1d2/0x210
> > (XEN) [<ffff82c480113d7e>] do_multicall+0x14e/0x340
> > (XEN) [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
> > (XEN)
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Xen BUG at mem_sharing.c:441
> > (XEN) ****************************************
> > (XEN)
> > (XEN) Manual reset required (''noreboot'' specified)
> > 
> > > Date: Mon, 17 Jan 2011 17:02:02 +0800
> > > Subject: Re: [PATCH] mem_sharing: fix race condition of nominate
and unshare
> > > From: juihaochiang@gmail.com
> > > To: tinnycloud@hotmail.com
> > > CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com
> > >
> > > Hi, tinnycloud:
> > >
> > > Do you have xenpaging tools running properly?
> > > I haven''t gone through that one, but it seems you have
run out of memory.
> > > When this case happens, mem_sharing will request memory to the
> > > xenpaging daemon, which tends to page out and free some memory.
> > > Otherwise, the allocation would fail.
> > > Is this your scenario?
> > >
> > > Bests,
> > > Jui-Hao
> > >
> > > 2011/1/17 MaoXiaoyun <tinnycloud@hotmail.com>:
> > > > Another failure on BUG() in mem_sharing_alloc_page()
> > > >
> > > > memset(&req, 0, sizeof(req));
> > > > if(must_succeed)
> > > > {
> > > > /* We do not support ''must_succeed'' any
more. External operations
> > > > such
> > > > * as grant table mappings may fail with OOM condition!
> > > > */
> > > > BUG();===================>bug here
> > > > }
> > > > else
> > > > {
> > > > /* All foreign attempts to unshare pages should be handled
through
> > > > * ''must_succeed'' case. */
> > > > ASSERT(v->domain->domain_id == d->domain_id);
> > > > vcpu_pause_nosync(v);
> > > > req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
> > > > }
> > > >
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Xen Platform Team
> Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi George:
 
       Beside the p2m_is_shared() need in p2m_pod_zero_check(), 
In mem_sharing_nominate_page() line 524, there is a page type check,
and line 566, page handle is set.
       So it looks like exists a competition on page type, 
mem_sharing_nominate_page()
may first find page is sharable and POD at same time put the page into it cache,
and later mem_sharing_nominate_page() set page handle, thus damage the page list
so we need to put mem_sharing_nominate_page into p2m_lock protect, right?
 
   491int mem_sharing_nominate_page(struct p2m_domain *p2m, 

   492                              unsigned long gfn,

   493                              int expected_refcnt,

   494                              shr_handle_t *phandle)

   495{

   496    p2m_type_t p2mt;

   497    mfn_t mfn;

   498    struct page_info *page;

   499    int ret;

   500    shr_handle_t handle;

   501    shr_hash_entry_t *hash_entry;

   502    struct gfn_info *gfn_info;

   503    struct domain *d = p2m->domain;

   504

   505    *phandle = 0UL;

   506

   507    shr_lock(); 

   508    mfn = gfn_to_mfn(p2m, gfn, &p2mt);

   509

   510    /* Check if mfn is valid */

   511    ret = -EINVAL;

   512    if (!mfn_valid(mfn))

   513        goto out;

   514

   515    /* Return the handle if the page is already shared */

   516    page = mfn_to_page(mfn);

   517    if (p2m_is_shared(p2mt)) {

   518        *phandle = page->shr_handle;

   519        ret = 0;

   520        goto out;

   521    }

   522

   523    /* Check p2m type */

   524    if (!p2m_is_sharable(p2mt))

   525        goto out;

   526

   527    /* Try to convert the mfn to the sharable type */

   528    ret = page_make_sharable(d, page, expected_refcnt); 

   529    if(ret) 

   530        goto out;

   531

   532    /* Create the handle */

   533    ret = -ENOMEM;

   534    handle = next_handle++;  

   535    if((hash_entry = mem_sharing_hash_insert(handle, mfn)) == NULL)

   536    {

   537        goto out;

   538    }

   539    if((gfn_info = mem_sharing_gfn_alloc()) == NULL)

   540    {

   541        mem_sharing_hash_destroy(hash_entry);

   542        goto out;

   543    }

   544

   545    /* Change the p2m type */

   546    if(p2m_change_type(p2m, gfn, p2mt, p2m_ram_shared) != p2mt) 

   547    {

   548        /* This is unlikely, as the type must have changed since
we''ve checked

   549         * it a few lines above.

   550         * The mfn needs to revert back to rw type. This should never
fail,

   551         * since no-one knew that the mfn was temporarily sharable */

   552        BUG_ON(page_make_private(d, page) != 0);

   553        mem_sharing_hash_destroy(hash_entry);

   554        mem_sharing_gfn_destroy(gfn_info, 0);

   555        goto out;

   556    }

   557

   558    /* Update m2p entry to SHARED_M2P_ENTRY */

   559    set_gpfn_from_mfn(mfn_x(mfn), SHARED_M2P_ENTRY);

   560

   561    INIT_LIST_HEAD(&hash_entry->gfns);

   562    INIT_LIST_HEAD(&gfn_info->list);

   563    list_add(&gfn_info->list, &hash_entry->gfns);

   564    gfn_info->gfn = gfn;

   565    gfn_info->domain = d->domain_id;

   566    page->shr_handle = handle;

   567    *phandle = handle;

   568

   569    ret = 0;

   570

   571out:

   572    shr_unlock();

   573    return ret;

   574}


 
> Date: Wed, 19 Jan 2011 09:11:37 +0000
> Subject: Re: [Xen-devel] RE: [PATCH] mem_sharing: fix race condition of
nominate and unshare
> From: George.Dunlap@eu.citrix.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com;
juihaochiang@gmail.com
> 
> Very likely. If you look in xen/arch/x86/mm/p2m.c, the two functions
> which check a page to see if it can be reclaimed are
> "p2m_pod_zero_check*()". A little ways into each function
there''s a
> giant "if()" which has all of the conditions for reclaiming a
page,
> starting with p2m_is_ram(). The easiest way to fix it is to add
> p2m_is_shared() to that "if" statement.
> 
> -George
> 
> 2011/1/19 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi George:
> >
> >        I am working on the xen mem_sharing,  I think the bug below is
> > related to POD.
> > (Test shows when POD is enable, it is easily hit the bug, when
disabled, no
> > bug occurs).
> >
> > As I know when domU starts will POD, it gets memory from POD cache,
and in
> > some
> > situation, POD cached will scan from Zero pages for reusing(link the
page
> > into POD
> > cache page list), and from the page_info define, list and handle share
same
> > posistion,
> > I think when reusing the page, POD doest''t check page type,
and if it is a
> > shared page
> > , it still can be put into POD cache, and thus handle is been
overwritten.
> >
> >       So maybe we need to check the page type before putting into
cache,
> > What''s your opinion?
> >       thanks.
> >
>
>>--------------------------------------------------------------------------------
> >>From: tinnycloud@hotmail.com
> >>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
> >>CC: tim.deegan@citrix.com
> >>Subject: RE: [PATCH] mem_sharing: fix race condition of nominate
and
> >> unshare
> >>Date: Tue, 18 Jan 2011 20:05:16 +0800
> >>
> >>Hi:
> >>
> >> It is later found that caused by below patch code and I am using
the
> >> blktap2.
> >>The handle retruned from here will later become ch in
> >> mem_sharing_share_pages, and then
> >>in mem_sharing_share_pages will have ch = sh, thus caused the
problem.
> >>
> >>+    /* Return the handle if the page is already shared */
> >>+    page = mfn_to_page(mfn);
> >>+    if (p2m_is_shared(p2mt)) {
> >>+        *phandle = page->shr_handle;
> >>+        ret = 0;
> >>+        goto out;
> >>+    }
> >>+
> >>
> >>But. after I  removed the code, tests still failed, and this
handle''s value
> >> is not make sence.
> >>
> >>
> >>(XEN) ===>total handles 17834 total gfns 255853
> >>(XEN) handle 13856642536914634
> >>(XEN) Debug for domain=1, gfn=19fed, Debug page: MFN=349c0a is
> >> ci=8000000000000008, ti=8400000000000007, owner_id=32755
> >>(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
> >>(XEN) CPU:    15
> >>(XEN) RIP:    e008:[<ffff82c4801bff4b>]
> >> mem_sharing_unshare_page+0x19b/0x720
> >>(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
> >>(XEN) rax: 0000000000000000   rbx: ffff83063fc67f28&nbsp ; 
rcx:
> >> 0000000000000092
> >>(XEN) rdx: 000000000000000a   rsi: 000000000000000a   rdi:
ffff82c48021e9c4
> >>(XEN) rbp: ffff830440000000   rsp: ffff83063fc67c48   r8: 
0000000000000001
> >>(XEN) r9:  0000000000000000   r10: 00000000fffffff8   r11:
0000000000000005
> >>(XEN) r12: 0000000000019fed   r13: 0000000000000000   r14:
0000000000000000
> >>(XEN) r15: ffff82f606938140   cr0: 000000008005003b   cr4:
00000000000026f0
> >>(XEN) cr3: 000000055513c000   cr2: 0000000000000018
> >>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs:
e008
> >>(XEN) Xen stack trace from rsp=ffff83063fc67c48:
> >>(XEN)    02c5f6c8b70fed66 39ef64058b487674 ffff82c4801a6082
> >> 0000000000000000
> >>(XEN)    00313a8b00313eca 0000000000000001 0000000000000009 ff
> >> ff830440000000
> >>(XEN)    ffff83063fc67cb8 ffff82c4801df6f9 0000000000000040
> >> ffff83063fc67d04
> >>(XEN)    0000000000019fed 0000000d000001ed ffff83055458d000
> >> ffff83063fc67f28
> >>(XEN)    0000000000019fed 0000000000349c0a 0000000000000030
> >> ffff83063fc67f28
> >>(XEN)    0000000000000030 ffff82c48019baa6 ffff82c4802519c0
> >> 0000000d8016838e
> >>(XEN)    0000000000000000 00000000000001aa ffff8300bf554000
> >> ffff82c4801b3864
> >>(XEN)    ffff830440000348 ffff8300bf554000 ffff8300bf5557f0
> >> ffff8300bf5557e8
> >>(XEN)    00000032027b81f2 ffff82c48026f080 ffff82c4801a9337
> >> ffff8300bf448000
> >>(XEN)    ffff8300bf554000 ffff830000000000 0000000019fed000
> >> ffff8300bf2f2000
> >>(XEN)    ffff82c48019985d 0000000000000080 ffff8300bf554000
> >> 0000000000019fed
> >>(XEN)    ffff82c4801b08ba 000000000001e000 ffff82c48014931f ff
> >> ff8305570c6d50
> >>(XEN)    ffff82c480251080 00000032027b81f2 ffff8305570c6d50
> >> ffff83052f3e2200
> >>(XEN)    0000000f027b7de0 ffff82c48011e07a 000000000000000f
> >> ffff82c48026f0a0
> >>(XEN)    0000000000000082 0000000000000000 0000000000000000
> >> 0000000000009e44
> >>(XEN)    ffff8300bf554000 ffff8300bf2f2000 ffff82c48011e07a
> >> 000000000000000f
> >>(XEN)    ffff8300bf555760 0000000000000292 ffff82c48011afca
> >> 00000032028a8fc0
> >>(XEN)    0000000000000292 ffff82c4801a93c3 00000000000000ef
> >> ffff8300bf554000
> >>(XEN)    ffff8300bf554000 ffff8300bf5557e8 ffff82c4801a6082
> >> ffff8300bf554000
> >>(XEN)    0000000000000000 ffff82c4801a0cc8 ffff8300bf554000
> >> ffff8300bf554000
> >>(XEN) Xen call trace:
> >>(XEN)    [<ffff82c4801bff4b>]
mem_sharing_unshare_page+0x19b/0x720
> >>(XEN)    [<ffff82c4801a6082>] v
lapic_has_pending_irq+0x42/0x70
> >>(XEN)    [<ffff82c4801df6f9>] ept_get_entry+0xa9/0x1c0
> >>(XEN)    [<ffff82c48019baa6>]
hvm_hap_nested_page_fault+0xd6/0x190
> >>(XEN)    [<ffff82c4801b3864>] vmx_vmexit_handler+0x304/0x1a90
> >>(XEN)    [<ffff82c4801a9337>] pt_restore_timer+0x57/0xb0
> >>(XEN)    [<ffff82c48019985d>] hvm_do_resume+0x1d/0x130
> >>(XEN)    [<ffff82c4801b08ba>] vmx_do_resume+0x11a/0x1c0
> >>(XEN)    [<ffff82c48014931f>] context_switch+0x76f/0xf00
> >>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
> >>(XEN)    [<ffff82c48011e07a>] add_entry+0x3a/0xb0
> >>(XEN)    [<ffff82c48011afca>] schedule+0x1ea/0x500
> >>(XEN)    [<ffff82c4801a93c3>] pt_update_irq+0x33/0x1e0
> >>(XEN)    [&lt ;ffff82c4801a6082>]
vlapic_has_pending_irq+0x42/0x70
> >>(XEN)    [<ffff82c4801a0cc8>]
hvm_vcpu_has_pending_irq+0x88/0xa0
> >>(XEN)    [<ffff82c4801b267b>] vmx_vmenter_helper+0x5b/0x150
> >>(XEN)    [<ffff82c4801adaa3>] vmx_asm_do_vmentry+0x0/0xdd
> >>(XEN)
> >>(XEN) Pagetable walk from 0000000000000018:
> >>(XEN)  L4[0x000] = 0000000000000000 ffffffffffffffff
> >>(XEN)
> >>(XEN) ****************************************
> >>(XEN) Panic on CPU 15:
> >>(XEN) FATAL PAGE FAULT
> >>(XEN) [error_code=0000]
> >>(XEN) Faulting linear address: 0000000000000018
> >>(XEN) ****************************************
> >>(XEN)
> >>(XEN) Manual reset required (''noreboot''
specified)
> >>
> >>
> >>
> >>
> >>
> >>
---------------------------------------------------------------------------------------------------
> >>>From: tinnycloud@hotmail.com
> >>>To: juihaochiang@gmail.com; xen-devel@lists.xensource.com
> >>>CC: tim.deegan@citrix.com
> >>>Subject: RE: [PATCH] mem_sharing: fix race condition of
nominate and
> >>> unshare
> >>>Date: Tue, 18 Jan 2011 17:42:32 +0800
> >>
> >>>Hi Tim & Jui-Hao:
> >>
> >> >     When I use Linux HVM instead of Windows HVM, more bug
shows up.
> >>
> >>>      I only start on VM, and when I destroy it , xen crashed
on
> >>> mem_sharing_unshare_page()
> >>>which in line709, hash_entry is NULL. Later I found the handle
has been
> >>> removed in
> >>>mem_sharing_share_pages(), please refer logs below.
> >>
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> >
 		 	   		  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi George:

 

         We have observed abnormal CPU utilization behavior on our HVMS.

 

         There are two same physical hosts, with exactly same Xen and dom0
kernel installed,

The host owns 24G memory and 16 CPUS. 

On each of the host, a HVM is running.

(WIN2008, 16G MEM, 8VCPUS, derive from same base VHD image)

 

That is  we have same HVMA, HVMB on different host.

 

Some days after, HVMA behaves quite abnormal, every time I start an
application inside VM, 

its CPU utilization  grows and drops very quickly(up to 80%), so it feels
that the system is very lag. 

While, HVMB works fine.

 

So it makes me to guess that processes on HVMA consume more CPU than HVMB.

 

Later, I run superPI tests on both HVMS, calculate 320million PI.

Here is the result.

 

                                                       HVMA
HVMB

End of initialization. Time:  2min, 54 Sec.                12sec

         1st time completion:            5min, 05 Sec
49 sec

         2nd time completion:            6min, 14 Sec
1min, 32sec

          3rd time completion:            7min,  17Sec
2min 14 sec

         4th time completion :            8min  22, sec
2min 57 

         5th time completion :            9min , 48 sec
3min 40

                          ...                              ..
..

          24th time completion             29min 52
16min 59

         

 

         From the data above, the initialization  time of superPI for HVMA
is much slower than HVMB.

         And every calculation of PI for HVMA is a bit slower than HVMB.

 

         How could this happen, could you enlighten me some?

         Many thanks.

 

 

 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel