Xen devel - Apr 2013 - [GIT PULL] (xen) stable/for-jens-3.10

Hey Jens,

Please in your spare time (if there is such a thing at a conference)
pull this branch:

 git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10

for your v3.10 branch. Sorry for being so late with this.

<blurb>
It has the ''feature-max-indirect-segments'' implemented in both
backend
and frontend. The current problem with the backend and frontend is that the
segment size is limited to 11 pages. It means we can at most squeeze in 44kB per
request. The ring can hold 32 (next power of two below 36) requests, meaning we
can do 1.4M of outstanding requests. Nowadays that is not enough.

The problem in the past was addressed in two ways - but neither one went
upstream.
The first solution to this proposed by Justin from Spectralogic was to negotiate
the segment size.  This means that the ‘struct blkif_sring_entry’ is now a
variable size.
It can expand from 112 bytes (cover 11 pages of data - 44kB) to 1580 bytes
(256 pages of data - so 1MB). It is a simple extension by just making the array
in the
request expand from 11 to a variable size negotiated. But it had limits: this
extension
still limits the number of segments per request to 255 (as the total number must
be
specified in the request, which only has an 8-bit field for that purpose).

The other solution (from Intel - Ronghui) was to create one extra ring that only
has the
‘struct blkif_request_segment’ in them. The ‘struct blkif_request’ would be
changed to have
an index in said ‘segment ring’. There is only one segment ring. This means that
the size of
the initial ring is still the same. The requests would point to the segment and
enumerate out
how many of the indexes it wants to use. The limit is of course the size of the
segment.
If one assumes a one-page segment this means we can in one request cover ~4MB.
Those patches were posted as RFC and the author never followed up on the ideas
on changing
it to be a bit more flexible.

There is yet another mechanism that could be employed  (which these patches
implement) - and it
borrows from VirtIO protocol. And that is the ‘indirect descriptors’. This very
similar to
what Intel suggests, but with a twist. The twist is to negotiate how many of
these
''segment'' pages (aka indirect descriptor pages) we want to
support (in reality we negotiate
how many entries in the segment we want to cover, and we module the number if it
is
bigger than the segment size).

This means that with the existing 36 slots in the ring (single page) we can
cover:
32 slots * each blkif_request_indirect covers: 512 * 4096 ~= 64M. Since we ample
space
in the blkif_request_indirect to span more than one indirect page, that number
(64M)
can be also multiplied by eight = 512MB. 

Roger Pau Monne took the idea and implemented them in these patches. They work
great and the corner cases (migration between backends with and without this
extension)
work nicely. The backend has a limit right now off how many indirect entries
it can handle: one indirect page, and at maximum 256 entries (out of 512 - so 
50% of the page
is used). That comes out to 32 slots * 256 entries in a indirect page * 1
indirect page
per request * 4096 = 32MB.

This is a conservative number that can change in the future. Right now it
strikes
a good balance between giving excellent performance, memory usage in the
backend, and
balancing the needs of many guests.

In the patchset there is also the split of the blkback structure to be per-VBD.
This means that the spinlock contention we had with many guests trying to do I/O
and
all the blkback threads hitting the same lock has been eliminated.
</blurb>

Anyhow, please pull and if possible include the nice overview I typed up in the
merge commit.

 Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
 drivers/block/xen-blkback/blkback.c            | 843 ++++++++++++++++---------
 drivers/block/xen-blkback/common.h             | 145 ++++-
 drivers/block/xen-blkback/xenbus.c             |  38 ++
 drivers/block/xen-blkfront.c                   | 490 +++++++++++---
 include/xen/interface/io/blkif.h               |  53 ++
 6 files changed, 1188 insertions(+), 399 deletions(-)

Roger Pau Monne (7):
      xen-blkback: print stats about persistent grants
      xen-blkback: use balloon pages for all mappings
      xen-blkback: implement LRU mechanism for persistent grants
      xen-blkback: move pending handles list from blkbk to pending_req
      xen-blkback: make the queue of free requests per backend
      xen-blkback: expand map/unmap functions
      xen-block: implement indirect descriptors

Friday, April 19, 2013, 4:44:01 PM, you wrote:
> Hey Jens,
> Please in your spare time (if there is such a thing at a conference)
> pull this branch:
>  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
> for your v3.10 branch. Sorry for being so late with this.
<big snip></big snip>
> Anyhow, please pull and if possible include the nice overview I typed up in
the
> merge commit.
>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>  drivers/block/xen-blkback/common.h             | 145 ++++-
>  drivers/block/xen-blkback/xenbus.c             |  38 ++
>  drivers/block/xen-blkfront.c                   | 490 +++++++++++---
>  include/xen/interface/io/blkif.h               |  53 ++
>  6 files changed, 1188 insertions(+), 399 deletions(-)
> Roger Pau Monne (7):
>       xen-blkback: print stats about persistent grants
>       xen-blkback: use balloon pages for all mappings
>       xen-blkback: implement LRU mechanism for persistent grants
>       xen-blkback: move pending handles list from blkbk to pending_req
>       xen-blkback: make the queue of free requests per backend
>       xen-blkback: expand map/unmap functions
>       xen-block: implement indirect descriptors

Hi Konrad / Roger,

I tried this pull on top of latest Linus latest linux-3.9 tree, but although it
seems to boot and work fine at first, i seem to get trouble after running for
about a day.
Without this pull it runs fine for several days.

Trying to start a new guest I ended up with the splat below. In the output of
xl-dmesg i seem to see more of these than before:
(XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1) grant table
from (9) to (10) frames

Attached xl-dmesg and dmesg.

--
Sander


[18496.013743] xenwatch: page allocation failure: order:7, mode:0x10c0d0
[18496.031948] Pid: 54, comm: xenwatch Not tainted 3.9.0-rc8-20130424-jens+ #1
[18496.049897] Call Trace:
[18496.067674]  [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
[18496.085453]  [<ffffffff810b25ed>] ? trace_hardirqs_on+0xd/0x10
[18496.102951]  [<ffffffff810bdb24>] ? on_each_cpu_mask+0x94/0xd0
[18496.120270]  [<ffffffff811028af>] __alloc_pages_nodemask+0x69f/0x960
[18496.137306]  [<ffffffff8113a161>] alloc_pages_current+0xb1/0x160
[18496.154051]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
[18496.170579]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
[18496.186921]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
[18496.202963]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
[18496.218714]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
[18496.234237]  [<ffffffff8156a151>] driver_probe_device+0x81/0x220
[18496.249605]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
[18496.264681]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
[18496.279500]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
[18496.294138]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
[18496.308553]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
[18496.322694]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
[18496.336640]  [<ffffffff81567c80>] device_add+0x650/0x720
[18496.350209]  [<ffffffff81573103>] ? device_pm_sleep_init+0x43/0x70
[18496.363602]  [<ffffffff81567d69>] device_register+0x19/0x20
[18496.376721]  [<ffffffff8147495b>] xenbus_probe_node+0x14b/0x160
[18496.389611]  [<ffffffff815685b4>] ? bus_for_each_dev+0xa4/0xb0
[18496.402298]  [<ffffffff81474b2c>] xenbus_dev_changed+0x1bc/0x1c0
[18496.414732]  [<ffffffff810b67f7>] ? lock_release+0x117/0x260
[18496.426904]  [<ffffffff81474f66>] backend_changed+0x16/0x20
[18496.438835]  [<ffffffff81472f5e>] xenwatch_thread+0x4e/0x150
[18496.450579]  [<ffffffff8108abb0>] ? wake_up_bit+0x40/0x40
[18496.462048]  [<ffffffff81472f10>] ? xs_watch+0x60/0x60
[18496.473286]  [<ffffffff8108a546>] kthread+0xd6/0xe0
[18496.484235]  [<ffffffff8108a470>] ? __init_kthread_worker+0x70/0x70
[18496.494987]  [<ffffffff819979bc>] ret_from_fork+0x7c/0xb0
[18496.505560]  [<ffffffff8108a470>] ? __init_kthread_worker+0x70/0x70
[18496.515949] Mem-Info:
[18496.526267] Node 0 DMA per-cpu:
[18496.536085] CPU    0: hi:    0, btch:   1 usd:   0
[18496.545692] CPU    1: hi:    0, btch:   1 usd:   0
[18496.554949] CPU    2: hi:    0, btch:   1 usd:   0
[18496.563969] CPU    3: hi:    0, btch:   1 usd:   0
[18496.572690] CPU    4: hi:    0, btch:   1 usd:   0
[18496.581067] CPU    5: hi:    0, btch:   1 usd:   0
[18496.589120] Node 0 DMA32 per-cpu:
[18496.596950] CPU    0: hi:  186, btch:  31 usd:  62
[18496.604583] CPU    1: hi:  186, btch:  31 usd:  85
[18496.611890] CPU    2: hi:  186, btch:  31 usd:  63
[18496.618950] CPU    3: hi:  186, btch:  31 usd:  84
[18496.625743] CPU    4: hi:  186, btch:  31 usd: 100
[18496.632205] CPU    5: hi:  186, btch:  31 usd:  32
[18496.638361] active_anon:2681 inactive_anon:11161 isolated_anon:0
[18496.638361]  active_file:30213 inactive_file:148645 isolated_file:0
[18496.638361]  unevictable:540 dirty:509 writeback:0 unstable:0
[18496.638361]  free:4756 slab_reclaimable:10388 slab_unreclaimable:10266
[18496.638361]  mapped:4093 shmem:316 pagetables:1177 bounce:0
[18496.638361]  free_cma:0
[18496.670912] Node 0 DMA free:3864kB min:64kB low:80kB high:96kB
active_anon:0kB inactive_anon:0kB active_file:140kB inactive_file:9440kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:60kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:2168kB slab_unreclaimable:256kB kernel_stack:8kB pagetables:4kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
[18496.686284] lowmem_reserve[]: 0 884 884 884
[18496.691452] Node 0 DMA32 free:15184kB min:3772kB low:4712kB high:5656kB
active_anon:10612kB inactive_anon:44648kB active_file:120756kB
inactive_file:585244kB unevictable:2068kB isolated(anon):0kB isolated(file):0kB
present:1032192kB managed:905896kB mlocked:2068kB dirty:1988kB writeback:0kB
mapped:16232kB shmem:1268kB slab_reclaimable:39384kB slab_unreclaimable:40812kB
kernel_stack:2176kB pagetables:4684kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[18496.714687] lowmem_reserve[]: 0 0 0 0
[18496.720726] Node 0 DMA: 18*4kB (UM) 16*8kB (UM) 15*16kB (UM) 21*32kB (UMR)
19*64kB (UMR) 4*128kB (MR) 4*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB =
3864kB
[18496.727233] Node 0 DMA32: 2014*4kB (UEMR) 549*8kB (UEMR) 39*16kB (EMR)
14*32kB (ER) 10*64kB (R) 4*128kB (R) 2*256kB (R) 0*512kB 0*1024kB 0*2048kB
0*4096kB = 15184kB
[18496.740356] 179631 total pagecache pages
[18496.746941] 0 pages in swap cache
[18496.753528] Swap cache stats: add 142, delete 142, find 67/73
[18496.760194] Free swap  = 2097148kB
[18496.766821] Total swap = 2097148kB
[18496.777073] 262143 pages RAM
[18496.783661] 28027 pages reserved
[18496.790185] 381067 pages shared
[18496.796701] 119219 pages non-shared
[18496.803221] vbd vbd-18-768: 12 creating block interface
[18496.811469] vbd vbd-18-768: 12 xenbus_dev_probe on backend/vbd/18/768
[18496.818901] vbd: probe of vbd-18-768 failed with error -12








#:/mnt/backup# vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
 0  1      0  14116 642816 131760    0    0   297   254  334  383  1  3 89  7

#:/mnt/backup# vmstat -s
       936464 K total memory
       924468 K used memory
       131572 K active memory
       645052 K inactive memory
        11996 K free memory
       123128 K buffer memory
       601596 K swap cache
      2097148 K total swap
            0 K used swap
      2097148 K free swap
        70635 non-nice user cpu ticks
        22910 nice user cpu ticks
       277581 system cpu ticks
      9765363 idle cpu ticks
       769650 IO-wait cpu ticks
          208 IRQ cpu ticks
        43718 softirq cpu ticks
        51929 stolen cpu ticks
     32661673 pages paged in
     27922685 pages paged out
           41 pages swapped in
           93 pages swapped out
     36733683 interrupts
     42159895 CPU context switches
   1366806417 boot time
       252828 forks

#:/mnt/backup# vmstat -m
Cache                       Num  Total   Size  Pages
ext4_groupinfo_4k         23382  23382    216     18
xt_hashlimit                  0      0    136     30
nf_conntrack_ffffffff81ec7b80   1486   1620    400     20
nf_conntrack_expect           0      0    288     28
dm_snap_pending_exception      0      0    104     39
kcopyd_job                    0      0   4176      7
dm_rq_target_io               0      0    416     19
blkif_cache                 100    100    800     20
cfq_io_cq                   204    204    120     34
cfq_queue                   153    153    232     17
bsg_cmd                       0      0    312     26
ceph_cap                      0      0    128     32
ceph_inode_info               0      0   2032     16
gfs2_mblk                     0      0    432     18
gfs2_bufdata              11048  11224     88     46
gfs2_inode                    0      0   1280     25
gfs2_glock(aspace)            0      0    960     17
gfs2_glock                    0      0    592     27
btrfs_delayed_data_ref        0      0     96     42
btrfs_delayed_ref_head        0      0    232     17
btrfs_delayed_node            0      0    392     20
btrfs_ordered_extent          0      0    464     17
btrfs_extent_buffer           0      0    584     28
btrfs_delalloc_work           0      0    184     22
btrfs_path                    0      0    144     28
btrfs_transaction             0      0    656     24
btrfs_trans_handle            0      0    160     25
btrfs_inode                   0      0   1944     16
fuse_request                  0      0    472     17
fuse_inode                    0      0   1152     28
ntfs_big_inode_cache          0      0   1664     19
ntfs_inode_cache              0      0    696     23
cifs_small_rq                36     36    448     18
cifs_request                  4      4  16512      1
cifs_inode_cache              0      0   1104     29
isofs_inode_cache             0      0    960     17
fat_inode_cache               0      0   1168     28
fat_cache                     0      0     40    102
hugetlbfs_inode_cache        16     16    976     16
jbd2_transaction_s          375    375    320     25
jbd2_journal_handle         336    336     72     56
journal_handle                0      0     56     73
journal_head               1296   1296    112     36
revoke_record               384    384     32    128
ext4_inode_cache           4598   8137   1696     19
ext4_free_data             1408   1408     64     64
ext4_allocation_context     180    180    136     30
ext4_prealloc_space         156    156    152     26
ext4_io_end                 390    420   1088     30
ext4_io_page               4096   4096     16    256
ext4_extent_status         8576   8772     40    102
ext3_inode_cache              0      0   1312     24
ext3_xattr                    0      0     88     46
dquot                         0      0    384     21
dio                          92     92    704     23
pid_namespace                 0      0   2200     14
posix_timers_cache          116    116    280     29
UNIX                        176    176   1472     22
UDP-Lite                      0      0   1280     25
ip_fib_trie                 292    292     56     73
Cache                       Num  Total   Size  Pages
RAW                         156    156   1216     26
UDP                         150    150   1280     25
tw_sock_TCP                 378    378    192     21
TCP                         117    117   2368     13
fscache_cookie_jar            0      0    184     22
sgpool-128                   68     90   5120      6
sgpool-64                    72     72   2560     12
sgpool-32                   175    175   1280     25
sgpool-16                   275    275    640     25
blkdev_integrity              0      0    112     36
blkdev_queue                143    143   2808     11
blkdev_requests             346    346    376     21
fsnotify_event_holder         0      0     24    170
bip-256                       7      7   4224      7
bip-128                       0      0   2176     15
bip-16                        0      0    384     21
sock_inode_cache            323    323    960     17
file_lock_cache             102    102    240     17
net_namespace                 0      0   1984     16
shmem_inode_cache          1736   1736   1144     28
Acpi-State                  306    306     80     51
Acpi-Namespace             1632   1632     40    102
task_delay_info             960    960    168     24
taskstats                   144    144    328     24
proc_inode_cache           1440   1440    976     16
sigqueue                    300    300    160     25
bdev_cache                  144    144   1344     24
sysfs_dir_cache           22512  22512    144     28
filp                       2134   2450    320     25
inode_cache                1260   1972    912     17
dentry                     7006  11888    248     16
buffer_head              104786 107328    104     39
vm_area_struct             4131   4444    184     22
mm_struct                   280    280   1152     28
files_cache                 273    273    768     21
signal_cache                483    529   1408     23
sighand_cache               387    406   2240     14
task_struct                 297    329   4208      7
anon_vma                   4147   4536    144     28
shared_policy_node          510    510     48     85
numa_policy                3360   3360     72     56
radix_tree_node            5470   5740    568     28
idr_layer_cache             330    330   2112     15
dma-kmalloc-8192              0      0   8192      4
dma-kmalloc-4096              0      0   4096      8
dma-kmalloc-2048              0      0   2048     16
dma-kmalloc-1024              0      0   1024     16
dma-kmalloc-512               0      0    512     16
dma-kmalloc-256               0      0    256     16
dma-kmalloc-128               0      0    128     32
dma-kmalloc-64                0      0     64     64
dma-kmalloc-32                0      0     32    128
dma-kmalloc-16                0      0     16    256
dma-kmalloc-8                 0      0      8    512
dma-kmalloc-192               0      0    192     21
dma-kmalloc-96                0      0     96     42
kmalloc-8192                 28     28   8192      4
kmalloc-4096                909    928   4096      8
kmalloc-2048                646    704   2048     16
kmalloc-1024               1803   1808   1024     16
Cache                       Num  Total   Size  Pages
kmalloc-512                1057   1152    512     16
kmalloc-256                4175   4304    256     16
kmalloc-128                2948   3456    128     32
kmalloc-64                13726  15040     64     64
kmalloc-32                 1536   1536     32    128
kmalloc-16                 4096   4096     16    256
kmalloc-8                  7168   7168      8    512
kmalloc-192               68687  69111    192     21
kmalloc-96                 3704   4032     96     42
kmem_cache_node             192    192    128     32
kmem_cache                  160    160    256     16


# vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
 0  1      0  14116 642816 131760    0    0   297   254  334  383  1  3 89  7

# vmstat -s
       936464 K total memory
       924468 K used memory
       131572 K active memory
       645052 K inactive memory
        11996 K free memory
       123128 K buffer memory
       601596 K swap cache
      2097148 K total swap
            0 K used swap
      2097148 K free swap
        70635 non-nice user cpu ticks
        22910 nice user cpu ticks
       277581 system cpu ticks
      9765363 idle cpu ticks
       769650 IO-wait cpu ticks
          208 IRQ cpu ticks
        43718 softirq cpu ticks
        51929 stolen cpu ticks
     32661673 pages paged in
     27922685 pages paged out
           41 pages swapped in
           93 pages swapped out
     36733683 interrupts
     42159895 CPU context switches
   1366806417 boot time
       252828 forks

# vmstat -m
Cache                       Num  Total   Size  Pages
ext4_groupinfo_4k         23382  23382    216     18
xt_hashlimit                  0      0    136     30
nf_conntrack_ffffffff81ec7b80   1486   1620    400     20
nf_conntrack_expect           0      0    288     28
dm_snap_pending_exception      0      0    104     39
kcopyd_job                    0      0   4176      7
dm_rq_target_io               0      0    416     19
blkif_cache                 100    100    800     20
cfq_io_cq                   204    204    120     34
cfq_queue                   153    153    232     17
bsg_cmd                       0      0    312     26
ceph_cap                      0      0    128     32
ceph_inode_info               0      0   2032     16
gfs2_mblk                     0      0    432     18
gfs2_bufdata              11048  11224     88     46
gfs2_inode                    0      0   1280     25
gfs2_glock(aspace)            0      0    960     17
gfs2_glock                    0      0    592     27
btrfs_delayed_data_ref        0      0     96     42
btrfs_delayed_ref_head        0      0    232     17
btrfs_delayed_node            0      0    392     20
btrfs_ordered_extent          0      0    464     17
btrfs_extent_buffer           0      0    584     28
btrfs_delalloc_work           0      0    184     22
btrfs_path                    0      0    144     28
btrfs_transaction             0      0    656     24
btrfs_trans_handle            0      0    160     25
btrfs_inode                   0      0   1944     16
fuse_request                  0      0    472     17
fuse_inode                    0      0   1152     28
ntfs_big_inode_cache          0      0   1664     19
ntfs_inode_cache              0      0    696     23
cifs_small_rq                36     36    448     18
cifs_request                  4      4  16512      1
cifs_inode_cache              0      0   1104     29
isofs_inode_cache             0      0    960     17
fat_inode_cache               0      0   1168     28
fat_cache                     0      0     40    102
hugetlbfs_inode_cache        16     16    976     16
jbd2_transaction_s          375    375    320     25
jbd2_journal_handle         336    336     72     56
journal_handle                0      0     56     73
journal_head               1296   1296    112     36
revoke_record               384    384     32    128
ext4_inode_cache           4598   8137   1696     19
ext4_free_data             1408   1408     64     64
ext4_allocation_context     180    180    136     30
ext4_prealloc_space         156    156    152     26
ext4_io_end                 390    420   1088     30
ext4_io_page               4096   4096     16    256
ext4_extent_status         8576   8772     40    102
ext3_inode_cache              0      0   1312     24
ext3_xattr                    0      0     88     46
dquot                         0      0    384     21
dio                          92     92    704     23
pid_namespace                 0      0   2200     14
posix_timers_cache          116    116    280     29
UNIX                        176    176   1472     22
UDP-Lite                      0      0   1280     25
ip_fib_trie                 292    292     56     73
Cache                       Num  Total   Size  Pages
RAW                         156    156   1216     26
UDP                         150    150   1280     25
tw_sock_TCP                 378    378    192     21
TCP                         117    117   2368     13
fscache_cookie_jar            0      0    184     22
sgpool-128                   68     90   5120      6
sgpool-64                    72     72   2560     12
sgpool-32                   175    175   1280     25
sgpool-16                   275    275    640     25
blkdev_integrity              0      0    112     36
blkdev_queue                143    143   2808     11
blkdev_requests             346    346    376     21
fsnotify_event_holder         0      0     24    170
bip-256                       7      7   4224      7
bip-128                       0      0   2176     15
bip-16                        0      0    384     21
sock_inode_cache            323    323    960     17
file_lock_cache             102    102    240     17
net_namespace                 0      0   1984     16
shmem_inode_cache          1736   1736   1144     28
Acpi-State                  306    306     80     51
Acpi-Namespace             1632   1632     40    102
task_delay_info             960    960    168     24
taskstats                   144    144    328     24
proc_inode_cache           1440   1440    976     16
sigqueue                    300    300    160     25
bdev_cache                  144    144   1344     24
sysfs_dir_cache           22512  22512    144     28
filp                       2134   2450    320     25
inode_cache                1260   1972    912     17
dentry                     7006  11888    248     16
buffer_head              104786 107328    104     39
vm_area_struct             4131   4444    184     22
mm_struct                   280    280   1152     28
files_cache                 273    273    768     21
signal_cache                483    529   1408     23
sighand_cache               387    406   2240     14
task_struct                 297    329   4208      7
anon_vma                   4147   4536    144     28
shared_policy_node          510    510     48     85
numa_policy                3360   3360     72     56
radix_tree_node            5470   5740    568     28
idr_layer_cache             330    330   2112     15
dma-kmalloc-8192              0      0   8192      4
dma-kmalloc-4096              0      0   4096      8
dma-kmalloc-2048              0      0   2048     16
dma-kmalloc-1024              0      0   1024     16
dma-kmalloc-512               0      0    512     16
dma-kmalloc-256               0      0    256     16
dma-kmalloc-128               0      0    128     32
dma-kmalloc-64                0      0     64     64
dma-kmalloc-32                0      0     32    128
dma-kmalloc-16                0      0     16    256
dma-kmalloc-8                 0      0      8    512
dma-kmalloc-192               0      0    192     21
dma-kmalloc-96                0      0     96     42
kmalloc-8192                 28     28   8192      4
kmalloc-4096                909    928   4096      8
kmalloc-2048                646    704   2048     16
kmalloc-1024               1803   1808   1024     16
Cache                       Num  Total   Size  Pages
kmalloc-512                1057   1152    512     16
kmalloc-256                4175   4304    256     16
kmalloc-128                2948   3456    128     32
kmalloc-64                13726  15040     64     64
kmalloc-32                 1536   1536     32    128
kmalloc-16                 4096   4096     16    256
kmalloc-8                  7168   7168      8    512
kmalloc-192               68687  69111    192     21
kmalloc-96                 3704   4032     96     42
kmem_cache_node             192    192    128     32
kmem_cache                  160    160    256     16


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 24/04/13 20:16, Sander Eikelenboom wrote:
> Friday, April 19, 2013, 4:44:01 PM, you wrote:
> 
>> Hey Jens,
> 
>> Please in your spare time (if there is such a thing at a conference)
>> pull this branch:
> 
>>  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
> 
>> for your v3.10 branch. Sorry for being so late with this.
> 
> <big snip></big snip>
> 
>> Anyhow, please pull and if possible include the nice overview I typed
up in the
>> merge commit.
> 
>>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>>  drivers/block/xen-blkback/common.h             | 145 ++++-
>>  drivers/block/xen-blkback/xenbus.c             |  38 ++
>>  drivers/block/xen-blkfront.c                   | 490 +++++++++++---
>>  include/xen/interface/io/blkif.h               |  53 ++
>>  6 files changed, 1188 insertions(+), 399 deletions(-)
> 
>> Roger Pau Monne (7):
>>       xen-blkback: print stats about persistent grants
>>       xen-blkback: use balloon pages for all mappings
>>       xen-blkback: implement LRU mechanism for persistent grants
>>       xen-blkback: move pending handles list from blkbk to pending_req
>>       xen-blkback: make the queue of free requests per backend
>>       xen-blkback: expand map/unmap functions
>>       xen-block: implement indirect descriptors
> 
> 
> Hi Konrad / Roger,
> 
> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
> Without this pull it runs fine for several days.
> 
> Trying to start a new guest I ended up with the splat below. In the output
of xl-dmesg i seem to see more of these than before:
> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1) grant
table from (9) to (10) frames
Hello Sander,

Thanks for the report, it is expected to see more messages regarding
grant table expansion with this patch, since we are using up to 1056
persistent grants for each backend. Could you try lowering down the
maximum number of persistent grants to see if that prevents running out
of memory:

# echo 384 > /sys/module/xen_blkback/parameters/max_persistent_grants

This will set the maximum number of persistent grants to the previous value.

Thursday, April 25, 2013, 10:35:43 AM, you wrote:
> On 24/04/13 20:16, Sander Eikelenboom wrote:
>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>> 
>>> Hey Jens,
>> 
>>> Please in your spare time (if there is such a thing at a
conference)
>>> pull this branch:
>> 
>>>  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>> 
>>> for your v3.10 branch. Sorry for being so late with this.
>> 
>> <big snip></big snip>
>> 
>>> Anyhow, please pull and if possible include the nice overview I
typed up in the
>>> merge commit.
>> 
>>>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>>>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>>>  drivers/block/xen-blkback/common.h             | 145 ++++-
>>>  drivers/block/xen-blkback/xenbus.c             |  38 ++
>>>  drivers/block/xen-blkfront.c                   | 490
+++++++++++---
>>>  include/xen/interface/io/blkif.h               |  53 ++
>>>  6 files changed, 1188 insertions(+), 399 deletions(-)
>> 
>>> Roger Pau Monne (7):
>>>       xen-blkback: print stats about persistent grants
>>>       xen-blkback: use balloon pages for all mappings
>>>       xen-blkback: implement LRU mechanism for persistent grants
>>>       xen-blkback: move pending handles list from blkbk to
pending_req
>>>       xen-blkback: make the queue of free requests per backend
>>>       xen-blkback: expand map/unmap functions
>>>       xen-block: implement indirect descriptors
>> 
>> 
>> Hi Konrad / Roger,
>> 
>> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
>> Without this pull it runs fine for several days.
>> 
>> Trying to start a new guest I ended up with the splat below. In the
output of xl-dmesg i seem to see more of these than before:
>> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1)
grant table from (9) to (10) frames
> Hello Sander,
> Thanks for the report, it is expected to see more messages regarding
> grant table expansion with this patch, since we are using up to 1056
> persistent grants for each backend. Could you try lowering down the
> maximum number of persistent grants to see if that prevents running out
> of memory:
# echo 384 >> /sys/module/xen_blkback/parameters/max_persistent_grants
> This will set the maximum number of persistent grants to the previous
value.
Sure will give that a shot, does it require some special mem ?
Because it doesn''t seem to invoke any oom-killer etc.

I have dom restricted with dom0_mem=1024M,max:1024M, that was previously enough
for running say 16 guests with 1 disk and 1 swap (and a few guests which have a
extra disk).

--

Sander

On 25/04/13 10:35, Roger Pau Monné wrote:
> On 24/04/13 20:16, Sander Eikelenboom wrote:
>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>>
>>> Hey Jens,
>>
>>> Please in your spare time (if there is such a thing at a
conference)
>>> pull this branch:
>>
>>>  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>>
>>> for your v3.10 branch. Sorry for being so late with this.
>>
>> <big snip></big snip>
>>
>>> Anyhow, please pull and if possible include the nice overview I
typed up in the
>>> merge commit.
>>
>>>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>>>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>>>  drivers/block/xen-blkback/common.h             | 145 ++++-
>>>  drivers/block/xen-blkback/xenbus.c             |  38 ++
>>>  drivers/block/xen-blkfront.c                   | 490
+++++++++++---
>>>  include/xen/interface/io/blkif.h               |  53 ++
>>>  6 files changed, 1188 insertions(+), 399 deletions(-)
>>
>>> Roger Pau Monne (7):
>>>       xen-blkback: print stats about persistent grants
>>>       xen-blkback: use balloon pages for all mappings
>>>       xen-blkback: implement LRU mechanism for persistent grants
>>>       xen-blkback: move pending handles list from blkbk to
pending_req
>>>       xen-blkback: make the queue of free requests per backend
>>>       xen-blkback: expand map/unmap functions
>>>       xen-block: implement indirect descriptors
>>
>>
>> Hi Konrad / Roger,
>>
>> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
>> Without this pull it runs fine for several days.
>>
>> Trying to start a new guest I ended up with the splat below. In the
output of xl-dmesg i seem to see more of these than before:
>> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1)
grant table from (9) to (10) frames
> 
> Hello Sander,
> 
> Thanks for the report, it is expected to see more messages regarding
> grant table expansion with this patch, since we are using up to 1056
> persistent grants for each backend. Could you try lowering down the
> maximum number of persistent grants to see if that prevents running out
> of memory:
> 
> # echo 384 > /sys/module/xen_blkback/parameters/max_persistent_grants
And the number of free pages keep in blkback cache:

# echo 256 > /sys/module/xen_blkback/parameters/max_buffer_pages

On 24/04/13 19:16, Sander Eikelenboom wrote:
> 
> Hi Konrad / Roger,
> 
> I tried this pull on top of latest Linus latest linux-3.9 tree, but
> although it seems to boot and work fine at first, i seem to get trouble
> after running for about a day.
> Without this pull it runs fine for several days.
> 
> Trying to start a new guest I ended up with the splat below. In the
> output of xl-dmesg i seem to see more of these than before:
> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1)
> grant table from (9) to (10) frames
> [18496.013743] xenwatch: page allocation failure: order:7, mode:0x10c0d0
I think this is the allocation of blkif->pending_reqs in
xen_blkif_alloc() which is now enormous requiring a 512 KiB contiguous
allocation.

All the arrays in struct pending_req need to be turned into individual
allocations so no individual allocation is larger than a page.

David
> [18496.031948] Pid: 54, comm: xenwatch Not tainted 3.9.0-rc8-20130424-jens+
#1
> [18496.049897] Call Trace:
> [18496.067674]  [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
> [18496.085453]  [<ffffffff810b25ed>] ? trace_hardirqs_on+0xd/0x10
> [18496.102951]  [<ffffffff810bdb24>] ? on_each_cpu_mask+0x94/0xd0
> [18496.120270]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
> [18496.137306]  [<ffffffff8113a161>] alloc_pages_current+0xb1/0x160
> [18496.154051]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
> [18496.170579]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
> [18496.186921]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
> [18496.202963]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
> [18496.218714]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> [18496.234237]  [<ffffffff8156a151>] driver_probe_device+0x81/0x220
> [18496.249605]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
> [18496.264681]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> [18496.279500]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
> [18496.294138]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
> [18496.308553]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
> [18496.322694]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
> [18496.336640]  [<ffffffff81567c80>] device_add+0x650/0x720
> [18496.350209]  [<ffffffff81573103>] ? device_pm_sleep_init+0x43/0x70
> [18496.363602]  [<ffffffff81567d69>] device_register+0x19/0x20
> [18496.376721]  [<ffffffff8147495b>] xenbus_probe_node+0x14b/0x160
> [18496.389611]  [<ffffffff815685b4>] ? bus_for_each_dev+0xa4/0xb0
> [18496.402298]  [<ffffffff81474b2c>] xenbus_dev_changed+0x1bc/0x1c0
> [18496.414732]  [<ffffffff810b67f7>] ? lock_release+0x117/0x260
> [18496.426904]  [<ffffffff81474f66>] backend_changed+0x16/0x20
> [18496.438835]  [<ffffffff81472f5e>] xenwatch_thread+0x4e/0x150
> [18496.450579]  [<ffffffff8108abb0>] ? wake_up_bit+0x40/0x40
> [18496.462048]  [<ffffffff81472f10>] ? xs_watch+0x60/0x60
> [18496.473286]  [<ffffffff8108a546>] kthread+0xd6/0xe0
> [18496.484235]  [<ffffffff8108a470>] ?
__init_kthread_worker+0x70/0x70
> [18496.494987]  [<ffffffff819979bc>] ret_from_fork+0x7c/0xb0
> [18496.505560]  [<ffffffff8108a470>] ?
__init_kthread_worker+0x70/0x70

Thursday, April 25, 2013, 10:43:33 AM, you wrote:
> On 25/04/13 10:35, Roger Pau Monné wrote:
>> On 24/04/13 20:16, Sander Eikelenboom wrote:
>>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>>>
>>>> Hey Jens,
>>>
>>>> Please in your spare time (if there is such a thing at a
conference)
>>>> pull this branch:
>>>
>>>>  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>>>
>>>> for your v3.10 branch. Sorry for being so late with this.
>>>
>>> <big snip></big snip>
>>>
>>>> Anyhow, please pull and if possible include the nice overview I
typed up in the
>>>> merge commit.
>>>
>>>>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>>>>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>>>>  drivers/block/xen-blkback/common.h             | 145 ++++-
>>>>  drivers/block/xen-blkback/xenbus.c             |  38 ++
>>>>  drivers/block/xen-blkfront.c                   | 490
+++++++++++---
>>>>  include/xen/interface/io/blkif.h               |  53 ++
>>>>  6 files changed, 1188 insertions(+), 399 deletions(-)
>>>
>>>> Roger Pau Monne (7):
>>>>       xen-blkback: print stats about persistent grants
>>>>       xen-blkback: use balloon pages for all mappings
>>>>       xen-blkback: implement LRU mechanism for persistent
grants
>>>>       xen-blkback: move pending handles list from blkbk to
pending_req
>>>>       xen-blkback: make the queue of free requests per backend
>>>>       xen-blkback: expand map/unmap functions
>>>>       xen-block: implement indirect descriptors
>>>
>>>
>>> Hi Konrad / Roger,
>>>
>>> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
>>> Without this pull it runs fine for several days.
>>>
>>> Trying to start a new guest I ended up with the splat below. In the
output of xl-dmesg i seem to see more of these than before:
>>> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1)
grant table from (9) to (10) frames
>> 
>> Hello Sander,
>> 
>> Thanks for the report, it is expected to see more messages regarding
>> grant table expansion with this patch, since we are using up to 1056
>> persistent grants for each backend. Could you try lowering down the
>> maximum number of persistent grants to see if that prevents running out
>> of memory:
>> 
>> # echo 384 >
/sys/module/xen_blkback/parameters/max_persistent_grants
> And the number of free pages keep in blkback cache:
# echo 256 >> /sys/module/xen_blkback/parameters/max_buffer_pages

With both set .. it still bails out after sometime when trying to start a new
guest.


[ 9871.923198] Pid: 54, comm: xenwatch Not tainted 3.9.0-rc8-20130424-jens+ #1
[ 9871.934278] Call Trace:
[ 9871.945146]  [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
[ 9871.956094]  [<ffffffff811021f1>] ?
__alloc_pages_direct_compact+0x211/0x230
[ 9871.967048]  [<ffffffff811028af>] __alloc_pages_nodemask+0x69f/0x960
[ 9871.978092]  [<ffffffff8113a161>] alloc_pages_current+0xb1/0x160
[ 9871.989065]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
[ 9871.999999]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
[ 9872.010845]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
[ 9872.021667]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
[ 9872.032542]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
[ 9872.043453]  [<ffffffff8156a151>] driver_probe_device+0x81/0x220
[ 9872.054115]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
[ 9872.064454]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
[ 9872.074610]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
[ 9872.084541]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
[ 9872.094282]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
[ 9872.103751]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
[ 9872.113158]  [<ffffffff81567c80>] device_add+0x650/0x720
[ 9872.122379]  [<ffffffff81573103>] ? device_pm_sleep_init+0x43/0x70
[ 9872.131304]  [<ffffffff81567d69>] device_register+0x19/0x20
[ 9872.139948]  [<ffffffff8147495b>] xenbus_probe_node+0x14b/0x160
[ 9872.148414]  [<ffffffff815685b4>] ? bus_for_each_dev+0xa4/0xb0
[ 9872.156603]  [<ffffffff81474b2c>] xenbus_dev_changed+0x1bc/0x1c0
[ 9872.164631]  [<ffffffff810b67f7>] ? lock_release+0x117/0x260
[ 9872.172551]  [<ffffffff81474f66>] backend_changed+0x16/0x20
[ 9872.180427]  [<ffffffff81472f5e>] xenwatch_thread+0x4e/0x150
[ 9872.188238]  [<ffffffff8108abb0>] ? wake_up_bit+0x40/0x40
[ 9872.196032]  [<ffffffff81472f10>] ? xs_watch+0x60/0x60
[ 9872.203841]  [<ffffffff8108a546>] kthread+0xd6/0xe0
[ 9872.211567]  [<ffffffff8108a470>] ? __init_kthread_worker+0x70/0x70
[ 9872.219075]  [<ffffffff819979bc>] ret_from_fork+0x7c/0xb0
[ 9872.226329]  [<ffffffff8108a470>] ? __init_kthread_worker+0x70/0x70
[ 9872.233416] Mem-Info:
[ 9872.241071] Node 0 DMA per-cpu:
[ 9872.248137] CPU    0: hi:    0, btch:   1 usd:   0
[ 9872.255108] CPU    1: hi:    0, btch:   1 usd:   0
[ 9872.262090] CPU    2: hi:    0, btch:   1 usd:   0
[ 9872.269069] CPU    3: hi:    0, btch:   1 usd:   0
[ 9872.275890] CPU    4: hi:    0, btch:   1 usd:   0
[ 9872.282629] CPU    5: hi:    0, btch:   1 usd:   0
[ 9872.289393] Node 0 DMA32 per-cpu:
[ 9872.296163] CPU    0: hi:  186, btch:  31 usd:  53
[ 9872.302701] CPU    1: hi:  186, btch:  31 usd:  72
[ 9872.308924] CPU    2: hi:  186, btch:  31 usd:  66
[ 9872.314937] CPU    3: hi:  186, btch:  31 usd:  30
[ 9872.320649] CPU    4: hi:  186, btch:  31 usd: 110
[ 9872.326032] CPU    5: hi:  186, btch:  31 usd: 163
[ 9872.331185] active_anon:4510 inactive_anon:10674 isolated_anon:0
[ 9872.331185]  active_file:21063 inactive_file:161965 isolated_file:0
[ 9872.331185]  unevictable:519 dirty:127 writeback:0 unstable:0
[ 9872.331185]  free:3448 slab_reclaimable:8061 slab_unreclaimable:10395
[ 9872.331185]  mapped:3916 shmem:321 pagetables:1249 bounce:0
[ 9872.331185]  free_cma:0
[ 9872.358911] Node 0 DMA free:3836kB min:64kB low:80kB high:96kB
active_anon:0kB inactive_anon:4kB active_file:76kB inactive_file:10748kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:1004kB slab_unreclaimable:224kB kernel_stack:16kB
pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
[ 9872.374281] lowmem_reserve[]: 0 884 884 884
[ 9872.379566] Node 0 DMA32 free:9916kB min:3772kB low:4712kB high:5656kB
active_anon:17968kB inactive_anon:42692kB active_file:84192kB
inactive_file:637180kB unevictable:2084kB isolated(anon):0kB isolated(file):0kB
present:1032192kB managed:905896kB mlocked:2084kB dirty:524kB writeback:0kB
mapped:15800kB shmem:1284kB slab_reclaimable:31240kB slab_unreclaimable:41352kB
kernel_stack:2160kB pagetables:5016kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9872.402980] lowmem_reserve[]: 0 0 0 0
[ 9872.409005] Node 0 DMA: 5*4kB (M) 13*8kB (M) 104*16kB (M) 4*32kB (MR) 2*64kB
(R) 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 3836kB
[ 9872.415521] Node 0 DMA32: 1665*4kB (UEMR) 206*8kB (MR) 10*16kB (UMR) 5*32kB
(R) 0*64kB 4*128kB (R) 1*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB =
9908kB
[ 9872.428594] 183858 total pagecache pages
[ 9872.435128] 0 pages in swap cache
[ 9872.441781] Swap cache stats: add 7, delete 7, find 3/3
[ 9872.448409] Free swap  = 2097148kB
[ 9872.455042] Total swap = 2097148kB
[ 9872.465414] 262143 pages RAM
[ 9872.471913] 28027 pages reserved
[ 9872.478443] 295127 pages shared
[ 9872.484909] 207989 pages non-shared
[ 9872.491387] vbd vbd-20-768: 12 creating block interface
[ 9872.499259] vbd vbd-20-768: 12 xenbus_dev_probe on backend/vbd/20/768
[ 9872.506942] vbd: probe of vbd-20-768 failed with error -12

On 25/04/13 14:32, Sander Eikelenboom wrote:
> 
> Thursday, April 25, 2013, 10:43:33 AM, you wrote:
> 
>> On 25/04/13 10:35, Roger Pau Monné wrote:
>>> On 24/04/13 20:16, Sander Eikelenboom wrote:
>>>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>>>>
>>>>> Hey Jens,
>>>>
>>>>> Please in your spare time (if there is such a thing at a
conference)
>>>>> pull this branch:
>>>>
>>>>> 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>>>>
>>>>> for your v3.10 branch. Sorry for being so late with this.
>>>>
>>>> <big snip></big snip>
>>>>
>>>>> Anyhow, please pull and if possible include the nice
overview I typed up in the
>>>>> merge commit.
>>>>
>>>>>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>>>>>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>>>>>  drivers/block/xen-blkback/common.h             | 145 ++++-
>>>>>  drivers/block/xen-blkback/xenbus.c             |  38 ++
>>>>>  drivers/block/xen-blkfront.c                   | 490
+++++++++++---
>>>>>  include/xen/interface/io/blkif.h               |  53 ++
>>>>>  6 files changed, 1188 insertions(+), 399 deletions(-)
>>>>
>>>>> Roger Pau Monne (7):
>>>>>       xen-blkback: print stats about persistent grants
>>>>>       xen-blkback: use balloon pages for all mappings
>>>>>       xen-blkback: implement LRU mechanism for persistent
grants
>>>>>       xen-blkback: move pending handles list from blkbk to
pending_req
>>>>>       xen-blkback: make the queue of free requests per
backend
>>>>>       xen-blkback: expand map/unmap functions
>>>>>       xen-block: implement indirect descriptors
>>>>
>>>>
>>>> Hi Konrad / Roger,
>>>>
>>>> I tried this pull on top of latest Linus latest linux-3.9 tree,
but although it seems to boot and work fine at first, i seem to get trouble
after running for about a day.
>>>> Without this pull it runs fine for several days.
>>>>
>>>> Trying to start a new guest I ended up with the splat below. In
the output of xl-dmesg i seem to see more of these than before:
>>>> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom
(1) grant table from (9) to (10) frames
>>>
>>> Hello Sander,
>>>
>>> Thanks for the report, it is expected to see more messages
regarding
>>> grant table expansion with this patch, since we are using up to
1056
>>> persistent grants for each backend. Could you try lowering down the
>>> maximum number of persistent grants to see if that prevents running
out
>>> of memory:
>>>
>>> # echo 384 >
/sys/module/xen_blkback/parameters/max_persistent_grants
> 
>> And the number of free pages keep in blkback cache:
> 
> # echo 256 >> /sys/module/xen_blkback/parameters/max_buffer_pages
> 
> With both set .. it still bails out after sometime when trying to start a
new guest.
OK, will work on a patch to split memory allocation instead of doing it
all in a big chunk.
> 
> 
> [ 9871.923198] Pid: 54, comm: xenwatch Not tainted 3.9.0-rc8-20130424-jens+
#1
> [ 9871.934278] Call Trace:
> [ 9871.945146]  [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
> [ 9871.956094]  [<ffffffff811021f1>] ?
__alloc_pages_direct_compact+0x211/0x230
> [ 9871.967048]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
> [ 9871.978092]  [<ffffffff8113a161>] alloc_pages_current+0xb1/0x160
> [ 9871.989065]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
> [ 9871.999999]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
> [ 9872.010845]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
> [ 9872.021667]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
> [ 9872.032542]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> [ 9872.043453]  [<ffffffff8156a151>] driver_probe_device+0x81/0x220
> [ 9872.054115]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
> [ 9872.064454]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> [ 9872.074610]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
> [ 9872.084541]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
> [ 9872.094282]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
> [ 9872.103751]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
> [ 9872.113158]  [<ffffffff81567c80>] device_add+0x650/0x720
> [ 9872.122379]  [<ffffffff81573103>] ? device_pm_sleep_init+0x43/0x70
> [ 9872.131304]  [<ffffffff81567d69>] device_register+0x19/0x20
> [ 9872.139948]  [<ffffffff8147495b>] xenbus_probe_node+0x14b/0x160
> [ 9872.148414]  [<ffffffff815685b4>] ? bus_for_each_dev+0xa4/0xb0
> [ 9872.156603]  [<ffffffff81474b2c>] xenbus_dev_changed+0x1bc/0x1c0
> [ 9872.164631]  [<ffffffff810b67f7>] ? lock_release+0x117/0x260
> [ 9872.172551]  [<ffffffff81474f66>] backend_changed+0x16/0x20
> [ 9872.180427]  [<ffffffff81472f5e>] xenwatch_thread+0x4e/0x150
> [ 9872.188238]  [<ffffffff8108abb0>] ? wake_up_bit+0x40/0x40
> [ 9872.196032]  [<ffffffff81472f10>] ? xs_watch+0x60/0x60
> [ 9872.203841]  [<ffffffff8108a546>] kthread+0xd6/0xe0
> [ 9872.211567]  [<ffffffff8108a470>] ?
__init_kthread_worker+0x70/0x70
> [ 9872.219075]  [<ffffffff819979bc>] ret_from_fork+0x7c/0xb0
> [ 9872.226329]  [<ffffffff8108a470>] ?
__init_kthread_worker+0x70/0x70
> [ 9872.233416] Mem-Info:
> [ 9872.241071] Node 0 DMA per-cpu:
> [ 9872.248137] CPU    0: hi:    0, btch:   1 usd:   0
> [ 9872.255108] CPU    1: hi:    0, btch:   1 usd:   0
> [ 9872.262090] CPU    2: hi:    0, btch:   1 usd:   0
> [ 9872.269069] CPU    3: hi:    0, btch:   1 usd:   0
> [ 9872.275890] CPU    4: hi:    0, btch:   1 usd:   0
> [ 9872.282629] CPU    5: hi:    0, btch:   1 usd:   0
> [ 9872.289393] Node 0 DMA32 per-cpu:
> [ 9872.296163] CPU    0: hi:  186, btch:  31 usd:  53
> [ 9872.302701] CPU    1: hi:  186, btch:  31 usd:  72
> [ 9872.308924] CPU    2: hi:  186, btch:  31 usd:  66
> [ 9872.314937] CPU    3: hi:  186, btch:  31 usd:  30
> [ 9872.320649] CPU    4: hi:  186, btch:  31 usd: 110
> [ 9872.326032] CPU    5: hi:  186, btch:  31 usd: 163
> [ 9872.331185] active_anon:4510 inactive_anon:10674 isolated_anon:0
> [ 9872.331185]  active_file:21063 inactive_file:161965 isolated_file:0
> [ 9872.331185]  unevictable:519 dirty:127 writeback:0 unstable:0
> [ 9872.331185]  free:3448 slab_reclaimable:8061 slab_unreclaimable:10395
> [ 9872.331185]  mapped:3916 shmem:321 pagetables:1249 bounce:0
> [ 9872.331185]  free_cma:0
> [ 9872.358911] Node 0 DMA free:3836kB min:64kB low:80kB high:96kB
active_anon:0kB inactive_anon:4kB active_file:76kB inactive_file:10748kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:1004kB slab_unreclaimable:224kB kernel_stack:16kB
pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
> [ 9872.374281] lowmem_reserve[]: 0 884 884 884
> [ 9872.379566] Node 0 DMA32 free:9916kB min:3772kB low:4712kB high:5656kB
active_anon:17968kB inactive_anon:42692kB active_file:84192kB
inactive_file:637180kB unevictable:2084kB isolated(anon):0kB isolated(file):0kB
present:1032192kB managed:905896kB mlocked:2084kB dirty:524kB writeback:0kB
mapped:15800kB shmem:1284kB slab_reclaimable:31240kB slab_unreclaimable:41352kB
kernel_stack:2160kB pagetables:5016kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [ 9872.402980] lowmem_reserve[]: 0 0 0 0
> [ 9872.409005] Node 0 DMA: 5*4kB (M) 13*8kB (M) 104*16kB (M) 4*32kB (MR)
2*64kB (R) 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB =
3836kB
> [ 9872.415521] Node 0 DMA32: 1665*4kB (UEMR) 206*8kB (MR) 10*16kB (UMR)
5*32kB (R) 0*64kB 4*128kB (R) 1*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB
= 9908kB
> [ 9872.428594] 183858 total pagecache pages
> [ 9872.435128] 0 pages in swap cache
> [ 9872.441781] Swap cache stats: add 7, delete 7, find 3/3
> [ 9872.448409] Free swap  = 2097148kB
> [ 9872.455042] Total swap = 2097148kB
> [ 9872.465414] 262143 pages RAM
> [ 9872.471913] 28027 pages reserved
> [ 9872.478443] 295127 pages shared
> [ 9872.484909] 207989 pages non-shared
> [ 9872.491387] vbd vbd-20-768: 12 creating block interface
> [ 9872.499259] vbd vbd-20-768: 12 xenbus_dev_probe on backend/vbd/20/768
> [ 9872.506942] vbd: probe of vbd-20-768 failed with error -12
>

On 25/04/13 14:39, Roger Pau Monné wrote:
> On 25/04/13 14:32, Sander Eikelenboom wrote:
>>
>> Thursday, April 25, 2013, 10:43:33 AM, you wrote:
>>
>>> On 25/04/13 10:35, Roger Pau Monné wrote:
>>>> On 24/04/13 20:16, Sander Eikelenboom wrote:
>>>>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>>>>>
>>>>>> Hey Jens,
>>>>>
>>>>>> Please in your spare time (if there is such a thing at
a conference)
>>>>>> pull this branch:
>>>>>
>>>>>> 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>>>>>
>>>>>> for your v3.10 branch. Sorry for being so late with
this.
>>>>>
>>>>> <big snip></big snip>
>>>>>
>>>>>> Anyhow, please pull and if possible include the nice
overview I typed up in the
>>>>>> merge commit.
>>>>>
>>>>>>  Documentation/ABI/stable/sysfs-bus-xen-backend |  18 +
>>>>>>  drivers/block/xen-blkback/blkback.c            | 843
++++++++++++++++---------
>>>>>>  drivers/block/xen-blkback/common.h             | 145
++++-
>>>>>>  drivers/block/xen-blkback/xenbus.c             |  38
++
>>>>>>  drivers/block/xen-blkfront.c                   | 490
+++++++++++---
>>>>>>  include/xen/interface/io/blkif.h               |  53
++
>>>>>>  6 files changed, 1188 insertions(+), 399 deletions(-)
>>>>>
>>>>>> Roger Pau Monne (7):
>>>>>>       xen-blkback: print stats about persistent grants
>>>>>>       xen-blkback: use balloon pages for all mappings
>>>>>>       xen-blkback: implement LRU mechanism for
persistent grants
>>>>>>       xen-blkback: move pending handles list from blkbk
to pending_req
>>>>>>       xen-blkback: make the queue of free requests per
backend
>>>>>>       xen-blkback: expand map/unmap functions
>>>>>>       xen-block: implement indirect descriptors
>>>>>
>>>>>
>>>>> Hi Konrad / Roger,
>>>>>
>>>>> I tried this pull on top of latest Linus latest linux-3.9
tree, but although it seems to boot and work fine at first, i seem to get
trouble after running for about a day.
>>>>> Without this pull it runs fine for several days.
>>>>>
>>>>> Trying to start a new guest I ended up with the splat
below. In the output of xl-dmesg i seem to see more of these than before:
>>>>> (XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding
dom (1) grant table from (9) to (10) frames
>>>>
>>>> Hello Sander,
>>>>
>>>> Thanks for the report, it is expected to see more messages
regarding
>>>> grant table expansion with this patch, since we are using up to
1056
>>>> persistent grants for each backend. Could you try lowering down
the
>>>> maximum number of persistent grants to see if that prevents
running out
>>>> of memory:
>>>>
>>>> # echo 384 >
/sys/module/xen_blkback/parameters/max_persistent_grants
>>
>>> And the number of free pages keep in blkback cache:
>>
>> # echo 256 >> /sys/module/xen_blkback/parameters/max_buffer_pages
>>
>> With both set .. it still bails out after sometime when trying to start
a new guest.
> 
> OK, will work on a patch to split memory allocation instead of doing it
> all in a big chunk.
Could you try the following patch?

---
From db0163f74dafb085457d843c6ab7935d5dc7bd65 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Thu, 25 Apr 2013 17:48:32 +0200
Subject: [PATCH] xen-blkback: allocate list of pending reqs in small chunks
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Allocate pending requests in smaller chunks instead of allocating them
all at the same time.

Also remove the global array of pending_reqs, it is no longer
necessary.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 drivers/block/xen-blkback/common.h |    2 -
 drivers/block/xen-blkback/xenbus.c |   41 ++++++++++++++++++++++-------------
 2 files changed, 26 insertions(+), 17 deletions(-)

diff --git a/drivers/block/xen-blkback/common.h
b/drivers/block/xen-blkback/common.h
index 1ac53da..2bbda8637 100644
--- a/drivers/block/xen-blkback/common.h
+++ b/drivers/block/xen-blkback/common.h
@@ -297,8 +297,6 @@ struct xen_blkif {
 	int			free_pages_num;
 	struct list_head	free_pages;
 
-	/* Allocation of pending_reqs */
-	struct pending_req	*pending_reqs;
 	/* List of all ''pending_req'' available */
 	struct list_head	pending_free;
 	/* And its spinlock. */
diff --git a/drivers/block/xen-blkback/xenbus.c
b/drivers/block/xen-blkback/xenbus.c
index afab208..ceefbe4 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -105,6 +105,7 @@ static void xen_update_blkif_status(struct xen_blkif *blkif)
 static struct xen_blkif *xen_blkif_alloc(domid_t domid)
 {
 	struct xen_blkif *blkif;
+	struct pending_req *req, *n;
 	int i;
 
 	BUILD_BUG_ON(MAX_INDIRECT_PAGES > BLKIF_MAX_INDIRECT_PAGES_PER_REQUEST);
@@ -127,22 +128,29 @@ static struct xen_blkif *xen_blkif_alloc(domid_t domid)
 	blkif->free_pages_num = 0;
 	atomic_set(&blkif->persistent_gnt_in_use, 0);
 
-	blkif->pending_reqs = kcalloc(XEN_BLKIF_REQS,
-	                              sizeof(blkif->pending_reqs[0]),
-	                              GFP_KERNEL);
-	if (!blkif->pending_reqs) {
-		kmem_cache_free(xen_blkif_cachep, blkif);
-		return ERR_PTR(-ENOMEM);
-	}
 	INIT_LIST_HEAD(&blkif->pending_free);
+
+	for (i = 0; i < XEN_BLKIF_REQS; i++) {
+		req = kzalloc(sizeof(*req), GFP_KERNEL);
+		if (!req)
+			goto fail;
+		list_add_tail(&req->free_list,
+		              &blkif->pending_free);
+	}
 	spin_lock_init(&blkif->pending_free_lock);
 	init_waitqueue_head(&blkif->pending_free_wq);
 
-	for (i = 0; i < XEN_BLKIF_REQS; i++)
-		list_add_tail(&blkif->pending_reqs[i].free_list,
-			      &blkif->pending_free);
-
 	return blkif;
+
+fail:
+	list_for_each_entry_safe(req, n, &blkif->pending_free, free_list) {
+		list_del(&req->free_list);
+		kfree(req);
+	}
+
+	kmem_cache_free(xen_blkif_cachep, blkif);
+
+	return ERR_PTR(-ENOMEM);
 }
 
 static int xen_blkif_map(struct xen_blkif *blkif, unsigned long shared_page,
@@ -221,18 +229,21 @@ static void xen_blkif_disconnect(struct xen_blkif *blkif)
 
 static void xen_blkif_free(struct xen_blkif *blkif)
 {
-	struct pending_req *req;
+	struct pending_req *req, *n;
 	int i = 0;
 
 	if (!atomic_dec_and_test(&blkif->refcnt))
 		BUG();
 
 	/* Check that there is no request in use */
-	list_for_each_entry(req, &blkif->pending_free, free_list)
+	list_for_each_entry_safe(req, n, &blkif->pending_free, free_list) {
+		list_del(&req->free_list);
+		kfree(req);
 		i++;
-	BUG_ON(i != XEN_BLKIF_REQS);
+	}
+
+	WARN_ON(i != XEN_BLKIF_REQS);
 
-	kfree(blkif->pending_reqs);
 	kmem_cache_free(xen_blkif_cachep, blkif);
 }
 
-- 
1.7.7.5 (Apple Git-26)




--------------010106070701060804080203
Content-Type: text/plain; charset="UTF-8"; x-mac-type=0;
x-mac-creator=0;
	name="0001-xen-blkback-allocate-list-of-pending-reqs-in-small-c.patch"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment;
	filename*0="0001-xen-blkback-allocate-list-of-pending-reqs-in-small-c.pa";
	filename*1="tch"

From db0163f74dafb085457d843c6ab7935d5dc7bd65 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Thu, 25 Apr 2013 17:48:32 +0200
Subject: [PATCH] xen-blkback: allocate list of pending reqs in small chunks
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Allocate pending requests in smaller chunks instead of allocating them
all at the same time.

Also remove the global array of pending_reqs, it is no longer
necessary.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 drivers/block/xen-blkback/common.h |    2 -
 drivers/block/xen-blkback/xenbus.c |   41 ++++++++++++++++++++++-------------
 2 files changed, 26 insertions(+), 17 deletions(-)

diff --git a/drivers/block/xen-blkback/common.h
b/drivers/block/xen-blkback/common.h
index 1ac53da..2bbda8637 100644
--- a/drivers/block/xen-blkback/common.h
+++ b/drivers/block/xen-blkback/common.h
@@ -297,8 +297,6 @@ struct xen_blkif {
 	int			free_pages_num;
 	struct list_head	free_pages;
 
-	/* Allocation of pending_reqs */
-	struct pending_req	*pending_reqs;
 	/* List of all ''pending_req'' available */
 	struct list_head	pending_free;
 	/* And its spinlock. */
diff --git a/drivers/block/xen-blkback/xenbus.c
b/drivers/block/xen-blkback/xenbus.c
index afab208..ceefbe4 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -105,6 +105,7 @@ static void xen_update_blkif_status(struct xen_blkif *blkif)
 static struct xen_blkif *xen_blkif_alloc(domid_t domid)
 {
 	struct xen_blkif *blkif;
+	struct pending_req *req, *n;
 	int i;
 
 	BUILD_BUG_ON(MAX_INDIRECT_PAGES > BLKIF_MAX_INDIRECT_PAGES_PER_REQUEST);
@@ -127,22 +128,29 @@ static struct xen_blkif *xen_blkif_alloc(domid_t domid)
 	blkif->free_pages_num = 0;
 	atomic_set(&blkif->persistent_gnt_in_use, 0);
 
-	blkif->pending_reqs = kcalloc(XEN_BLKIF_REQS,
-	                              sizeof(blkif->pending_reqs[0]),
-	                              GFP_KERNEL);
-	if (!blkif->pending_reqs) {
-		kmem_cache_free(xen_blkif_cachep, blkif);
-		return ERR_PTR(-ENOMEM);
-	}
 	INIT_LIST_HEAD(&blkif->pending_free);
+
+	for (i = 0; i < XEN_BLKIF_REQS; i++) {
+		req = kzalloc(sizeof(*req), GFP_KERNEL);
+		if (!req)
+			goto fail;
+		list_add_tail(&req->free_list,
+		              &blkif->pending_free);
+	}
 	spin_lock_init(&blkif->pending_free_lock);
 	init_waitqueue_head(&blkif->pending_free_wq);
 
-	for (i = 0; i < XEN_BLKIF_REQS; i++)
-		list_add_tail(&blkif->pending_reqs[i].free_list,
-			      &blkif->pending_free);
-
 	return blkif;
+
+fail:
+	list_for_each_entry_safe(req, n, &blkif->pending_free, free_list) {
+		list_del(&req->free_list);
+		kfree(req);
+	}
+
+	kmem_cache_free(xen_blkif_cachep, blkif);
+
+	return ERR_PTR(-ENOMEM);
 }
 
 static int xen_blkif_map(struct xen_blkif *blkif, unsigned long shared_page,
@@ -221,18 +229,21 @@ static void xen_blkif_disconnect(struct xen_blkif *blkif)
 
 static void xen_blkif_free(struct xen_blkif *blkif)
 {
-	struct pending_req *req;
+	struct pending_req *req, *n;
 	int i = 0;
 
 	if (!atomic_dec_and_test(&blkif->refcnt))
 		BUG();
 
 	/* Check that there is no request in use */
-	list_for_each_entry(req, &blkif->pending_free, free_list)
+	list_for_each_entry_safe(req, n, &blkif->pending_free, free_list) {
+		list_del(&req->free_list);
+		kfree(req);
 		i++;
-	BUG_ON(i != XEN_BLKIF_REQS);
+	}
+
+	WARN_ON(i != XEN_BLKIF_REQS);
 
-	kfree(blkif->pending_reqs);
 	kmem_cache_free(xen_blkif_cachep, blkif);
 }
 
-- 
1.7.7.5 (Apple Git-26)


--------------010106070701060804080203
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

--------------010106070701060804080203--

On 25/04/13 16:52, Roger Pau Monné wrote:
> 
> [PATCH] xen-blkback: allocate list of pending reqs in small chunks
> 
> Allocate pending requests in smaller chunks instead of allocating them
> all at the same time.
> 
> Also remove the global array of pending_reqs, it is no longer
> necessary.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
>  drivers/block/xen-blkback/common.h |    2 -
>  drivers/block/xen-blkback/xenbus.c |   41
++++++++++++++++++++++-------------
>  2 files changed, 26 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/block/xen-blkback/common.h
b/drivers/block/xen-blkback/common.h
> index 1ac53da..2bbda8637 100644
> --- a/drivers/block/xen-blkback/common.h
> +++ b/drivers/block/xen-blkback/common.h
> @@ -297,8 +297,6 @@ struct xen_blkif {
>  	int			free_pages_num;
>  	struct list_head	free_pages;
>  
> -	/* Allocation of pending_reqs */
> -	struct pending_req	*pending_reqs;
>  	/* List of all ''pending_req'' available */
>  	struct list_head	pending_free;
>  	/* And its spinlock. */
> diff --git a/drivers/block/xen-blkback/xenbus.c
b/drivers/block/xen-blkback/xenbus.c
> index afab208..ceefbe4 100644
> --- a/drivers/block/xen-blkback/xenbus.c
> +++ b/drivers/block/xen-blkback/xenbus.c
> @@ -105,6 +105,7 @@ static void xen_update_blkif_status(struct xen_blkif
*blkif)
>  static struct xen_blkif *xen_blkif_alloc(domid_t domid)
>  {
>  	struct xen_blkif *blkif;
> +	struct pending_req *req, *n;
>  	int i;
>  
>  	BUILD_BUG_ON(MAX_INDIRECT_PAGES >
BLKIF_MAX_INDIRECT_PAGES_PER_REQUEST);
> @@ -127,22 +128,29 @@ static struct xen_blkif *xen_blkif_alloc(domid_t
domid)
>  	blkif->free_pages_num = 0;
>  	atomic_set(&blkif->persistent_gnt_in_use, 0);
>  
> -	blkif->pending_reqs = kcalloc(XEN_BLKIF_REQS,
> -	                              sizeof(blkif->pending_reqs[0]),
> -	                              GFP_KERNEL);
> -	if (!blkif->pending_reqs) {
> -		kmem_cache_free(xen_blkif_cachep, blkif);
> -		return ERR_PTR(-ENOMEM);
> -	}
>  	INIT_LIST_HEAD(&blkif->pending_free);
> +
> +	for (i = 0; i < XEN_BLKIF_REQS; i++) {
> +		req = kzalloc(sizeof(*req), GFP_KERNEL);
This still requires an order 2 allocation, right?  Can you make further
changes to make all allocations order 0?
> +		if (!req)
> +			goto fail;
> +		list_add_tail(&req->free_list,
> +		              &blkif->pending_free);
> +	}
David

On Wed, Apr 24, 2013 at 08:16:37PM +0200, Sander Eikelenboom
wrote:
> Friday, April 19, 2013, 4:44:01 PM, you wrote:
> 
> > Hey Jens,
> 
> > Please in your spare time (if there is such a thing at a conference)
> > pull this branch:
> 
> >  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
> 
.. snip..
> Hi Konrad / Roger,
> 
> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
> Without this pull it runs fine for several days.
.. snipp.
> [18496.013743] xenwatch: page allocation failure: order:7, mode:0x10c0d0
> [18496.031948] Pid: 54, comm: xenwatch Not tainted 3.9.0-rc8-20130424-jens+
#1
> [18496.049897] Call Trace:
> [18496.067674]  [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
> [18496.085453]  [<ffffffff810b25ed>] ? trace_hardirqs_on+0xd/0x10
> [18496.102951]  [<ffffffff810bdb24>] ? on_each_cpu_mask+0x94/0xd0
> [18496.120270]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
> [18496.137306]  [<ffffffff8113a161>] alloc_pages_current+0xb1/0x160
> [18496.154051]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
> [18496.170579]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
> [18496.186921]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
> [18496.202963]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
> [18496.218714]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> [18496.234237]  [<ffffffff8156a151>] driver_probe_device+0x81/0x220
> [18496.249605]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
> [18496.264681]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> [18496.279500]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
> [18496.294138]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
> [18496.308553]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
> [18496.322694]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
> [18496.336640]  [<ffffffff81567c80>] device_add+0x650/0x720
.. snip..

Jens,
I don''t know if you had pulled this git tree yet (I don''t see
it in
your for-3.10/* branches).

But if you have, Sander has found a bug (and Roger has a fix for it).

Whether you would like to wait until v3.11 to pull it (and me sending
the git pull around a month) is OK. Or pull it now and we will fix the
bugs in the -rc''s as they creep up.

Thanks!

Monday, April 29, 2013, 5:46:23 PM, you wrote:
> On Wed, Apr 24, 2013 at 08:16:37PM +0200, Sander Eikelenboom wrote:
>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>> 
>> > Hey Jens,
>> 
>> > Please in your spare time (if there is such a thing at a
conference)
>> > pull this branch:
>> 
>> >  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>> 
> .. snip..
>> Hi Konrad / Roger,
>> 
>> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
>> Without this pull it runs fine for several days.
> .. snipp.
>> [18496.013743] xenwatch: page allocation failure: order:7,
mode:0x10c0d0
>> [18496.031948] Pid: 54, comm: xenwatch Not tainted
3.9.0-rc8-20130424-jens+ #1
>> [18496.049897] Call Trace:
>> [18496.067674]  [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
>> [18496.085453]  [<ffffffff810b25ed>] ? trace_hardirqs_on+0xd/0x10
>> [18496.102951]  [<ffffffff810bdb24>] ? on_each_cpu_mask+0x94/0xd0
>> [18496.120270]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
>> [18496.137306]  [<ffffffff8113a161>]
alloc_pages_current+0xb1/0x160
>> [18496.154051]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
>> [18496.170579]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
>> [18496.186921]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
>> [18496.202963]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
>> [18496.218714]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
>> [18496.234237]  [<ffffffff8156a151>]
driver_probe_device+0x81/0x220
>> [18496.249605]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
>> [18496.264681]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
>> [18496.279500]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
>> [18496.294138]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
>> [18496.308553]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
>> [18496.322694]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
>> [18496.336640]  [<ffffffff81567c80>] device_add+0x650/0x720
> .. snip..
> Jens,
> I don''t know if you had pulled this git tree yet (I don''t
see it in
> your for-3.10/* branches).
> But if you have, Sander has found a bug (and Roger has a fix for it).
> Whether you would like to wait until v3.11 to pull it (and me sending
> the git pull around a month) is OK. Or pull it now and we will fix the
> bugs in the -rc''s as they creep up.
Roger''s fix seems to work for me ..

--
Sander
> Thanks!

On Mon, Apr 29 2013, Konrad Rzeszutek Wilk wrote:
> On Wed, Apr 24, 2013 at 08:16:37PM +0200, Sander Eikelenboom wrote:
> > Friday, April 19, 2013, 4:44:01 PM, you wrote:
> > 
> > > Hey Jens,
> > 
> > > Please in your spare time (if there is such a thing at a
conference)
> > > pull this branch:
> > 
> > >  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
> > 
> 
> .. snip..
> > Hi Konrad / Roger,
> > 
> > I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
> > Without this pull it runs fine for several days.
> .. snipp.
> 
> > [18496.013743] xenwatch: page allocation failure: order:7,
mode:0x10c0d0
> > [18496.031948] Pid: 54, comm: xenwatch Not tainted
3.9.0-rc8-20130424-jens+ #1
> > [18496.049897] Call Trace:
> > [18496.067674]  [<ffffffff81100c51>]
warn_alloc_failed+0xf1/0x140
> > [18496.085453]  [<ffffffff810b25ed>] ?
trace_hardirqs_on+0xd/0x10
> > [18496.102951]  [<ffffffff810bdb24>] ?
on_each_cpu_mask+0x94/0xd0
> > [18496.120270]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
> > [18496.137306]  [<ffffffff8113a161>]
alloc_pages_current+0xb1/0x160
> > [18496.154051]  [<ffffffff81100679>] __get_free_pages+0x9/0x40
> > [18496.170579]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
> > [18496.186921]  [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
> > [18496.202963]  [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
> > [18496.218714]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> > [18496.234237]  [<ffffffff8156a151>]
driver_probe_device+0x81/0x220
> > [18496.249605]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
> > [18496.264681]  [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
> > [18496.279500]  [<ffffffff8156a3db>] __device_attach+0x4b/0x50
> > [18496.294138]  [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
> > [18496.308553]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
> > [18496.322694]  [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
> > [18496.336640]  [<ffffffff81567c80>] device_add+0x650/0x720
> 
> .. snip..
> 
> Jens,
> I don''t know if you had pulled this git tree yet (I don''t
see it in
> your for-3.10/* branches).
> 
> But if you have, Sander has found a bug (and Roger has a fix for it).
> 
> Whether you would like to wait until v3.11 to pull it (and me sending
> the git pull around a month) is OK. Or pull it now and we will fix the
> bugs in the -rc''s as they creep up.
Thanks for letting me know. Lets postpone it for 3.11. If you have more
critical bits in your current tree, you can send them in over this week.

-- 
Jens Axboe

Hello Sander,

Monday, April 29, 2013, 6:05:20 PM, you wrote:

> Monday, April 29, 2013, 5:46:23 PM, you wrote:
>> On Wed, Apr 24, 2013 at 08:16:37PM +0200, Sander Eikelenboom wrote:
>>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>>> 
>>> > Hey Jens,
>>> 
>>> > Please in your spare time (if there is such a thing at a
conference)
>>> > pull this branch:
>>> 
>>> >  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>>> 
>> .. snip..
>>> Hi Konrad / Roger,
>>> 
>>> I tried this pull on top of latest Linus latest linux-3.9 tree, but
although it seems to boot and work fine at first, i seem to get trouble after
running for about a day.
>>> Without this pull it runs fine for several days.
>> .. snipp.
>>> [18496.013743] xenwatch: page allocation failure: order:7,
mode:0x10c0d0
>>> [18496.031948] Pid: 54, comm: xenwatch Not tainted
3.9.0-rc8-20130424-jens+ #1
>>> [18496.049897] Call Trace:
>>> [18496.067674]  [<ffffffff81100c51>]
warn_alloc_failed+0xf1/0x140
>>> [18496.085453]  [<ffffffff810b25ed>] ?
trace_hardirqs_on+0xd/0x10
>>> [18496.102951]  [<ffffffff810bdb24>] ?
on_each_cpu_mask+0x94/0xd0
>>> [18496.120270]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
>>> [18496.137306]  [<ffffffff8113a161>]
alloc_pages_current+0xb1/0x160
>>> [18496.154051]  [<ffffffff81100679>]
__get_free_pages+0x9/0x40
>>> [18496.170579]  [<ffffffff81142af4>] __kmalloc+0x134/0x160
>>> [18496.186921]  [<ffffffff815832d0>]
xen_blkbk_probe+0x170/0x2f0
>>> [18496.202963]  [<ffffffff81474ce7>]
xenbus_dev_probe+0x77/0x130
>>> [18496.218714]  [<ffffffff8156a390>] ?
__driver_attach+0xa0/0xa0
>>> [18496.234237]  [<ffffffff8156a151>]
driver_probe_device+0x81/0x220
>>> [18496.249605]  [<ffffffff8198198c>] ? klist_next+0x8c/0x110
>>> [18496.264681]  [<ffffffff8156a390>] ?
__driver_attach+0xa0/0xa0
>>> [18496.279500]  [<ffffffff8156a3db>]
__device_attach+0x4b/0x50
>>> [18496.294138]  [<ffffffff815684e8>]
bus_for_each_drv+0x68/0x90
>>> [18496.308553]  [<ffffffff8156a0c9>] device_attach+0x89/0x90
>>> [18496.322694]  [<ffffffff81569258>]
bus_probe_device+0xa8/0xd0
>>> [18496.336640]  [<ffffffff81567c80>] device_add+0x650/0x720
>> .. snip..
>> Jens,
>> I don''t know if you had pulled this git tree yet (I
don''t see it in
>> your for-3.10/* branches).
>> But if you have, Sander has found a bug (and Roger has a fix for it).
>> Whether you would like to wait until v3.11 to pull it (and me sending
>> the git pull around a month) is OK. Or pull it now and we will fix the
>> bugs in the -rc''s as they creep up.
> Roger''s fix seems to work for me ..
Hmm although it takes longer, i still see my memory getting souped-up.
Will see what setting:

echo 384 > /sys/module/xen_blkback/parameters/max_persistent_grants
echo 256 >> /sys/module/xen_blkback/parameters/max_buffer_pages

will do this time around.

--
Sander


(XEN) [2013-05-03 23:00:21] grant_table.c:1250:d1 Expanding dom (1) grant table
from (7) to (8) frames.
(XEN) [2013-05-03 23:00:41] grant_table.c:1250:d1 Expanding dom (1) grant table
from (8) to (9) frames.
(XEN) [2013-05-03 23:01:01] grant_table.c:1250:d1 Expanding dom (1) grant table
from (9) to (10) frames.
(XEN) [2013-05-03 23:01:01] grant_table.c:1250:d1 Expanding dom (1) grant table
from (10) to (11) frames.
(XEN) [2013-05-04 03:15:32] grant_table.c:289:d0 Increased maptrack size to 10
frames
(XEN) [2013-05-04 03:15:34] grant_table.c:1250:d9 Expanding dom (9) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:34] grant_table.c:1250:d9 Expanding dom (9) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:34] grant_table.c:1250:d9 Expanding dom (9) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:34] grant_table.c:289:d0 Increased maptrack size to 11
frames
(XEN) [2013-05-04 03:15:36] grant_table.c:1250:d8 Expanding dom (8) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:36] grant_table.c:1250:d8 Expanding dom (8) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:36] grant_table.c:289:d0 Increased maptrack size to 12
frames
(XEN) [2013-05-04 03:15:36] grant_table.c:289:d0 Increased maptrack size to 13
frames
(XEN) [2013-05-04 03:15:37] grant_table.c:1250:d4 Expanding dom (4) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:37] grant_table.c:1250:d4 Expanding dom (4) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:37] grant_table.c:1250:d4 Expanding dom (4) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:37] grant_table.c:289:d0 Increased maptrack size to 14
frames
(XEN) [2013-05-04 03:15:37] grant_table.c:289:d0 Increased maptrack size to 15
frames
(XEN) [2013-05-04 03:15:39] grant_table.c:1250:d3 Expanding dom (3) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:39] grant_table.c:1250:d3 Expanding dom (3) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to 16
frames
(XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to 17
frames
(XEN) [2013-05-04 03:15:39] grant_table.c:1250:d13 Expanding dom (13) grant
table from (4) to (5) frames.
(XEN) [2013-05-04 03:15:39] grant_table.c:1250:d13 Expanding dom (13) grant
table from (5) to (6) frames.
(XEN) [2013-05-04 03:15:39] grant_table.c:1250:d13 Expanding dom (13) grant
table from (6) to (7) frames.
(XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to 18
frames
(XEN) [2013-05-04 03:15:39] grant_table.c:1250:d3 Expanding dom (3) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to 19
frames
(XEN) [2013-05-04 03:15:40] grant_table.c:1250:d8 Expanding dom (8) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:42] grant_table.c:1250:d2 Expanding dom (2) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:42] grant_table.c:1250:d2 Expanding dom (2) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:42] grant_table.c:289:d0 Increased maptrack size to 20
frames
(XEN) [2013-05-04 03:15:42] grant_table.c:289:d0 Increased maptrack size to 21
frames
(XEN) [2013-05-04 03:15:42] grant_table.c:1250:d2 Expanding dom (2) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:43] grant_table.c:1250:d10 Expanding dom (10) grant
table from (4) to (5) frames.
(XEN) [2013-05-04 03:15:44] grant_table.c:1250:d10 Expanding dom (10) grant
table from (5) to (6) frames.
(XEN) [2013-05-04 03:15:44] grant_table.c:289:d0 Increased maptrack size to 22
frames
(XEN) [2013-05-04 03:15:44] grant_table.c:1250:d10 Expanding dom (10) grant
table from (6) to (7) frames.
(XEN) [2013-05-04 03:15:45] grant_table.c:1250:d5 Expanding dom (5) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:45] grant_table.c:1250:d5 Expanding dom (5) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:45] grant_table.c:1250:d5 Expanding dom (5) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:45] grant_table.c:289:d0 Increased maptrack size to 23
frames
(XEN) [2013-05-04 03:15:45] grant_table.c:289:d0 Increased maptrack size to 24
frames
(XEN) [2013-05-04 03:15:50] grant_table.c:1250:d7 Expanding dom (7) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 03:15:50] grant_table.c:1250:d7 Expanding dom (7) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 03:15:50] grant_table.c:1250:d7 Expanding dom (7) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 03:15:50] grant_table.c:289:d0 Increased maptrack size to 25
frames
(XEN) [2013-05-04 03:15:52] grant_table.c:289:d0 Increased maptrack size to 26
frames
(XEN) [2013-05-04 03:16:04] grant_table.c:1250:d12 Expanding dom (12) grant
table from (4) to (5) frames.
(XEN) [2013-05-04 03:16:05] grant_table.c:1250:d12 Expanding dom (12) grant
table from (5) to (6) frames.
(XEN) [2013-05-04 03:16:05] grant_table.c:289:d0 Increased maptrack size to 27
frames
(XEN) [2013-05-04 03:16:05] grant_table.c:1250:d12 Expanding dom (12) grant
table from (6) to (7) frames.
(XEN) [2013-05-04 05:15:05] grant_table.c:289:d0 Increased maptrack size to 28
frames
(XEN) [2013-05-04 05:15:09] grant_table.c:1250:d6 Expanding dom (6) grant table
from (4) to (5) frames.
(XEN) [2013-05-04 05:15:09] grant_table.c:1250:d6 Expanding dom (6) grant table
from (5) to (6) frames.
(XEN) [2013-05-04 05:15:09] grant_table.c:1250:d6 Expanding dom (6) grant table
from (6) to (7) frames.
(XEN) [2013-05-04 05:15:09] grant_table.c:289:d0 Increased maptrack size to 29
frames

> --
> Sander
>> Thanks!

On 04/05/13 09:34, Sander Eikelenboom wrote:
> Hello Sander,
> 
> Monday, April 29, 2013, 6:05:20 PM, you wrote:
> 
> 
>> Monday, April 29, 2013, 5:46:23 PM, you wrote:
> 
>>> On Wed, Apr 24, 2013 at 08:16:37PM +0200, Sander Eikelenboom wrote:
>>>> Friday, April 19, 2013, 4:44:01 PM, you wrote:
>>>>
>>>>> Hey Jens,
>>>>
>>>>> Please in your spare time (if there is such a thing at a
conference)
>>>>> pull this branch:
>>>>
>>>>> 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
stable/for-jens-3.10
>>>>
> 
>>> .. snip..
>>>> Hi Konrad / Roger,
>>>>
>>>> I tried this pull on top of latest Linus latest linux-3.9 tree,
but although it seems to boot and work fine at first, i seem to get trouble
after running for about a day.
>>>> Without this pull it runs fine for several days.
>>> .. snipp.
> 
>>>> [18496.013743] xenwatch: page allocation failure: order:7,
mode:0x10c0d0
>>>> [18496.031948] Pid: 54, comm: xenwatch Not tainted
3.9.0-rc8-20130424-jens+ #1
>>>> [18496.049897] Call Trace:
>>>> [18496.067674]  [<ffffffff81100c51>]
warn_alloc_failed+0xf1/0x140
>>>> [18496.085453]  [<ffffffff810b25ed>] ?
trace_hardirqs_on+0xd/0x10
>>>> [18496.102951]  [<ffffffff810bdb24>] ?
on_each_cpu_mask+0x94/0xd0
>>>> [18496.120270]  [<ffffffff811028af>]
__alloc_pages_nodemask+0x69f/0x960
>>>> [18496.137306]  [<ffffffff8113a161>]
alloc_pages_current+0xb1/0x160
>>>> [18496.154051]  [<ffffffff81100679>]
__get_free_pages+0x9/0x40
>>>> [18496.170579]  [<ffffffff81142af4>]
__kmalloc+0x134/0x160
>>>> [18496.186921]  [<ffffffff815832d0>]
xen_blkbk_probe+0x170/0x2f0
>>>> [18496.202963]  [<ffffffff81474ce7>]
xenbus_dev_probe+0x77/0x130
>>>> [18496.218714]  [<ffffffff8156a390>] ?
__driver_attach+0xa0/0xa0
>>>> [18496.234237]  [<ffffffff8156a151>]
driver_probe_device+0x81/0x220
>>>> [18496.249605]  [<ffffffff8198198c>] ?
klist_next+0x8c/0x110
>>>> [18496.264681]  [<ffffffff8156a390>] ?
__driver_attach+0xa0/0xa0
>>>> [18496.279500]  [<ffffffff8156a3db>]
__device_attach+0x4b/0x50
>>>> [18496.294138]  [<ffffffff815684e8>]
bus_for_each_drv+0x68/0x90
>>>> [18496.308553]  [<ffffffff8156a0c9>]
device_attach+0x89/0x90
>>>> [18496.322694]  [<ffffffff81569258>]
bus_probe_device+0xa8/0xd0
>>>> [18496.336640]  [<ffffffff81567c80>]
device_add+0x650/0x720
> 
>>> .. snip..
> 
>>> Jens,
>>> I don''t know if you had pulled this git tree yet (I
don''t see it in
>>> your for-3.10/* branches).
> 
>>> But if you have, Sander has found a bug (and Roger has a fix for
it).
> 
>>> Whether you would like to wait until v3.11 to pull it (and me
sending
>>> the git pull around a month) is OK. Or pull it now and we will fix
the
>>> bugs in the -rc''s as they creep up.
> 
>> Roger''s fix seems to work for me ..
> 
> Hmm although it takes longer, i still see my memory getting souped-up.
> Will see what setting:
> 
> echo 384 > /sys/module/xen_blkback/parameters/max_persistent_grants
> echo 256 >> /sys/module/xen_blkback/parameters/max_buffer_pages
> 
> will do this time around.
The error message is the same?
> 
> --
> Sander
> 
> 
> (XEN) [2013-05-03 23:00:21] grant_table.c:1250:d1 Expanding dom (1) grant
table from (7) to (8) frames.
> (XEN) [2013-05-03 23:00:41] grant_table.c:1250:d1 Expanding dom (1) grant
table from (8) to (9) frames.
> (XEN) [2013-05-03 23:01:01] grant_table.c:1250:d1 Expanding dom (1) grant
table from (9) to (10) frames.
> (XEN) [2013-05-03 23:01:01] grant_table.c:1250:d1 Expanding dom (1) grant
table from (10) to (11) frames.
> (XEN) [2013-05-04 03:15:32] grant_table.c:289:d0 Increased maptrack size to
10 frames
> (XEN) [2013-05-04 03:15:34] grant_table.c:1250:d9 Expanding dom (9) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:34] grant_table.c:1250:d9 Expanding dom (9) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:34] grant_table.c:1250:d9 Expanding dom (9) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:34] grant_table.c:289:d0 Increased maptrack size to
11 frames
> (XEN) [2013-05-04 03:15:36] grant_table.c:1250:d8 Expanding dom (8) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:36] grant_table.c:1250:d8 Expanding dom (8) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:36] grant_table.c:289:d0 Increased maptrack size to
12 frames
> (XEN) [2013-05-04 03:15:36] grant_table.c:289:d0 Increased maptrack size to
13 frames
> (XEN) [2013-05-04 03:15:37] grant_table.c:1250:d4 Expanding dom (4) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:37] grant_table.c:1250:d4 Expanding dom (4) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:37] grant_table.c:1250:d4 Expanding dom (4) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:37] grant_table.c:289:d0 Increased maptrack size to
14 frames
> (XEN) [2013-05-04 03:15:37] grant_table.c:289:d0 Increased maptrack size to
15 frames
> (XEN) [2013-05-04 03:15:39] grant_table.c:1250:d3 Expanding dom (3) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:39] grant_table.c:1250:d3 Expanding dom (3) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to
16 frames
> (XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to
17 frames
> (XEN) [2013-05-04 03:15:39] grant_table.c:1250:d13 Expanding dom (13) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:39] grant_table.c:1250:d13 Expanding dom (13) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:39] grant_table.c:1250:d13 Expanding dom (13) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to
18 frames
> (XEN) [2013-05-04 03:15:39] grant_table.c:1250:d3 Expanding dom (3) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:39] grant_table.c:289:d0 Increased maptrack size to
19 frames
> (XEN) [2013-05-04 03:15:40] grant_table.c:1250:d8 Expanding dom (8) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:42] grant_table.c:1250:d2 Expanding dom (2) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:42] grant_table.c:1250:d2 Expanding dom (2) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:42] grant_table.c:289:d0 Increased maptrack size to
20 frames
> (XEN) [2013-05-04 03:15:42] grant_table.c:289:d0 Increased maptrack size to
21 frames
> (XEN) [2013-05-04 03:15:42] grant_table.c:1250:d2 Expanding dom (2) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:43] grant_table.c:1250:d10 Expanding dom (10) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:44] grant_table.c:1250:d10 Expanding dom (10) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:44] grant_table.c:289:d0 Increased maptrack size to
22 frames
> (XEN) [2013-05-04 03:15:44] grant_table.c:1250:d10 Expanding dom (10) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:45] grant_table.c:1250:d5 Expanding dom (5) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:45] grant_table.c:1250:d5 Expanding dom (5) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:45] grant_table.c:1250:d5 Expanding dom (5) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:45] grant_table.c:289:d0 Increased maptrack size to
23 frames
> (XEN) [2013-05-04 03:15:45] grant_table.c:289:d0 Increased maptrack size to
24 frames
> (XEN) [2013-05-04 03:15:50] grant_table.c:1250:d7 Expanding dom (7) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:15:50] grant_table.c:1250:d7 Expanding dom (7) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:15:50] grant_table.c:1250:d7 Expanding dom (7) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 03:15:50] grant_table.c:289:d0 Increased maptrack size to
25 frames
> (XEN) [2013-05-04 03:15:52] grant_table.c:289:d0 Increased maptrack size to
26 frames
> (XEN) [2013-05-04 03:16:04] grant_table.c:1250:d12 Expanding dom (12) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 03:16:05] grant_table.c:1250:d12 Expanding dom (12) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 03:16:05] grant_table.c:289:d0 Increased maptrack size to
27 frames
> (XEN) [2013-05-04 03:16:05] grant_table.c:1250:d12 Expanding dom (12) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 05:15:05] grant_table.c:289:d0 Increased maptrack size to
28 frames
> (XEN) [2013-05-04 05:15:09] grant_table.c:1250:d6 Expanding dom (6) grant
table from (4) to (5) frames.
> (XEN) [2013-05-04 05:15:09] grant_table.c:1250:d6 Expanding dom (6) grant
table from (5) to (6) frames.
> (XEN) [2013-05-04 05:15:09] grant_table.c:1250:d6 Expanding dom (6) grant
table from (6) to (7) frames.
> (XEN) [2013-05-04 05:15:09] grant_table.c:289:d0 Increased maptrack size to
29 frames
All this messages are expected.