[Xen-users] No buffer space available - loses network connectivity

Sherin George
Thu, 01 Sep 2011 04:36:44 -0000

Hi,

I have a centos 5.6 xen vps which loses network connectivity once in a
while with following error.

========================================-bash-3.2# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
========================================
All my investigation so far led me to believe that it is because
skbuff cache getting full.

========================================================================
PROC-SLABINFO
skbuff_fclone_cache    227    308    512    7    1 : tunables   54
27    8 : slabdata     44     44      0
skbuff_head_cache   1574   1650    256   15    1 : tunables  120   60
 8 : slabdata    110    110      0

SLAB-TOP
 Active / Total Objects (% used)    : 2140910 / 2200115 (97.3%)
 Active / Total Slabs (% used)      : 139160 / 139182 (100.0%)
 Active / Total Caches (% used)     : 88 / 136 (64.7%)
 Active / Total Size (% used)       : 512788.94K / 520252.14K (98.6%)
 Minimum / Average / Maximum Object : 0.02K / 0.24K / 128.00K

 OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
664000 620290  93%    0.09K  16600       40     66400K buffer_head
409950 408396  99%    0.21K  22775       18     91100K dentry_cache
343056 340307  99%    0.08K   7147       48     28588K selinux_inode_security
338590 336756  99%    0.74K  67718        5    270872K ext3_inode_cache
143665 143363  99%    0.06K   2435       59      9740K size-64
 99540  99407  99%    0.25K   6636       15     26544K size-256
 96450  96447  99%    0.12K   3215       30     12860K size-128
 60858  60858 100%    0.52K   8694        7     34776K radix_tree_node
 12420  11088  89%    0.16K    540       23      2160K vm_area_struct
 5895   4185  70%    0.25K    393       15      1572K filp
 4816   3355  69%    0.03K     43      112       172K size-32
 2904   2810  96%    0.09K     66       44       264K sysfs_dir_cache
 2058   1937  94%    0.58K    343        6      1372K proc_inode_cache
 1728   1215  70%    0.02K     12      144        48K anon_vma
 1650   1590  96%    0.25K    110       15       440K skbuff_head_cache
 1498   1493  99%    2.00K    749        2      2996K size-2048
 1050   1032  98%    0.55K    150        7       600K inode_cache
  792    767  96%    1.00K    198        4       792K size-1024
  649    298  45%    0.06K     11       59        44K pid
  600    227  37%    0.09K     15       40        60K journal_head
  590    298  50%    0.06K     10       59        40K delayacct_cache
  496    424  85%    0.50K     62        8       248K size-512
  413    156  37%    0.06K      7       59        28K fs_cache
  404     44  10%    0.02K      2      202         8K biovec-1
  390    293  75%    0.12K     13       30        52K bio
  327    327 100%    4.00K    327        1      1308K size-4096
  320    190  59%    0.38K     32       10       128K ip_dst_cache
  308    227  73%    0.50K     44        7       176K skbuff_fclone_cache
  258    247  95%    0.62K     43        6       172K sock_inode_cache
  254    254 100%    1.84K    127        2       508K task_struct
  252    225  89%    0.81K     28        9       224K signal_cache
  240    203  84%    0.73K     48        5       192K shmem_inode_cache
  204    204 100%    2.06K     68        3       544K sighand_cache
  202      4   1%    0.02K      1      202         4K revoke_table
  195    194  99%    0.75K     39        5       156K UDP
  159     77  48%    0.07K      3       53        12K eventpoll_pwq
  145    139  95%    0.75K     29        5       116K files_cache
  144     41  28%    0.02K      1      144         4K journal_handle
  140    140 100%    0.88K     35        4       140K mm_struct
  140     77  55%    0.19K      7       20        28K eventpoll_epi
  135    135 100%    2.12K    135        1       540K kmem_cache
  121     45  37%    0.69K     11       11        88K UNIX
  119    114  95%    0.52K     17        7        68K idr_layer_cache
  118     41  34%    0.06K      2       59         8K blkdev_ioc
  112     32  28%    0.03K      1      112         4K tcp_bind_bucket
  110     56  50%    0.17K      5       22        20K file_lock_cache
  106     35  33%    0.07K      2       53         8K avc_node
  105     98  93%    1.50K     21        5       168K TCP
  105    100  95%    1.04K     15        7       120K bio_map_info
   92      1   1%    0.04K      1       92         4K dnotify_cache
   80     18  22%    0.19K      4       20        16K tw_sock_TCP
   70     44  62%    0.27K      5       14        20K blkdev_requests
   59     23  38%    0.06K      1       59         4K biovec-4
   59     13  22%    0.06K      1       59         4K fib6_nodes
   59     11  18%    0.06K      1       59         4K ip_fib_hash
   59     11  18%    0.06K      1       59         4K ip_fib_alias
   53     53 100%    0.07K      1       53         4K taskstats_cache
   53      1   1%    0.07K      1       53         4K inotify_watch_cache
   48     48 100%    0.16K      2       24         8K sigqueue
   48      8  16%    0.08K      1       48         4K crq_pool
   45     27  60%    0.25K      3       15        12K mnt_cache
   45     29  64%    0.25K      3       15        12K dquot
   45     32  71%    0.25K      3       15        12K sgpool-8
   40     19  47%    0.19K      2       20         8K key_jar
   32     32 100%    0.50K      4        8        16K sgpool-16
   32     32 100%    1.00K      8        4        32K sgpool-32
   32     32 100%    2.00K     16        2        64K sgpool-64
   32     32 100%    4.00K     32        1       128K sgpool-128
   31     31 100%    8.00K     31        1       248K size-8192
   30     27  90%    1.54K      6        5        48K blkdev_queue
   30     14  46%    0.12K      1       30         4K request_sock_TCP
   30      3  10%    0.12K      1       30         4K inet_peer_cache

FREE-M
            total       used       free     shared    buffers     cached
Mem:          8192       4821       3370          0        500       2793
-/+ buffers/cache:       1527       6664
Swap:         8191          0       8191

PROC-MEMINFO
MemTotal:      8388608 kB
MemFree:       3451384 kB
Buffers:        512352 kB
Cached:        2860580 kB
SwapCached:          0 kB
Active:        2971812 kB
Inactive:      1178908 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8388608 kB
LowFree:       3451384 kB
SwapTotal:     8388600 kB
SwapFree:      8388600 kB
Dirty:             644 kB
Writeback:           0 kB
AnonPages:      777524 kB
Mapped:          24816 kB
Slab:           557716 kB
PageTables:      16300 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  12582904 kB
Committed_AS:  1624128 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      4556 kB
VmallocChunk: 34359733771 kB
==========================================================
Can any one provide any insights into a possible solution of this issue.

--
Kind Regards,
Sherin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com