Using 2-node clustered file system on DELL/EMC SAN/RHEL 2.6.9-34.0.2.ELsmp x86_64. Config: O2CB_HEARTBEAT_THRESHOLD=30 Kernel param: elavator=deadline (per FAQ) These log items appear and the server crashes. Has happened twice now at three week intervals, each time during a heavy IO operation: Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_setup_one_bio:371 ERROR: Could not alloc slots BIO! Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_read_slots:507 ERROR: status = -12 Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_do_disk_heartbeat:973 ERROR: status = -12 Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_setup_one_bio:371 ERROR: Could not alloc slots BIO! Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_read_slots:507 ERROR: status = -12 Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_do_disk_heartbeat:973 ERROR: status Can't find much on any of these errors?what is 507 ERROR status = -12? Any help appreciated -- Brian
Looks to be running out of lowmem. # date # cat /proc/meminfo # cat /proc/slabinfo Run a script that dumps the above every 1 to 5 mins. That should help explain the cause. Brian Sieler wrote:> Using 2-node clustered file system on DELL/EMC SAN/RHEL > 2.6.9-34.0.2.ELsmp x86_64. > > Config: > > O2CB_HEARTBEAT_THRESHOLD=30 > > Kernel param: elavator=deadline (per FAQ) > > These log items appear and the server crashes. Has happened twice now > at three week intervals, each time during a heavy IO operation: > > Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_setup_one_bio:371 ERROR: > Could not alloc slots BIO! > Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_read_slots:507 ERROR: > status = -12 > Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_do_disk_heartbeat:973 > ERROR: status = -12 > Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_setup_one_bio:371 ERROR: > Could not alloc slots BIO! > Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_read_slots:507 ERROR: > status = -12 > Jan 15 16:08:29 db100 kernel: (3898,6):o2hb_do_disk_heartbeat:973 > ERROR: status > > Can't find much on any of these errors?what is 507 ERROR status = -12? > > Any help appreciated >