Ward Fenton
2009-Mar-09 22:52 UTC
[Ocfs2-users] Experiencing occasional system crashes with RHEL5 and ocfs2 1.2.9
We have been experiencing unplanned outages on a subset of the clustered
systems we have deployed to support SAP. The following captured information
came from one of three ocfs2 clusters which handle SAP SEM/BW functionality.
Each of those has experienced multiple kernel panics which get reported as
dlm_drop_lockres_ref and dlm_defer_lockres_handler errors.
This particular environment is deployed on two identical HP DL580-G5 servers
running RHEL5.2 with kernel 2.6.18-92.1.1.el5. It should be noted that we
have encountered similar issues with similar frequency on kernel
2.6.18-92.1.22.el5. We are running ocfs2 1.2.9. These two servers support
Oracle RAC (10.2.0.2) and they also sharing through NFS two ocfs2 file
systems to associated SAP application servers. Database devices and RMAN
backups are supported by ocfs2 file systems with "datavolume,nointr"
options. The additional cluster file systems which support Oracle and SAP do
not contain those options.
We have encountered recurring unplanned reboots which appear to be initiated
by the crashing of either node in the cluster. This particular cluster had
experienced a crashed of the other cluster member one week previous to this
most recent crash. Both crashes had occurred at times with very minimal
activity.
Support for netconsole is implemented and we are redirecting the kernel's
console output to a central syslog server. The logs below originated from
netconsole. Though not yet in place at the time of this crash, kexec/kdump
functionality was also introduced.
Thanks,
Ward
################################################################################
netconsole capture of rebooted server:
################################################################################
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_drop_lockres_ref:2298 ERROR:
while dropping ref on
F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26 (master=1)
got -22.
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_one_lock_resource:461
lockres: M00000000000000003c1c868d5afc26, owner=1, state=64
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:476
lockres: M00000000000000003c1c868d5afc26, owner=1, state=64
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:478
last used: 4728024828, on purge list: yes
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_lockres_refmap:444
refmap nodes: [
Mar 7 07:45:25 sapbdb-uat-d2 ], inflight=0
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:480
granted queue:
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:495
converting queue:
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:510
blocked queue:
Mar 7 07:45:25 sapbdb-uat-d2 ----------- [cut here ] --------- [please bite
here ] ---------
Mar 7 07:45:25 sapbdb-uat-d2 Kernel BUG at
...mushran/BUILD/ocfs2-1.2.9/fs/ocfs2/dlm/dlmmaster.c:2300
Mar 7 07:45:25 sapbdb-uat-d2 invalid opcode: 0000 [1]
Mar 7 07:45:25 sapbdb-uat-d2 SMP
Mar 7 07:45:25 sapbdb-uat-d2
Mar 7 07:45:25 sapbdb-uat-d2 last sysfs file:
/devices/pci0000:00/0000:00:00.0/irq
Mar 7 07:45:25 sapbdb-uat-d2 CPU 2
Mar 7 07:45:25 sapbdb-uat-d2
Mar 7 07:45:25 sapbdb-uat-d2 Modules linked in:
Mar 7 07:45:25 sapbdb-uat-d2 iptable_filter
Mar 7 07:45:26 sapbdb-uat-d2 ip_tables
Mar 7 07:45:26 sapbdb-uat-d2 x_tables
Mar 7 07:45:26 sapbdb-uat-d2 netconsole
Mar 7 07:45:26 sapbdb-uat-d2 nfsd
Mar 7 07:45:26 sapbdb-uat-d2 exportfs
Mar 7 07:45:26 sapbdb-uat-d2 auth_rpcgss
Mar 7 07:45:26 sapbdb-uat-d2 autofs4
Mar 7 07:45:26 sapbdb-uat-d2 hidp
Mar 7 07:45:26 sapbdb-uat-d2 ocfs2(U)
Mar 7 07:45:26 sapbdb-uat-d2 nfs
Mar 7 07:45:26 sapbdb-uat-d2 lockd
Mar 7 07:45:26 sapbdb-uat-d2 fscache
Mar 7 07:45:26 sapbdb-uat-d2 nfs_acl
Mar 7 07:45:26 sapbdb-uat-d2 rfcomm
Mar 7 07:45:26 sapbdb-uat-d2 l2cap
Mar 7 07:45:26 sapbdb-uat-d2 bluetooth
Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlmfs(U)
Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlm(U)
Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_nodemanager(U)
Mar 7 07:45:26 sapbdb-uat-d2 lock_dlm(U)
Mar 7 07:45:26 sapbdb-uat-d2 gfs2(U)
Mar 7 07:45:26 sapbdb-uat-d2 dlm
Mar 7 07:45:26 sapbdb-uat-d2 configfs
Mar 7 07:45:26 sapbdb-uat-d2 sunrpc
Mar 7 07:45:26 sapbdb-uat-d2 bonding
Mar 7 07:45:26 sapbdb-uat-d2 ipv6
Mar 7 07:45:26 sapbdb-uat-d2 xfrm_nalgo
Mar 7 07:45:26 sapbdb-uat-d2 crypto_api
Mar 7 07:45:26 sapbdb-uat-d2 dm_round_robin
Mar 7 07:45:26 sapbdb-uat-d2 dm_emc
Mar 7 07:45:26 sapbdb-uat-d2 dm_multipath
Mar 7 07:45:26 sapbdb-uat-d2 video
Mar 7 07:45:26 sapbdb-uat-d2 sbs
Mar 7 07:45:26 sapbdb-uat-d2 backlight
Mar 7 07:45:26 sapbdb-uat-d2 i2c_ec
Mar 7 07:45:26 sapbdb-uat-d2 i2c_core
Mar 7 07:45:26 sapbdb-uat-d2 button
Mar 7 07:45:26 sapbdb-uat-d2 battery
Mar 7 07:45:26 sapbdb-uat-d2 asus_acpi
Mar 7 07:45:26 sapbdb-uat-d2 acpi_memhotplug
Mar 7 07:45:26 sapbdb-uat-d2 ac
Mar 7 07:45:26 sapbdb-uat-d2 parport_pc
Mar 7 07:45:26 sapbdb-uat-d2 lp
Mar 7 07:45:26 sapbdb-uat-d2 parport
Mar 7 07:45:26 sapbdb-uat-d2 joydev
Mar 7 07:45:26 sapbdb-uat-d2 ata_piix
Mar 7 07:45:26 sapbdb-uat-d2 libata
Mar 7 07:45:26 sapbdb-uat-d2 sg
Mar 7 07:45:26 sapbdb-uat-d2 ide_cd
Mar 7 07:45:26 sapbdb-uat-d2 shpchp
Mar 7 07:45:27 sapbdb-uat-d2 cdrom
Mar 7 07:45:27 sapbdb-uat-d2 bnx2
Mar 7 07:45:27 sapbdb-uat-d2 pcspkr
Mar 7 07:45:27 sapbdb-uat-d2 serio_raw
Mar 7 07:45:27 sapbdb-uat-d2 dm_snapshot
Mar 7 07:45:27 sapbdb-uat-d2 dm_zero
Mar 7 07:45:27 sapbdb-uat-d2 dm_mirror
Mar 7 07:45:27 sapbdb-uat-d2 dm_mod
Mar 7 07:45:27 sapbdb-uat-d2 qla2xxx
Mar 7 07:45:27 sapbdb-uat-d2 scsi_transport_fc
Mar 7 07:45:27 sapbdb-uat-d2 cciss
Mar 7 07:45:27 sapbdb-uat-d2 sd_mod
Mar 7 07:45:27 sapbdb-uat-d2 scsi_mod
Mar 7 07:45:27 sapbdb-uat-d2 ext3
Mar 7 07:45:27 sapbdb-uat-d2 jbd
Mar 7 07:45:27 sapbdb-uat-d2 uhci_hcd
Mar 7 07:45:27 sapbdb-uat-d2 ohci_hcd
Mar 7 07:45:27 sapbdb-uat-d2 ehci_hcd
Mar 7 07:45:27 sapbdb-uat-d2
Mar 7 07:45:27 sapbdb-uat-d2 Pid: 11241, comm: dlm_thread Tainted: G
2.6.18-92.1.1.el5 #1
Mar 7 07:45:27 sapbdb-uat-d2 RIP: 0010:[<ffffffff885ba6a0>]
Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885ba6a0>]
:ocfs2_dlm:dlm_drop_lockres_ref+0x1dc/0x1f5
Mar 7 07:45:27 sapbdb-uat-d2 RSP: 0018:ffff8105e1395de0 EFLAGS: 00010246
Mar 7 07:45:27 sapbdb-uat-d2 RAX: ffff81005ef3fe38 RBX: 0000000000000000
RCX: ffffffff802ed9a8
Mar 7 07:45:27 sapbdb-uat-d2 RDX: ffffffff802ed9a8 RSI: 0000000000000000
RDI: ffffffff802ed9a0
Mar 7 07:45:27 sapbdb-uat-d2 RBP: 000000000000001f R08: ffffffff802ed9a8
R09: 0000000000000046
Mar 7 07:45:27 sapbdb-uat-d2 R10: 0000000000000000 R11: 0000000000000280
R12: ffff81005ef3fe00
Mar 7 07:45:27 sapbdb-uat-d2 R13: ffff8105e227fc00 R14: ffff81002aec2d40
R15: ffffffff8009dced
Mar 7 07:45:27 sapbdb-uat-d2 FS: 0000000000000000(0000)
GS:ffff81102ff1ce40(0000) knlGS:0000000000000000
Mar 7 07:45:27 sapbdb-uat-d2 CS: 0010 DS: 0018 ES: 0018 CR0:
000000008005003b
Mar 7 07:45:27 sapbdb-uat-d2 CR2: 00000000006bdc5c CR3: 0000000594e3c000
CR4: 00000000000006e0
Mar 7 07:45:27 sapbdb-uat-d2 Process dlm_thread (pid: 11241, threadinfo
ffff8105e1394000, task ffff8105e30a2820)
Mar 7 07:45:27 sapbdb-uat-d2 Stack:
Mar 7 07:45:27 sapbdb-uat-d2 1f02000000000000
Mar 7 07:45:27 sapbdb-uat-d2 303030303030304d
Mar 7 07:45:27 sapbdb-uat-d2 3030303030303030
Mar 7 07:45:27 sapbdb-uat-d2 3836386331633330
Mar 7 07:45:27 sapbdb-uat-d2
Mar 7 07:45:27 sapbdb-uat-d2 0036326366613564
Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000
Mar 7 07:45:27 sapbdb-uat-d2 last message repeated 2 times
Mar 7 07:45:27 sapbdb-uat-d2
Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000
Mar 7 07:45:27 sapbdb-uat-d2 ffffffea00100100
Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe48
Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe00
Mar 7 07:45:27 sapbdb-uat-d2
Mar 7 07:45:27 sapbdb-uat-d2 Call Trace:
Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cd72b>]
:ocfs2_dlm:dlm_purge_lockres+0x175/0x34f
Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdb98>]
:ocfs2_dlm:dlm_thread+0xd7/0x579
Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff8009df05>]
autoremove_wake_function+0x0/0x2e
Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdac1>]
:ocfs2_dlm:dlm_thread+0x0/0x579
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003253d>] kthread+0xfe/0x132
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfb1>] child_rip+0xa/0x11
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8009dced>]
keventd_create_kthread+0x0/0xc4
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003243f>] kthread+0x0/0x132
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfa7>] child_rip+0x0/0x11
Mar 7 07:45:28 sapbdb-uat-d2
Mar 7 07:45:28 sapbdb-uat-d2
Mar 7 07:45:28 sapbdb-uat-d2 Code:
Mar 7 07:45:28 sapbdb-uat-d2 0f
Mar 7 07:45:28 sapbdb-uat-d2 0b
Mar 7 07:45:28 sapbdb-uat-d2 68
Mar 7 07:45:28 sapbdb-uat-d2 2d
Mar 7 07:45:28 sapbdb-uat-d2 10
Mar 7 07:45:28 sapbdb-uat-d2 5d
Mar 7 07:45:28 sapbdb-uat-d2 88
Mar 7 07:45:28 sapbdb-uat-d2 c2
Mar 7 07:45:28 sapbdb-uat-d2 fc
################################################################################
netconsole capture for remaining server:
################################################################################
Mar 7 07:45:25 sapbdb-uat-d1 (9917,0):dlm_deref_lockres_handler:2340 ERROR:
F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26: bad
lockres name
Mar 7 07:45:40 sapbdb-uat-d1 dlm: closing connection to node 2
Mar 7 07:45:55 sapbdb-uat-d1 o2net: connection to node sapbdb-uat-d2 (num
2) at 192.168.1.2:7777 has been idle for 30.0 seconds, shutting it down.
Mar 7 07:45:55 sapbdb-uat-d1 (0,0):o2net_idle_timer:1426 here are some
times that might help debug the situation: (tmr 1236429925.620147 now
1236429955.618766 dr 1236429925.620142 adv
1236429925.622211:1236429925.622212 func (90648c68:507)
1236429925.620148:1236429925.622206)
Mar 7 07:45:55 sapbdb-uat-d1 o2net: no longer connected to node
sapbdb-uat-d2 (num 2) at 192.168.1.2:7777
Mar 7 07:45:55 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293 ERROR:
status = -112
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -112
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -112
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_do_master_request:1418 ERROR:
link to 2 went down!
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:995 ERROR:
status = -112
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR: status
= -112
Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293 ERROR:
status = -107
Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR: status
= -107
Mar 7 07:46:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 ERROR: no
connection established with node 2 after 30.0 seconds, giving up and
returning errors.
Mar 7 07:46:55 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 ERROR: no
connection established with node 2 after 30.0 seconds, giving up and
returning errors.
Mar 7 07:47:01 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:01 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:06 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:06 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:11 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:11 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:17 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:17 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:22 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:22 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 ERROR: no
connection established with node 2 after 30.0 seconds, giving up and
returning errors.
Mar 7 07:47:27 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:27 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32060,5):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32060,5):dlm_wait_for_node_death:365
EF2FF243631540E0A9418F491C21D661: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32062,11):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32062,11):dlm_wait_for_node_death:365
02C0423D592746DD89EAC08217172364: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32063,11):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32063,11):dlm_wait_for_node_death:365
988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32064,11):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32064,11):dlm_wait_for_node_death:365
D82D1AFEC06F4A5591C051A8AD55F6B8: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32065,11):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32065,11):dlm_wait_for_node_death:365
F1EC618443464E5295BAEC6CB6A0289A: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32066,11):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32066,11):dlm_wait_for_node_death:365
F96CA86A377E4562A237A572D5AAE337: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32068,11):dlm_send_remote_convert_request:398
ERROR: status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32068,11):dlm_wait_for_node_death:365
0715F21DDB174D4FA4F3AC717751BB8A: waiting 5000ms for notification of death
of node 2
Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_do_master_request:1418 ERROR:
link to 2 went down!
Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_get_lock_resource:995 ERROR:
status = -107
Mar 7 07:47:28 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:921
3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:921
D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_do_master_request:1418 ERROR:
link to 2 went down!
Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_get_lock_resource:995 ERROR:
status = -107
Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:921
87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:921
549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32053,1):dlm_get_lock_resource:921
3DF97CEDF6C04E9AAECE7665905F655C:M000000000000000000008fd0deae0b: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:921
FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_do_master_request:1418 ERROR:
link to 2 went down!
Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_get_lock_resource:995 ERROR:
status = -107
Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:921
C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):dlm_get_lock_resource:921
EB739A1B7BCE4948A408D8ACBAC2FCE4:M000000000000000000008f9047acff: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32059,1):dlm_get_lock_resource:921
87868F0E137342B0A1C5C525717DA24B:M000000000000000000008fef7ad4cb: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:921
EB739A1B7BCE4948A408D8ACBAC2FCE4:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:955
EB739A1B7BCE4948A408D8ACBAC2FCE4: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_do_recovery:519 (10569) Node 1
is the Recovery Master for the Dead Node 2 for Domain
EB739A1B7BCE4948A408D8ACBAC2FCE4
Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:921
87868F0E137342B0A1C5C525717DA24B:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:955
87868F0E137342B0A1C5C525717DA24B: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_do_recovery:519 (10726) Node 1
is the Recovery Master for the Dead Node 2 for Domain
87868F0E137342B0A1C5C525717DA24B
Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:921
D82D1AFEC06F4A5591C051A8AD55F6B8:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:955
D82D1AFEC06F4A5591C051A8AD55F6B8: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_do_recovery:519 (10883) Node 1
is the Recovery Master for the Dead Node 2 for Domain
D82D1AFEC06F4A5591C051A8AD55F6B8
Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:921
17D5F5A73FA94B59A2C3CD91E32DCA99:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:955
17D5F5A73FA94B59A2C3CD91E32DCA99: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_do_recovery:519 (11040) Node 1
is the Recovery Master for the Dead Node 2 for Domain
17D5F5A73FA94B59A2C3CD91E32DCA99
Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:921
3DF97CEDF6C04E9AAECE7665905F655C:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:955
3DF97CEDF6C04E9AAECE7665905F655C: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_do_recovery:519 (10480) Node 1
is the Recovery Master for the Dead Node 2 for Domain
3DF97CEDF6C04E9AAECE7665905F655C
Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:976
D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:976
87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:976
549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32053,8):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,42)
Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:976
FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:976
C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,43)
Mar 7 07:47:29 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:976
3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (32059,8):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,49)
Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:921
C2FD0D8F366742D4A9F399FDFF914A78:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:955
C2FD0D8F366742D4A9F399FDFF914A78: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_do_recovery:519 (10637) Node 1
is the Recovery Master for the Dead Node 2 for Domain
C2FD0D8F366742D4A9F399FDFF914A78
Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:921
988B694C74FC44FBB22F5D59CAA5686F:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:955
988B694C74FC44FBB22F5D59CAA5686F: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_do_recovery:519 (10794) Node 1
is the Recovery Master for the Dead Node 2 for Domain
988B694C74FC44FBB22F5D59CAA5686F
Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:921
0715F21DDB174D4FA4F3AC717751BB8A:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:955
0715F21DDB174D4FA4F3AC717751BB8A: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_do_recovery:519 (10951) Node 1
is the Recovery Master for the Dead Node 2 for Domain
0715F21DDB174D4FA4F3AC717751BB8A
Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_restart_lock_mastery:1301 ERROR:
node down! 2
Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_wait_for_lock_mastery:1118 ERROR:
status = -11
Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:921
FD5BCEBA9AB041E2AC257C529FEAF692:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:955
FD5BCEBA9AB041E2AC257C529FEAF692: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_do_recovery:519 (10463) Node 1
is the Recovery Master for the Dead Node 2 for Domain
FD5BCEBA9AB041E2AC257C529FEAF692
Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:921
549A03816D5A4BEA912BC04E7F118822:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:955
549A03816D5A4BEA912BC04E7F118822: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_do_recovery:519 (10620) Node 1
is the Recovery Master for the Dead Node 2 for Domain
549A03816D5A4BEA912BC04E7F118822
Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:921
02C0423D592746DD89EAC08217172364:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:955
02C0423D592746DD89EAC08217172364: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_do_recovery:519 (10777) Node 1
is the Recovery Master for the Dead Node 2 for Domain
02C0423D592746DD89EAC08217172364
Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:921
B0A423E24D5A47DEB93D3448C0B43073:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:955
B0A423E24D5A47DEB93D3448C0B43073: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_do_recovery:519 (10934) Node 1
is the Recovery Master for the Dead Node 2 for Domain
B0A423E24D5A47DEB93D3448C0B43073
Mar 7 07:47:31 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:976
EF2FF243631540E0A9418F491C21D661:D000000000000000002060e659b79cc: at least
one node (2) torecover before lock mastery can begin
Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:921
87038E39DCFC4111A179DE517E1B71FC:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:955
87038E39DCFC4111A179DE517E1B71FC: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_do_recovery:519 (10603) Node 1
is the Recovery Master for the Dead Node 2 for Domain
87038E39DCFC4111A179DE517E1B71FC
Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:921
D213C5F3386B474FA09DA88744CB0B3F:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:955
D213C5F3386B474FA09DA88744CB0B3F: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_do_recovery:519 (10760) Node 1
is the Recovery Master for the Dead Node 2 for Domain
D213C5F3386B474FA09DA88744CB0B3F
Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:921
F96CA86A377E4562A237A572D5AAE337:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:955
F96CA86A377E4562A237A572D5AAE337: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_do_recovery:519 (10917) Node 1
is the Recovery Master for the Dead Node 2 for Domain
F96CA86A377E4562A237A572D5AAE337
Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:921
3269F5BC97A64188821CB931CAA5F005:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:955
3269F5BC97A64188821CB931CAA5F005: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_do_recovery:519 (10586) Node 1
is the Recovery Master for the Dead Node 2 for Domain
3269F5BC97A64188821CB931CAA5F005
Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:921
EF2FF243631540E0A9418F491C21D661:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:955
EF2FF243631540E0A9418F491C21D661: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_do_recovery:519 (10743) Node 1
is the Recovery Master for the Dead Node 2 for Domain
EF2FF243631540E0A9418F491C21D661
Mar 7 07:47:33 sapbdb-uat-d1 (32060,8):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,32)
Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:921
F1EC618443464E5295BAEC6CB6A0289A:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:955
F1EC618443464E5295BAEC6CB6A0289A: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_do_recovery:519 (10900) Node 1
is the Recovery Master for the Dead Node 2 for Domain
F1EC618443464E5295BAEC6CB6A0289A
Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:921
2950FD80D24445598E284B386255DEA2:$RECOVERY: at least one node (2) torecover
before lock mastery can begin
Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:955
2950FD80D24445598E284B386255DEA2: recovery map is not empty, but must master
$RECOVERY lock now
Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_do_recovery:519 (11057) Node 1
is the Recovery Master for the Dead Node 2 for Domain
2950FD80D24445598E284B386255DEA2
Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_restart_lock_mastery:1301
ERROR: node down! 2
Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_wait_for_lock_mastery:1118
ERROR: status = -11
Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_restart_lock_mastery:1301
ERROR: node down! 2
Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_wait_for_lock_mastery:1118
ERROR: status = -11
Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_restart_lock_mastery:1301
ERROR: node down! 2
Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_wait_for_lock_mastery:1118
ERROR: status = -11
Mar 7 07:47:33 sapbdb-uat-d1 (32068,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,37)
Mar 7 07:47:33 sapbdb-uat-d1 (32065,2):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,39)
Mar 7 07:47:33 sapbdb-uat-d1 (32062,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,33)
Mar 7 07:47:33 sapbdb-uat-d1 (32066,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,35)
Mar 7 07:47:33 sapbdb-uat-d1 (32063,8):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,40)
Mar 7 07:47:33 sapbdb-uat-d1 (32064,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,38)
Mar 7 07:47:34 sapbdb-uat-d1 kjournald starting. Commit interval 5 seconds
Mar 7 07:47:34 sapbdb-uat-d1 last message repeated 2 times
Mar 7 07:47:34 sapbdb-uat-d1 (32070,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,41)
Mar 7 07:47:34 sapbdb-uat-d1 (32069,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,34)
Mar 7 07:47:34 sapbdb-uat-d1 (32067,11):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,36)
Mar 7 07:47:36 sapbdb-uat-d1 kjournald starting. Commit interval 5 seconds
Mar 7 07:47:40 sapbdb-uat-d1 last message repeated 8 times
Mar 7 07:47:40 sapbdb-uat-d1 (32056,6):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,46)
Mar 7 07:47:40 sapbdb-uat-d1 (32057,15):ocfs2_replay_journal:1238
Recovering node 2 from slot 1 on device (253,47)
Mar 7 07:47:40 sapbdb-uat-d1 (32052,6):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,44)
Mar 7 07:47:40 sapbdb-uat-d1 (32058,2):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,48)
Mar 7 07:47:40 sapbdb-uat-d1 (32061,5):ocfs2_replay_journal:1238 Recovering
node 2 from slot 0 on device (253,31)
Mar 7 07:47:40 sapbdb-uat-d1 (32055,4):ocfs2_replay_journal:1238 Recovering
node 2 from slot 1 on device (253,45)
Mar 7 07:47:44 sapbdb-uat-d1 kjournald starting. Commit interval 5 seconds
Mar 7 07:47:47 sapbdb-uat-d1 last message repeated 6 times
Mar 7 07:49:49 sapbdb-uat-d1 dlm: connecting to 2
Mar 7 07:49:49 sapbdb-uat-d1 dlm: got connection from 2
Mar 7 07:49:59 sapbdb-uat-d1 o2net: accepted connection from node
sapbdb-uat-d2 (num 2) at 192.168.1.2:7777
Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
FD5BCEBA9AB041E2AC257C529FEAF692
Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("FD5BCEBA9AB041E2AC257C529FEAF692"):
Mar 7 07:50:03 sapbdb-uat-d1 1
Mar 7 07:50:04 sapbdb-uat-d1 2
Mar 7 07:50:04 sapbdb-uat-d1
Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
3DF97CEDF6C04E9AAECE7665905F655C
Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("3DF97CEDF6C04E9AAECE7665905F655C"):
Mar 7 07:50:08 sapbdb-uat-d1 1
Mar 7 07:50:08 sapbdb-uat-d1 2
Mar 7 07:50:08 sapbdb-uat-d1
Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
EB739A1B7BCE4948A408D8ACBAC2FCE4
Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("EB739A1B7BCE4948A408D8ACBAC2FCE4"):
Mar 7 07:50:12 sapbdb-uat-d1 1
Mar 7 07:50:12 sapbdb-uat-d1 2
Mar 7 07:50:12 sapbdb-uat-d1
Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
3269F5BC97A64188821CB931CAA5F005
Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("3269F5BC97A64188821CB931CAA5F005"):
Mar 7 07:50:16 sapbdb-uat-d1 1
Mar 7 07:50:16 sapbdb-uat-d1 2
Mar 7 07:50:16 sapbdb-uat-d1
Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
87038E39DCFC4111A179DE517E1B71FC
Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("87038E39DCFC4111A179DE517E1B71FC"):
Mar 7 07:50:20 sapbdb-uat-d1 1
Mar 7 07:50:20 sapbdb-uat-d1 2
Mar 7 07:50:20 sapbdb-uat-d1
Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
549A03816D5A4BEA912BC04E7F118822
Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("549A03816D5A4BEA912BC04E7F118822"):
Mar 7 07:50:24 sapbdb-uat-d1 1
Mar 7 07:50:24 sapbdb-uat-d1 2
Mar 7 07:50:24 sapbdb-uat-d1
Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
C2FD0D8F366742D4A9F399FDFF914A78
Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("C2FD0D8F366742D4A9F399FDFF914A78"):
Mar 7 07:50:28 sapbdb-uat-d1 1
Mar 7 07:50:28 sapbdb-uat-d1 2
Mar 7 07:50:28 sapbdb-uat-d1
Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
87868F0E137342B0A1C5C525717DA24B
Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("87868F0E137342B0A1C5C525717DA24B"):
Mar 7 07:50:32 sapbdb-uat-d1 1
Mar 7 07:50:32 sapbdb-uat-d1 2
Mar 7 07:50:32 sapbdb-uat-d1
Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
EF2FF243631540E0A9418F491C21D661
Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("EF2FF243631540E0A9418F491C21D661"):
Mar 7 07:50:36 sapbdb-uat-d1 1
Mar 7 07:50:36 sapbdb-uat-d1 2
Mar 7 07:50:36 sapbdb-uat-d1
Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
D213C5F3386B474FA09DA88744CB0B3F
Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("D213C5F3386B474FA09DA88744CB0B3F"):
Mar 7 07:50:40 sapbdb-uat-d1 1
Mar 7 07:50:40 sapbdb-uat-d1 2
Mar 7 07:50:40 sapbdb-uat-d1
Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
02C0423D592746DD89EAC08217172364
Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("02C0423D592746DD89EAC08217172364"):
Mar 7 07:50:44 sapbdb-uat-d1 1
Mar 7 07:50:44 sapbdb-uat-d1 2
Mar 7 07:50:44 sapbdb-uat-d1
Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
988B694C74FC44FBB22F5D59CAA5686F
Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("988B694C74FC44FBB22F5D59CAA5686F"):
Mar 7 07:50:48 sapbdb-uat-d1 1
Mar 7 07:50:48 sapbdb-uat-d1 2
Mar 7 07:50:48 sapbdb-uat-d1
Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
D82D1AFEC06F4A5591C051A8AD55F6B8
Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("D82D1AFEC06F4A5591C051A8AD55F6B8"):
Mar 7 07:50:52 sapbdb-uat-d1 1
Mar 7 07:50:52 sapbdb-uat-d1 2
Mar 7 07:50:52 sapbdb-uat-d1
Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
F1EC618443464E5295BAEC6CB6A0289A
Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("F1EC618443464E5295BAEC6CB6A0289A"):
Mar 7 07:50:56 sapbdb-uat-d1 1
Mar 7 07:50:56 sapbdb-uat-d1 2
Mar 7 07:50:56 sapbdb-uat-d1
Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
F96CA86A377E4562A237A572D5AAE337
Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("F96CA86A377E4562A237A572D5AAE337"):
Mar 7 07:51:00 sapbdb-uat-d1 1
Mar 7 07:51:00 sapbdb-uat-d1 2
Mar 7 07:51:00 sapbdb-uat-d1
Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
B0A423E24D5A47DEB93D3448C0B43073
Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("B0A423E24D5A47DEB93D3448C0B43073"):
Mar 7 07:51:04 sapbdb-uat-d1 1
Mar 7 07:51:04 sapbdb-uat-d1 2
Mar 7 07:51:04 sapbdb-uat-d1
Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
0715F21DDB174D4FA4F3AC717751BB8A
Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("0715F21DDB174D4FA4F3AC717751BB8A"):
Mar 7 07:51:09 sapbdb-uat-d1 1
Mar 7 07:51:09 sapbdb-uat-d1 2
Mar 7 07:51:09 sapbdb-uat-d1
Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
17D5F5A73FA94B59A2C3CD91E32DCA99
Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("17D5F5A73FA94B59A2C3CD91E32DCA99"):
Mar 7 07:51:13 sapbdb-uat-d1 1
Mar 7 07:51:13 sapbdb-uat-d1 2
Mar 7 07:51:13 sapbdb-uat-d1
Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
2950FD80D24445598E284B386255DEA2
Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
("2950FD80D24445598E284B386255DEA2"):
Mar 7 07:51:17 sapbdb-uat-d1 1
Mar 7 07:51:17 sapbdb-uat-d1 2
Mar 7 07:51:17 sapbdb-uat-d1
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090309/ba67ba56/attachment-0001.html
Sunil Mushran
2009-Mar-10 01:29 UTC
[Ocfs2-users] Experiencing occasional system crashes with RHEL5 and ocfs2 1.2.9
Known issue. We have a potential fix for it. It is in testing. How often do you hit this? On Mon, Mar 09, 2009 at 06:52:57PM -0400, Ward Fenton wrote:> We have been experiencing unplanned outages on a subset of the > clustered systems we have deployed to support SAP. The following > captured information came from one of three ocfs2 clusters which handle > SAP SEM/BW functionality. Each of those has experienced multiple kernel > panics which get reported as dlm_drop_lockres_ref and > dlm_defer_lockres_handler errors. > This particular environment is deployed on two identical HP DL580-G5 > servers running RHEL5.2 with kernel 2.6.18-92.1.1.el5. It should be > noted that we have encountered similar issues with similar frequency on > kernel 2.6.18-92.1.22.el5. We are running ocfs2 1.2.9. These two > servers support Oracle RAC (10.2.0.2) and they also sharing through NFS > two ocfs2 file systems to associated SAP application servers. Database > devices and RMAN backups are supported by ocfs2 file systems with > "datavolume,nointr" options. The additional cluster file systems which > support Oracle and SAP do not contain those options. > We have encountered recurring unplanned reboots which appear to be > initiated by the crashing of either node in the cluster. This > particular cluster had experienced a crashed of the other cluster > member one week previous to this most recent crash. Both crashes had > occurred at times with very minimal activity. > Support for netconsole is implemented and we are redirecting the > kernel's console output to a central syslog server. The logs below > originated from netconsole. Though not yet in place at the time of this > crash, kexec/kdump functionality was also introduced. > Thanks, > Ward > ####################################################################### > ######### > netconsole capture of rebooted server: > ####################################################################### > ######### > Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_drop_lockres_ref:2298 > ERROR: while dropping ref on > F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26 > (master=1) got -22. > Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_one_lock_resource:461 > lockres: M00000000000000003c1c868d5afc26, owner=1, state=64 > Mar 7 07:45:25 sapbdb-uat-d2 > (11241,2):__dlm_print_one_lock_resource:476 lockres: > M00000000000000003c1c868d5afc26, owner=1, state=64 > Mar 7 07:45:25 sapbdb-uat-d2 > (11241,2):__dlm_print_one_lock_resource:478 last used: 4728024828, on > purge list: yes > Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_lockres_refmap:444 > refmap nodes: [ > Mar 7 07:45:25 sapbdb-uat-d2 ], inflight=0 > Mar 7 07:45:25 sapbdb-uat-d2 > (11241,2):__dlm_print_one_lock_resource:480 granted queue: > Mar 7 07:45:25 sapbdb-uat-d2 > (11241,2):__dlm_print_one_lock_resource:495 converting queue: > Mar 7 07:45:25 sapbdb-uat-d2 > (11241,2):__dlm_print_one_lock_resource:510 blocked queue: > Mar 7 07:45:25 sapbdb-uat-d2 ----------- [cut here ] --------- [please > bite here ] --------- > Mar 7 07:45:25 sapbdb-uat-d2 Kernel BUG at > ...mushran/BUILD/ocfs2-1.2.9/fs/ocfs2/dlm/dlmmaster.c:2300 > Mar 7 07:45:25 sapbdb-uat-d2 invalid opcode: 0000 [1] > Mar 7 07:45:25 sapbdb-uat-d2 SMP > Mar 7 07:45:25 sapbdb-uat-d2 > Mar 7 07:45:25 sapbdb-uat-d2 last sysfs file: > /devices/pci0000:00/0000:00:00.0/irq > Mar 7 07:45:25 sapbdb-uat-d2 CPU 2 > Mar 7 07:45:25 sapbdb-uat-d2 > Mar 7 07:45:25 sapbdb-uat-d2 Modules linked in: > Mar 7 07:45:25 sapbdb-uat-d2 iptable_filter > Mar 7 07:45:26 sapbdb-uat-d2 ip_tables > Mar 7 07:45:26 sapbdb-uat-d2 x_tables > Mar 7 07:45:26 sapbdb-uat-d2 netconsole > Mar 7 07:45:26 sapbdb-uat-d2 nfsd > Mar 7 07:45:26 sapbdb-uat-d2 exportfs > Mar 7 07:45:26 sapbdb-uat-d2 auth_rpcgss > Mar 7 07:45:26 sapbdb-uat-d2 autofs4 > Mar 7 07:45:26 sapbdb-uat-d2 hidp > Mar 7 07:45:26 sapbdb-uat-d2 ocfs2(U) > Mar 7 07:45:26 sapbdb-uat-d2 nfs > Mar 7 07:45:26 sapbdb-uat-d2 lockd > Mar 7 07:45:26 sapbdb-uat-d2 fscache > Mar 7 07:45:26 sapbdb-uat-d2 nfs_acl > Mar 7 07:45:26 sapbdb-uat-d2 rfcomm > Mar 7 07:45:26 sapbdb-uat-d2 l2cap > Mar 7 07:45:26 sapbdb-uat-d2 bluetooth > Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlmfs(U) > Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlm(U) > Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_nodemanager(U) > Mar 7 07:45:26 sapbdb-uat-d2 lock_dlm(U) > Mar 7 07:45:26 sapbdb-uat-d2 gfs2(U) > Mar 7 07:45:26 sapbdb-uat-d2 dlm > Mar 7 07:45:26 sapbdb-uat-d2 configfs > Mar 7 07:45:26 sapbdb-uat-d2 sunrpc > Mar 7 07:45:26 sapbdb-uat-d2 bonding > Mar 7 07:45:26 sapbdb-uat-d2 ipv6 > Mar 7 07:45:26 sapbdb-uat-d2 xfrm_nalgo > Mar 7 07:45:26 sapbdb-uat-d2 crypto_api > Mar 7 07:45:26 sapbdb-uat-d2 dm_round_robin > Mar 7 07:45:26 sapbdb-uat-d2 dm_emc > Mar 7 07:45:26 sapbdb-uat-d2 dm_multipath > Mar 7 07:45:26 sapbdb-uat-d2 video > Mar 7 07:45:26 sapbdb-uat-d2 sbs > Mar 7 07:45:26 sapbdb-uat-d2 backlight > Mar 7 07:45:26 sapbdb-uat-d2 i2c_ec > Mar 7 07:45:26 sapbdb-uat-d2 i2c_core > Mar 7 07:45:26 sapbdb-uat-d2 button > Mar 7 07:45:26 sapbdb-uat-d2 battery > Mar 7 07:45:26 sapbdb-uat-d2 asus_acpi > Mar 7 07:45:26 sapbdb-uat-d2 acpi_memhotplug > Mar 7 07:45:26 sapbdb-uat-d2 ac > Mar 7 07:45:26 sapbdb-uat-d2 parport_pc > Mar 7 07:45:26 sapbdb-uat-d2 lp > Mar 7 07:45:26 sapbdb-uat-d2 parport > Mar 7 07:45:26 sapbdb-uat-d2 joydev > Mar 7 07:45:26 sapbdb-uat-d2 ata_piix > Mar 7 07:45:26 sapbdb-uat-d2 libata > Mar 7 07:45:26 sapbdb-uat-d2 sg > Mar 7 07:45:26 sapbdb-uat-d2 ide_cd > Mar 7 07:45:26 sapbdb-uat-d2 shpchp > Mar 7 07:45:27 sapbdb-uat-d2 cdrom > Mar 7 07:45:27 sapbdb-uat-d2 bnx2 > Mar 7 07:45:27 sapbdb-uat-d2 pcspkr > Mar 7 07:45:27 sapbdb-uat-d2 serio_raw > Mar 7 07:45:27 sapbdb-uat-d2 dm_snapshot > Mar 7 07:45:27 sapbdb-uat-d2 dm_zero > Mar 7 07:45:27 sapbdb-uat-d2 dm_mirror > Mar 7 07:45:27 sapbdb-uat-d2 dm_mod > Mar 7 07:45:27 sapbdb-uat-d2 qla2xxx > Mar 7 07:45:27 sapbdb-uat-d2 scsi_transport_fc > Mar 7 07:45:27 sapbdb-uat-d2 cciss > Mar 7 07:45:27 sapbdb-uat-d2 sd_mod > Mar 7 07:45:27 sapbdb-uat-d2 scsi_mod > Mar 7 07:45:27 sapbdb-uat-d2 ext3 > Mar 7 07:45:27 sapbdb-uat-d2 jbd > Mar 7 07:45:27 sapbdb-uat-d2 uhci_hcd > Mar 7 07:45:27 sapbdb-uat-d2 ohci_hcd > Mar 7 07:45:27 sapbdb-uat-d2 ehci_hcd > Mar 7 07:45:27 sapbdb-uat-d2 > Mar 7 07:45:27 sapbdb-uat-d2 Pid: 11241, comm: dlm_thread Tainted: > G 2.6.18-92.1.1.el5 #1 > Mar 7 07:45:27 sapbdb-uat-d2 RIP: 0010:[<ffffffff885ba6a0>] > Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885ba6a0>] > :ocfs2_dlm:dlm_drop_lockres_ref+0x1dc/0x1f5 > Mar 7 07:45:27 sapbdb-uat-d2 RSP: 0018:ffff8105e1395de0 EFLAGS: > 00010246 > Mar 7 07:45:27 sapbdb-uat-d2 RAX: ffff81005ef3fe38 RBX: > 0000000000000000 RCX: ffffffff802ed9a8 > Mar 7 07:45:27 sapbdb-uat-d2 RDX: ffffffff802ed9a8 RSI: > 0000000000000000 RDI: ffffffff802ed9a0 > Mar 7 07:45:27 sapbdb-uat-d2 RBP: 000000000000001f R08: > ffffffff802ed9a8 R09: 0000000000000046 > Mar 7 07:45:27 sapbdb-uat-d2 R10: 0000000000000000 R11: > 0000000000000280 R12: ffff81005ef3fe00 > Mar 7 07:45:27 sapbdb-uat-d2 R13: ffff8105e227fc00 R14: > ffff81002aec2d40 R15: ffffffff8009dced > Mar 7 07:45:27 sapbdb-uat-d2 FS: 0000000000000000(0000) > GS:ffff81102ff1ce40(0000) knlGS:0000000000000000 > Mar 7 07:45:27 sapbdb-uat-d2 CS: 0010 DS: 0018 ES: 0018 CR0: > 000000008005003b > Mar 7 07:45:27 sapbdb-uat-d2 CR2: 00000000006bdc5c CR3: > 0000000594e3c000 CR4: 00000000000006e0 > Mar 7 07:45:27 sapbdb-uat-d2 Process dlm_thread (pid: 11241, > threadinfo ffff8105e1394000, task ffff8105e30a2820) > Mar 7 07:45:27 sapbdb-uat-d2 Stack: > Mar 7 07:45:27 sapbdb-uat-d2 1f02000000000000 > Mar 7 07:45:27 sapbdb-uat-d2 303030303030304d > Mar 7 07:45:27 sapbdb-uat-d2 3030303030303030 > Mar 7 07:45:27 sapbdb-uat-d2 3836386331633330 > Mar 7 07:45:27 sapbdb-uat-d2 > Mar 7 07:45:27 sapbdb-uat-d2 0036326366613564 > Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000 > Mar 7 07:45:27 sapbdb-uat-d2 last message repeated 2 times > Mar 7 07:45:27 sapbdb-uat-d2 > Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000 > Mar 7 07:45:27 sapbdb-uat-d2 ffffffea00100100 > Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe48 > Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe00 > Mar 7 07:45:27 sapbdb-uat-d2 > Mar 7 07:45:27 sapbdb-uat-d2 Call Trace: > Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cd72b>] > :ocfs2_dlm:dlm_purge_lockres+0x175/0x34f > Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdb98>] > :ocfs2_dlm:dlm_thread+0xd7/0x579 > Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff8009df05>] > autoremove_wake_function+0x0/0x2e > Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdac1>] > :ocfs2_dlm:dlm_thread+0x0/0x579 > Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003253d>] kthread+0xfe/0x132 > Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfb1>] child_rip+0xa/0x11 > Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8009dced>] > keventd_create_kthread+0x0/0xc4 > Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003243f>] kthread+0x0/0x132 > Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfa7>] child_rip+0x0/0x11 > Mar 7 07:45:28 sapbdb-uat-d2 > Mar 7 07:45:28 sapbdb-uat-d2 > Mar 7 07:45:28 sapbdb-uat-d2 Code: > Mar 7 07:45:28 sapbdb-uat-d2 0f > Mar 7 07:45:28 sapbdb-uat-d2 0b > Mar 7 07:45:28 sapbdb-uat-d2 68 > Mar 7 07:45:28 sapbdb-uat-d2 2d > Mar 7 07:45:28 sapbdb-uat-d2 10 > Mar 7 07:45:28 sapbdb-uat-d2 5d > Mar 7 07:45:28 sapbdb-uat-d2 88 > Mar 7 07:45:28 sapbdb-uat-d2 c2 > Mar 7 07:45:28 sapbdb-uat-d2 fc > ####################################################################### > ######### > netconsole capture for remaining server: > ####################################################################### > ######### > Mar 7 07:45:25 sapbdb-uat-d1 (9917,0):dlm_deref_lockres_handler:2340 > ERROR: > F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26: bad > lockres name > Mar 7 07:45:40 sapbdb-uat-d1 dlm: closing connection to node 2 > Mar 7 07:45:55 sapbdb-uat-d1 o2net: connection to node sapbdb-uat-d2 > (num 2) at [1]192.168.1.2:7777 has been idle for 30.0 seconds, shutting > it down. > Mar 7 07:45:55 sapbdb-uat-d1 (0,0):o2net_idle_timer:1426 here are some > times that might help debug the situation: (tmr 1236429925.620147 now > 1236429955.618766 dr 1236429925.620142 adv > 1236429925.622211:1236429925.622212 func (90648c68:507) > 1236429925.620148:1236429925.622206) > Mar 7 07:45:55 sapbdb-uat-d1 o2net: no longer connected to node > sapbdb-uat-d2 (num 2) at [2]192.168.1.2:7777 > Mar 7 07:45:55 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293 > ERROR: status = -112 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -112 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -112 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_do_master_request:1418 > ERROR: link to 2 went down! > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:995 ERROR: > status = -112 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR: > status = -112 > Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293 > ERROR: status = -107 > Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR: > status = -107 > Mar 7 07:46:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 > ERROR: no connection established with node 2 after 30.0 seconds, giving > up and returning errors. > Mar 7 07:46:55 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 > ERROR: no connection established with node 2 after 30.0 seconds, giving > up and returning errors. > Mar 7 07:47:01 sapbdb-uat-d1 > (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:01 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:06 sapbdb-uat-d1 > (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:06 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:11 sapbdb-uat-d1 > (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:11 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:17 sapbdb-uat-d1 > (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:17 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:22 sapbdb-uat-d1 > (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:22 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 > ERROR: no connection established with node 2 after 30.0 seconds, giving > up and returning errors. > Mar 7 07:47:27 sapbdb-uat-d1 > (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:27 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32060,5):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32060,5):dlm_wait_for_node_death:365 > EF2FF243631540E0A9418F491C21D661: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32062,11):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32062,11):dlm_wait_for_node_death:365 > 02C0423D592746DD89EAC08217172364: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32063,11):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32063,11):dlm_wait_for_node_death:365 > 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32064,11):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32064,11):dlm_wait_for_node_death:365 > D82D1AFEC06F4A5591C051A8AD55F6B8: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32065,11):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32065,11):dlm_wait_for_node_death:365 > F1EC618443464E5295BAEC6CB6A0289A: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32066,11):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32066,11):dlm_wait_for_node_death:365 > F96CA86A377E4562A237A572D5AAE337: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 > (32068,11):dlm_send_remote_convert_request:398 ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32068,11):dlm_wait_for_node_death:365 > 0715F21DDB174D4FA4F3AC717751BB8A: waiting 5000ms for notification of > death of node 2 > Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_do_master_request:1418 > ERROR: link to 2 went down! > Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_get_lock_resource:995 > ERROR: status = -107 > Mar 7 07:47:28 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:921 > 3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:921 > D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_do_master_request:1418 > ERROR: link to 2 went down! > Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_get_lock_resource:995 > ERROR: status = -107 > Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:921 > 87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:921 > 549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32053,1):dlm_get_lock_resource:921 > 3DF97CEDF6C04E9AAECE7665905F655C:M000000000000000000008fd0deae0b: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:921 > FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_do_master_request:1418 > ERROR: link to 2 went down! > Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_get_lock_resource:995 > ERROR: status = -107 > Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:921 > C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):dlm_get_lock_resource:921 > EB739A1B7BCE4948A408D8ACBAC2FCE4:M000000000000000000008f9047acff: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32059,1):dlm_get_lock_resource:921 > 87868F0E137342B0A1C5C525717DA24B:M000000000000000000008fef7ad4cb: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:921 > EB739A1B7BCE4948A408D8ACBAC2FCE4:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:955 > EB739A1B7BCE4948A408D8ACBAC2FCE4: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_do_recovery:519 (10569) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > EB739A1B7BCE4948A408D8ACBAC2FCE4 > Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:921 > 87868F0E137342B0A1C5C525717DA24B:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:955 > 87868F0E137342B0A1C5C525717DA24B: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_do_recovery:519 (10726) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 87868F0E137342B0A1C5C525717DA24B > Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:921 > D82D1AFEC06F4A5591C051A8AD55F6B8:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:955 > D82D1AFEC06F4A5591C051A8AD55F6B8: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_do_recovery:519 (10883) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > D82D1AFEC06F4A5591C051A8AD55F6B8 > Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:921 > 17D5F5A73FA94B59A2C3CD91E32DCA99:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:955 > 17D5F5A73FA94B59A2C3CD91E32DCA99: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_do_recovery:519 (11040) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 17D5F5A73FA94B59A2C3CD91E32DCA99 > Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:921 > 3DF97CEDF6C04E9AAECE7665905F655C:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:955 > 3DF97CEDF6C04E9AAECE7665905F655C: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_do_recovery:519 (10480) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 3DF97CEDF6C04E9AAECE7665905F655C > Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:976 > D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:976 > 87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:976 > 549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32053,8):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,42) > Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:976 > FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:976 > C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,43) > Mar 7 07:47:29 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:976 > 3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (32059,8):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,49) > Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:921 > C2FD0D8F366742D4A9F399FDFF914A78:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:955 > C2FD0D8F366742D4A9F399FDFF914A78: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_do_recovery:519 (10637) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > C2FD0D8F366742D4A9F399FDFF914A78 > Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:921 > 988B694C74FC44FBB22F5D59CAA5686F:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:955 > 988B694C74FC44FBB22F5D59CAA5686F: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_do_recovery:519 (10794) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 988B694C74FC44FBB22F5D59CAA5686F > Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:921 > 0715F21DDB174D4FA4F3AC717751BB8A:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:955 > 0715F21DDB174D4FA4F3AC717751BB8A: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_do_recovery:519 (10951) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 0715F21DDB174D4FA4F3AC717751BB8A > Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_restart_lock_mastery:1301 > ERROR: node down! 2 > Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_wait_for_lock_mastery:1118 > ERROR: status = -11 > Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:921 > FD5BCEBA9AB041E2AC257C529FEAF692:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:955 > FD5BCEBA9AB041E2AC257C529FEAF692: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_do_recovery:519 (10463) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > FD5BCEBA9AB041E2AC257C529FEAF692 > Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:921 > 549A03816D5A4BEA912BC04E7F118822:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:955 > 549A03816D5A4BEA912BC04E7F118822: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_do_recovery:519 (10620) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 549A03816D5A4BEA912BC04E7F118822 > Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:921 > 02C0423D592746DD89EAC08217172364:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:955 > 02C0423D592746DD89EAC08217172364: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_do_recovery:519 (10777) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 02C0423D592746DD89EAC08217172364 > Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:921 > B0A423E24D5A47DEB93D3448C0B43073:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:955 > B0A423E24D5A47DEB93D3448C0B43073: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_do_recovery:519 (10934) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > B0A423E24D5A47DEB93D3448C0B43073 > Mar 7 07:47:31 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:976 > EF2FF243631540E0A9418F491C21D661:D000000000000000002060e659b79cc: at > least one node (2) torecover before lock mastery can begin > Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:921 > 87038E39DCFC4111A179DE517E1B71FC:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:955 > 87038E39DCFC4111A179DE517E1B71FC: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_do_recovery:519 (10603) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 87038E39DCFC4111A179DE517E1B71FC > Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:921 > D213C5F3386B474FA09DA88744CB0B3F:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:955 > D213C5F3386B474FA09DA88744CB0B3F: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_do_recovery:519 (10760) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > D213C5F3386B474FA09DA88744CB0B3F > Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:921 > F96CA86A377E4562A237A572D5AAE337:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:955 > F96CA86A377E4562A237A572D5AAE337: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_do_recovery:519 (10917) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > F96CA86A377E4562A237A572D5AAE337 > Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:921 > 3269F5BC97A64188821CB931CAA5F005:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:955 > 3269F5BC97A64188821CB931CAA5F005: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_do_recovery:519 (10586) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 3269F5BC97A64188821CB931CAA5F005 > Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:921 > EF2FF243631540E0A9418F491C21D661:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:955 > EF2FF243631540E0A9418F491C21D661: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_do_recovery:519 (10743) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > EF2FF243631540E0A9418F491C21D661 > Mar 7 07:47:33 sapbdb-uat-d1 (32060,8):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,32) > Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:921 > F1EC618443464E5295BAEC6CB6A0289A:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:955 > F1EC618443464E5295BAEC6CB6A0289A: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_do_recovery:519 (10900) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > F1EC618443464E5295BAEC6CB6A0289A > Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:921 > 2950FD80D24445598E284B386255DEA2:$RECOVERY: at least one node (2) > torecover before lock mastery can begin > Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:955 > 2950FD80D24445598E284B386255DEA2: recovery map is not empty, but must > master $RECOVERY lock now > Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_do_recovery:519 (11057) > Node 1 is the Recovery Master for the Dead Node 2 for Domain > 2950FD80D24445598E284B386255DEA2 > Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_restart_lock_mastery:1301 > ERROR: node down! 2 > Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_wait_for_lock_mastery:1118 > ERROR: status = -11 > Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_restart_lock_mastery:1301 > ERROR: node down! 2 > Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_wait_for_lock_mastery:1118 > ERROR: status = -11 > Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_restart_lock_mastery:1301 > ERROR: node down! 2 > Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_wait_for_lock_mastery:1118 > ERROR: status = -11 > Mar 7 07:47:33 sapbdb-uat-d1 (32068,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,37) > Mar 7 07:47:33 sapbdb-uat-d1 (32065,2):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,39) > Mar 7 07:47:33 sapbdb-uat-d1 (32062,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,33) > Mar 7 07:47:33 sapbdb-uat-d1 (32066,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,35) > Mar 7 07:47:33 sapbdb-uat-d1 (32063,8):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,40) > Mar 7 07:47:33 sapbdb-uat-d1 (32064,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,38) > Mar 7 07:47:34 sapbdb-uat-d1 kjournald starting. Commit interval 5 > seconds > Mar 7 07:47:34 sapbdb-uat-d1 last message repeated 2 times > Mar 7 07:47:34 sapbdb-uat-d1 (32070,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,41) > Mar 7 07:47:34 sapbdb-uat-d1 (32069,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,34) > Mar 7 07:47:34 sapbdb-uat-d1 (32067,11):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,36) > Mar 7 07:47:36 sapbdb-uat-d1 kjournald starting. Commit interval 5 > seconds > Mar 7 07:47:40 sapbdb-uat-d1 last message repeated 8 times > Mar 7 07:47:40 sapbdb-uat-d1 (32056,6):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,46) > Mar 7 07:47:40 sapbdb-uat-d1 (32057,15):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,47) > Mar 7 07:47:40 sapbdb-uat-d1 (32052,6):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,44) > Mar 7 07:47:40 sapbdb-uat-d1 (32058,2):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,48) > Mar 7 07:47:40 sapbdb-uat-d1 (32061,5):ocfs2_replay_journal:1238 > Recovering node 2 from slot 0 on device (253,31) > Mar 7 07:47:40 sapbdb-uat-d1 (32055,4):ocfs2_replay_journal:1238 > Recovering node 2 from slot 1 on device (253,45) > Mar 7 07:47:44 sapbdb-uat-d1 kjournald starting. Commit interval 5 > seconds > Mar 7 07:47:47 sapbdb-uat-d1 last message repeated 6 times > Mar 7 07:49:49 sapbdb-uat-d1 dlm: connecting to 2 > Mar 7 07:49:49 sapbdb-uat-d1 dlm: got connection from 2 > Mar 7 07:49:59 sapbdb-uat-d1 o2net: accepted connection from node > sapbdb-uat-d2 (num 2) at [3]192.168.1.2:7777 > Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > FD5BCEBA9AB041E2AC257C529FEAF692 > Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("FD5BCEBA9AB041E2AC257C529FEAF692"): > Mar 7 07:50:03 sapbdb-uat-d1 1 > Mar 7 07:50:04 sapbdb-uat-d1 2 > Mar 7 07:50:04 sapbdb-uat-d1 > Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 3DF97CEDF6C04E9AAECE7665905F655C > Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("3DF97CEDF6C04E9AAECE7665905F655C"): > Mar 7 07:50:08 sapbdb-uat-d1 1 > Mar 7 07:50:08 sapbdb-uat-d1 2 > Mar 7 07:50:08 sapbdb-uat-d1 > Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > EB739A1B7BCE4948A408D8ACBAC2FCE4 > Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("EB739A1B7BCE4948A408D8ACBAC2FCE4"): > Mar 7 07:50:12 sapbdb-uat-d1 1 > Mar 7 07:50:12 sapbdb-uat-d1 2 > Mar 7 07:50:12 sapbdb-uat-d1 > Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 3269F5BC97A64188821CB931CAA5F005 > Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("3269F5BC97A64188821CB931CAA5F005"): > Mar 7 07:50:16 sapbdb-uat-d1 1 > Mar 7 07:50:16 sapbdb-uat-d1 2 > Mar 7 07:50:16 sapbdb-uat-d1 > Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 87038E39DCFC4111A179DE517E1B71FC > Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("87038E39DCFC4111A179DE517E1B71FC"): > Mar 7 07:50:20 sapbdb-uat-d1 1 > Mar 7 07:50:20 sapbdb-uat-d1 2 > Mar 7 07:50:20 sapbdb-uat-d1 > Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 549A03816D5A4BEA912BC04E7F118822 > Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("549A03816D5A4BEA912BC04E7F118822"): > Mar 7 07:50:24 sapbdb-uat-d1 1 > Mar 7 07:50:24 sapbdb-uat-d1 2 > Mar 7 07:50:24 sapbdb-uat-d1 > Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > C2FD0D8F366742D4A9F399FDFF914A78 > Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("C2FD0D8F366742D4A9F399FDFF914A78"): > Mar 7 07:50:28 sapbdb-uat-d1 1 > Mar 7 07:50:28 sapbdb-uat-d1 2 > Mar 7 07:50:28 sapbdb-uat-d1 > Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 87868F0E137342B0A1C5C525717DA24B > Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("87868F0E137342B0A1C5C525717DA24B"): > Mar 7 07:50:32 sapbdb-uat-d1 1 > Mar 7 07:50:32 sapbdb-uat-d1 2 > Mar 7 07:50:32 sapbdb-uat-d1 > Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > EF2FF243631540E0A9418F491C21D661 > Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("EF2FF243631540E0A9418F491C21D661"): > Mar 7 07:50:36 sapbdb-uat-d1 1 > Mar 7 07:50:36 sapbdb-uat-d1 2 > Mar 7 07:50:36 sapbdb-uat-d1 > Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > D213C5F3386B474FA09DA88744CB0B3F > Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("D213C5F3386B474FA09DA88744CB0B3F"): > Mar 7 07:50:40 sapbdb-uat-d1 1 > Mar 7 07:50:40 sapbdb-uat-d1 2 > Mar 7 07:50:40 sapbdb-uat-d1 > Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 02C0423D592746DD89EAC08217172364 > Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("02C0423D592746DD89EAC08217172364"): > Mar 7 07:50:44 sapbdb-uat-d1 1 > Mar 7 07:50:44 sapbdb-uat-d1 2 > Mar 7 07:50:44 sapbdb-uat-d1 > Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 988B694C74FC44FBB22F5D59CAA5686F > Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("988B694C74FC44FBB22F5D59CAA5686F"): > Mar 7 07:50:48 sapbdb-uat-d1 1 > Mar 7 07:50:48 sapbdb-uat-d1 2 > Mar 7 07:50:48 sapbdb-uat-d1 > Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > D82D1AFEC06F4A5591C051A8AD55F6B8 > Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("D82D1AFEC06F4A5591C051A8AD55F6B8"): > Mar 7 07:50:52 sapbdb-uat-d1 1 > Mar 7 07:50:52 sapbdb-uat-d1 2 > Mar 7 07:50:52 sapbdb-uat-d1 > Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > F1EC618443464E5295BAEC6CB6A0289A > Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("F1EC618443464E5295BAEC6CB6A0289A"): > Mar 7 07:50:56 sapbdb-uat-d1 1 > Mar 7 07:50:56 sapbdb-uat-d1 2 > Mar 7 07:50:56 sapbdb-uat-d1 > Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > F96CA86A377E4562A237A572D5AAE337 > Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("F96CA86A377E4562A237A572D5AAE337"): > Mar 7 07:51:00 sapbdb-uat-d1 1 > Mar 7 07:51:00 sapbdb-uat-d1 2 > Mar 7 07:51:00 sapbdb-uat-d1 > Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > B0A423E24D5A47DEB93D3448C0B43073 > Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("B0A423E24D5A47DEB93D3448C0B43073"): > Mar 7 07:51:04 sapbdb-uat-d1 1 > Mar 7 07:51:04 sapbdb-uat-d1 2 > Mar 7 07:51:04 sapbdb-uat-d1 > Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 0715F21DDB174D4FA4F3AC717751BB8A > Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("0715F21DDB174D4FA4F3AC717751BB8A"): > Mar 7 07:51:09 sapbdb-uat-d1 1 > Mar 7 07:51:09 sapbdb-uat-d1 2 > Mar 7 07:51:09 sapbdb-uat-d1 > Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 17D5F5A73FA94B59A2C3CD91E32DCA99 > Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("17D5F5A73FA94B59A2C3CD91E32DCA99"): > Mar 7 07:51:13 sapbdb-uat-d1 1 > Mar 7 07:51:13 sapbdb-uat-d1 2 > Mar 7 07:51:13 sapbdb-uat-d1 > Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain > 2950FD80D24445598E284B386255DEA2 > Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain > ("2950FD80D24445598E284B386255DEA2"): > Mar 7 07:51:17 sapbdb-uat-d1 1 > Mar 7 07:51:17 sapbdb-uat-d1 2 > Mar 7 07:51:17 sapbdb-uat-d1 > > References > > 1. http://192.168.1.2:7777/ > 2. http://192.168.1.2:7777/ > 3. http://192.168.1.2:7777/> _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users