search for: do_recovery

...dbd[6602]: CTDB_WAIT_UNTIL_RECOVERED 2018/02/12 19:38:57.885800 ctdb-recoverd[7182]: Election period ended 2018/02/12 19:38:57.886134 ctdb-recoverd[7182]: Node:1 was in recovery mode. Start recovery process 2018/02/12 19:38:57.886160 ctdb-recoverd[7182]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery 2018/02/12 19:38:57.886187 ctdb-recoverd[7182]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) 2018/02/12 19:38:57.886243 ctdb-recoverd[7182]: Set cluster mutex helper to "/usr/libexec/ctdb/ctdb_mutex_fcntl_helper" 2018/02/12 19:38:57.899722 ctdb-recoverd[7182]: Un...

...dbd[6602]: CTDB_WAIT_UNTIL_RECOVERED 2018/02/12 19:38:57.885800 ctdb-recoverd[7182]: Election period ended 2018/02/12 19:38:57.886134 ctdb-recoverd[7182]: Node:1 was in recovery mode. Start recovery process 2018/02/12 19:38:57.886160 ctdb-recoverd[7182]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery 2018/02/12 19:38:57.886187 ctdb-recoverd[7182]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) 2018/02/12 19:38:57.886243 ctdb-recoverd[7182]: Set cluster mutex helper to "/usr/libexec/ctdb/ctdb_mutex_fcntl_helper" 2018/02/12 19:38:57.899722 ctdb-recoverd[7182]: Un...

...dbd[6602]: CTDB_WAIT_UNTIL_RECOVERED 2018/02/12 19:38:57.885800 ctdb-recoverd[7182]: Election period ended 2018/02/12 19:38:57.886134 ctdb-recoverd[7182]: Node:1 was in recovery mode. Start recovery process 2018/02/12 19:38:57.886160 ctdb-recoverd[7182]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery 2018/02/12 19:38:57.886187 ctdb-recoverd[7182]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) 2018/02/12 19:38:57.886243 ctdb-recoverd[7182]: Set cluster mutex helper to "/usr/libexec/ctdb/ctdb_mutex_fcntl_helper" 2018/02/12 19:38:57.899722 ctdb-recoverd[7182]: Un...

...414369 ctdbd[10129]: Recovery has started 2018/09/04 04:35:06.414944 ctdbd[10129]: connect() failed, errno=111 2018/09/04 04:35:06.415076 ctdbd[10129]: Unable to run startrecovery event node2: repeat logs: 2018/09/04 04:35:09.412368 ctdb-recoverd[9437]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery 2018/09/04 04:35:09.412689 ctdb-recoverd[9437]: Already holding recovery lock 2018/09/04 04:35:09.412700 ctdb-recoverd[9437]: ../ctdb/server/ctdb_recoverd.c:1326 Recovery initiated due to problem with node 1 2018/09/04 04:35:09.412974 ctdb-recoverd[9437]: ../ctdb/server/ctdb_recoverd.c:1351 Recover...

...dbd[6602]: CTDB_WAIT_UNTIL_RECOVERED 2018/02/12 19:38:57.885800 ctdb-recoverd[7182]: Election period ended 2018/02/12 19:38:57.886134 ctdb-recoverd[7182]: Node:1 was in recovery mode. Start recovery process 2018/02/12 19:38:57.886160 ctdb-recoverd[7182]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery 2018/02/12 19:38:57.886187 ctdb-recoverd[7182]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) 2018/02/12 19:38:57.886243 ctdb-recoverd[7182]: Set cluster mutex helper to "/usr/libexec/ctdb/ctdb_mutex_fcntl_helper" 2018/02/12 19:38:57.899722 ctdb-recoverd[7182]: Un...

...9/04 04:29:55.469404 ctdb-recoverd[11302]: Node 2 has changed flags - now 0x8 was 0x0 > 2018/09/04 04:29:55.469475 ctdb-recoverd[11302]: Remote node 2 had flags 0x8, local had 0x0 - updating local > 2018/09/04 04:29:55.469514 ctdb-recoverd[11302]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery > 2018/09/04 04:29:55.469525 ctdb-recoverd[11302]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) > 2018/09/04 04:29:55.563522 ctdb-recoverd[11302]: Unable to take recovery lock - contention > 2018/09/04 04:29:55.563573 ctdb-recoverd[11302]: Unable to get recovery l...

...9/04 04:29:55.469404 ctdb-recoverd[11302]: Node 2 has changed flags - now 0x8 was 0x0 > 2018/09/04 04:29:55.469475 ctdb-recoverd[11302]: Remote node 2 had flags 0x8, local had 0x0 - updating local > 2018/09/04 04:29:55.469514 ctdb-recoverd[11302]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery > 2018/09/04 04:29:55.469525 ctdb-recoverd[11302]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) > 2018/09/04 04:29:55.563522 ctdb-recoverd[11302]: Unable to take recovery lock - contention > 2018/09/04 04:29:55.563573 ctdb-recoverd[11302]: Unable to get recovery l...

...- force an election ctdbd[1220]: Recovery mode set to ACTIVE ctdbd[1220]: This node (1) is now the recovery master ctdb-recoverd[1236]: Election period ended ctdb-recoverd[1236]: Node:1 was in recovery mode. Start recovery process ctdb-recoverd[1236]: ../../ctdb/server/ctdb_recoverd.c:1347 Starting do_recovery ctdb-recoverd[1236]: Attempting to take recovery lock (!/usr/local/bin/lockctl elect --endpoints REDACTED:2379 SM ctdbd[1220]: High RECLOCK latency 4.268180s for operation recd reclock ctdb-recoverd[1236]: Recovery lock taken successfully ctdb-recoverd[1236]: ../../ctdb/server/ctdb_recoverd.c:1422...

...9/04 04:29:55.469404 ctdb-recoverd[11302]: Node 2 has changed flags - now 0x8 was 0x0 > 2018/09/04 04:29:55.469475 ctdb-recoverd[11302]: Remote node 2 had flags 0x8, local had 0x0 - updating local > 2018/09/04 04:29:55.469514 ctdb-recoverd[11302]: ../ctdb/server/ctdb_recoverd.c:1267 Starting do_recovery > 2018/09/04 04:29:55.469525 ctdb-recoverd[11302]: Attempting to take recovery lock (/share-fs/export/ctdb/.ctdb/reclock) > 2018/09/04 04:29:55.563522 ctdb-recoverd[11302]: Unable to take recovery lock - contention > 2018/09/04 04:29:55.563573 ctdb-recoverd[11302]: Unable to get recovery l...

...de 0 - force takeover run 2013/04/09 16:10:03.258805 [recoverd:30648]: Trigger takeoverrun 2013/04/09 16:10:03.259041 [recoverd:30648]: server/ctdb_recoverd.c:2702 Node:0 was in recovery mode. Restart recovery process 2013/04/09 16:10:03.259071 [recoverd:30648]: server/ctdb_recoverd.c:1555 Starting do_recovery 2013/04/09 16:10:03.259085 [recoverd:30648]: Taking out recovery lock from recovery daemon 2013/04/09 16:10:03.259108 [recoverd:30648]: Take the recovery lock 2013/04/09 16:10:03.267903 [recoverd:30648]: Recovery lock taken successfully 2013/04/09 16:10:03.267933 [recoverd:30648]: ctdb_recovery_loc...

Very helpful. Thank you, Martin. I'd like to share the information below with you and solicit your fine feedback :-) I provide additional detail in case there is something else you feel strongly we should consider. We made some changes last night, let me share those with you. The error that is repeating itself and causing these failures is: Takeover run starting RELEASE_IP 10.200.1.230

Hello, at this time i want to install a CTDB Cluster with SAMBA 4.7.7 from SOURCE! I compiled samba as follow: |./configure| |--with-cluster-support ||--with-shared-modules=idmap_rid,idmap_tdb2,idmap_ad| The whole SAMBA enviroment is located in /usr/local/samba/. CTDB is located in /usr/local/samba/etc/ctdb. I guess right that the correct path of ctdbd.conf (node file, public address file

...d[2093]: CTDB_WAIT_UNTIL_RECOVERED 2018/05/07 15:31:44.103288 ctdb-recoverd[2160]: Election period ended 2018/05/07 15:31:44.105712 ctdb-recoverd[2160]: Node:1 was in recovery mode. Start recovery process 2018/05/07 15:31:44.105826 ctdb-recoverd[2160]: ../ctdb/server/ctdb_recoverd.c:1268 Starting do_recovery 2018/05/07 15:31:44.105909 ctdb-recoverd[2160]: ../ctdb/server/ctdb_recoverd.c:1327 Recovery initiated due to problem with node 0 2018/05/07 15:31:44.106152 ctdb-recoverd[2160]: ../ctdb/server/ctdb_recoverd.c:1352 Recovery - created remote databases 2018/05/07 15:31:44.106892 ctdb-recoverd[2160]...

...or db brlock.tdb on node 1, ret=110 2017/04/19 10:42:51.463230 [recoverd: 7632]: recovery: 19 of 21 databases recovered 2017/04/19 10:42:51.463234 [recoverd: 7632]: recovery: database recovery failed, ret=5 2017/04/19 10:42:51.466924 [recoverd: 7632]: ../ctdb/server/ctdb_recoverd.c:2013 Starting do_recovery It does look like we have some database corruption. What may have caused this, and is there any way to resolve it? Our samba and CTDB version is Version 4.4.9-SerNet-RedHat-37.el7, and I've checked that are consistent. Any help would be most gratefully received. Alex -- This message is i...