Chris St. Pierre
2007-Aug-31 15:33 UTC
[Fedora-directory-users] MMR broken, can''t get it started again
I noticed today that replication between a few of my four FDS machines (all MMR agreements going every which way) had failed with errors like this: [30/Aug/2007:00:02:04 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Incremental protocol: event update_window_opened should not occur in state start_backoff [31/Aug/2007:00:03:59 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Incremental protocol: event update_window_opened should not occur in state start_backoff [31/Aug/2007:07:35:59 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Incremental update failed and requires administrator action I tried using mmr.pl to reinitialize the agreements, but that failed miserably and wiped all of the data from the database I was re-initializing. I got the following errors on the supplier: [31/Aug/2007:10:24:56 -0500] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn="Replication to chico (o=isp)"" (chico:389)". [31/Aug/2007:10:25:36 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Failed to send extended operation: LDAP error 81 (Can''t contact LDAP server) [31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error 89: NULL for total update operation [31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error 89: NULL for total update operation [31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error 89: NULL for total update operation [31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error 89: NULL for total update operation [31/Aug/2007:10:25:39 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error 89: NULL for total update operation [31/Aug/2007:10:25:39 -0500] NSMMReplicationPlugin - agmt="cn="Replication to chico (o=isp)"" (chico:389): Warning: unable to send endReplication extended operation (Bad parameter to an ldap routine) And on the consumer: [31/Aug/2007:10:24:54 -0500] NSMMReplicationPlugin - multimaster_be_state_change: replica o=isp is going offline; disabling replication [31/Aug/2007:10:24:56 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [31/Aug/2007:10:25:16 -0500] - import userRoot: Processed 2036 entries -- average rate 101.8/sec, recent rate 101.8/sec, hit ratio 0% [31/Aug/2007:10:25:24 -0500] - ERROR bulk import abandoned [31/Aug/2007:10:25:24 -0500] - import userRoot: Aborting all import threads... [31/Aug/2007:10:25:31 -0500] - import userRoot: Import threads aborted. [31/Aug/2007:10:25:31 -0500] - import userRoot: Closing files... [31/Aug/2007:10:25:35 -0500] - libdb: userRoot/owner.db4: unable to flush: No such file or directory [...lots of lines like that...] [31/Aug/2007:10:25:36 -0500] - libdb: userRoot/id2entry.db4: unable to flush: No such file or directory [31/Aug/2007:10:25:36 -0500] - import userRoot: Import failed. [31/Aug/2007:10:25:36 -0500] - process_bulk_import_op: NULL backend At that point, the supplier crashes and we get the "Can''t contact LDAP server" error in the consumer. This looked really similar to an error I''d had before, in this thread: http://www.mail-archive.com/fedora-directory-users@redhat.com/msg04969.html I had eventually resolved that by dismantling all replication information, using db2ldif to import the database from the supplier to the consumer while the consumer was down, bringing the consumer back up, and reinitializing the replication agreements. This has not worked this time; I get the exact same errors. Anyone have any suggestions? Thanks! Chris St. Pierre Unix Systems Administrator Nebraska Wesleyan University