Hello list, I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file: tdb> info Size of file/data: 3388084000/1050098055 Header offset/logical size: 0/3388084000 Number of records: 669737 Incompatible hash: no Active/supported feature flags: 0x00000000/0x00000001 Robust mutexes locking: no Smallest/average/largest keys: 12/57/242 Smallest/average/largest data: 72/1510/1235987 Smallest/average/largest padding: 5/412/252927 Number of dead records: 2 Smallest/average/largest dead records: 399912712/661866360/923820008 Number of free records: 38999 Smallest/average/largest free records: 12/18485/715170720 Number of hash chains: 10000 Smallest/average/largest hash chains: 26/66/116 Number of uncoalesced records: 253 Smallest/average/largest uncoalesced runs: 1/1/2 Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 1/30/8/21/39/1/0 it ends with this error: tdb_recovery_allocate: overflow recovery area tdb_transaction_prepare_commit: failed to setup recovery data ../lib/tdb/common/tdb.c:1055 Failed to commit I follow the code and found this line: transaction.c:783 /* New head will be at end of file. */ recovery_head = tdb->map_size; In my case it means following: my filesize is 3,3GB, old recovery_head was on 2064351232, size of old recovery was about 923820008, size of new recovery should be 1323794228, after tdb_free of old recovery area and setting the new recovery head to end of file(tdb->map_size), it is no more place for new recovery area, so the next check returns "overflow recovery area". I'm now thinking about how it should be fixed? I thinking about to improve the checks and maybe search for last data entry and set new recovery head behind it? Andrej
On Thu, 2 Aug 2018 15:53:37 +0200 Andrej Gessel via samba <samba at lists.samba.org> wrote:> Hello list, > > I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:Would this be under the 'sam.ldb.d' directory ? If so, I haven't really got a clue what you are trying to achieve, but I would advise that you stop doing it now. You should only carry out operations on 'sam.ldb' and never on the files under 'sam.ldb.d' Rowland
On Thu, 2018-08-02 at 15:53 +0200, Andrej Gessel via samba wrote:> Hello list, > > I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file: > > tdb> info > Size of file/data: 3388084000/1050098055 > Header offset/logical size: 0/3388084000 > Number of records: 669737 > Incompatible hash: no > Active/supported feature flags: 0x00000000/0x00000001 > Robust mutexes locking: no > Smallest/average/largest keys: 12/57/242 > Smallest/average/largest data: 72/1510/1235987 > Smallest/average/largest padding: 5/412/252927 > Number of dead records: 2 > Smallest/average/largest dead records: 399912712/661866360/923820008 > Number of free records: 38999 > Smallest/average/largest free records: 12/18485/715170720 > Number of hash chains: 10000 > Smallest/average/largest hash chains: 26/66/116 > Number of uncoalesced records: 253 > Smallest/average/largest uncoalesced runs: 1/1/2 > Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: > 1/30/8/21/39/1/0 > > > it ends with this error: > > tdb_recovery_allocate: overflow recovery area > tdb_transaction_prepare_commit: failed to setup recovery data > ../lib/tdb/common/tdb.c:1055 Failed to commit > > I follow the code and found this line: > > transaction.c:783 > > /* New head will be at end of file. */ > recovery_head = tdb->map_size; > > In my case it means following: > > my filesize is 3,3GB, old recovery_head was on 2064351232, size of old > recovery was about 923820008, size of new recovery should be 1323794228, > after tdb_free of old recovery area and setting the new recovery head to > end of file(tdb->map_size), it is no more place for new recovery area, > so the next check returns "overflow recovery area". > > > I'm now thinking about how it should be fixed? I thinking about to > improve the checks and maybe search for last data entry and set new > recovery head behind it?In the short term: Take the DC offline, and then run tdbbackup on each backend file, then replace them one by one. That will do the repack, but into a new file rather than with a transaction. In the medium term patches to improve this TDB behaviour would be most welcome, with tests, so we can avoid folks getting stuck. This will help us bridge the gap to LMDB more safely. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
Hello, as Andrew has already recognized, the idea behind is to reduce file size of database in filesystem. In my case this database needs to be mmaped on 2/2GB split 32bit arm machine, so I don't have address space for it, but I could have space, if file size could be equal to data size. Thanks Andrew to show me the offline workaround. After tdbbackup I got this information, but I did not start my DC yet. tdb> info Size of file/data: 1350275072/1050098055 Header offset/logical size: 0/1350275072 Number of records: 669737 Incompatible hash: no Active/supported feature flags: 0x00000000/0x00000001 Robust mutexes locking: no Smallest/average/largest keys: 12/57/242 Smallest/average/largest data: 72/1510/1235987 Smallest/average/largest padding: 25/397/309007 Number of dead records: 0 Smallest/average/largest dead records: 0/0/0 Number of free records: 25 Smallest/average/largest free records: 12/725751/18142352 Number of hash chains: 10000 Smallest/average/largest hash chains: 26/66/116 Number of uncoalesced records: 0 Smallest/average/largest uncoalesced runs: 0/0/0 Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 3/75/20/1/0/1/0 File size is as expected after repack. For tdbbackup I need to take my DC offline, how about repack? If my database would be smaller, and repack could work, can I run it online, or should I take my DC offline as well? Andrej ________________________________ Von: Andrew Bartlett <abartlet at samba.org> Gesendet: Donnerstag, 2. August 2018 20:14 An: Andrej Gessel; 'samba' Betreff: Re: [Samba] tdbtool repack fails On Thu, 2018-08-02 at 15:53 +0200, Andrej Gessel via samba wrote:> Hello list, > > I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file: > > tdb> info > Size of file/data: 3388084000/1050098055 > Header offset/logical size: 0/3388084000 > Number of records: 669737 > Incompatible hash: no > Active/supported feature flags: 0x00000000/0x00000001 > Robust mutexes locking: no > Smallest/average/largest keys: 12/57/242 > Smallest/average/largest data: 72/1510/1235987 > Smallest/average/largest padding: 5/412/252927 > Number of dead records: 2 > Smallest/average/largest dead records: 399912712/661866360/923820008 > Number of free records: 38999 > Smallest/average/largest free records: 12/18485/715170720 > Number of hash chains: 10000 > Smallest/average/largest hash chains: 26/66/116 > Number of uncoalesced records: 253 > Smallest/average/largest uncoalesced runs: 1/1/2 > Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: > 1/30/8/21/39/1/0 > > > it ends with this error: > > tdb_recovery_allocate: overflow recovery area > tdb_transaction_prepare_commit: failed to setup recovery data > ../lib/tdb/common/tdb.c:1055 Failed to commit > > I follow the code and found this line: > > transaction.c:783 > > /* New head will be at end of file. */ > recovery_head = tdb->map_size; > > In my case it means following: > > my filesize is 3,3GB, old recovery_head was on 2064351232, size of old > recovery was about 923820008, size of new recovery should be 1323794228, > after tdb_free of old recovery area and setting the new recovery head to > end of file(tdb->map_size), it is no more place for new recovery area, > so the next check returns "overflow recovery area". > > > I'm now thinking about how it should be fixed? I thinking about to > improve the checks and maybe search for last data entry and set new > recovery head behind it?In the short term: Take the DC offline, and then run tdbbackup on each backend file, then replace them one by one. That will do the repack, but into a new file rather than with a transaction. In the medium term patches to improve this TDB behaviour would be most welcome, with tests, so we can avoid folks getting stuck. This will help us bridge the gap to LMDB more safely. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba