thr3ads.net - samba - [Samba] tdbtool repack fails [Aug 2018]

If this information is useful, please help other people find it:
Share via:

Andrej Gessel

2018-Aug-02 13:53 UTC

[Samba] tdbtool repack fails

Hello list,

I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:

tdb> info
Size of file/data: 3388084000/1050098055
Header offset/logical size: 0/3388084000
Number of records: 669737
Incompatible hash: no
Active/supported feature flags: 0x00000000/0x00000001
Robust mutexes locking: no
Smallest/average/largest keys: 12/57/242
Smallest/average/largest data: 72/1510/1235987
Smallest/average/largest padding: 5/412/252927
Number of dead records: 2
Smallest/average/largest dead records: 399912712/661866360/923820008
Number of free records: 38999
Smallest/average/largest free records: 12/18485/715170720
Number of hash chains: 10000
Smallest/average/largest hash chains: 26/66/116
Number of uncoalesced records: 253
Smallest/average/largest uncoalesced runs: 1/1/2
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 
1/30/8/21/39/1/0


it ends with this error:

tdb_recovery_allocate: overflow recovery area
tdb_transaction_prepare_commit: failed to setup recovery data
../lib/tdb/common/tdb.c:1055 Failed to commit

I follow the code and found this line:

transaction.c:783

         /* New head will be at end of file. */
         recovery_head = tdb->map_size;

In my case it means following:

my filesize is 3,3GB, old recovery_head was on 2064351232, size of old 
recovery was about 923820008, size of new recovery should be 1323794228, 
after tdb_free of old recovery area and setting the new recovery head to 
end of file(tdb->map_size), it is no more place for new recovery area, 
so the next check returns "overflow recovery area".


I'm now thinking about how it should be fixed? I thinking about to 
improve the checks and maybe search for last data entry and set new 
recovery head behind it?


Andrej

Rowland Penny

2018-Aug-02 14:25 UTC

head link

[Samba] tdbtool repack fails

On Thu, 2 Aug 2018 15:53:37 +0200
Andrej Gessel via samba <samba at lists.samba.org> wrote:
> Hello list,
> 
> I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:
Would this be under the 'sam.ldb.d' directory ?
If so, I haven't really got a clue what you are trying to achieve, but
I would advise that you stop doing it now. You should only carry out
operations on 'sam.ldb' and never on the files under 'sam.ldb.d'

Rowland

Andrew Bartlett

2018-Aug-02 18:14 UTC

head link

[Samba] tdbtool repack fails

On Thu, 2018-08-02 at 15:53 +0200, Andrej Gessel via samba
wrote:> Hello list,
> 
> I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:
> 
> tdb> info
> Size of file/data: 3388084000/1050098055
> Header offset/logical size: 0/3388084000
> Number of records: 669737
> Incompatible hash: no
> Active/supported feature flags: 0x00000000/0x00000001
> Robust mutexes locking: no
> Smallest/average/largest keys: 12/57/242
> Smallest/average/largest data: 72/1510/1235987
> Smallest/average/largest padding: 5/412/252927
> Number of dead records: 2
> Smallest/average/largest dead records: 399912712/661866360/923820008
> Number of free records: 38999
> Smallest/average/largest free records: 12/18485/715170720
> Number of hash chains: 10000
> Smallest/average/largest hash chains: 26/66/116
> Number of uncoalesced records: 253
> Smallest/average/largest uncoalesced runs: 1/1/2
> Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 
> 1/30/8/21/39/1/0
> 
> 
> it ends with this error:
> 
> tdb_recovery_allocate: overflow recovery area
> tdb_transaction_prepare_commit: failed to setup recovery data
> ../lib/tdb/common/tdb.c:1055 Failed to commit
> 
> I follow the code and found this line:
> 
> transaction.c:783
> 
>          /* New head will be at end of file. */
>          recovery_head = tdb->map_size;
> 
> In my case it means following:
> 
> my filesize is 3,3GB, old recovery_head was on 2064351232, size of old 
> recovery was about 923820008, size of new recovery should be 1323794228, 
> after tdb_free of old recovery area and setting the new recovery head to 
> end of file(tdb->map_size), it is no more place for new recovery area, 
> so the next check returns "overflow recovery area".
> 
> 
> I'm now thinking about how it should be fixed? I thinking about to 
> improve the checks and maybe search for last data entry and set new 
> recovery head behind it?
In the short term:

Take the DC offline, and then run tdbbackup on each backend file, then
replace them one by one.  That will do the repack, but into a new file
rather than with a transaction. 

In the medium term patches to improve this TDB behaviour would be most
welcome, with tests, so we can avoid folks getting stuck.  This will
help us bridge the gap to LMDB more safely. 

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Andrej Gessel

2018-Aug-02 23:20 UTC

head link

[Samba] tdbtool repack fails

Hello,


as Andrew has already recognized, the idea behind is to reduce file size of
database in filesystem. In my case this database needs to be mmaped on 2/2GB
split 32bit arm machine, so I don't have address space for it, but I could
have space, if file size could be equal to data size.

Thanks Andrew to show me the offline workaround.


After tdbbackup I got this information, but I did not start my DC yet.

tdb> info
Size of file/data: 1350275072/1050098055
Header offset/logical size: 0/1350275072
Number of records: 669737
Incompatible hash: no
Active/supported feature flags: 0x00000000/0x00000001
Robust mutexes locking: no
Smallest/average/largest keys: 12/57/242
Smallest/average/largest data: 72/1510/1235987
Smallest/average/largest padding: 25/397/309007
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 25
Smallest/average/largest free records: 12/725751/18142352
Number of hash chains: 10000
Smallest/average/largest hash chains: 26/66/116
Number of uncoalesced records: 0
Smallest/average/largest uncoalesced runs: 0/0/0
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
3/75/20/1/0/1/0

File size is as expected after repack.

For tdbbackup I need to take my DC offline, how about repack? If my database
would be smaller, and repack could work, can I run it online, or should I take
my DC offline as well?

Andrej


________________________________
Von: Andrew Bartlett <abartlet at samba.org>
Gesendet: Donnerstag, 2. August 2018 20:14
An: Andrej Gessel; 'samba'
Betreff: Re: [Samba] tdbtool repack fails

On Thu, 2018-08-02 at 15:53 +0200, Andrej Gessel via samba
wrote:> Hello list,
>
> I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:
>
> tdb> info
> Size of file/data: 3388084000/1050098055
> Header offset/logical size: 0/3388084000
> Number of records: 669737
> Incompatible hash: no
> Active/supported feature flags: 0x00000000/0x00000001
> Robust mutexes locking: no
> Smallest/average/largest keys: 12/57/242
> Smallest/average/largest data: 72/1510/1235987
> Smallest/average/largest padding: 5/412/252927
> Number of dead records: 2
> Smallest/average/largest dead records: 399912712/661866360/923820008
> Number of free records: 38999
> Smallest/average/largest free records: 12/18485/715170720
> Number of hash chains: 10000
> Smallest/average/largest hash chains: 26/66/116
> Number of uncoalesced records: 253
> Smallest/average/largest uncoalesced runs: 1/1/2
> Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
> 1/30/8/21/39/1/0
>
>
> it ends with this error:
>
> tdb_recovery_allocate: overflow recovery area
> tdb_transaction_prepare_commit: failed to setup recovery data
> ../lib/tdb/common/tdb.c:1055 Failed to commit
>
> I follow the code and found this line:
>
> transaction.c:783
>
>          /* New head will be at end of file. */
>          recovery_head = tdb->map_size;
>
> In my case it means following:
>
> my filesize is 3,3GB, old recovery_head was on 2064351232, size of old
> recovery was about 923820008, size of new recovery should be 1323794228,
> after tdb_free of old recovery area and setting the new recovery head to
> end of file(tdb->map_size), it is no more place for new recovery area,
> so the next check returns "overflow recovery area".
>
>
> I'm now thinking about how it should be fixed? I thinking about to
> improve the checks and maybe search for last data entry and set new
> recovery head behind it?
In the short term:

Take the DC offline, and then run tdbbackup on each backend file, then
replace them one by one.  That will do the repack, but into a new file
rather than with a transaction.

In the medium term patches to improve this TDB behaviour would be most
welcome, with tests, so we can avoid folks getting stuck.  This will
help us bridge the gap to LMDB more safely.

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Possibly Parallel Threads

Search for more possibly parallel threads

samba - Aug 2018 - tdbtool repack fails

[Samba] tdbtool repack fails

[Samba] tdbtool repack fails

[Samba] tdbtool repack fails

[Samba] tdbtool repack fails

Possibly Parallel Threads