samba - Sep 2019 - Repacking database from v1 to v2 format: how long does it take?

On 23/09/2019 20:41, Andrew Bartlett wrote:
> On Mon, 2019-09-23 at 11:25 +0200, Francesco Malvezzi via samba wrote:
>> hi all,
>>
>> I updated a small domain with 8k object to samba-4.11.0 and the
>> database
>> conversion from v1 to v2 didn't take a noticeable time.
>>
>> On the other hand, in a larger domain with 67k object, where the
>> sudo ./bin/samba-tool dbcheck --cross-ncs --fix
>> takes ~40 minutes, 2 hours and half were not enough to complete the
>> conversion.
>>
>> Is it a couple of hours something expected if dbcheck takes so much
>> time?
>>
>> Is it maybe my AD db to be pathologic?
> 
> Ouch!  We did not see such long runtimes during our testing, which is
> why no major warnings were made.  Are you on TDB or LMDB?  Does the
> host have enough RAM?
It's TDB. I am unaware of the chance to convert to LMDB.
> 
> At your scale I would suggest say 10G, being 4GB for the main TDB, 4GB
> for the transaction copy for the repack, and extra.
It's close: 8GB, but while doing repack memory's usage was never higher
than 20%; it was high on cpu, one core out of two was stuck at 100%.
According to top, there was no iowait, so I supposed it was not a memory
issue.

I'm going to check more carefully,
> 
> Sorry!
no problem,

thank you,

franz

On Tue, 2019-09-24 at 11:44 +0200, Francesco Malvezzi via samba
wrote:
> On 23/09/2019 20:41, Andrew Bartlett wrote:
> > On Mon, 2019-09-23 at 11:25 +0200, Francesco Malvezzi via samba
> > wrote:
> > > hi all,
> > > 
> > > I updated a small domain with 8k object to samba-4.11.0 and the
> > > database
> > > conversion from v1 to v2 didn't take a noticeable time.
> > > 
> > > On the other hand, in a larger domain with 67k object, where the
> > > sudo ./bin/samba-tool dbcheck --cross-ncs --fix
> > > takes ~40 minutes, 2 hours and half were not enough to complete
> > > the
> > > conversion.
> > > 
> > > Is it a couple of hours something expected if dbcheck takes so
> > > much
> > > time?
> > > 
> > > Is it maybe my AD db to be pathologic?
> > 
> > Ouch!  We did not see such long runtimes during our testing, which
> > is
> > why no major warnings were made.  Are you on TDB or LMDB?  Does the
> > host have enough RAM?
> 
> It's TDB. I am unaware of the chance to convert to LMDB.
Thanks for confirming that.  Currently the only way to convert is to
join a new DC using LMDB. 

We certainly could write a conversion tool, and perhaps we should, but
for now that is the only process.
> > 
> > At your scale I would suggest say 10G, being 4GB for the main TDB,
> > 4GB
> > for the transaction copy for the repack, and extra.
> 
> It's close: 8GB, but while doing repack memory's usage was never
> higher
> than 20%; it was high on cpu, one core out of two was stuck at 100%.
> According to top, there was no iowait, so I supposed it was not a
> memory
> issue.
Thanks for confirming that.  I think you are probably hitting some
pathological behaviours at the TDB layer when handling so many records,
perhaps in the free list.

The way to debug this, if you are keen to, is to do a debug build of
Samba run get a flame graph per:

http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions

That will get us a better idea of which routines are 'hot' during your
in-place upgrade.

I do wish you the very best with this and am keen to ensure this works
really well for you.

Andrew Bartlett
-- 
Andrew Bartlett                       https://samba.org/~abartlet/
Authentication Developer, Samba Team  https://samba.org
Samba Developer, Catalyst IT          
https://catalyst.net.nz/services/samba

[...]
> 
> Thanks for confirming that.  I think you are probably hitting some
> pathological behaviours at the TDB layer when handling so many records,
> perhaps in the free list.
> 
> The way to debug this, if you are keen to, is to do a debug build of
> Samba run get a flame graph per:
> 
> http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions
> 
> That will get us a better idea of which routines are 'hot' during
your
> in-place upgrade.
I am not sure whether I got the correct graph:

https://send.firefox.com/download/161a3c864f996b32/#7REkvXE5e6dVxB8422YGWg

It was taken while the cpu was hot, five minutes log, during repack.
It's a 4 core box so it's correct there is a 25% cpu usage while the
others are idle.

If the graph is what you would expect, do you need a longer time slice?

thank you,

Franz