I have a set of CoS objects I am importing in and their add times are
extremely slow about 1 a second. There are about 500k objects in the
directory currently and its broken down in a hierarchical format.
A simple ASCII drawing would be.
ou=top
|
- ou=First
|
+ ou=Second
|
+ cn=Final
This is representation of the data but for ease of explanation this
should work.
There are lots of "First" object and they have the possibility of lots
of "second" objects. Its the also the same for "Second"
object they
could have a lot of "Final" objects. The idea is to use CoS at the
First and Second level to reduce the amount look ups and redundant data
as final objects need some info from the second objects and first objects.
Hopefully I explained that in a way it is easy to understand.
My question is are CoS objects not supposed to be used this way? Also
are lots of CoS objects used in a hierarchical tree this way bad? Is
there a way to make these imports faster? And last am I just doing
something completely wrong and there is a better way that I should work
to my end goal.
Thanks
Edward
Edward "Koko" Konetzko wrote:> I have a set of CoS objects I am importing in and their add times are > extremely slow about 1 a second.What platform? What 389-ds-base version? By import do you mean ldif2db or ldap add?> There are about 500k objects in the directory currently and its broken > down in a hierarchical format. > A simple ASCII drawing would be. > > ou=top > | > - ou=First > | > + ou=Second > | > + cn=Final > > > This is representation of the data but for ease of explanation this > should work. > > There are lots of "First" object and they have the possibility of lots > of "second" objects. Its the also the same for "Second" object they > could have a lot of "Final" objects. The idea is to use CoS at the > First and Second level to reduce the amount look ups and redundant > data as final objects need some info from the second objects and first > objects. > Hopefully I explained that in a way it is easy to understand. > > My question is are CoS objects not supposed to be used this way? Also > are lots of CoS objects used in a hierarchical tree this way bad? Is > there a way to make these imports faster? And last am I just doing > something completely wrong and there is a better way that I should > work to my end goal. > > Thanks > Edward > > > > -- > 389 users mailing list > 389-users@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-users
Rich Megginson wrote:> Edward "Koko" Konetzko wrote: >> I have a set of CoS objects I am importing in and their add times are >> extremely slow about 1 a second. > What platform? What 389-ds-base version? By import do you mean > ldif2db or ldap add?RHEL 5 64 bit, RHDS 8.1 and using ldapadd. Hardware is HP DL385 with 16 gigs of ram, raid1 for os and raid 10 for /var/lib/dirsrv. I have tried with the import buffer(?) set to auto and 2 gigs.>> There are about 500k objects in the directory currently and its >> broken down in a hierarchical format. >> A simple ASCII drawing would be. >> >> ou=top >> | >> - ou=First >> | >> + ou=Second >> | >> + cn=Final >> >> >> This is representation of the data but for ease of explanation this >> should work. >> >> There are lots of "First" object and they have the possibility of >> lots of "second" objects. Its the also the same for "Second" object >> they could have a lot of "Final" objects. The idea is to use CoS at >> the First and Second level to reduce the amount look ups and >> redundant data as final objects need some info from the second >> objects and first objects. >> Hopefully I explained that in a way it is easy to understand. >> >> My question is are CoS objects not supposed to be used this way? >> Also are lots of CoS objects used in a hierarchical tree this way >> bad? Is there a way to make these imports faster? And last am I >> just doing something completely wrong and there is a better way that >> I should work to my end goal. >> >> Thanks >> Edward >> >> >> >> -- >> 389 users mailing list >> 389-users@redhat.com >> https://www.redhat.com/mailman/listinfo/fedora-directory-users >
Edward "Koko" Konetzko wrote:> Rich Megginson wrote: >> Edward "Koko" Konetzko wrote: >>> I have a set of CoS objects I am importing in and their add times >>> are extremely slow about 1 a second. >> What platform? What 389-ds-base version? By import do you mean >> ldif2db or ldap add? > RHEL 5 64 bit, RHDS 8.1 and using ldapadd. Hardware is HP DL385 with > 16 gigs of ram, raid1 for os and raid 10 for /var/lib/dirsrv. I have > tried with the import buffer(?) set to auto and 2 gigs.Since you are using Red Hat Directory Server, you should contact Red Hat support. import buffer is only when using ldif2db. LDAP Add is much slower than ldif2db. I think part of it is that CoS is optimized for searching, but not so much for adding new CoS definitions - it builds a cache in memory to make searches go very quickly, but building the cache takes time during each add or modify operation.>>> There are about 500k objects in the directory currently and its >>> broken down in a hierarchical format. >>> A simple ASCII drawing would be. >>> >>> ou=top >>> | >>> - ou=First >>> | >>> + ou=Second >>> | >>> + cn=Final >>> >>> >>> This is representation of the data but for ease of explanation this >>> should work. >>> >>> There are lots of "First" object and they have the possibility of >>> lots of "second" objects. Its the also the same for "Second" object >>> they could have a lot of "Final" objects. The idea is to use CoS at >>> the First and Second level to reduce the amount look ups and >>> redundant data as final objects need some info from the second >>> objects and first objects. >>> Hopefully I explained that in a way it is easy to understand. >>> >>> My question is are CoS objects not supposed to be used this way? >>> Also are lots of CoS objects used in a hierarchical tree this way >>> bad? Is there a way to make these imports faster? And last am I >>> just doing something completely wrong and there is a better way that >>> I should work to my end goal. >>> >>> Thanks >>> Edward >>> >>> >>> >>> -- >>> 389 users mailing list >>> 389-users@redhat.com >>> https://www.redhat.com/mailman/listinfo/fedora-directory-users >> >
Rich Megginson wrote:> Edward "Koko" Konetzko wrote: >> Rich Megginson wrote: >>> Edward "Koko" Konetzko wrote: >>>> I have a set of CoS objects I am importing in and their add times >>>> are extremely slow about 1 a second. >>> What platform? What 389-ds-base version? By import do you mean >>> ldif2db or ldap add? >> RHEL 5 64 bit, RHDS 8.1 and using ldapadd. Hardware is HP DL385 with >> 16 gigs of ram, raid1 for os and raid 10 for /var/lib/dirsrv. I >> have tried with the import buffer(?) set to auto and 2 gigs. > Since you are using Red Hat Directory Server, you should contact Red > Hat support. > > import buffer is only when using ldif2db. LDAP Add is much slower > than ldif2db. I think part of it is that CoS is optimized for > searching, but not so much for adding new CoS definitions - it builds > a cache in memory to make searches go very quickly, but building the > cache takes time during each add or modify operation.Can''t call Redhat as its centos-ds, I screwed up calling it RHDS since they are the same. Thanks for the help Ill try a few other things.>>>> There are about 500k objects in the directory currently and its >>>> broken down in a hierarchical format. >>>> A simple ASCII drawing would be. >>>> >>>> ou=top >>>> | >>>> - ou=First >>>> | >>>> + ou=Second >>>> | >>>> + cn=Final >>>> >>>> >>>> This is representation of the data but for ease of explanation this >>>> should work. >>>> >>>> There are lots of "First" object and they have the possibility of >>>> lots of "second" objects. Its the also the same for "Second" object >>>> they could have a lot of "Final" objects. The idea is to use CoS >>>> at the First and Second level to reduce the amount look ups and >>>> redundant data as final objects need some info from the second >>>> objects and first objects. >>>> Hopefully I explained that in a way it is easy to understand. >>>> >>>> My question is are CoS objects not supposed to be used this way? >>>> Also are lots of CoS objects used in a hierarchical tree this way >>>> bad? Is there a way to make these imports faster? And last am I >>>> just doing something completely wrong and there is a better way >>>> that I should work to my end goal. >>>> >>>> Thanks >>>> Edward >>>> >>>> >>>> >>>> -- >>>> 389 users mailing list >>>> 389-users@redhat.com >>>> https://www.redhat.com/mailman/listinfo/fedora-directory-users >>> >> >