Fedora directory users - Jan 2007 - big searches dont return anything

Hello,

I am moving our old Sun One setup to Fedora Directory server and I am pretty 
impressed.

Just one thing does not work: searches that find a lot of entries:

e.g. if I search for "smith", nothing comes back, no error either
(from the
console). I have around 10 000 users, so I''d say there are a lot of
Smith''s
since this is London...

However, if I make the search more precise, like "John Smith", it
returns the
results.

It must have something to do with search limits, I tried a couple of things in 
the console but no result at all.

I think I remember a param called "admin limit" but I cannot find it
in the
console. I tuned "lookthrough limit" and "size limit" with
no luck.

Any ideas?

This is FDS 1.0.4 running on Fedora Core 6, x86_64.

Thanks for helping,
-- 
Stéphane Konstantaropoulos <skonstant@sgul.ac.uk>
-- Web Developer - Computing Services
--- St George''s University of London

Le lundi 08 jan 2007 17:58, Richard Megginson a écrit :
> Stéphane Konstantaropoulos wrote:
> > Hello,
> >
> > I am moving our old Sun One setup to Fedora Directory server and I am
> > pretty impressed.
> >
> > Just one thing does not work: searches that find a lot of entries:
> >
> > e.g. if I search for "smith", nothing comes back, no error
either (from
> > the console). I have around 10 000 users, so I''d say there
are a lot of
> > Smith''s since this is London...
> >
> > However, if I make the search more precise, like "John
Smith", it returns
> > the results.
> >
> > It must have something to do with search limits, I tried a couple of
> > things in the console but no result at all.
> >
> > I think I remember a param called "admin limit" but I cannot
find it in
> > the console. I tuned "lookthrough limit" and "size
limit" with no luck.
>
> http://www.redhat.com/docs/manuals/dir-server/pdf/ds71cli.pdf
> Look for nsslapd-sizelimit, nsslapd-timelimit, and nsslapd-lookthroughlimit
>
> Also, please post the RESULT line from the access log for these searches
> which do not return results.
>

OK,

here is a search log, for a search that does not return anything:

[09/Jan/2007:11:08:15 +0000] conn=914 op=7 SRCH base="o=sghms.ac.uk"
scope=2
filter="(|(&(objectClass=person)(cn=*smith*))(&(objectClass=person)
(uid=smith)))" attrs=ALL
[09/Jan/2007:11:08:15 +0000] conn=914 op=7 RESULT err=0 tag=101 nentries=0 
etime=0

seems to be no error at all.

size limit is set to 10000, time limit to 3600 and lookthrough to 20000 (is 
that  right? it''s on the LDBM plug-in settings).

Does this make sense to anybody?

-- 
Stéphane Konstantaropoulos <skonstant@sgul.ac.uk>
-- Web Developer - Computing Services
--- St George''s University of London

Stéphane Konstantaropoulos wrote:
>[09/Jan/2007:11:08:15 +0000] conn=914 op=7 SRCH
base="o=sghms.ac.uk" scope=2
>filter="(|(&(objectClass=person)(cn=*smith*))(&(objectClass=person)
>(uid=smith)))" attrs=ALL
>[09/Jan/2007:11:08:15 +0000] conn=914 op=7 RESULT err=0 tag=101 nentries=0 
>etime=0
>
>seems to be no error at all.
>
>size limit is set to 10000, time limit to 3600 and lookthrough to 20000 (is 
>that  right? it''s on the LDBM plug-in settings).
>
>Does this make sense to anybody?
>
>  
>
Not yet. Can you post a successful search and an example of one of the 
entries returned ?
The above log seems to indicate that there really are no target entries 
in the database,
but there could be other explainations such as ACL preventing their 
access, database corrupt, etc.

Le mardi 09 jan 2007 18:50, David Boreham a écrit :
> Stéphane Konstantaropoulos wrote:
> >[09/Jan/2007:11:08:15 +0000] conn=914 op=7 SRCH
base="o=sghms.ac.uk"
> > scope=2
> >
filter="(|(&(objectClass=person)(cn=*smith*))(&(objectClass=person)
> > (uid=smith)))" attrs=ALL
> >[09/Jan/2007:11:08:15 +0000] conn=914 op=7 RESULT err=0 tag=101
nentries=0
> >etime=0
> >
> >seems to be no error at all.
> >
> >size limit is set to 10000, time limit to 3600 and lookthrough to 20000
> > (is that  right? it''s on the LDBM plug-in settings).
> >
> >Does this make sense to anybody?
>
> Not yet. Can you post a successful search and an example of one of the
> entries returned ?
> The above log seems to indicate that there really are no target entries
> in the database,
> but there could be other explainations such as ACL preventing their
> access, database corrupt, etc.
Here you go, I first search on "k", then "ko", then
"kon", then "kons" returns
results.

Yes, i think my db may be a bit funny, I just deleted all the indexes and 
tried re-creating them, no change tho still. I set timelimit, sizelimit and 
lookthrough limit to -1, so as to have no limit at all now.

[10/Jan/2007:13:24:26 +0000] conn=24 op=62 SRCH 
base="o=sghms.ac.uk,o=sghms.ac.uk" scope=2
filter="(|(&(objectClass=person)
(cn=*k*))(&(objectClass=person)(uid=k)))" attrs=ALL
[10/Jan/2007:13:24:26 +0000] conn=24 op=62 RESULT err=0 tag=101 nentries=0 
etime=0
[10/Jan/2007:13:24:29 +0000] conn=24 op=64 SRCH base="" scope=0
filter="(|
(objectClass=*)(objectClass=ldapsubentry))"
attrs="nsBackendSuffix"
[10/Jan/2007:13:24:29 +0000] conn=24 op=64 RESULT err=0 tag=101 nentries=1 
etime=0
[10/Jan/2007:13:24:29 +0000] conn=24 op=65 SRCH base="cn=userRoot,cn=ldbm 
database, cn=plugins, cn=config" scope=1
filter="(objectClass=vlvSearch)"
attrs=ALL
[10/Jan/2007:13:24:29 +0000] conn=24 op=65 RESULT err=0 tag=101 nentries=4 
etime=0
[10/Jan/2007:13:24:29 +0000] conn=24 op=66 SRCH 
base="o=sghms.ac.uk,o=sghms.ac.uk" scope=2
filter="(|(&(objectClass=person)
(cn=*kon*))(&(objectClass=person)(uid=kon)))" attrs=ALL
[10/Jan/2007:13:24:29 +0000] conn=24 op=66 RESULT err=0 tag=101 nentries=0 
etime=0
[10/Jan/2007:13:24:31 +0000] conn=24 op=67 SRCH base="" scope=0
filter="(|
(objectClass=*)(objectClass=ldapsubentry))"
attrs="nsBackendSuffix"
[10/Jan/2007:13:24:31 +0000] conn=24 op=67 RESULT err=0 tag=101 nentries=1 
etime=0
[10/Jan/2007:13:24:31 +0000] conn=24 op=68 SRCH base="cn=userRoot,cn=ldbm 
database, cn=plugins, cn=config" scope=1
filter="(objectClass=vlvSearch)"
attrs=ALL
[10/Jan/2007:13:24:31 +0000] conn=24 op=68 RESULT err=0 tag=101 nentries=4 
etime=0
[10/Jan/2007:13:24:31 +0000] conn=24 op=69 SRCH 
base="o=sghms.ac.uk,o=sghms.ac.uk" scope=2
filter="(|(&(objectClass=person)
(cn=*kons*))(&(objectClass=person)(uid=kons)))" attrs=ALL
[10/Jan/2007:13:24:31 +0000] conn=24 op=69 RESULT err=0 tag=101 nentries=4 
etime=0


-- 
Stéphane Konstantaropoulos <skonstant@sgul.ac.uk>
-- Web Developer - Computing Services
--- St George''s University of London

Le mercredi 10 jan 2007 15:18, Richard Megginson a
écrit :
> Stéphane Konstantaropoulos wrote:
> > Le mardi 09 jan 2007 18:50, David Boreham a écrit :
> >> Stéphane Konstantaropoulos wrote:
> >>> [09/Jan/2007:11:08:15 +0000] conn=914 op=7 SRCH
base="o=sghms.ac.uk"
> >>> scope=2
> >>>
filter="(|(&(objectClass=person)(cn=*smith*))(&(objectClass=person)
> >>> (uid=smith)))" attrs=ALL
> >>> [09/Jan/2007:11:08:15 +0000] conn=914 op=7 RESULT err=0
tag=101
> >>> nentries=0 etime=0
> >>>
> >>> seems to be no error at all.
> >>>
> >>> size limit is set to 10000, time limit to 3600 and lookthrough
to 20000
> >>> (is that  right? it''s on the LDBM plug-in settings).
> >>>
> >>> Does this make sense to anybody?
> >>
> >> Not yet. Can you post a successful search and an example of one of
the
> >> entries returned ?
> >> The above log seems to indicate that there really are no target
entries
> >> in the database,
> >> but there could be other explainations such as ACL preventing
their
> >> access, database corrupt, etc.
> >
> > Here you go, I first search on "k", then "ko",
then "kon", then "kons"
> > returns results.
> >
> > Yes, i think my db may be a bit funny, I just deleted all the indexes
and
> > tried re-creating them, no change tho still. I set timelimit,
sizelimit
> > and lookthrough limit to -1, so as to have no limit at all now.
> >
> > [10/Jan/2007:13:24:26 +0000] conn=24 op=62 SRCH
> > base="o=sghms.ac.uk,o=sghms.ac.uk" scope=2
> > filter="(|(&(objectClass=person)
> > (cn=*k*))(&(objectClass=person)(uid=k)))" attrs=ALL
> > [10/Jan/2007:13:24:26 +0000] conn=24 op=62 RESULT err=0 tag=101
> > nentries=0 etime=0
> > [10/Jan/2007:13:24:29 +0000] conn=24 op=64 SRCH base=""
scope=0
> > filter="(| (objectClass=*)(objectClass=ldapsubentry))"
> > attrs="nsBackendSuffix" [10/Jan/2007:13:24:29 +0000] conn=24
op=64 RESULT
> > err=0 tag=101 nentries=1 etime=0
> > [10/Jan/2007:13:24:29 +0000] conn=24 op=65 SRCH
base="cn=userRoot,cn=ldbm
> > database, cn=plugins, cn=config" scope=1
filter="(objectClass=vlvSearch)"
> > attrs=ALL
> > [10/Jan/2007:13:24:29 +0000] conn=24 op=65 RESULT err=0 tag=101
> > nentries=4 etime=0
> > [10/Jan/2007:13:24:29 +0000] conn=24 op=66 SRCH
> > base="o=sghms.ac.uk,o=sghms.ac.uk" scope=2
> > filter="(|(&(objectClass=person)
> > (cn=*kon*))(&(objectClass=person)(uid=kon)))" attrs=ALL
> > [10/Jan/2007:13:24:29 +0000] conn=24 op=66 RESULT err=0 tag=101
> > nentries=0 etime=0
>
> Hmm - no unindexed notes in the results.  Do you know how many entries
> match *kon*?  How many match *kons*?
>
"kons" matched 5 entries, "kon" not sure. Anyway, it was
definitely a database
corruption problem, I emptied the directory and re-populated it with a script 
I have and now it finds everybody and returns an ADMINLIMIT_EXCEEDED error 
when it reaches the limit, I set it to no limit now because I have a 40k 
entries.

It''d be nice if it noticed by itself that the db is corrupted. I also
tried to
delete all the indexes and then re-create them, which it did but that made no 
difference.

Anyway, thanks for helping,

-- 
Stéphane Konstantaropoulos <skonstant@sgul.ac.uk>
-- Web Developer - Computing Services
--- St George''s University of London

Stéphane Konstantaropoulos wrote:
>It''d be nice if it noticed by itself that the db is corrupted. 
>
Unfortunately that''s something of an AI problem :(
There is some code in the server that can compare the
results of an indexed vs an unindexed execution of the same
query (used in the past to debug query optimizations). Someone
could develop that into a kind of index inconsistency tool.
All out corruption (someone writes random c**p over the
database pages _will_ be detected). It sounds like you had
some inconsistency between the primary and secondary
indices. I''m not sure how that could have happened
(it shouldn''t).
>I also tried to 
>delete all the indexes and then re-create them, which it did but that made
no
>difference.
>  
>

Is it possible for DB corruption to be replicated?
In other words, if a master replica''s DB goes corrupt, how likely is
that to
corrupt the DB on the consumers (if at all)?
Thanks,
-- George

----- Original Message ----- 
From: "David Boreham" <david_list@boreham.org>
To: "General discussion list for the Fedora Directory server project."
<fedora-directory-users@redhat.com>
Sent: Wednesday, January 10, 2007 8:07 AM
Subject: Re: [Fedora-directory-users] big searches dont return anything


Stéphane Konstantaropoulos wrote:
>It''d be nice if it noticed by itself that the db is corrupted.
Unfortunately that''s something of an AI problem :(
There is some code in the server that can compare the
results of an indexed vs an unindexed execution of the same
query (used in the past to debug query optimizations). Someone
could develop that into a kind of index inconsistency tool.
All out corruption (someone writes random c**p over the
database pages _will_ be detected). It sounds like you had
some inconsistency between the primary and secondary
indices. I''m not sure how that could have happened
(it shouldn''t).

George Holbert wrote:
> Is it possible for DB corruption to be replicated?
> In other words, if a master replica''s DB goes corrupt, how likely
is
> that to corrupt the DB on the consumers (if at all)?
In general this can''t happen. Replication is done at the
directory entry semantic level, so each server re-creates
its own underlying database content to reflect replicated
entries (unlike for example a transaction log shipping type
replication that you might see in some relational databases).

However, if there were a database corruption bug present
somewhere in the server, it is possible, even likely that the
same bug would be triggered in multiple replicating servers
that contain the same data.