samba - Apr 2020 - Status of Windows Search Protocol support in Samba?

(resending, apologies if you got it already but I got a rejection message)

Hi All


On 21/04/2020 01:48, Jeremy Allison via samba wrote:
> On Sun, Apr 19, 2020 at 10:49:55AM +0100, Mark Rousell via samba wrote:
>> On 19/04/2020 10:25, Ralph Boehme via samba wrote:
>>> Am 4/17/20 um 5:55 PM schrieb Mark Rousell via samba:
>>>> I've been trying to Google this but have not found any
definitive
>>>> answers. Does Samba currently support Windows Search
Protocol[1]?
>>> short answer: no.
>>>
>>> Last time I talked to Noel this is still a multi-weeks effort
(iirc)
>>> getting it finished, polished and integrated into master.
>>>
>>> -slow
>> Ralph,
>>
>> Thank you very much. I appreciate the response.
>>
>> Yes, I can imagine that integrating this is a non-trivial effort.
>>
>> Oddly enough, perhaps, I'd have thought that one of the hardest
parts
>> would be interfacing with a host's index/search tool and figuring
out
>> how to degrade sensibly if there is no compatible tool.
>>
>> On Linux operating systems (and perhaps BSD too, not sure), Tracker,
>> Baloo, Nepomuk (on old installations), Beagle (on yet older
>> installations), or Recoll might need to be interface targets.
Presumably
>> Spotlight on Mac allows querying by API but I know nothing of it. And
>> Windows Search on Windows does allow querying by multiple APIs but, in
>> practice, would be the lowest priority to act as a server where WSP
over
>> Samba is concerned.
>>
>> I feel I should be contributing to getting this done but my C skills
are
>> extremely limited.
> Yes, I think Noel has it working, but he needs some help
> getting it reviewerd and integrated.
>
> Noel, does that accurately describe the state of affairs ?
>
yep, WSP server that translates protocol to tracker sparql queries is
working (with caveats [1])

The code would need some effort to get it in shape, as you might imagine
there has been quite a bit of bitrot especially of late as recent dce
rework broke it (although I have a WIP version working against current
master (even more hacky than usual)

I've also started to look at elastic search as a backend, and am going
to talk about that a little at sambaxp. Probably that talk would maybe
explain more (and in more detail) about some of these questions. I
intend to talk about state of the WSP work, revisit again and explain
how it works, what works, what doesn't,? what is provided, talk a bit
about experience of experimenting with elasticsearch how that affect
what needs to be done with the current implementation etc. (or something
like that :-))


Noel

[1]

WSP is complex, generic and very windows centric, there is a large
impedance? mismatch between the features that WSP seems to be able to
provide and query for and what say is available from tracker. (mileage
may vary with other indexers). The WSP support that I have worked on has
concentrated mainly on supporting the subset of queries that are emitted
when using the search UI ribbon in windows explorer. Additionally there
is a simple WSP samba search client that allows simple queries to
constructed to be targeted at any WSP server.

On 21/04/2020 12:38, Noel Power via samba wrote:
> (resending, apologies if you got it already but I got a rejection message)
This one got through on the Samba list.

Thank you for your helpful reply.
> yep, WSP server that translates protocol to tracker sparql queries is
> working (with caveats [1])
>
> The code would need some effort to get it in shape, as you might imagine
> there has been quite a bit of bitrot especially of late as recent dce
> rework broke it (although I have a WIP version working against current
> master (even more hacky than usual)
>
> I've also started to look at elastic search as a backend, and am going
> to talk about that a little at sambaxp. Probably that talk would maybe
> explain more (and in more detail) about some of these questions. I
> intend to talk about state of the WSP work, revisit again and explain
> how it works, what works, what doesn't,  what is provided, talk a bit
> about experience of experimenting with elasticsearch how that affect
> what needs to be done with the current implementation etc. (or something
> like that :-))
>
>
> Noel
>
> [1]
>
> WSP is complex, generic and very windows centric, there is a large
> impedance  mismatch between the features that WSP seems to be able to
> provide and query for and what say is available from tracker. (mileage
> may vary with other indexers). The WSP support that I have worked on has
> concentrated mainly on supporting the subset of queries that are emitted
> when using the search UI ribbon in windows explorer. Additionally there
> is a simple WSP samba search client that allows simple queries to
> constructed to be targeted at any WSP server.
Good idea, I think, about focussing on likely core searches from the
Windows search UI (at least to begin with).

I think that Elasticsearch vs. Tracker (or Baloo, or Recoll, or others)
is largely a use case suitability issue. If one is querying a file
server or a NAS then Elasticsearch is probably an ideal server-side
indexing solution. On the other hand, if one is querying a desktop box
(which is still a valid use case for WSP) then it would make more sense
to query into whatever the user is already running (e.g. Tracker on
Gnome, Baloo on KDE, Recoll on anything, Nepomuk on old KDE, and so on).
However, I appreciate that this is all easier said than done.

May I ask, is the index/search integration modular? What I mean is:
Would it be possible/feasible to create a plugin structure that allowed
third parties to create a plugin for an arbitrary indexing/search tool?
(And yes, I realise that creating a plugin infrastructure is even more
work than integrating directly with Elasticsearch, Tracker, etc., so I'm
just asking how things are. :-) ).


-- 
Mark Rousell

On 21/04/2020 13:14, Mark Rousell via samba wrote:
> On 21/04/2020 12:38, Noel Power via samba wrote:
>> (resending, apologies if you got it already but I got a rejection
message)
> This one got through on the Samba list.
>
> Thank you for your helpful reply.
>
>> yep, WSP server that translates protocol to tracker sparql queries is
>> working (with caveats [1])
>>
>> The code would need some effort to get it in shape, as you might
imagine
>> there has been quite a bit of bitrot especially of late as recent dce
>> rework broke it (although I have a WIP version working against current
>> master (even more hacky than usual)
>>
>> I've also started to look at elastic search as a backend, and am
going
>> to talk about that a little at sambaxp. Probably that talk would maybe
>> explain more (and in more detail) about some of these questions. I
>> intend to talk about state of the WSP work, revisit again and explain
>> how it works, what works, what doesn't,  what is provided, talk a
bit
>> about experience of experimenting with elasticsearch how that affect
>> what needs to be done with the current implementation etc. (or
something
>> like that :-))
>>
>>
>> Noel
>>
>> [1]
>>
>> WSP is complex, generic and very windows centric, there is a large
>> impedance  mismatch between the features that WSP seems to be able to
>> provide and query for and what say is available from tracker. (mileage
>> may vary with other indexers). The WSP support that I have worked on
has
>> concentrated mainly on supporting the subset of queries that are
emitted
>> when using the search UI ribbon in windows explorer. Additionally there
>> is a simple WSP samba search client that allows simple queries to
>> constructed to be targeted at any WSP server.
> Good idea, I think, about focussing on likely core searches from the
> Windows search UI (at least to begin with).
>
> I think that Elasticsearch vs. Tracker (or Baloo, or Recoll, or others)
> is largely a use case suitability issue. If one is querying a file
> server or a NAS then Elasticsearch is probably an ideal server-side
> indexing solution. On the other hand, if one is querying a desktop box
> (which is still a valid use case for WSP) then it would make more sense
> to query into whatever the user is already running (e.g. Tracker on
> Gnome, Baloo on KDE, Recoll on anything, Nepomuk on old KDE, and so on).
> However, I appreciate that this is all easier said than done.
>
> May I ask, is the index/search integration modular? What I mean is:
> Would it be possible/feasible to create a plugin structure that allowed
> third parties to create a plugin for an arbitrary indexing/search tool?
yes and no, there is an initial separation layer built into the server
implementation, currently only one concrete implementation
(tracker/sparql). Part of the rational to try use elasticsearch was to
really see if the separation layer is workable/suitable and to more
clearly identify parts of the code that could be shared for use by other
'backends'
> (And yes, I realise that creating a plugin infrastructure is even more
> work than integrating directly with Elasticsearch, Tracker, etc., so
I'm
> just asking how things are. :-) ).
>
so currently I am not looking at a plugin infrastructure, I think it is
a bit early to nail api(s) in stone. I think when the code has
stabilised and there is more than one real concrete backend implemented
it might be a more suitable time to think about a plugin infrastructure.
As you say it's alot of work to get a plugin framework nailed down
(api(s) etc.) and currently this is just a hobby project for me

Noel