similar to: GSOC-2016 Project : Clustering of search results

On Sun, Mar 6, 2016 at 7:17 AM, James Aylett <james-xapian at tartarus.org> wrote: > On Sat, Mar 05, 2016 at 10:58:43PM +0530, Richhiey Thomas wrote: > > K-Means or something related certainly seems like a viable approach, > so what you'll need to do is to come up with a proposal of how you'd > implement this in Xapian (either with reference to the previous work, >

Hello, I've been working on the KMeans clustering algorithm recently and since the past week, I have been stuck on a problem which I'm not able to find a solution to. Since we are representing documents as Tf-idf vectors, they are really sparse vectors (a usual corpus can have around 5000 terms). So it gets really difficult to represent these sparse vectors in a way that would be

Hey Parth, Thanks for the reply. I am considering implementing a cosine distance metric too, along with euclidian distance because of the dimensionality issue that comes in with K-Means and euclidian distance metric. That does help when we deal with sparse vectors for documents. The particular problem I'm having is representing centroids in an efficient way. For example, when we find the mean

On Mon, Mar 07, 2016 at 01:36:43AM +0530, Richhiey Thomas wrote: > My questions are: > 1) Can you direct me on how to convert this raw idea into a proposal in > context to Xapian with more detail? What areas do I focus on? Our GSoC guide has an application template <https://trac.xapian.org/wiki/GSoCApplicationTemplate> which you should use to structure your proposal. It has some

On Sat, Mar 12, 2016 at 04:27:55PM +0530, Richhiey Thomas wrote: > Below I write a raw version of my proposal for Clustering of Search Results > based on our previous mails. Hi, Richhiey. Thanks for putting this together ahead of the formal start of applications, and sharing it with us -- and it's really not too long! Project proposals for something that will last the summer are

Hello devs. I would like to propose how I plan to go about improving and getting a system that can be integrated into Xapian in this GSoC for the clustering branch. I have identified three areas of work which were not touched last time. 1) Automated Performance Analysis I had roughly implemented 2 evaluation techniques previously (Distance b/w document and centroids within clusters and

On 18 Aug 2016, at 23:59, Richhiey Thomas <richhiey.thomas at gmail.com> wrote: > I've currently added a few classes which don't really belong to the public API (currently) into private headers and used PIMPL with the Cluster class. I'm having difficulty reading your changes, because you aren't keeping to one complete change per commit. So for instance you've added a

Hello, I've recently finished with an implementation of KMeans with two initialization techniques, random initialization and KMeans++. I would like to share my findings after evaluating the same. I have tested this implementation of KMeans with a BBC news article dataset. I am currently working on evaluating the same with FIRE datasets. Currently, clustering more than 500 documents

On Mon, Mar 14, 2016 at 02:09:13AM +0530, Richhiey Thomas wrote: > The way the paper has been written I guess is the main source of your > confusion. Let me provide a paper that explains this same concept in a way > that is easier to understand. I was confused by eq (3) that you mentioned > too. Here it is : > http://www.sau.ac.in/~vivek/softcomp/clustering%20PSO+K-means.pdf Ah,

Hello, Thanks James for the reply. That cleared a few things out. Apologies for replying late because of exams going on. I was going through the previous clustering API to understand how it worked and it seems like the the approach for construction of the termlists which are used for distance metrics use TF-IDF weighting with cosine similarity, which is very similar to the approach I would need

> > > > Actually, you're doing something slightly unusual there: making the > internal member public. Protected would be better, and private is I think > most usual; library clients aren't going to have access to the Internal > class declaration, so they can't call things on it. This means it's > actually difficult right now to subclass Feature. > > I

Hello devs, My name is Richhiey Thomas.and I've been selected for GSoC 2016 for the project Clustering of Search Results. I would like to thank the Xapian GSoC admin's for giving me this opportunity and James and Olly to help me with my first merge request. In the next two to three days, I'll critically examine all the aspects of the project that I could have any doubts in and clear

Before going ahead with the tests as you mentioned above, I would just like to clarify a few higher level things that I am still in doubt about. 1) As discussed during the IRC interview, I was suggested about first implementing a normal K-means clustering implementation and then adding on the PSO module as a functionality that can be used to improve quality of clustering for speed as a trade off.

> How long does 200?300 documents take to cluster? How does it grow as more > documents are included in the MSet? We'd expect an MSet of 1000 documents > to take longer to cluster than one with 100, but the important thing is > _how_ the time increases as the number of documents grows. > > Currently, the number of seconds taken for clustering a set of documents for varying

On Wed, Aug 17, 2016 at 7:23 PM, James Aylett <james-xapian at tartarus.org> wrote: > >> How long does 200?300 documents take to cluster? How does it grow as > more documents are included in the MSet? We'd expect an MSet of 1000 > documents to take longer to cluster than one with 100, but the important > thing is _how_ the time increases as the number of documents

Hi, I am currently pursuing my computing science bachelors degree at university of Alberta, Canada. My speciality lie in Information retrieval, machine learning and data mining. In order to get hands on experience with real world information retrieval systems, I would like to contribute to the Xapian project. I have been going through some of the project ideas in

Dear list, first apologies for this is not strictly an R question but a theoretical one. I have read that use of k-means clustering assumes sphericity of data distribution. Can anyone explain me what this means? My statistical background is too poor. Is it another kind of distribution, like gaussian or binomial? What does it happen if the distribution is not spherical? Could you give me an

Xapian 1.4.2 can now be downloaded from: https://xapian.org/download The wiki has a summary of the most notable changes: https://trac.xapian.org/wiki/ReleaseOverview/1.4.2 A big thanks to the following people for helping to make this release a reality: Germán M. Bravo, Vivek Pal, Emmanuel Engelhart, Assem Chelli, Lucas Nussbaum, Alex Dunn, James Aylett, catkin, William Fulton. If I've

Helo there. I am using kmeans of base package to cluster my customers. As the results of kmeans is dependent on the initial centroid, may I know: 1) how can we specify the centroid in the R function? (I don't want random starting pt) 2) how to determine the optimal (if not, a good) centroid to start with? (I am not after the fixed seed solution as it only ensure that the

I'm happy to be able to announce that Xapian 1.3.5 is now available. Please note that 1.3.x releases are development releases - they are made to encourage earlier and wider use and testing of new and changed code. We're closing in on a stable 1.4 release series, so please consider trying this out with your code if you haven't already. There's not a set release date for 1.4.0,