[Xapian-discuss] Always returning ALL the documents matching a query

[James Aylett]

On Mon, Dec 29, 2008 at 02:26:37PM -0500, tata 668 wrote:

> > You could try paging the Xapian result sets, and grabbing more page by
> > page until you get enough (after removing false positives) to fill the
> > page you need. That should be more efficient than grabbing all results
> > on the first page and caching, I'd have thought.
> 
> The problem I see using this method is that the sorting has to be
> done by Xapian, right? If I want to manage the sorting outside
> Xapian, I have to get all the posts resulting of the search in the
> first place.

Yes.

> In your opinion the problem will be memory? Even if I only store the
> ids of the posts as the document "data" and I have no "vales" at
> all? An PHP array of, let's say, 5000-10000 ids won't bust my server
> memory except if a lot of searches are done at the same time, no? 
> Where is the bootleneck exactly, in your opinion?

I have no opinion here without knowing a lot more about how your
system is set up. But memory is a concern in the context of
disk-backed databases (such as Xapian uses), for instance. But really
you just need to measure this and figure out where your problems, if
any, lie.

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org