[Xapian-discuss] Re: DFS solution

[Andrey]

Hi James

thx for your reply.
I am planning to use xapian to index multiple sites (say 100+), and each 
site have their own xapian.Database.
For preformance/ scalability issues, I like to build it on something which 
is easy to scale in future.
I've looked in apache's Hadoop, thats exactly what I needed, but its 
writtern in java =( and I had a hard time
in mounting the HDFS(hadoop DFS) to Linux readable file system for xapian..

I dont know if this is the right track or not (DFS)...if there is something 
capable to spread the IO, highly scalable, and
readable by xapian

appreciated for any suggestion =)
thanks again for this helpful mail list
sorry for a bit off-topic
Andrey



"James Aylett" <james-xapian at tartarus.org> wrote in message 
news:20070710085636.GD7807 at tartarus.org...
> On Mon, Jul 09, 2007 at 12:20:08PM -0700, Andrey wrote:
>
>> I am currently doing some research on running xapian on Distributed
>> File Sysyem(DFS), try to find some cheap solution to scale up the
>> storage, and, without any luck...
>>
>> I am wondering anyone will give me some suggestions and point me to
>> a better direction...
>
> Andrey - why are you looking into distributed file systems? That's a
> huge amount of infrastructure! What's your actual problem? How big a
> database are you looking to support? There's a point (although I'm not
> quite sure where it is) at which moving from (say) a linux clustering
> solution towards an enterprise box (or boxes) may turn out to be more
> cost effective. It's difficult to advise you without knowing what kind
> of scale and constraints you're working with.
>
> In general though, providing a file system has normal unix semantics,
> Xapian should run over it. For Xapian you'd probably want to cluster
> at the block level I suspect (ideally you probably want to cluster at
> the btree node level, but that's impractical :-). Olly or Richard may
> be able to advise further if you do need to go down this route.
>
> J
>
> -- 
> /--------------------------------------------------------------------------\
>  James Aylett                                                  xapian.org
>  james at tartarus.org                               uncertaintydivision.org