[Xapian-discuss] Searching log files?

[Moazam Raja]

Well, what I'm wondering is more along the lines of how fast can
Xapian handle incoming data, index it, and make it available for
search. Is it real time, near real time, or 1-5 minute lag?

-Moazam


On Mon, Jan 18, 2010 at 3:21 AM, Olly Betts <olly at survex.com> wrote:
> On Fri, Jan 15, 2010 at 10:20:33PM -0800, Moazam Raja wrote:
>> Hi all, has anyone used Xapian/Omega to index and search large amounts
>> of (Unix) server logs?
>
> I'm not aware of anyone doing it, but then a lot of people quietly use
> Xapian without telling us what for so that doesn't mean much.
>
>> I'm looking to create a search application which will allow me to
>> index and search logs from roughly 20-100 servers but I'm not sure
>> which engine to use that will provide near real time indexing and good
>> search performance. Is this something that can be accomplished with
>> Xapian/Omega?
>
> I can't see why not.  You probably want to make the "document" a single logical
> log entry (i.e. probably a single line in the log, but it might consist of
> several lines logged at once in some log formats).
>
> Cheers,
>    Olly
>