[Xapian-discuss] Searching log files?

[Andreas Marienborg]

Depends on how you handle it. We handle a smaller data-set than log-entries (news articles), but we handle a few thousand a day I think (I haven't been directly involved in that project for some time).

We index every minute, which is fine, so long as you handle the database is changed exception that gets thrown if the database changes mid-query.

You could also keep several DBs, one for historical, one for last hour or so, and make it search both.

- andreas

On 19 Jan 2010, at 05:16, Moazam Raja wrote:

> Well, what I'm wondering is more along the lines of how fast can
> Xapian handle incoming data, index it, and make it available for
> search. Is it real time, near real time, or 1-5 minute lag?
> 
> -Moazam
> 
> 
> On Mon, Jan 18, 2010 at 3:21 AM, Olly Betts <olly at survex.com> wrote:
>> On Fri, Jan 15, 2010 at 10:20:33PM -0800, Moazam Raja wrote:
>>> Hi all, has anyone used Xapian/Omega to index and search large amounts
>>> of (Unix) server logs?
>> 
>> I'm not aware of anyone doing it, but then a lot of people quietly use
>> Xapian without telling us what for so that doesn't mean much.
>> 
>>> I'm looking to create a search application which will allow me to
>>> index and search logs from roughly 20-100 servers but I'm not sure
>>> which engine to use that will provide near real time indexing and good
>>> search performance. Is this something that can be accomplished with
>>> Xapian/Omega?
>> 
>> I can't see why not.  You probably want to make the "document" a single logical
>> log entry (i.e. probably a single line in the log, but it might consist of
>> several lines logged at once in some log formats).
>> 
>> Cheers,
>>    Olly
>> 
> 
> _______________________________________________
> Xapian-discuss mailing list
> Xapian-discuss at lists.xapian.org
> http://lists.xapian.org/mailman/listinfo/xapian-discuss