[REQUEST] Search

[k0nsl]

It would be very useful to have a extensive, or even just a rudimentary, search functionality incorporated into Logbot.

[godfat]

Fulltext search? Since we're storing in redis, this probably can't be done.

[k0nsl]

I think keyword-based search functionality would be perfectly acceptable if fulltext search is impossible or unavailable in Redis.

[godfat]

Do you mean for example that we define we want to search against "keyword", and from now on we index each messages which contain "keyword"?

If so that would work, but we need to define the keyword upfront and cannot look back. (unless we try to traverse through the dataset and build the index for the past, which could be expensive)

[k0nsl]

@godfat: Yes, that sounds like it could be at least a step in the right direction. I wonder how difficult or time consuming it would be to switch from Redis to MariaDB (SQL) or even something like MongoDB? The latter options have excellent option for fulltext search functionality. But SQL might be considerably slow in comparison to Redis, and it would probably be demanding on resources too.

[godfat]

Since Logbot is still relatively simple, and Redis is also quite simple, I think it won't be difficult to migrate to another database if we just do offline migrations. I actually think that we should probably do that at some point, too, as Redis can't be the primary database to scale out. All data would need to fit into memory for Redis, which certainly won't scale if we have huge data.

I am more familiar with PostgreSQL, so that would be my choice if we're going to change database. I don't like MySQL (so MariaDB) but if people are choosing it I am not against it. I am less confident with MongoDB, but sure people could give it a try.

As for the performance, we could always optimize that later. If data still fits inside Redis, surely a relational database could still easily manage that. It might take a bit more resource to run a relational database, but I think it should be ok? It's like SQLite is surely faster given smaller dataset, but no serious service would store data in SQLite.

If Logbot is just a personal toy, then Redis is a perfect pick. Otherwise, we need to find another primary database.

[poga]

https://github.com/RedisLabsModules/RediSearch might help.

But you need Redis 4.0, which is still in RC, to work with it.

[godfat]

Impressive. We should probably try it out