Akumuli advice

[arrowwood49]

I'm evaluating Akumuli to replace existing time-series implementation which is based on a relational database.

We have around 20 different stats of an object which are modeled as 20 columns in a table. There are total 50,000 objects and each object sends 20 stats every 20 seconds.

20 seconds stats are rolled up to -> 5 minutes -> 1 hour -> 1 day.
20 seconds stats and each roll ups are stored into a separate table. The followings are the retention periods:
20 seconds stats - 1 hour
5 min roll-up stats - 1 day
1 hour roll-up stats - 30 days
1 day roll-up stats - 2 years-

All 20 stats for a given object can be queried either from 20 seconds or from one of the roll-up table using time range as filter.

Based on Akumui data model, each stat of an object will be a separate series and object's uuid and its location will be used as tags:

<object_type>.<stats name> <timestamp> <value> object_id=13cf0ad8-f022-4ca0-8ec7-b0b32133f1ea location_id=L1

What is the recommendations for disk space, memory and cpu?
What are the backup and recovery options?
Do you a general advice on what to consider when migrating from relational database?

Thanks in advance.

[Lazin]

First of all, Akumuli doesn't support rollup-based retention. You can't set up different retention periods for different time-series. Also, Akumuli doesn't implement rollups the same way Grafana does. It won't pre-generate that 5-minute, 1-hour, and 1-day rollups as a separate time-series. Instead, it has group-aggregate query that can be used to get data with the step you need on demand. Of cause you can generate rollups in your application and write them but 'group-aggregate' is still a recommended way.

The back of the envelope calculations for the capacity:

• cardinality - 50 000*20 - 1M unique time-series;
• write rate - 50 000 / sec;
• space depends on data a lot, 1 576 800 000 000 data-points per year could be stored using less than 1.5TB of disk space if the values are small integers. If the data-points are floats or large integers then it could take anywhere from >1.5TB (low variation) up to 12TB (mostly random data). If the data has a lot of duplicates this can take way less than 1.5TB. But it's better to experiment and load some data to see how compressible it is.

[arrowwood49]

Of cause you can generate rollups in your application and write them but 'group-aggregate' is still a recommended way.

We are using rollups into separate table to keep disk space usage low. Looks like it will be difficult to do the same in Akumuli even if I create rollups in application since Akumuli doesn't support different retention periods for different time-series. Is there a way to implement custom deletion/retention in application for different time-series?

I'll run some test with real data to find the disk space usage from Akumuli.

If we use Akumuli group-aggregate instead of rollups then what is the memory requirement for query? Assuming user can run only 5 max group-aggregate queries concurrently.
What is the memory requirement to support 50,000 writes /sec?

[Lazin]

It is possible to use multiple akumulid instances with different configs. One for raw data with 20s step and short retention and another one for pre-aggregated data. Akumuli uses space based retention for operational simplicity. Because of that it's not possible to use different retention policies for different series.

The 50000 writes/sec rate is quite small TBH. But the total cardinality is 1M and this means that you will need around 8-10GB of RAM. The query doesn't require a lot of RAM if you query series by one. And if you query many series at once you can use "order-by": "series" to minimize memory usage.

[arrowwood49]

Thanks @Lazin for the response.

The 50000 writes/sec rate is quite small TBH. But the total cardinality is 1M and this means that you will need around 8-10GB of RAM. The query doesn't require a lot of RAM if you query series by one. And if you query many series at once you can use "order-by": "series" to minimize memory usage.

Could you explain how you estimated 8-10 GB of RAM for 1M series? Will it help to reduce memory footprint if data to Akumuli is added in chunks instead of all 1M series together?

Regarding query, I need it query using object-id tag to retrieve at least 10-15 stats (time series) for a given time range which can be last 1 hour. Other group-aggregate queries to simulate current roll-ups using object-id tag.

I'll take a look at "order-by": "series".

[Lazin]

Could you explain how you estimated 8-10 GB of RAM for 1M series? Will it help to reduce memory footprint if data to Akumuli is added in chunks instead of all 1M series together?

Memory use depend on cardinality. To handle every time-series Akumuli needs 1-10KB of RAM (depending on the size of the series).

Regarding query, I need it query using object-id tag to retrieve at least 10-15 stats (time series) for a given time range which can be last 1 hour. Other group-aggregate queries to simulate current roll-ups using object-id tag.

This should be a light weight query. To see high memory use by queries you need to query tens/hundreds thousands of metrics. It's because query processor caches 4KB of data for every time-series that is involved in the query so when you have a lot of them you will see higher memory use. Also, if query returns data ordered by time the memory use is also higher because it needs to join many series together. 15 stats is not a big deal.

[arrowwood49]

Thanks @Lazin for the explanation. I'll test queries to check the memory usage.