Incorrect backup usage when VM is removed then re-assign to backup offering

[olivierlemasle]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Usage

CLOUDSTACK VERSION

4.14.1.0
4.15.0.0

CONFIGURATION

Backup framework enabled

OS / ENVIRONMENT

N/A

SUMMARY

When a VM is removed from a backup offering, then re-assigned to the same backup offering, the usage record generated for the period in which these events occur has an invalid usage value, equal to the sum of the sizes of the backups.

STEPS TO REPRODUCE

Example scenario:

1. 02:30 PM: Create a VM and assign it to a backup offering
2. 02:35 PM: Make backups of the VM (for a size of e.g. 1 GiB)
3. 03:15 PM: Remove the backup offering, with the "force" option (to delete existing backups)
4. 03:30 PM: Re-assign the VM to the backup offering
5. 03:35 PM: Make a backup of the VM (for a size of e.g. 0.8 GiB)

To reproduce the bug, if the usage aggregation range is e.g 1 Hour (usage.stats.job.aggregation.range = 60), make sure the steps 3, 4 and 5 are executed in the same hour.

EXPECTED RESULTS

Option 1: I would expect CloudStack usage server to:

• generate two usage records for the period between 03:00 PM and 04:00 PM (one for each period with a backup offering)
• use the fields raw_usage / usage_display to show the duration of usage, and the field size to show the size of the backups (that's the way storage usage records are generated for volumes, templates, ISOs and snashots)

+------------+---------------+------------+------------+----------+----------+------------------------+
|  raw_usage | usage_display | size       | start_date | end_date | usage_id | description            |
+------------+---------------+------------+------------+----------+----------+------------------------+
|        0.5 | 0.5 Hrs       | 1073741824 | 14:00:00   | 14:59:59 |       67 | Backup usage VM ID: 67 |
|       0.25 | 0.25 Hrs      | 1073741824 | 15:00:00   | 15:59:59 |       67 | Backup usage VM ID: 67 |
|        0.5 | 0.5 Hrs       |  858993460 | 15:00:00   | 15:59:59 |       67 | Backup usage VM ID: 67 |
+------------+---------------+------------+------------+----------+----------+------------------------+

Option 2: This could also be ok (even if we loose information):

+------------+---------------+------------+------------+----------+----------+------------------------+
|  raw_usage | usage_display | size       | start_date | end_date | usage_id | description            |
+------------+---------------+------------+------------+----------+----------+------------------------+
| 1073741824 | 1.0 GiB       | 1073741824 | 14:00:00   | 14:59:59 |       67 | Backup usage VM ID: 67 |
|  858993460 | 0.8 GiB       |  858993460 | 15:00:00   | 15:59:59 |       67 | Backup usage VM ID: 67 |
+------------+---------------+------------+------------+----------+----------+------------------------+

ACTUAL RESULTS

Usage records:

+------------+---------------+------------+------------+----------+----------+------------------------+
|  raw_usage | usage_display | size       | start_date | end_date | usage_id | description            |
+------------+---------------+------------+------------+----------+----------+------------------------+
| 1073741824 | 1.0 GiB       | 1073741824 | 14:00:00   | 14:59:59 |       67 | Backup usage VM ID: 67 |
| 1932735284 | 1.8 GiB       | 1932735284 | 15:00:00   | 15:59:59 |       67 | Backup usage VM ID: 67 |
+------------+---------------+------------+------------+----------+----------+------------------------+

Here, the raw_usage for the period from 3 PM to 4 PM is the sum of the backup sizes, which is irrelevant (the 1 GiB backups are deleted before the creation of the 0.8 GiB backup).

[yadvr]

Tentatively assigned to 4.15.1
cc @shwstppr @davidjumani @Spaceman1984 @Pearl1594

[Pearl1594]

@olivierlemasle Over the time-period of 15:00:00 - 15:59:59, since there was a backup taken before removing the offering from the VM and reassigning another backup offering, as per my understanding, in that one hour time frame, the cumulative size (backup size taken between 3PM to 3:15 and after the disassociation and re-assign of another backup offering) would be the usage. Subsequent backup usage will not consider the cumulative usage. cc @rhtyd Could you please confirm if this is the expected behavior?

[olivierlemasle]

Actually, the bug is to sum the backup sizes over the time frame.

I think that what I've called the Option 1 (cf above) is the best way to handle backup usages. In that option, the usage value is the duration (in hours) over the time frame. This is aligned with other storage usage types in CloudStack (volumes, ISOs, templates, snapshots). For all these usage types, the size of the volume/ISO/template/snapshot is displayed in "size", and the duration is displayed "raw_usage" / "usage_display".

The Option 2 is less accurate. E.g. it does not show that from 14:00:00 to 14:59:59, the VM was assigned to the backup offering during only half the time. However, this option is also less a breaking change.
It consists of just using as the "size" the last known size over the time frame.

I really think these options should be discussed before we implement any change. I would prefer the Option 1 but I'm interested to hear your opinion / the opinion of the community.

On a side node, you wrote:

... before removing the offering from the VM and reassigning another backup offering ...

It may not be important, but in my scenario above, the VM was re-assigned to the same backup offering. This is not a far-fetched scenario: actually, removing the backup offering and re-assigning the VM to the backup offering is the only way to attach or detach a volume to the VM.

[yadvr]

We may also need to investigate how usage records are generated for any resource where the usage is less than the aggregation interval. In general I agree a new backup record can be created for every offering change. However the usage records shouldn't club the usage but show individual record per offering (say someone switched from platinum to gold offering and we didn't bill/record usage for when they were on platinum offering).

[Pearl1594]

@olivierlemasle @rhtyd Did some tests on VM instance usage to compare its behavior against the way backup usage currently works and observed the following:
Note: The usage aggregation time interval ( usage.stats.job.aggregation.range) is set to 5 minutes

Case 1: When VM is destroyed before the aggregation time interval:

Usage record is generated for only the time period the VM was present: For example:

mysql> select * from usage_vm_instance where usage_type = 1;
+------------+---------+------------+----------------+---------+---------------------+-------------+-----------------+---------------------+---------------------+-----------+-----------+--------+
| usage_type | zone_id | account_id | vm_instance_id | vm_name | service_offering_id | template_id | hypervisor_type | start_date          | end_date            | cpu_speed | cpu_cores | memory |
+------------+---------+------------+----------------+---------+---------------------+-------------+-----------------+---------------------+---------------------+-----------+-----------+--------+
|          1 |       1 |          2 |              3 | test    |                   1 |         111 | Simulator       | 2021-05-10 04:35:23 | 2021-05-10 07:40:34 |      NULL |      NULL |   NULL |
+------------+---------+------------+----------------+---------+---------------------+-------------+-----------------+---------------------+---------------------+-----------+-----------+--------+

Corresponding usage record for that time period:
mysql> select id,zone_id,account_id,usage_display,usage_type,raw_usage,start_date,end_date from cloud_usage where usage_type = 1;

| 219 |       1 |          2 | 0.083334 Hrs  |          1 | 0.08333361148834229 | 2021-05-10 07:33:13 | 2021-05-10 07:38:13 | <-- 0.083334 Hrs corresponds to 5mins
| 225 |       1 |          2 | 0.038991 Hrs  |          1 | 0.03899139165878296 | 2021-05-10 07:38:13 | 2021-05-10 07:43:13 | <-- VM instance deleted at 7:40:34 ; 0.038992 corresponds to approx. 2mins
+-----+---------+------------+---------------+------------+---------------------+---------------------+---------------------+

Same holds true for backups as well, when a backup offering is removed, usage record is generated only for the specific duration of the time-interval (aggregation interval) - #5005 handles displaying duration of usage in the raw_usage / usage_display fields of the cloud_usage table as opposed to size

Case 2: When the VM is scaled i.e., service offering is changed, 2 records are generated

2 records are generated for the time interval when the compute offering was changed

mysql> select vm_name,service_offering_id,start_date,end_date from usage_vm_instance where usage_type = 1 and vm_instance_id = 6;
+---------+---------------------+---------------------+---------------------+
| vm_name | service_offering_id | start_date          | end_date            |
+---------+---------------------+---------------------+---------------------+
| iso1    |                   1 | 2021-05-10 08:00:54 | 2021-05-10 08:06:12 |
| iso1    |                   2 | 2021-05-10 08:06:12 | NULL                |
+---------+---------------------+---------------------+---------------------+


Corresponding usage record when service offering was changed
mysql> select usage_display,raw_usage,start_date,end_date,vm_name,offering_id from cloud_usage where usage_type = 1;
| 0.038787 Hrs  |  0.03878694400191307 | 2021-05-10 07:58:13 | 2021-05-10 08:03:13 | iso1    |           1 |
| 0.033787 Hrs  | 0.033786945044994354 | 2021-05-10 08:03:13 | 2021-05-10 08:08:13 | iso1    |           2 | <--- 0.033787 Hrs corresponds to 2mins ; @ 8:06 the Service offering was changed hence 2 records generated within the time interval of 8:03 - 8:08
| 0.049547 Hrs  |  0.04954666644334793 | 2021-05-10 08:03:13 | 2021-05-10 08:08:13 | iso1    |           1 | <---- 0.049547 Hrs corresponds to 3 mins

Case 2 holds true for backups as well, i.e., if the backup offering is changed to another one within the stipulated aggregation interval, 2 records are generated for that time period.
At 8:45 the backup offering was unassigned from a VM and a new backup offering was assigned to it:

mysql> select vm_id,backup_offering_id,created,removed from usage_backup where vm_id =6;
+-------+--------------------+---------------------+---------------------+
| vm_id | backup_offering_id | created             | removed             |
+-------+--------------------+---------------------+---------------------+
|     6 |                  1 | 2021-05-10 08:34:05 | 2021-05-10 08:45:27 | <-- Backup Offering removed
|     6 |                  2 | 2021-05-10 08:45:38 | NULL                | <-- New Backup offering assigned to the VM
+-------+--------------------+---------------------+---------------------+


mysql> select id,offering_id,usage_display,raw_usage,size,start_date,end_date from cloud_usage where usage_type = 28 and vm_instance_id = 6;
+-----+-------------+---------------+----------------------+------+---------------------+---------------------+
| id  | offering_id | usage_display | raw_usage            | size | start_date          | end_date            |
+-----+-------------+---------------+----------------------+------+---------------------+---------------------+
| 296 |           1 | 0.069064 Hrs  |  0.06906416267156601 |    0 | 2021-05-10 08:33:13 | 2021-05-10 08:38:13 |
| 303 |           1 | 0.083329 Hrs  |  0.08332943916320801 | 1000 | 2021-05-10 08:38:13 | 2021-05-10 08:43:13 |
| 310 |           1 | 0.037051 Hrs  | 0.037051115185022354 | 1000 | 2021-05-10 08:43:13 | 2021-05-10 08:48:13 | <--- usage record for backup with offering ID = 1
| 311 |           2 | 0.043229 Hrs  |  0.04322861135005951 |    0 | 2021-05-10 08:43:13 | 2021-05-10 08:48:13 | <--- usage record for backup with offering ID = 2
| 318 |           2 | 0.083334 Hrs  |  0.08333417028188705 | 1000 | 2021-05-10 08:48:13 | 2021-05-10 08:53:13 |

However, when the backup offering is removed from the VM and the VM is then re-assigned to the SAME backup offering, the usage gets aggregated, which in my opinion is the way it should behave.

[olivierlemasle]

Yes, I also think we should aggregate the usages over timerange when the VM is re-assigned to the same backup offering, IF we sum durations, not sizes (#5005). 👍
So #5005 (not reviewed yet) should indeed fix the issue.

[yadvr]

Let's discuss on the mailing list, sounds like a new behaviour/change that may require consensus on dev/users ML. cc @Pearl1594 pl start a discussion thread, thanks.

[yadvr]

This may require a more broad fix, should we move it to the next milestone @olivierlemasle @Pearl1594 ?

[Pearl1594]

I agree, targeting it for the next milestone would be better, since we are looking at also changing the way data is to be stored for backup usage records.

[yadvr]

Alright un-assigning the milestone, we can look into 4.15.2.0 or 4.16.0.0 based on when the PR is ready for review/testing.

[nvazquez]

@Pearl1594 @rhtyd @olivierlemasle tentatively assigning it to 4.16 milestone