Resource reservation framework by DaanHoogland · Pull Request #6694 · apache/cloudstack

DaanHoogland · 2022-08-31T08:14:33Z

Description

This PR addresses parallel resource allocation as a generalization of the problem and solution described in #6644. Instead of the Global lock on the resources a reservation record is created which is added in the resource check count in the ResourceLimitService/ResourceLimitManagerImpl. As a convenience a CheckedReservation is created. This is an implementation of AutoClosable and can be used as a guard in a try-with-resource fashion. The close method of the CheckedReservation wil delete the reservation record.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

in a cloud with three Management Servers

create a domain limited
set domain limits in limited (for instance user_vm == 2)
create an account luser
create a network for luser
create a first vm for this account in this network
execute a similar script as:

for i in 1 2 3 4 5
do
  for  profile in first second third
  do
    cmk -p $profile deploy virtualmachine templateid=82fb9e32-291b-11ed-89fb-1e001b000173 serviceofferingid=1009e2e0-1bd1-4142-a539-1a95a2835467 zoneid=8fb4602d-87cb-4942-ad64-140a2e53d686 domainid=9676aa4b-a218-4adb-99cd-3179b9cff867 account=luser &
  done
done

acs-robot · 2022-08-31T08:15:36Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-31T08:16:02Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-08-31T08:25:45Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6694 (SL-JID-2259)

acs-robot · 2022-08-31T09:20:27Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-31T09:21:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-08-31T09:26:28Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6694 (SL-JID-2260)

acs-robot · 2022-08-31T11:27:59Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-31T11:37:22Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6694 (SL-JID-2261)

codecov · 2022-08-31T11:40:01Z

Codecov Report

Merging #6694 (02dc903) into main (78b68fd) will decrease coverage by 0.00%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main    #6694      +/-   ##
============================================
- Coverage      5.89%    5.89%   -0.01%     
+ Complexity     3945     3943       -2     
============================================
  Files          2454     2457       +3     
  Lines        242975   243080     +105     
  Branches      38036    38041       +5     
============================================
- Hits          14323    14319       -4     
- Misses       227064   227173     +109     
  Partials       1588     1588

Impacted Files	Coverage Δ
...g/apache/cloudstack/reservation/ReservationVO.java	`0.00% <0.00%> (ø)`
...cloudstack/reservation/dao/ReservationDaoImpl.java	`0.00% <0.00%> (ø)`
...va/com/cloud/resourcelimit/CheckedReservation.java	`0.00% <0.00%> (ø)`
.../cloud/resourcelimit/ResourceLimitManagerImpl.java	`0.00% <0.00%> (ø)`
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	`0.00% <0.00%> (ø)`
...apache/cloudstack/syslog/AlertsSyslogAppender.java	`56.49% <0.00%> (-2.26%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

acs-robot · 2022-08-31T12:25:40Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-31T12:26:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-09-13T09:13:03Z

@rohityadavcloud a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2022-09-13T09:52:44Z

Trillian Build Failed (tid-4882)

DaanHoogland · 2022-09-13T14:05:32Z

@blueorangutan test centos7 vmware65u2

blueorangutan · 2022-09-13T14:06:03Z

@DaanHoogland unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

yadvr · 2022-09-13T16:33:13Z

@blueorangutan test

blueorangutan · 2022-09-13T16:34:03Z

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-09-13T16:45:24Z

Trillian Build Failed (tid-4890)

yadvr · 2022-09-13T16:50:20Z

@blueorangutan test

blueorangutan · 2022-09-13T16:51:03Z

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-09-13T17:01:47Z

Trillian Build Failed (tid-4891)

blueorangutan · 2022-09-13T20:37:16Z

Trillian test result (tid-4880)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39675 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6694-t4880-xenserver-71.zip
Smoke tests completed. 102 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2022-09-13T20:38:24Z

Trillian test result (tid-4881)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39623 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6694-t4881-kvm-centos7.zip
Smoke tests completed. 97 look OK, 5 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	157.09	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	265.69	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	141.79	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	43.75	test_vm_life_cycle.py
test_01_add_primary_storage_disabled_host	`Error`	0.59	test_primary_storage.py
test_01_primary_storage_nfs	`Error`	0.12	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	`Error`	0.21	test_primary_storage.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	26.71	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	67.63	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	39.98	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	35.91	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	120.10	test_kubernetes_clusters.py
test_02_list_snapshots_with_removed_data_store	`Error`	9.46	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	9.46	test_snapshots.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	305.81	test_hostha_kvm.py

yadvr · 2022-09-14T07:08:42Z

@blueorangutan package

borisstoyanov · 2022-09-14T11:24:26Z

@blueorangutan package

blueorangutan · 2022-09-14T11:25:04Z

@borisstoyanov a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-09-14T12:04:45Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4188

blueorangutan · 2022-09-15T07:59:47Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4195

yadvr · 2022-09-15T08:10:47Z

@blueorangutan test

blueorangutan · 2022-09-15T08:11:03Z

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-09-15T20:35:43Z

Trillian test result (tid-4918)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43363 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6694-t4918-kvm-centos7.zip
Smoke tests completed. 103 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

borisstoyanov

LGTM, manually tested

DaanHoogland added 9 commits August 29, 2022 11:20

reservation table

a958052

try-with-resource

6234f19

domain added to reservation

3076155

default constructor

87788ba

proper field naming

c0ddb01

amount getter and injection fallback

e44c1f9

conditional try with resource

938c520

integration test for parallel vm creations

69585fc

licences

7da9051

boring-cyborg Bot added component:api component:integration-test component:vpc Python Warning... Python code Ahead! labels Aug 31, 2022

add cpu and mem checks

576d514

bug hunting and sonar issues

3a5a28d

apache deleted a comment from blueorangutan Aug 31, 2022

logic error

04b14cf

apache deleted a comment from blueorangutan Sep 14, 2022

borisstoyanov approved these changes Sep 16, 2022

View reviewed changes

weizhouapache approved these changes Sep 16, 2022

View reviewed changes

DaanHoogland marked this pull request as ready for review September 16, 2022 09:55

yadvr approved these changes Sep 16, 2022

View reviewed changes

yadvr merged commit bbc1260 into apache:main Sep 16, 2022

weizhouapache mentioned this pull request Aug 24, 2023

Race condition when automatically assigning IPs to VMs #7907

Closed

Conversation

DaanHoogland commented Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

acs-robot commented Aug 31, 2022

Uh oh!

blueorangutan commented Aug 31, 2022

Uh oh!

blueorangutan commented Aug 31, 2022

Uh oh!

acs-robot commented Aug 31, 2022

Uh oh!

blueorangutan commented Aug 31, 2022

Uh oh!

blueorangutan commented Aug 31, 2022

Uh oh!

acs-robot commented Aug 31, 2022

Uh oh!

blueorangutan commented Aug 31, 2022

Uh oh!

codecov Bot commented Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

acs-robot commented Aug 31, 2022

Uh oh!

blueorangutan commented Aug 31, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

DaanHoogland commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

yadvr commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

yadvr commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

blueorangutan commented Sep 13, 2022

Uh oh!

yadvr commented Sep 14, 2022

Uh oh!

borisstoyanov commented Sep 14, 2022

Uh oh!

blueorangutan commented Sep 14, 2022

Uh oh!

blueorangutan commented Sep 14, 2022

Uh oh!

blueorangutan commented Sep 15, 2022

Uh oh!

yadvr commented Sep 15, 2022

Uh oh!

blueorangutan commented Sep 15, 2022

Uh oh!

blueorangutan commented Sep 15, 2022

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

DaanHoogland commented Aug 31, 2022 •

edited

Loading

codecov Bot commented Aug 31, 2022 •

edited

Loading