systemvmtemplate: new template for 4.15.1 by yadvr · Pull Request #4793 · apache/cloudstack

yadvr · 2021-03-11T07:57:11Z

Update new systemvmtemplate for 4.15.1.0; synced:
http://download.cloudstack.org/systemvm/4.15/

A new template is necessary due to many security fixes over the last year, the 4.15.0 systemvmtemplate was created about a year ago.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2021-03-11T07:58:22Z

@blueorangutan package

blueorangutan · 2021-03-11T07:59:02Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

weizhouapache · 2021-03-11T08:19:22Z

+
+        for (final Map.Entry<Hypervisor.HypervisorType, String> hypervisorAndTemplateName : NewTemplateNameList.entrySet()) {
+            LOG.debug("Updating " + hypervisorAndTemplateName.getKey() + " System Vms");
+            try (PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1")) {


@rhtyd
if an user (not root admin) register a template with the same name, will it be a security issue ?

Good catch - we've been copying the same code/logic; however I think it may not be a big issue as it picks the last template that was added before the upgrade, even if user may have added a template, the admin likely would have added the most recent or last template. We may be saved by the order by id desc limit 1. Or, do you want me to add a check for account_idj = 2?

@rhtyd
is it possible that cloudstack user uses other admin account than root admin (account_id=2) ?

would it be better to use this ?

select id from `cloud`.`vm_template` where name = ? and removed is null and account_id in (select id from account where role_id=1 and removed is null) order by id desc limit 1;

Thanks, I'll test and include it

... but what is the account is of admin role type but not role ID 1?

what would be the issue, @rhtyd? except that the account used will fail registering systemvm templates?

Looks like we can use Wei's query with type=1 (instead of role_id); type1 is root admin role.

blueorangutan · 2021-03-11T08:21:23Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 83

yadvr · 2021-03-11T08:29:40Z

@blueorangutan test matrix

blueorangutan · 2021-03-11T08:30:02Z

@rhtyd a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests [S]

blueorangutan · 2021-03-11T18:52:25Z

[S] Trillian test result (tid-80)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34834 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t80-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	66.15	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	51.16	test_vm_life_cycle.py

DaanHoogland · 2021-03-12T09:57:41Z

      "net_device": "virtio-net",
-      "iso_url": "https://download.cloudstack.org/systemvm/debian/debian-10.5.0-amd64-netinst.iso",
-      "iso_checksum": "0a6aee1d9aafc1ed095105c052f9fdd65ed00ea9274188c9cd0072c8e6838ab40e246d45a1e6956d74ef1b04a1fc042151762f25412e9ff0cbf49418eef7992e",
+      "iso_url": "https://cdimage.debian.org/debian-cd/10.8.0/amd64/iso-cd/debian-10.8.0-amd64-netinst.iso",


debian tends to move their ISOs, is this the best way, or should we keep a copy as we used to?

It's either this or someone manually copying/updating the template on download.cloudstack.org and then using it here. One option is to use the archive links, but Debian doesn't put the latest version/release isos there https://cdimage.debian.org/mirror/cdimage/archive/

blueorangutan · 2021-03-12T18:58:31Z

[S] Trillian test result (tid-95)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40585 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t95-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 83 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_10_traceroute_in_vr	`Failure`	61.43	test_diagnostics.py
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	785.37	test_kubernetes_clusters.py
test_03_migrate_detached_volume	`Error`	96.19	test_vm_life_cycle.py
ContextSuite context=TestVAppsVM>:setup	`Error`	44.45	test_vm_life_cycle.py

blueorangutan · 2021-03-12T20:27:10Z

[S] Trillian test result (tid-96)
Environment: xcpng81 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45763 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t96-xcpng81.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	4.15	test_scale_vm.py
test_01_volume_usage	`Error`	96.50	test_usage.py

blueorangutan · 2021-03-15T17:34:58Z

[S] Trillian test result (tid-134)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37627 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t134-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	13.41	test_scale_vm.py
test_02_VPC_default_routes	`Failure`	196.50	test_vpc_router_nics.py

blueorangutan · 2021-03-16T14:05:55Z

[S] Trillian test result (tid-111)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 338722 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t111-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_affinity_groups_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_affinity_groups.py
Intermittent failure detected: /marvin/tests/smoke/test_async_job.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_extra_config_data.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vms_with_varied_deploymentplanners.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_with_userdata.py
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_domain_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_domain_vpc_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_network_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_portforwardingrules.py
Intermittent failure detected: /marvin/tests/smoke/test_primary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_deployment_planner.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 37 look OK, 49 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

yadvr · 2021-03-17T12:14:15Z

@blueorangutan package

blueorangutan · 2021-03-17T12:15:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

blueorangutan · 2021-03-17T12:51:03Z

Packaging result: ✔️ centos7 ✖️ centos8 ✖️ debian. SL-JID 157

yadvr · 2021-03-17T12:57:11Z

@blueorangutan test matrix

blueorangutan · 2021-03-17T12:58:03Z

@rhtyd a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests [S]

shwstppr · 2021-03-18T05:39:58Z

@blueorangutan package

blueorangutan · 2021-03-18T05:41:03Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

davidjumani · 2021-03-19T10:21:46Z

@blueorangutan package

blueorangutan · 2021-03-19T10:24:06Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

blueorangutan · 2021-03-19T10:25:05Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

blueorangutan · 2021-03-19T11:09:38Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 182

blueorangutan · 2021-03-19T11:09:39Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 185

yadvr · 2021-03-22T06:48:13Z

@blueorangutan package

blueorangutan · 2021-03-22T06:49:04Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-03-22T08:10:11Z

Packaging result: ✖️ centos7 ✖️ centos8 ✔️ debian. SL-JID 201

yadvr · 2021-03-23T08:11:55Z

@Pearl1594 to carry forward with the PR

…-template-fix

Pearl1594 · 2021-03-23T11:58:12Z

@blueorangutan package

blueorangutan · 2021-03-23T11:59:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-03-23T12:24:22Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 207

Pearl1594 · 2021-03-23T12:25:40Z

@blueorangutan test matrix

blueorangutan · 2021-03-23T12:26:02Z

@Pearl1594 a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2021-03-23T22:59:27Z

Trillian test result (tid-225)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35727 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t225-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	94.74	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	47.01	test_vm_life_cycle.py

blueorangutan · 2021-03-23T22:59:51Z

Trillian test result (tid-224)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36554 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t224-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	13.30	test_scale_vm.py

blueorangutan · 2021-03-24T01:32:28Z

Trillian test result (tid-226)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44682 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4793-t226-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	818.78	test_kubernetes_clusters.py
ContextSuite context=TestVAppsVM>:setup	`Error`	43.38	test_vm_life_cycle.py

yadvr · 2021-03-24T06:58:11Z

+
+        for (final Map.Entry<Hypervisor.HypervisorType, String> hypervisorAndTemplateName : NewTemplateNameList.entrySet()) {
+            LOG.debug("Updating " + hypervisorAndTemplateName.getKey() + " System Vms");
+            try (PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null and account_id in (select id from account where type = 1 and removed is NULL) order by id desc limit 1")) {


@weizhouapache @DaanHoogland the security issue fixed here by checking the template belongs to an account with type=1 (root admin)

yadvr · 2021-03-24T07:02:04Z

Tests LGTM, thanks for fixing the issue and kicking tests @Pearl1594

Pearl1594

LGTM.
Verified the security issue by registering a template with name "systemvm-vmware-4.15.1" as a user post root admin registering the template and then initiated upgrade. It picks up the right systemVM template, i.e., the one registered by the root admin and updates the required fields e.g. type

yadvr · 2021-03-24T11:38:27Z

Thanks Pearl for testing.

systemvmtemplate: new template for 4.15.1

bf75c86

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr added the component:systemvm-template label Mar 11, 2021

yadvr added this to the 4.15.1.0 milestone Mar 11, 2021

yadvr requested review from davidjumani and shwstppr March 11, 2021 07:57

weizhouapache reviewed Mar 11, 2021

View reviewed changes

DaanHoogland reviewed Mar 12, 2021

View reviewed changes

yadvr closed this Mar 22, 2021

yadvr reopened this Mar 22, 2021

yadvr assigned Pearl1594 Mar 23, 2021

Pearl Dsilva added 2 commits March 23, 2021 14:40

Merge branch '4.15' of https://github.com/apache/cloudstack into 4151…

f603980

…-template-fix

Find systemvm template registered by root admin

01fc4e3

yadvr requested review from DaanHoogland and weizhouapache March 24, 2021 06:57

yadvr commented Mar 24, 2021

View reviewed changes

Pearl1594 approved these changes Mar 24, 2021

View reviewed changes

shwstppr approved these changes Mar 24, 2021

View reviewed changes

yadvr merged commit 9b1d1e6 into apache:4.15 Mar 24, 2021

Conversation

yadvr commented Mar 11, 2021

Uh oh!

yadvr commented Mar 11, 2021

Uh oh!

blueorangutan commented Mar 11, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 11, 2021

Uh oh!

yadvr commented Mar 11, 2021

Uh oh!

blueorangutan commented Mar 11, 2021

Uh oh!

blueorangutan commented Mar 11, 2021

Uh oh!

DaanHoogland Mar 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 12, 2021

Uh oh!

blueorangutan commented Mar 12, 2021

Uh oh!

blueorangutan commented Mar 15, 2021

Uh oh!

blueorangutan commented Mar 16, 2021

Uh oh!

yadvr commented Mar 17, 2021

Uh oh!

blueorangutan commented Mar 17, 2021

Uh oh!

blueorangutan commented Mar 17, 2021

Uh oh!

yadvr commented Mar 17, 2021

Uh oh!

blueorangutan commented Mar 17, 2021

Uh oh!

shwstppr commented Mar 18, 2021

Uh oh!

blueorangutan commented Mar 18, 2021

Uh oh!

davidjumani commented Mar 19, 2021

Uh oh!

blueorangutan commented Mar 19, 2021

Uh oh!

blueorangutan commented Mar 19, 2021

Uh oh!

blueorangutan commented Mar 19, 2021

Uh oh!

blueorangutan commented Mar 19, 2021

Uh oh!

yadvr commented Mar 22, 2021

Uh oh!

blueorangutan commented Mar 22, 2021

Uh oh!

blueorangutan commented Mar 22, 2021

Uh oh!

yadvr commented Mar 23, 2021

Uh oh!

Pearl1594 commented Mar 23, 2021

Uh oh!

blueorangutan commented Mar 23, 2021

Uh oh!

blueorangutan commented Mar 23, 2021

DaanHoogland Mar 12, 2021 •

edited

Loading