Adding showunique parameter to list templates and isos by davidjumani · Pull Request #4140 · apache/cloudstack

davidjumani · 2020-06-10T11:59:05Z

Description

Adds a new parameter showunique to listTemplate and listIsos to return only unique templates / isos across all zones

Fixes #4041

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

davidjumani · 2020-06-10T11:59:27Z

@blueorangutan package

blueorangutan · 2020-06-10T12:00:00Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-06-10T12:18:47Z

Packaging result: ✔centos7 ✔debian. JID-1332

davidjumani · 2020-06-10T12:19:35Z

@blueorangutan test

blueorangutan · 2020-06-10T12:21:06Z

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-06-10T22:26:09Z

Trillian test result (tid-1669)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34408 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4140-t1669-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	237.41	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	213.80	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	288.37	test_privategw_acl.py

yadvr · 2020-06-11T09:37:02Z

@davidjumani have you checked if this also addresses the pagination issue?

To test, you can deploy two simulator zones and register some (fake) templates and isos. The expected outcome is listTemplates and listIsos should return unique templates/isos as well as honour pagination (test using page=1 and pagesize=1, page=2... vary both page and pagesize to see expected returns are seen).

yadvr

Changes/tests asked.

davidjumani · 2020-06-12T06:08:10Z

@blueorangutan test

blueorangutan · 2020-06-12T06:09:13Z

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2020-06-12T06:11:24Z

@davidjumani you need to package before you kick tests, esp after any code changes

davidjumani · 2020-06-12T06:14:03Z

@blueorangutan package

blueorangutan · 2020-06-12T06:15:07Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-06-12T06:47:50Z

Packaging result: ✔centos7 ✔debian. JID-1362

RodrigoDLopez · 2020-06-12T16:15:23Z

@davidjumani
I tested this enhancement with Cloudmonkey and used JQ to parse the output.
My Environment have 2 Zones and some fake templates named T copied into both zones

Test 01

cloudmonkey -p acs listtemplates \
templatefilter=self \
showunique=true \
| jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

Expected result: List all my templates without duplicate entries.
Result: the output brought all templates duplicates too

name:	T1	id: 422b3235-ad7f-49fa-93ec-2c2df9140569	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T2	id: 6d224f3a-057e-4be3-bf27-ea911a77b87d	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T3	id: f302a9f9-50e6-4990-a746-3185be9fa6fb	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T4	id: a10d6bb8-dea5-45a9-8e45-4fec5456c885	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T5	id: 584317d2-009e-46eb-a5b0-ef005231bd14	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T6	id: 07994afd-ec0d-4098-ba86-c5ae1a4198df	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T7	id: 09face5a-f496-47cf-a367-1e872bb1c659	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T8	id: f6e92bc2-5dbd-4c9a-bba6-1ec4f1402667	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T9	id: c79b03a0-c9c8-431a-a4a4-a11164ea2f09	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T10	id: c3c8e05c-f8b9-44f7-91c8-b9679de95b30	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T11	id: bd552d64-7354-4e51-9c57-b7991f5d3a95	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T12	id: b21ffae9-cf69-4c3e-b204-3ea9202a662a	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T3	id: f302a9f9-50e6-4990-a746-3185be9fa6fb	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T4	id: a10d6bb8-dea5-45a9-8e45-4fec5456c885	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T5	id: 584317d2-009e-46eb-a5b0-ef005231bd14	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T6	id: 07994afd-ec0d-4098-ba86-c5ae1a4198df	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T7	id: 09face5a-f496-47cf-a367-1e872bb1c659	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T8	id: f6e92bc2-5dbd-4c9a-bba6-1ec4f1402667	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T9	id: c79b03a0-c9c8-431a-a4a4-a11164ea2f09	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T10	id: c3c8e05c-f8b9-44f7-91c8-b9679de95b30	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T11	id: bd552d64-7354-4e51-9c57-b7991f5d3a95	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T12	id: b21ffae9-cf69-4c3e-b204-3ea9202a662a	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a

Test 02

cloudmonkey -p acs listtemplates \
templatefilter=self \
pagesize=5 \
page=1 \
showunique=true \
| jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

Expected result: List only 5 templates
Result: All good, as expected

name:	T1	id: 422b3235-ad7f-49fa-93ec-2c2df9140569	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T2	id: 6d224f3a-057e-4be3-bf27-ea911a77b87d	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T3	id: f302a9f9-50e6-4990-a746-3185be9fa6fb	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T4	id: a10d6bb8-dea5-45a9-8e45-4fec5456c885	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T5	id: 584317d2-009e-46eb-a5b0-ef005231bd14	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a

Test 03

cloudmonkey -p acs listtemplates \
templatefilter=self \
pagesize=7 \
page=1 \
showunique=true \
| jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

Expected result: List only 7 templates
Result: All good, as expected


name:	T1	id: 422b3235-ad7f-49fa-93ec-2c2df9140569	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T2	id: 6d224f3a-057e-4be3-bf27-ea911a77b87d	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T3	id: f302a9f9-50e6-4990-a746-3185be9fa6fb	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T4	id: a10d6bb8-dea5-45a9-8e45-4fec5456c885	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T5	id: 584317d2-009e-46eb-a5b0-ef005231bd14	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T6	id: 07994afd-ec0d-4098-ba86-c5ae1a4198df	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T7	id: 09face5a-f496-47cf-a367-1e872bb1c659	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a

Test 04

cloudmonkey -p acs listtemplates \
templatefilter=self \
pagesize=9 \
page=1 \
showunique=true \
| jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

Expected result: List only 9 templates
Result: the output brought 16 templates and some duplicate entries

name:	T1	id: 422b3235-ad7f-49fa-93ec-2c2df9140569	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T2	id: 6d224f3a-057e-4be3-bf27-ea911a77b87d	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T3	id: f302a9f9-50e6-4990-a746-3185be9fa6fb	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T4	id: a10d6bb8-dea5-45a9-8e45-4fec5456c885	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T5	id: 584317d2-009e-46eb-a5b0-ef005231bd14	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T6	id: 07994afd-ec0d-4098-ba86-c5ae1a4198df	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T7	id: 09face5a-f496-47cf-a367-1e872bb1c659	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T8	id: f6e92bc2-5dbd-4c9a-bba6-1ec4f1402667	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T9	id: c79b03a0-c9c8-431a-a4a4-a11164ea2f09	zoneid: bfa05e73-7948-419f-a47d-29b498bf3ef9
name:	T3	id: f302a9f9-50e6-4990-a746-3185be9fa6fb	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T4	id: a10d6bb8-dea5-45a9-8e45-4fec5456c885	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T5	id: 584317d2-009e-46eb-a5b0-ef005231bd14	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T6	id: 07994afd-ec0d-4098-ba86-c5ae1a4198df	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T7	id: 09face5a-f496-47cf-a367-1e872bb1c659	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T8	id: f6e92bc2-5dbd-4c9a-bba6-1ec4f1402667	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T9	id: c79b03a0-c9c8-431a-a4a4-a11164ea2f09	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a

Whithout showunique parameter, I receive duplicate entries as well, but the pagesize was respected

Test 05

cloudmonkey -p acs listtemplates \
templatefilter=self \
pagesize=10 \
page=1 \
showunique=true \
| jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

Expected result: List only 10 templates
Result: As the test 04, I received duplicate entries, and the pagesize was not respected.
But the second page seems to be okay.

cloudmonkey -p acs listtemplates \
templatefilter=self \
pagesize=10 \
page=2 \
showunique=true \
| jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

Expected result: List the remaining templates, without duplicates.
Result: all good, bring the remaining templates, without duplicate entries

name:	T11	id: bd552d64-7354-4e51-9c57-b7991f5d3a95	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a
name:	T12	id: b21ffae9-cf69-4c3e-b204-3ea9202a662a	zoneid: 8e4d0af1-fd82-4a25-bf90-e0adfb0a4e8a

davidjumani · 2020-06-12T16:43:42Z

@RodrigoDLopez could you try it on the latest commit. I ran it and it worked as expected :

Just the flag
cmk listtemplates templatefilter=all showunique=true | jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

name:	SystemVM Template (simulator)	id: 1e097a9f-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e
name:	CentOS 5.6 (64-bit) no GUI (Simulator)	id: 1e09b0c9-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e

With pagination
cmk listtemplates templatefilter=all showunique=true page=1 pagesize=1 | jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

name:	SystemVM Template (simulator)	id: 1e097a9f-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e

cmk listtemplates templatefilter=all showunique=true page=2 pagesize=1 | jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

name:	CentOS 5.6 (64-bit) no GUI (Simulator)	id: 1e09b0c9-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e

cmk listtemplates templatefilter=all showunique=true page=1 pagesize=2 | jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

name:	SystemVM Template (simulator)	id: 1e097a9f-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e
name:	CentOS 5.6 (64-bit) no GUI (Simulator)	id: 1e09b0c9-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e

All the templates

cmk listtemplates templatefilter=all | jq -r '.template[] | "name:\t\(.name)\tid: \(.id)\tzoneid: \(.zoneid)"'

name:	SystemVM Template (simulator)	id: 1e097a9f-aafb-11ea-a5c3-50eb71576f75	zoneid: 7eea393a-2738-46b2-a85e-12c25ccf5913
name:	SystemVM Template (simulator)	id: 1e097a9f-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e
name:	CentOS 5.6 (64-bit) no GUI (Simulator)	id: 1e09b0c9-aafb-11ea-a5c3-50eb71576f75	zoneid: 7eea393a-2738-46b2-a85e-12c25ccf5913
name:	CentOS 5.6 (64-bit) no GUI (Simulator)	id: 1e09b0c9-aafb-11ea-a5c3-50eb71576f75	zoneid: a82d7a28-e405-4456-b095-62c635951b0e

RodrigoDLopez · 2020-06-12T16:58:22Z

@davidjumani I'm already using the last commit.
I will retest. and update you

RodrigoDLopez · 2020-06-12T20:39:22Z

Hi @davidjumani
I did those tests again.
Everything as expected.

My first test was fail, because those templates was not in the right state, caused by an error into my environment. My misstake.

LGTM based on manual tests

yadvr · 2020-06-12T22:16:29Z

@blueorangutan test

blueorangutan · 2020-06-12T22:17:16Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2020-06-13T22:29:27Z

@blueorangutan test

davidjumani · 2020-06-17T04:45:09Z

@GabrielBrascher @rhtyd comments addressed!
@blueorangutan package

blueorangutan · 2020-06-17T04:46:30Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-06-17T05:03:42Z

Packaging result: ✖centos7 ✖debian. JID-1404

yadvr · 2020-06-17T05:05:04Z

@davidjumani can you merge latest master and try packaging again?

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

davidjumani · 2020-06-17T05:20:14Z

@blueorangutan package

blueorangutan · 2020-06-17T05:21:29Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-06-17T05:32:07Z

Packaging result: ✖centos7 ✖debian. JID-1406

yadvr · 2020-06-17T05:32:13Z

@blueorangutan package

blueorangutan · 2020-06-17T05:33:29Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-06-17T06:04:35Z

Packaging result: ✔centos7 ✔debian. JID-1409

yadvr · 2020-06-17T06:39:26Z

@blueorangutan test

blueorangutan · 2020-06-17T06:40:30Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-06-17T21:28:36Z

Trillian test result (tid-1754)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36757 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4140-t1754-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	252.54	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	240.65	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	310.45	test_privategw_acl.py

svenvogel added component:api type:enhancement labels Jun 10, 2020

yadvr added this to the 4.13.2.0 milestone Jun 11, 2020

yadvr added the component:UI label Jun 11, 2020

yadvr requested changes Jun 11, 2020

View reviewed changes

yadvr requested a review from DaanHoogland June 12, 2020 06:53

yadvr assigned GabrielBrascher, svenvogel and weizhouapache Jun 12, 2020

apache deleted a comment from blueorangutan Jun 13, 2020

davidjumani force-pushed the adding-unique-template-flag branch from 9c71dae to b271cb3 Compare June 17, 2020 04:42

yadvr approved these changes Jun 17, 2020

View reviewed changes

davidjumani force-pushed the adding-unique-template-flag branch from b271cb3 to 35b5bfa Compare June 17, 2020 05:07

davidjumani and others added 10 commits June 17, 2020 10:47

Adding showunique parameter to list templates and isos

b250145

Update ListIsosCmd.java

775e487

Update ListTemplatesCmd.java

77ba5a8

Update ListIsosCmd.java

e9f48ff

Fixing count bug

1997abf

Simplifying

8ccdfd1

Update ListTemplatesCmd.java

94963df

Update ListIsosCmd.java

b83a4c4

fix trailing white space - remind me to not use Github editor

83bd417

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Refactoring for showUnique templates

28137f6

davidjumani force-pushed the adding-unique-template-flag branch from 35b5bfa to 28137f6 Compare June 17, 2020 05:18

yadvr merged commit e9f59e2 into apache:4.13 Jun 18, 2020

davidjumani mentioned this pull request Jul 29, 2020

Templates are showing twice apache/cloudstack-primate#561

Closed

davidjumani deleted the adding-unique-template-flag branch October 27, 2020 14:49

Conversation

davidjumani commented Jun 10, 2020

Description

Types of changes

Uh oh!

davidjumani commented Jun 10, 2020

Uh oh!

blueorangutan commented Jun 10, 2020

Uh oh!

blueorangutan commented Jun 10, 2020

Uh oh!

davidjumani commented Jun 10, 2020

Uh oh!

blueorangutan commented Jun 10, 2020

Uh oh!

blueorangutan commented Jun 10, 2020

Uh oh!

yadvr commented Jun 11, 2020

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

davidjumani commented Jun 12, 2020

Uh oh!

blueorangutan commented Jun 12, 2020

Uh oh!

yadvr commented Jun 12, 2020

Uh oh!

davidjumani commented Jun 12, 2020

Uh oh!

blueorangutan commented Jun 12, 2020

Uh oh!

blueorangutan commented Jun 12, 2020

Uh oh!

RodrigoDLopez commented Jun 12, 2020

Uh oh!

davidjumani commented Jun 12, 2020

Uh oh!

RodrigoDLopez commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RodrigoDLopez commented Jun 12, 2020

Uh oh!

yadvr commented Jun 12, 2020

Uh oh!

blueorangutan commented Jun 12, 2020

Uh oh!

yadvr commented Jun 13, 2020

Uh oh!

davidjumani commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

yadvr commented Jun 17, 2020

Uh oh!

davidjumani commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

yadvr commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

yadvr commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

blueorangutan commented Jun 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

RodrigoDLopez commented Jun 12, 2020 •

edited

Loading

davidjumani commented Jun 17, 2020 •

edited

Loading