Datera storage plugin by manojkverma · Pull Request #3470 · apache/cloudstack

manojkverma · 2019-07-05T00:18:54Z

Features:

Zone-wide and cluster-wide primary storage support
VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
Rapid storage-native snapshot
Multiple managed primary storages can be created with a single Datera cluster to provide better management of
- Total provisioned capacity
- Default storage QoS values
- Replica size ( 1 to 5 )
- IP pool assignment for iSCSI target
- Volume Placement ( hybrid, single_flash, all_flash )
Volume snapshot to VM template
Volume to VM template
Volume size increase using service policy
Volume QoS change using service policy
Enabled KVM support
New Datera app_instance name format to include ACS volume name
VM live migration

manojkverma · 2019-07-05T00:53:21Z

This is same plugin from https://github.com/jcsdatera/cloudstack
since @jcsdatera is on vacation, I made a few fixes for him.

Features: 1. Zone-wide and cluster-wide primary storage support 2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume. 3. Rapid storage-native snapshot 4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of - Total provisioned capacity - Default storage QoS values - Replica size ( 1 to 5 ) - IP pool assignment for iSCSI target - Volume Placement ( hybrid, single_flash, all_flash ) 5. Volume snapshot to VM template 6. Volume to VM template 7. Volume size increase using service policy 8. Volume QoS change using service policy 9. Enabled KVM support 10. New Datera app_instance name format to include ACS volume name 11. VM live migration

yadvr · 2019-07-05T08:42:32Z

@blueorangutan package

blueorangutan · 2019-07-05T08:42:45Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-07-05T09:04:12Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-90

borisstoyanov · 2019-07-05T09:14:19Z

@blueorangutan test

blueorangutan · 2019-07-05T09:14:45Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov

Can you add integration tests and unit tests please @manojkverma

blueorangutan · 2019-07-05T17:18:31Z

Trillian test result (tid-105)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27179 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3470-t105-kvm-centos7.zip
Smoke tests completed. 71 look OK, 1 have error(s)
Only failed tests results shown below:

yadvr · 2019-07-05T17:21:55Z

Tests LGTM, we don't have datera infra to test the plugin. @swill @manojkverma can you share your code review and test details if any. As @borisstoyanov mentions, any tests are welcome.

manojkverma · 2019-07-09T00:49:20Z

Tests LGTM, we don't have datera infra to test the plugin. @swill @manojkverma can you share your code review and test details if any. As @borisstoyanov mentions, any tests are welcome.

Thanks @rhtyd, I am sorry, I am new to open source development model, so thanks for bearing with me as I am working on this. @jcsdatera had python based integration test cases, which he ran against datera storage hardware. I am looking into getting java unit test to work with mock objects.

yadvr · 2019-07-09T07:15:40Z

@jcsdatera can you work with @swill on top of this PR? Also, what we're looking for are list of tests results (see other smoketests for example). Ideally, if you'd prefer you can add marvin smoketests to test/integration/plugins/ of the source tree.

swill · 2019-07-10T14:01:35Z

@manojkverma & @jcsdatera, if you can post the results of the python integration tests which you ran against the datara storage hardware, that would go a long way towards helping the community understand the current state of the plugin. I agree with @rhtyd that adding marvin smoketests to test/integration/plugins/ is the ideal way to contribute the tests upstream.

yadvr · 2019-07-14T11:59:32Z

We're getting closer to 4.13.0.0 freeze date, ping @manojkverma @jcsdatera

shwstppr · 2019-07-18T05:54:50Z

@manojkverma @jcsdatera any update on this?

jcsdatera · 2019-07-18T09:33:34Z

We are still working on this and very close. Thank you

On Wed, Jul 17, 2019 at 10:55 PM Abhishek Kumar ***@***.***> wrote: @manojkverma <https://github.com/manojkverma> @jcsdatera <https://github.com/jcsdatera> any update on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3470?email_source=notifications&email_token=AE2HBTM4Y4ZHXMIRFFSC4ZTQAAATTA5CNFSM4H6CHRBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2HM4FQ#issuecomment-512675350>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE2HBTLO4GYR7ILJCG7VSWLQAAATTANCNFSM4H6CHRBA> .

-- James from iPhone

jcsdatera · 2019-07-23T04:42:14Z

Added integrated tests with the following test result:

 
Attach a volume to a stopped virtual machine, then start VM ... === TestName: test_01_attach_new_volume_to_stopped_VM | Status : SUCCESS ===
ok
Attach, detach, and attach volume to a running VM ... === TestName: test_02_attach_detach_attach_volume | Status : SUCCESS ===
ok
Attach volume to running VM, then reboot. ... === TestName: test_03_attached_volume_reboot_VM | Status : SUCCESS ===
ok

----------------------------------------------------------------------
XML: /data/git/cs1/cloudstack/test/integration/nosetests.xml
----------------------------------------------------------------------
Ran 3 tests in 885.349s

OK

yadvr · 2019-07-23T07:16:36Z

@blueorangutan package

blueorangutan · 2019-07-23T07:17:08Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-07-23T07:37:28Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-172

yadvr · 2019-07-23T07:37:43Z

@blueorangutan test

blueorangutan · 2019-07-23T07:38:07Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-07-23T18:14:03Z

Trillian test result (tid-199)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36279 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3470-t199-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	207.86	test_internal_lb.py
test_05_rvpc_multi_tiers	`Failure`	419.43	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	449.91	test_vpc_redundant.py

yadvr · 2019-07-23T18:19:11Z

@blueorangutan test

blueorangutan · 2019-07-23T18:20:10Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

jcsdatera · 2019-07-23T18:26:51Z

Ping @rhtyd, @swill
I noticed that this PR has been moved from milestone 4.13.0.0 to 4.14.0.0
Actually, we spent most of our time passing the tests until we apply the patch of #3497

If 4.13.0 deadline has passed, is it possible to mark this to the next 4.13.1 instead?

blueorangutan · 2019-07-24T03:12:46Z

Trillian test result (tid-209)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30048 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3470-t209-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_cancel_host_maintenace_with_no_migration_jobs	`Failure`	1.17	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	3.70	test_host_maintenance.py
test_hostha_enable_ha_when_host_disabled	`Error`	2.67	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	301.85	test_hostha_kvm.py

yadvr · 2019-07-24T06:37:03Z

I've re-added this to 4.13.0.0, need another lgtm/review. @swill ?

yadvr · 2019-07-25T07:01:10Z

@swill are you lgtm on it? We can merge, the regression tests show the plugin is isolated enough and doesn't bring in any regression. cc @borisstoyanov @shwstppr @anuragaw

borisstoyanov

LGTM, based on marvin tests results

[WIP] Add Datera storage plugin

f53036f

yadvr added this to the 4.13.0.0 milestone Jul 5, 2019

yadvr added the type:new-feature label Jul 5, 2019

add plugin to client pom.xml

7ec478c

borisstoyanov reviewed Jul 5, 2019

View reviewed changes

yadvr approved these changes Jul 5, 2019

View reviewed changes

manojkverma changed the title ~~[WIP] Add Datera storage plugin~~ Datera storage plugin Jul 8, 2019

yadvr modified the milestones: 4.13.0.0, 4.14.0.0 Jul 22, 2019

Add Datera volume plugin integration tests

c338a0d

yadvr modified the milestones: 4.14.0.0, 4.13.0.0 Jul 24, 2019

yadvr approved these changes Jul 24, 2019

View reviewed changes

yadvr requested a review from swill July 24, 2019 06:59

borisstoyanov approved these changes Jul 25, 2019

View reviewed changes

yadvr merged commit e3d70b7 into apache:master Jul 25, 2019

Conversation

manojkverma commented Jul 5, 2019

Uh oh!

manojkverma commented Jul 5, 2019

Uh oh!

yadvr commented Jul 5, 2019

Uh oh!

blueorangutan commented Jul 5, 2019

Uh oh!

blueorangutan commented Jul 5, 2019

Uh oh!

borisstoyanov commented Jul 5, 2019

Uh oh!

blueorangutan commented Jul 5, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jul 5, 2019

Uh oh!

yadvr commented Jul 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manojkverma commented Jul 9, 2019

Uh oh!

yadvr commented Jul 9, 2019

Uh oh!

swill commented Jul 10, 2019

Uh oh!

yadvr commented Jul 14, 2019

Uh oh!

shwstppr commented Jul 18, 2019

Uh oh!

jcsdatera commented Jul 18, 2019 via email

Uh oh!

jcsdatera commented Jul 23, 2019

Uh oh!

yadvr commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

yadvr commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

yadvr commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

jcsdatera commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 24, 2019

Uh oh!

yadvr commented Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Jul 25, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

yadvr commented Jul 5, 2019 •

edited

Loading

yadvr commented Jul 24, 2019 •

edited

Loading