python 3.9 comptibility fixes by DaanHoogland · Pull Request #7027 · apache/cloudstack

DaanHoogland · 2022-12-27T13:35:24Z

Description

This PR...

replace encodestring for encodebytes
...

Fixes: #6962

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

codecov · 2022-12-27T15:45:43Z

Codecov Report

Merging #7027 (3c01790) into 4.18 (ca7e1ac) will increase coverage by 1.95%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               4.18    #7027      +/-   ##
============================================
+ Coverage     11.31%   13.27%   +1.95%     
- Complexity     7334     9550    +2216     
============================================
  Files          2494     2732     +238     
  Lines        246787   296181   +49394     
  Branches      38533    57367   +18834     
============================================
+ Hits          27933    39323   +11390     
- Misses       215255   251851   +36596     
- Partials       3599     5007    +1408

see 497 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

DaanHoogland · 2022-12-28T08:04:02Z

@blueorangutan package

blueorangutan · 2022-12-28T08:05:05Z

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-12-28T09:02:59Z

Packaging result: ✔️ el7 ✔️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 5102

blueorangutan · 2022-12-28T10:27:15Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5113

DaanHoogland · 2022-12-28T11:12:22Z

@blueorangutan help

blueorangutan · 2022-12-28T11:13:03Z

@DaanHoogland I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [keepEnv] [qemuEv]
Mgmt OS options: ['centos7', 'centos6', 'suse15', 'alma8', 'ubuntu18', 'ubuntu22', 'ubuntu20', 'rocky8']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-rocky8', 'kvm-alma8', 'kvm-ubuntu18', 'kvm-ubuntu20', 'kvm-ubuntu22', 'kvm-suse15', 'vmware-55u3', 'vmware-60u2', 'vmware-65u2', 'vmware-67u3', 'vmware-70u1', 'vmware-70u2', 'vmware-70u3', 'xenserver-65sp1', 'xenserver-71', 'xenserver-74', 'xcpng74', 'xcpng76', 'xcpng80', 'xcpng81', 'xcpng82']
Note: when keepEnv is passed, you need to specify mgmt server os and hypervisor or use the matrix command.
when qemuEv is passed, it will deploy KVM hyperviosr hosts with qemu-kvm-ev, else it will default to stock qemu.
Package command usage: package [all(default value),kvm,xen,vmware,hyperv,ovm] - a comma separated list can be passed with package command to bundle the required hypervisor's systemVM templates. Not passing any argument will bundle all - kvm,xen and vmware templates.

Blessed contributors for kicking Trillian test jobs: ['rohityadavcloud', 'nvazquez', 'borisstoyanov', 'DaanHoogland', 'shwstppr', 'andrijapanicsb', 'rahulbcn27', 'davidjumani', 'harikrishna-patnala', 'vladimirpetrov', 'weizhouapache', 'NuxRo', 'mlsorensen']

blueorangutan · 2022-12-29T02:46:31Z

Trillian test result (tid-5673)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 53371 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7027-t5673-kvm-rocky8.zip
Smoke tests completed. 105 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	747.43	test_kubernetes_clusters.py

yadvr

LGTM, tested test_accounts with the diff/change in cloudstackConnection.py on EL9.

sonarqubecloud · 2023-01-23T10:21:25Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

DaanHoogland · 2023-03-23T14:27:29Z

@vladimirpetrov @borisstoyanov can you review this one?

yadvr · 2023-04-19T07:43:46Z

@blueorangutan package

blueorangutan · 2023-04-19T07:44:03Z

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-04-19T09:04:10Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5936

sonarqubecloud · 2023-04-19T09:04:42Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

kiranchavala · 2023-04-20T07:11:42Z

@DaanHoogland , I did a test run with python 3.9 and was observing this exception

kiran@kiran-ThinkPad-X1-Carbon-7th:~/Desktop/github-repos/shapeblue/cloudstack$ python --version
Python 3.9.16

kiran@kiran-ThinkPad-X1-Carbon-7th:~/Desktop/github-repos/shapeblue/cloudstack$ python3 --version
Python 3.9.16

kiran@kiran-ThinkPad-X1-Carbon-7th:~/Desktop/github-repos/shapeblue/cloudstack$ python tools/marvin/marvin/deployDataCenter.py -i setup/dev/advanced.cfg

==== Log Folder Path: /tmp/MarvinLogs/DeployDataCenter__Apr_20_2023_11_27_57_3O0NYX All logs will be available here ====

=== TestClient Creation Failed===


               
kiran@kiran-ThinkPad-X1-Carbon-7th:~/Desktop/github-repos/shapeblue/cloudstack$ cat /tmp/MarvinLogs/DeployDataCenter__Apr_20_2023_11_27_57_3O0NYX/runinfo.txt 
2023-04-20 11:27:57,572 - DEBUG - Parsing Test data successful
2023-04-20 11:27:57,572 - DEBUG - Payload: {'account': 'admin', 'command': 'listUsers', 'response': 'json'}
2023-04-20 11:27:57,572 - DEBUG - ========Sending GET Cmd : listUsers=======
2023-04-20 11:27:57,598 - ERROR - Exception:['Traceback (most recent call last):\n', '  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/site-packages/requests/models.py", line 971, in json\n    return complexjson.loads(self.text, **kwargs)\n', '  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/json/__init__.py", line 346, in loads\n    return _default_decoder.decode(s)\n', '  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/json/decoder.py", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n', '  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/json/decoder.py", line 355, in raw_decode\n    raise JSONDecodeError("Expecting value", s, err.value) from None\n', 'json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/site-packages/marvin/cloudstackConnection.py", line 309, in __parseAndGetResponse\n    cmd_response.json(),\n', '  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/site-packages/requests/models.py", line 975, in json\n    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)\n', 'requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n']
Traceback (most recent call last):
  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/site-packages/marvin/cloudstackConnection.py", line 309, in __parseAndGetResponse
    cmd_response.json(),
  File "/home/kiran/.pyenv/versions/3.9.16/lib/python3.9/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

DaanHoogland · 2023-04-28T07:49:31Z

@blueorangutan test keepEnv

blueorangutan · 2023-04-28T07:50:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland · 2023-04-28T09:47:29Z

@kiranchavala , i think you have an environmental issue:

$ python --version
Python 3.9.13
$ python3 --version
Python 3.9.13
$ python tools/marvin/marvin/deployDataCenter.py -i setup/dev/advanced.cfg 

==== Log Folder Path: /tmp/MarvinLogs/DeployDataCenter__Apr_28_2023_11_42_16_GILDD2 All logs will be available here ====

==== Deploy DC Started ====

=== Data Center Settings are dumped to /tmp/MarvinLogs/DeployDataCenter__Apr_28_2023_11_42_16_GILDD2/dc_entries.obj===

====Deploy DC Successful=====

blueorangutan · 2023-04-28T20:56:53Z

Trillian test result (tid-6484)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45713 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7027-t6484-kvm-centos7.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

yadvr · 2023-05-08T07:20:11Z

I think this should be merged @kiranchavala @DaanHoogland - would unblock a lot of people on newer distros and verions using py3 > python 3.8

replace encodestring for encodebytes

3c01790

boring-cyborg Bot added component:integration-test component:marvin Python Warning... Python code Ahead! labels Dec 27, 2022

yadvr approved these changes Jan 23, 2023

View reviewed changes

DaanHoogland closed this Jan 23, 2023

DaanHoogland reopened this Jan 23, 2023

DaanHoogland marked this pull request as ready for review February 24, 2023 14:40

yadvr added this to the 4.18.1.0 milestone Apr 19, 2023

yadvr changed the base branch from main to 4.18 April 19, 2023 07:43

yadvr closed this Apr 19, 2023

yadvr reopened this Apr 19, 2023

yadvr approved these changes May 8, 2023

View reviewed changes

yadvr merged commit 897dad6 into apache:4.18 May 8, 2023

yadvr deleted the python3.9 branch May 8, 2023 07:21

yadvr mentioned this pull request May 8, 2023

Marvin and CloudStack integration tests aren't fully Python v3.9-3.11 compatible #6962

Closed

Conversation

DaanHoogland commented Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov Bot commented Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland commented Dec 28, 2022

Uh oh!

blueorangutan commented Dec 28, 2022

Uh oh!

blueorangutan commented Dec 28, 2022

Uh oh!

blueorangutan commented Dec 28, 2022

Uh oh!

DaanHoogland commented Dec 28, 2022

Uh oh!

blueorangutan commented Dec 28, 2022

Uh oh!

blueorangutan commented Dec 29, 2022

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Jan 23, 2023

Uh oh!

DaanHoogland commented Mar 23, 2023

Uh oh!

yadvr commented Apr 19, 2023

Uh oh!

blueorangutan commented Apr 19, 2023

Uh oh!

blueorangutan commented Apr 19, 2023

Uh oh!

sonarqubecloud Bot commented Apr 19, 2023

Uh oh!

kiranchavala commented Apr 20, 2023

Uh oh!

DaanHoogland commented Apr 28, 2023

Uh oh!

blueorangutan commented Apr 28, 2023

Uh oh!

DaanHoogland commented Apr 28, 2023

Uh oh!

blueorangutan commented Apr 28, 2023

Uh oh!

yadvr commented May 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DaanHoogland commented Dec 27, 2022 •

edited

Loading

codecov Bot commented Dec 27, 2022 •

edited

Loading