fix(cloudstack): get domain name and vr information from network manager leases#6829
fix(cloudstack): get domain name and vr information from network manager leases#6829ani-sinha wants to merge 4 commits intocanonical:mainfrom
Conversation
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com>
d9e4324 to
0056499
Compare
|
@holmanb @TheRealFalcon please review |
blackboxsw
left a comment
There was a problem hiding this comment.
Thanks for this proposal @ani-sinha. I think it is almost there, but needs a bit more support awareness of multi-NIC environments with two active interfacees and test coverage in test_dhcp.py.
| line = line.split(":", 1)[-1].strip() | ||
| content.append(line) | ||
|
|
||
| return dict(configobj.ConfigObj(content, list_values=False)) |
There was a problem hiding this comment.
While this processing works for single interface systems, it falls over on systems that have multiple devices because we have output like the following containing non-unique keys and each separate device is only separated by a blank newline
DHCP4.OPTION[1]: dhcp_client_identifier = 01:6c:ff:dd:aa>
DHCP4.OPTION[2]: dhcp_lease_time = 86400
DHCP4.OPTION[3]: dhcp_server_identifier = 192.168.1.1
DHCP4.OPTION[4]: domain_name_servers = 192.168.1.1
DHCP4.OPTION[5]: expiry = 1776277372
DHCP4.OPTION[6]: ip_address = 192.168.1.44
DHCP4.OPTION[7]: requested_broadcast_address = 1
DHCP4.OPTION[8]: requested_domain_name = 1
DHCP4.OPTION[9]: requested_domain_name_servers = 1
DHCP4.OPTION[10]: requested_domain_search = 1
DHCP4.OPTION[11]: requested_host_name = 1
DHCP4.OPTION[12]: requested_interface_mtu = 1
DHCP4.OPTION[13]: requested_ms_classless_static_routes = 1
DHCP4.OPTION[14]: requested_nis_domain = 1
DHCP4.OPTION[15]: requested_nis_servers = 1
DHCP4.OPTION[16]: requested_ntp_servers = 1
DHCP4.OPTION[17]: requested_rfc3442_classless_static_rout>
DHCP4.OPTION[18]: requested_root_path = 1
DHCP4.OPTION[19]: requested_routers = 1
DHCP4.OPTION[20]: requested_static_routes = 1
DHCP4.OPTION[21]: requested_subnet_mask = 1
DHCP4.OPTION[22]: requested_time_offset = 1
DHCP4.OPTION[23]: requested_wpad = 1
DHCP4.OPTION[24]: routers = 192.168.1.1
DHCP4.OPTION[25]: subnet_mask = 255.255.255.0
DHCP4.OPTION[1]: dhcp_client_identifier = 01:38:cc:nn:aa>
DHCP4.OPTION[2]: dhcp_lease_time = 86400
DHCP4.OPTION[3]: dhcp_server_identifier = 192.168.1.1
DHCP4.OPTION[4]: domain_name_servers = 192.168.1.1
DHCP4.OPTION[5]: expiry = 1776255526
DHCP4.OPTION[6]: ip_address = 192.168.1.17
DHCP4.OPTION[7]: requested_broadcast_address = 1
DHCP4.OPTION[8]: requested_domain_name = 1
DHCP4.OPTION[9]: requested_domain_name_servers = 1
DHCP4.OPTION[10]: requested_domain_search = 1
DHCP4.OPTION[11]: requested_host_name = 1
DHCP4.OPTION[12]: requested_interface_mtu = 1
DHCP4.OPTION[13]: requested_ms_classless_static_routes = 1
DHCP4.OPTION[14]: requested_nis_domain = 1
DHCP4.OPTION[15]: requested_nis_servers = 1
DHCP4.OPTION[16]: requested_ntp_servers = 1
DHCP4.OPTION[17]: requested_rfc3442_classless_static_rout>
DHCP4.OPTION[18]: requested_root_path = 1
DHCP4.OPTION[19]: requested_routers = 1
DHCP4.OPTION[20]: requested_static_routes = 1
DHCP4.OPTION[21]: requested_subnet_mask = 1
DHCP4.OPTION[22]: requested_time_offset = 1
DHCP4.OPTION[23]: requested_wpad = 1
DHCP4.OPTION[24]: routers = 192.168.1.1
DHCP4.OPTION[25]: subnet_mask = 255.255.255.0
There was a problem hiding this comment.
This results in ConfgObj failures due to being unable to process the content:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./cloud-init/cloudinit/net/dhcp.py", line 187, in network_manager_load_leases
return dict(configobj.ConfigObj(content, list_values=False))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/configobj/__init__.py", line 1228, in __init__
self._load(infile, configspec)
File "/usr/lib/python3/dist-packages/configobj/__init__.py", line 1317, in _load
raise error
configobj.ConfigObjError: Parsing failed with several errors.
First error at line 27.
There was a problem hiding this comment.
Pull request overview
This PR extends CloudStack hostname/FQDN discovery by adding NetworkManager (nmcli) DHCP lease parsing helpers and using them as an additional fallback when determining the DHCP-provided domain name. It also adjusts CloudStack unit tests to avoid a mock leak and to cover the new NetworkManager fallback behavior.
Changes:
- Add
nmcli-based DHCP4/DHCP6 lease parsing helpers incloudinit/net/dhcp.py. - Update
DataSourceCloudStack._get_domainname()to attempt NetworkManager leases after ISC dhclient leases. - Fix/extend unit tests to properly mock ISC dhclient lease file selection and to validate the NetworkManager path.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
cloudinit/net/dhcp.py |
Adds helper functions to run nmcli and parse NetworkManager DHCP lease output. |
cloudinit/sources/DataSourceCloudStack.py |
Adds a NetworkManager lease fallback when determining domain name for FQDN construction. |
tests/unittests/sources/test_cloudstack.py |
Updates mocks to avoid platform-dependent behavior and adds a test for the NetworkManager fallback path. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
b95f3bd to
96a8a02
Compare
blackboxsw
left a comment
There was a problem hiding this comment.
Thank you @ani-sinha. I think what remains is a request for type hints for new functions and params and a couple of additional unittest requests.
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
96a8a02 to
f7a881b
Compare
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
f7a881b to
c7dc84d
Compare
holmanb
left a comment
There was a problem hiding this comment.
Thanks for the proposal. I haven't looked closely at the tests, but one thing about this change that stands out is that networkd code exists to select the virtual router and for domain name, but this implements only the domain name portion. Is there a plan for a followup to implement the other part?
| return domain_name.strip() | ||
|
|
||
| LOG.debug( | ||
| "Could not obtain FQDN from NM leases. Falling back to %s", |
There was a problem hiding this comment.
Why is this inserted before the client rather than after?
There was a problem hiding this comment.
Why is this inserted before the client rather than after?
You lost me here. Its how the debug logs are added for others like IscDHClient etc as well.
There was a problem hiding this comment.
I was referring to fallback order. This adds NetworkManager before the dhcp client.
There was a problem hiding this comment.
I was referring to fallback order. This adds NetworkManager before the dhcp client.
This is correct. This aligns with other dhcp clients in the function. Default client is dhcpcd but on RHEL, its network manager that primarily gives out the leases. We do not have full support for network manager as a client yet. There is a plan to implement it but won't happen anytime soon.
| # skip loopback | ||
| if dev_name == "lo": | ||
| continue | ||
| # if no device name is passed, use the first one found. |
There was a problem hiding this comment.
Do we know what order these will be printed in?
There was a problem hiding this comment.
Do we know what order these will be printed in?
From what I have seen, its mostly active devices, then loopback and then deactivated devices. But I am not sure if this order is true for all versions of nmcli across all distributions.
I can n look into it yes. The idea is to fix all the gaps where network manager support is inadequate |
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
…nical#6829) Red Hat distributions uses network manager to manage DHCP leases. This change adds support for obtaining vr address information from DHCP leases maintained by network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com>
e0451af to
ae8aa9e
Compare
I have added support for this in the 4th patch. |
…nical#6829) Red Hat distributions uses network manager to manage DHCP leases. This change adds support for obtaining vr address information from DHCP leases maintained by network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com>
ae8aa9e to
595624b
Compare
|
@blackboxsw ping. |
blackboxsw
left a comment
There was a problem hiding this comment.
Looks good @ani-sinha thank you for the updates and test improvements.
One minor test nit and this will be good to merge.
holmanb
left a comment
There was a problem hiding this comment.
This results in ConfgObj failures due to being unable to process the content:
I don't see any test for this behavior that was previously mentioned by @blackboxsw. Was it fixed?
Yes. Test cases were also added. |
Which test covers this case? |
find_correct_device_firstmatch() etc. they all deal with multi connected device cases. The output is specific to a device so it does not have to parse an output that contains results from more than one interface. Hence it does not result in the exception previously we were seeing. |
@holmanb, additionally @ani-sinha changed the original logic proposed by first calling |
|
@ani-sinha one final miss in testing is that we now have to re-enable the previously commented mock.patch or get_vr_address with something like the following diff. Without this patch we'll see issues from --- a/tests/unittests/sources/test_cloudstack.py
+++ b/tests/unittests/sources/test_cloudstack.py
@@ -744,11 +744,11 @@ class TestDataSourceCloudStackLocal:
MOD_PATH + ".EphemeralIPNetwork",
autospec=True,
)
- @mock.patch(MOD_PATH + ".net.find_fallback_nic")
- # @mock.patch(MOD_PATH + ".get_vr_address", return_value="10.1.37.131")
+ @mock.patch(MOD_PATH + ".net.find_fallback_nic", return_value="enp0s1")
+ @mock.patch(MOD_PATH + ".get_vr_address", return_value="10.1.37.131")
def test_local_datasource_success(
self,
- # m_get_vr_address,
+ m_get_vr_address,
m_find_fallback_nic,
m_dhcp,
m_wait_for_mds,
@@ -762,7 +762,6 @@ class TestDataSourceCloudStackLocal:
{}, distro, helpers.Paths({"run_dir": tmpdir})
)
- m_find_fallback_nic.return_value = "enp0s1"
m_dhcp.return_value.__enter__.side_effect = (None,)
m_wait_for_mds.return_value = (True,)
m_get_userdata.return_value = "ud" |
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
…nical#6829) Red Hat distributions uses network manager to manage DHCP leases. This change adds support for obtaining vr address information from DHCP leases maintained by network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com>
Fixed. |
|
For alternate distros, the failure seems to be in test infra For the integration test, not sure why this |
) DHCP leases can be directly obtained from network manager through appropriate command to the network manager cli. Add a couple of helper functions to get the lease information from network manager. In a subsequent patch, we will use the helper function from cloud stack datasource to get the lease information from network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
Without mocking out dhcp.IscDhclient.get_newest_lease_file_from_distro, the functions on some platforms returns None. This means get_key_from_latest_lease() bails out early without calling parse_leases() to parse the lease file. Fix it. Signed-off-by: Ani Sinha <anisinha@redhat.com>
…ses (canonical#6829) Red Hat uses network manager as the supported dhcp client. If network manager cli is available, we should try to get domain name information directly from the network manager leases before asking distro specific dhcp client (dhcpcd by default). Signed-off-by: Ani Sinha <anisinha@redhat.com> Co-authored-by: Chad Smith <chad.smith@canonical.com>
…nical#6829) Red Hat distributions uses network manager to manage DHCP leases. This change adds support for obtaining vr address information from DHCP leases maintained by network manager. Signed-off-by: Ani Sinha <anisinha@redhat.com>
This is a series of four independent commits.
The first commit adds helper functions in dhcp module to get dhcp lease information from network manager.
The second commit fixes a mock leak in existing cloudstack unit test code.
The third commit adds ability in cloud stack code trying to get domain name use network manager leases as well (in addition to trying networtkd and isa dhclient as it does today already).
The fourth commit adds support in cloud stack code to get VR information from network manager lease information.
This change has been tested by our (Red Hat) customer. After the change, we see logs of this type:
Proposed Commit Message
Commit 1:
Commit 2:
Commit 3:
Commit 4:
Merge type