fix: storage map accumulation and optimize datastore filtering

roles/mtv_management/README.md

@@ -197,6 +197,8 @@ Description: Management of the Migration Toolkit for Virtualization (MTV).
 | _mtv_storage_map ¦ Determine Destination Storage Class | `ansible.builtin.set_fact` | True |
 | _mtv_storage_map ¦ Query for Datastores from Destination Provider | `ansible.builtin.include_tasks` | True |
 | _mtv_storage_map ¦ Query for Datastores from Destination Provider | `ansible.builtin.include_tasks` | True |
+| _mtv_storage_map ¦ Extract included datastore IDs from overrides | `ansible.builtin.set_fact` | True |
+| _mtv_storage_map ¦ Extract excluded datastore IDs from overrides | `ansible.builtin.set_fact` | True |
 | _mtv_storage_map ¦ Process VMware Datastores | `ansible.builtin.include_tasks` | True |
 | _mtv_storage_map ¦ Process Ovirt Datastores | `ansible.builtin.include_tasks` | True |
 | _mtv_storage_map ¦ Template StorageMap Map | `ansible.builtin.set_fact` | False |
@@ -406,11 +408,13 @@ classDef rescue stroke:#665352,stroke-width:2px;
   _mtv_storage_map___Query_for_Storage_Classes_from_Destination_Provider_mtv_query_inventory_yml_2-->|Task| _mtv_storage_map___Determine_Destination_Storage_Class3[ mtv storage map   determine destination storage<br>class<br>When: **mtv management default storage class is not<br>defined or mtv management default storage class  <br>default     true    trim   length    0 and item<br>object metadata annotations is defined and   item<br>object metadata annotations  storageclass<br>kubernetes io is default class     default  false <br>   string   lower      true**]:::task
   _mtv_storage_map___Determine_Destination_Storage_Class3-->|Include task| _mtv_storage_map___Query_for_Datastores_from_Destination_Provider_mtv_query_inventory_yml_4[ mtv storage map   query for datastores from<br>destination provider<br>When: **ovirt  in provider**<br>include_task: mtv query inventory yml]:::includeTasks
   _mtv_storage_map___Query_for_Datastores_from_Destination_Provider_mtv_query_inventory_yml_4-->|Include task| _mtv_storage_map___Query_for_Datastores_from_Destination_Provider_mtv_query_inventory_yml_5[ mtv storage map   query for datastores from<br>destination provider<br>When: **vsphere  in provider**<br>include_task: mtv query inventory yml]:::includeTasks
-  _mtv_storage_map___Query_for_Datastores_from_Destination_Provider_mtv_query_inventory_yml_5-->|Include task| _mtv_storage_map___Process_VMware_Datastores__mtv_storage_map_process_datastore_yml_6[ mtv storage map   process vmware datastores<br>When: **vsphere  in provider and mtv destination<br>datastores is defined and mtv destination<br>datastores   length   0**<br>include_task:  mtv storage map process datastore yml]:::includeTasks
-  _mtv_storage_map___Process_VMware_Datastores__mtv_storage_map_process_datastore_yml_6-->|Include task| _mtv_storage_map___Process_Ovirt_Datastores__mtv_storage_map_process_datastore_yml_7[ mtv storage map   process ovirt datastores<br>When: **ovirt  in provider and mtv destination datastores<br>is defined and mtv destination datastores   length<br>  0**<br>include_task:  mtv storage map process datastore yml]:::includeTasks
-  _mtv_storage_map___Process_Ovirt_Datastores__mtv_storage_map_process_datastore_yml_7-->|Task| _mtv_storage_map___Template_StorageMap_Map8[ mtv storage map   template storagemap map]:::task
-  _mtv_storage_map___Template_StorageMap_Map8-->|Task| _mtv_storage_map___Create_Storage_Map9[ mtv storage map   create storage map]:::task
-  _mtv_storage_map___Create_Storage_Map9-->End
+  _mtv_storage_map___Query_for_Datastores_from_Destination_Provider_mtv_query_inventory_yml_5-->|Task| _mtv_storage_map___Extract_included_datastore_IDs_from_overrides6[ mtv storage map   extract included datastore ids<br>from overrides<br>When: **mtv management storage map overrides is defined<br>and mtv management storage map overrides   length <br> 0 and mtv management storage map overrides  <br>selectattr  include    defined     list   length  <br>0**]:::task
+  _mtv_storage_map___Extract_included_datastore_IDs_from_overrides6-->|Task| _mtv_storage_map___Extract_excluded_datastore_IDs_from_overrides7[ mtv storage map   extract excluded datastore ids<br>from overrides<br>When: **mtv management storage map overrides is defined<br>and mtv management storage map overrides   length <br> 0 and mtv management storage map overrides  <br>selectattr  exclude    defined     list   length  <br>0**]:::task
+  _mtv_storage_map___Extract_excluded_datastore_IDs_from_overrides7-->|Include task| _mtv_storage_map___Process_VMware_Datastores__mtv_storage_map_process_datastore_yml_8[ mtv storage map   process vmware datastores<br>When: **vsphere  in provider and mtv destination<br>datastores is defined and mtv destination<br>datastores   length   0**<br>include_task:  mtv storage map process datastore yml]:::includeTasks
+  _mtv_storage_map___Process_VMware_Datastores__mtv_storage_map_process_datastore_yml_8-->|Include task| _mtv_storage_map___Process_Ovirt_Datastores__mtv_storage_map_process_datastore_yml_9[ mtv storage map   process ovirt datastores<br>When: **ovirt  in provider and mtv destination datastores<br>is defined and mtv destination datastores   length<br>  0**<br>include_task:  mtv storage map process datastore yml]:::includeTasks
+  _mtv_storage_map___Process_Ovirt_Datastores__mtv_storage_map_process_datastore_yml_9-->|Task| _mtv_storage_map___Template_StorageMap_Map10[ mtv storage map   template storagemap map]:::task
+  _mtv_storage_map___Template_StorageMap_Map10-->|Task| _mtv_storage_map___Create_Storage_Map11[ mtv storage map   create storage map]:::task
+  _mtv_storage_map___Create_Storage_Map11-->End
 ```
 
 ### Graph for _mtv_storage_map_process_datastore.yml
@@ -434,8 +438,8 @@ classDef rescue stroke:#665352,stroke-width:2px;
   _mtv_storage_map_process_datastore___Set_Ovirt_StorageMap_StorageClass_from_Overrides3-->|Task| _mtv_storage_map_process_datastore___Verify_VMWare_Destination_Storage_Class4[ mtv storage map process datastore   verify vmware<br>destination storage class<br>When: **mtv vmware datastore is defined**]:::task
   _mtv_storage_map_process_datastore___Verify_VMWare_Destination_Storage_Class4-->|Task| _mtv_storage_map_process_datastore___Verify_Ovirt_Destination_Storage_Class5[ mtv storage map process datastore   verify ovirt<br>destination storage class<br>When: **mtv ovirt datastore is defined**]:::task
   _mtv_storage_map_process_datastore___Verify_Ovirt_Destination_Storage_Class5-->|Task| _mtv_storage_map_process_datastore___Template_StorageMap_Map6[ mtv storage map process datastore   template<br>storagemap map]:::task
-  _mtv_storage_map_process_datastore___Template_StorageMap_Map6-->|Task| _mtv_storage_map_process_datastore___Add_VMware_StorageMap_Map_to_Dict7[ mtv storage map process datastore   add vmware<br>storagemap map to dict<br>When: **mtv vmware datastore is defined and    include  in<br>mtv management mtv vmware datastore overrides and<br>mtv management storage map overrides   selectattr <br>include    defined     list   length   0  or  mtv<br>management storage map overrides   selectattr <br>include    defined     list   length    0 and mtv<br>management storage map overrides   selectattr <br>exclude    defined     list   length    0   or  <br>exclude  not in mtv management mtv vmware<br>datastore overrides and mtv management storage map<br>overrides   selectattr  exclude    defined    <br>list   length   0**]:::task
-  _mtv_storage_map_process_datastore___Add_VMware_StorageMap_Map_to_Dict7-->|Task| _mtv_storage_map_process_datastore___Add_Ovirt_StorageMap_Map_to_Dict8[ mtv storage map process datastore   add ovirt<br>storagemap map to dict<br>When: **mtv ovirt datastore is defined and    include  in<br>mtv management mtv ovirt datastore overrides and<br>mtv management storage map overrides   selectattr <br>include    defined     list   length   0  or  mtv<br>management storage map overrides   selectattr <br>include    defined     list   length    0 and mtv<br>management storage map overrides   selectattr <br>exclude    defined     list   length    0   or  <br>exclude  not in mtv management mtv ovirt datastore<br>overrides and mtv management storage map overrides<br>  selectattr  exclude    defined     list   length<br>  0**]:::task
+  _mtv_storage_map_process_datastore___Template_StorageMap_Map6-->|Task| _mtv_storage_map_process_datastore___Add_VMware_StorageMap_Map_to_Dict7[ mtv storage map process datastore   add vmware<br>storagemap map to dict<br>When: **mtv vmware datastore is defined**]:::task
+  _mtv_storage_map_process_datastore___Add_VMware_StorageMap_Map_to_Dict7-->|Task| _mtv_storage_map_process_datastore___Add_Ovirt_StorageMap_Map_to_Dict8[ mtv storage map process datastore   add ovirt<br>storagemap map to dict<br>When: **mtv ovirt datastore is defined**]:::task
   _mtv_storage_map_process_datastore___Add_Ovirt_StorageMap_Map_to_Dict8-->End
 ```
 

roles/mtv_management/tasks/_mtv_storage_map.yml

@@ -2,7 +2,7 @@
 
 - name: _mtv_storage_map | Initialize data structures
   ansible.builtin.set_fact:
-    mtv_management_mtv_storagemaps: {}
+    mtv_management_mtv_storagemap_maps: {}
 
 - name: _mtv_storage_map | Verify Storage Map Overrides do not contain both includes and excludes
   ansible.builtin.fail:
@@ -41,9 +41,48 @@
     mtv_management_inventory_query_result_var: mtv_destination_datastores
   when: "'vsphere' in provider"
 
+- name: _mtv_storage_map | Extract included datastore IDs from overrides
+  ansible.builtin.set_fact:
+    mtv_management_mtv_allowed_datastore_ids: >-
+      {{
+        mtv_management_storage_map_overrides
+        | selectattr('include', 'defined')
+        | selectattr('include', 'equalto', true)
+        | map(attribute='id')
+        | list
+      }}
+  when:
+    - mtv_management_storage_map_overrides is defined
+    - mtv_management_storage_map_overrides | length > 0
+    - mtv_management_storage_map_overrides | selectattr('include', 'defined') | list | length > 0
+
+- name: _mtv_storage_map | Extract excluded datastore IDs from overrides
+  ansible.builtin.set_fact:
+    mtv_management_mtv_excluded_datastore_ids: >-
+      {{
+        mtv_management_storage_map_overrides
+        | selectattr('exclude', 'defined')
+        | selectattr('exclude', 'equalto', true)
+        | map(attribute='id')
+        | list
+      }}
+  when:
+    - mtv_management_storage_map_overrides is defined
+    - mtv_management_storage_map_overrides | length > 0
+    - mtv_management_storage_map_overrides | selectattr('exclude', 'defined') | list | length > 0
+
 - name: _mtv_storage_map | Process VMware Datastores
   ansible.builtin.include_tasks: _mtv_storage_map_process_datastore.yml
-  loop: "{{ mtv_destination_datastores }}"
+  loop: >-
+    {{
+      (mtv_destination_datastores | selectattr('id', 'in', mtv_management_mtv_allowed_datastore_ids) | list)
+      if (mtv_management_mtv_allowed_datastore_ids | default([]) | length > 0)
+      else (mtv_destination_datastores
+        | rejectattr('id', 'in', mtv_management_mtv_excluded_datastore_ids | default([]))
+        | list)
+      if (mtv_management_mtv_excluded_datastore_ids | default([]) | length > 0)
+      else mtv_destination_datastores
+    }}
   loop_control:
     loop_var: mtv_vmware_datastore
   when:
@@ -53,7 +92,16 @@
 
 - name: _mtv_storage_map | Process Ovirt Datastores
   ansible.builtin.include_tasks: _mtv_storage_map_process_datastore.yml
-  loop: "{{ mtv_destination_datastores }}"
+  loop: >-
+    {{
+      (mtv_destination_datastores | selectattr('id', 'in', mtv_management_mtv_allowed_datastore_ids) | list)
+      if (mtv_management_mtv_allowed_datastore_ids | default([]) | length > 0)
+      else (mtv_destination_datastores
+        | rejectattr('id', 'in', mtv_management_mtv_excluded_datastore_ids | default([]))
+        | list)
+      if (mtv_management_mtv_excluded_datastore_ids | default([]) | length > 0)
+      else mtv_destination_datastores
+    }}
   loop_control:
     loop_var: mtv_ovirt_datastore
   when:

roles/mtv_management/tasks/_mtv_storage_map_process_datastore.yml

@@ -54,15 +54,13 @@
 
 - name: _mtv_storage_map_process_datastore | Add VMware StorageMap Map to Dict
   ansible.builtin.set_fact:
-    mtv_management_mtv_storagemap_maps: "{{ mtv_management_mtv_storagemap_maps | default({}) | ansible.builtin.combine({mtv_vmware_datastore.selfLink: mtv_management_storage_map_map}, recursive=true) }}" # noqa: yaml[line-length]
+    mtv_management_mtv_storagemap_maps: "{{ mtv_management_mtv_storagemap_maps | default({}) | ansible.builtin.combine({(mtv_vmware_datastore.selfLink + '-' + _mtv_management_storage_map_storage_class): mtv_management_storage_map_map}, recursive=true) }}" # noqa: yaml[line-length]
   when:
     - mtv_vmware_datastore is defined
-    - "(('include' in mtv_management_mtv_vmware_datastore_overrides and mtv_management_storage_map_overrides | selectattr('include', 'defined') | list | length > 0) or (mtv_management_storage_map_overrides | selectattr('include', 'defined') | list | length == 0 and mtv_management_storage_map_overrides | selectattr('exclude', 'defined') | list | length == 0)) or ('exclude' not in mtv_management_mtv_vmware_datastore_overrides and mtv_management_storage_map_overrides | selectattr('exclude', 'defined') | list | length > 0)" # noqa: yaml[line-length]
 
 - name: _mtv_storage_map_process_datastore | Add Ovirt StorageMap Map to Dict
   ansible.builtin.set_fact:
-    mtv_management_mtv_storagemap_maps: "{{ mtv_management_mtv_storagemap_maps | default({}) | ansible.builtin.combine({mtv_ovirt_datastore.selfLink: mtv_management_storage_map_map}, recursive=true) }}" # noqa: yaml[line-length]
+    mtv_management_mtv_storagemap_maps: "{{ mtv_management_mtv_storagemap_maps | default({}) | ansible.builtin.combine({(mtv_ovirt_datastore.selfLink + '-' + _mtv_management_storage_map_storage_class): mtv_management_storage_map_map}, recursive=true) }}" # noqa: yaml[line-length]
   when:
     - mtv_ovirt_datastore is defined
-    - "(('include' in mtv_management_mtv_ovirt_datastore_overrides and mtv_management_storage_map_overrides | selectattr('include', 'defined') | list | length > 0) or (mtv_management_storage_map_overrides | selectattr('include', 'defined') | list | length == 0 and mtv_management_storage_map_overrides | selectattr('exclude', 'defined') | list | length == 0)) or ('exclude' not in mtv_management_mtv_ovirt_datastore_overrides and mtv_management_storage_map_overrides | selectattr('exclude', 'defined') | list | length > 0)" # noqa: yaml[line-length]
 ...