Anti affinity Groups not work sometimes , Start VM in batches

[havengit]

ISSUE TYPE

• Bug Report

COMPONENT NAME

affinity Groups

CLOUDSTACK VERSION

CloudStack 4.15.0.0

CONFIGURATION

OS / ENVIRONMENT

CloudStack 4.15.0.0
host OS version: Ubuntu 20.04

SUMMARY

Start virtual machines in batches, anti affinity Groups not work sometimes

STEPS TO REPRODUCE

It doesn't happen every time

1.Two host in one cluster
2.add three VM to a anti-affinity group
3.VM in stopped state
4.start vm with WEB UI, Check three VM to start
5.Three VMs will start successful

logs ：
vm id : 158 159 160
host id: 3 9
job-5595:
2021-09-29 09:35:14,379 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-31:ctx-ff549a9e job-5595 ctx-e1872415) (logid:ec8a48c2) Processing affinity group gumd-test for VM Id: 158
2021-09-29 09:35:14,382 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-31:ctx-ff549a9e job-5595 ctx-e1872415) (logid:ec8a48c2) Added host 3 to avoid set, since VM 159 is present on the host, in Stopped state but has reserved capacity
2021-09-29 09:35:14,383 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-31:ctx-ff549a9e job-5595 ctx-e1872415) (logid:ec8a48c2) Added host 9 to avoid set, since VM 160 is present on the host, in Stopped state but has reserved capacity

job-5598
2021-09-29 09:35:14,594 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-32:ctx-acdc74f8 job-5598 ctx-3ce8dd3b) (logid:1d2445d0) Processing affinity group gumd-test for VM Id: 159
2021-09-29 09:35:14,597 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-32:ctx-acdc74f8 job-5598 ctx-3ce8dd3b) (logid:1d2445d0) Added host 9 to avoid set, since VM 158 is present on the host, in Stopped state but has reserved capacity
2021-09-29 09:35:14,600 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-32:ctx-acdc74f8 job-5598 ctx-3ce8dd3b) (logid:1d2445d0) Added host 9 to avoid set, since VM 160 is present on the host, in Stopped state but has reserved capacity

job-5592
2021-09-29 09:35:14,498 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-30:ctx-307add2b job-5592 ctx-0802d14f) (logid:c430fdd5) Processing affinity group gumd-test for VM Id: 160
2021-09-29 09:35:14,500 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-30:ctx-307add2b job-5592 ctx-0802d14f) (logid:c430fdd5) Added host 9 to avoid set, since VM 158 is present on the host, in Stopped state but has reserved capacity
2021-09-29 09:35:14,502 DEBUG [o.a.c.a.HostAntiAffinityProcessor] (API-Job-Executor-30:ctx-307add2b job-5592 ctx-0802d14f) (logid:c430fdd5) Added host 3 to avoid set, since VM 159 is present on the host, in Stopped state but has reserved capacity

EXPECTED RESULTS

Two VMs start success , one VM start failed

ACTUAL RESULTS

Three VMs start success 

[yadvr]

cc @nvazquez

[nvazquez]

Hi @havengit I have tried some scenarios on KVM with a cluster of 2 hosts:

Case 1: SUCCESS
Deploy VM1 - select host anti-affinity -> VM running on host 1
Deploy VM2 - select host anti-affinity -> VM running on host 2
Deploy VM3 - select host anti-affinity -> ERROR

Case 2: ERROR
Deploy 3 VMs without selecting any affinity group -> VMs distributed across the 2 hosts
Stop 3 VMs
Add VMs to anti affinity group
Start 3 VMs at the same time -> 3 VMs running (even starting them one by one)

Will work on it

[nvazquez]

@Pearl1594 @davidjumani @weizhouapache @rohityadavcloud @DaanHoogland @andrijapanicsb this will a proper definition, VM deployments are always failing considering the anti-affinity groups, however if the VMs are stopped/started belonging to the anti-affinity group it is possible to have VMs running on the same host as this issue describes (anti affinity for 3 VMs with 2 hosts). Should stop/start also fail if the anti-affinity rules are not met? Looking forward to hearing your input

[DaanHoogland]

I think anti affinity groups need to have more attributes. A strict anti-affinity might require cluster spreading or even pod spreading for a VM. In those cases a VM should not be started at all if any VM in the group is already running on each host/cluster/pod. If the anti-affinity is not strict one might argue for instance, "no third VM may be started if any resource (host/cluster/pod) doesn't host two VMs from the group yet".

of course there are monsters hiding in all details for this.

@Pearl1594 @davidjumani @weizhouapache @rohityadavcloud @DaanHoogland @andrijapanicsb this will a proper definition, VM deployments are always failing considering the anti-affinity groups, however if the VMs are stopped/started belonging to the anti-affinity group it is possible to have VMs running on the same host as this issue describes (anti affinity for 3 VMs with 2 hosts). Should stop/start also fail if the anti-affinity rules are not met? Looking forward to hearing your input

[weizhouapache]

I think anti affinity groups need to have more attributes. A strict anti-affinity might require cluster spreading or even pod spreading for a VM. In those cases a VM should not be started at all if any VM in the group is already running on each host/cluster/pod. If the anti-affinity is not strict one might argue for instance, "no third VM may be started if any resource (host/cluster/pod) doesn't host two VMs from the group yet".

@DaanHoogland
this is how it works in cloudstack for now.
VMs with same host anti-affinity can not be deployed to the same host. It leads to an issue that user cannot deploy VMs if there are not enough hosts.

we could rename the current "host anti-affinity" to "Strict host anti-affinity" , and create a type like "Loose anti-host affinity". The new type works like guest os preference. VMs are preferably allocated to different hosts, but if there are not enough hosts, VMs can be deployed to same host as other vm with same affinity group.

anyway, it is not the root cause of this issue I believe.

of course there are monsters hiding in all details for this.

@Pearl1594 @davidjumani @weizhouapache @rohityadavcloud @DaanHoogland @andrijapanicsb this will a proper definition, VM deployments are always failing considering the anti-affinity groups, however if the VMs are stopped/started belonging to the anti-affinity group it is possible to have VMs running on the same host as this issue describes (anti affinity for 3 VMs with 2 hosts). Should stop/start also fail if the anti-affinity rules are not met? Looking forward to hearing your input

[weizhouapache]

Hi @havengit I have tried some scenarios on KVM with a cluster of 2 hosts:

Case 1: SUCCESS Deploy VM1 - select host anti-affinity -> VM running on host 1 Deploy VM2 - select host anti-affinity -> VM running on host 2 Deploy VM3 - select host anti-affinity -> ERROR

Case 2: ERROR Deploy 3 VMs without selecting any affinity group -> VMs distributed across the 2 hosts Stop 3 VMs Add VMs to anti affinity group Start 3 VMs at the same time -> 3 VMs running (even starting them one by one)

Will work on it

@nvazquez
I suspect, the root cause of this issue is, affinity group is not checked when start a vm with last host id.

I have fixed a similar issue with guest os preference in #4338,
unfortunately it seems to be broken by #3600 which reverted the change cc @GabrielBrascher .
I have created a PR to fix it #6000

we need more logs to confirm it. if so, we need to add check for affinity group in checkVmProfileAndHost method.

[nvazquez]

Thanks @weizhouapache checking it

[nvazquez]

@weizhouapache I tested the fix but still getting all the 3 VMs running even though they belong to an anti-affinity group. Attaching the logs antiaffinity.log

[weizhouapache]

@weizhouapache I tested the fix but still getting all the 3 VMs running even though they belong to an anti-affinity group. Attaching the logs antiaffinity.log

@nvazquez
I had a quick check on logs, it seems affinity group check has been processed in DeploymentPlanningManagerImpl, so my suspicion is not correct.

I think the issue is because 3 vms are started in parallel.

the original vm state:
(job_107) VM 1: Stopped, host_id: NULL, last host_id: 2
(job_108) VM 2: Stopped, host_id: NULL, last host_id: 2
(job_109) VM 3: Stopped, host_id: NULL, last host_id: 1

after first try:
(job_107) VM 1: Starting, host_id: NULL, last host_id: 2
(job_108) VM 2: Starting, host_id: NULL, last host_id: 2
(job_109) VM 3: Starting, host_id: 1, last host_id: 1

after second try:
(job_107) VM 1: Starting, host_id: 2, last host_id: 2
(job_108) VM 2: Starting, host_id: 2, last host_id: 2
(job_109) VM 3: Starting, host_id: 1, last host_id: 1

After first try, VM 1 and VM2 are Starting state and do not have host_id. in second try, affinity group does not work well. see code below

cloudstack/plugins/affinity-group-processors/host-anti-affinity/src/main/java/org/apache/cloudstack/affinity/HostAntiAffinityProcessor.java

Lines 82 to 98 in 478e244

	if (groupVM != null && !groupVM.isRemoved()) {
	if (groupVM.getHostId() != null) {
	avoid.addHost(groupVM.getHostId());
	if (s_logger.isDebugEnabled()) {
	s_logger.debug("Added host " + groupVM.getHostId() + " to avoid set, since VM " + groupVM.getId() + " is present on the host");
	}
	} else if (VirtualMachine.State.Stopped.equals(groupVM.getState()) && groupVM.getLastHostId() != null) {
	long secondsSinceLastUpdate = (DateUtil.currentGMTTime().getTime() - groupVM.getUpdateTime().getTime()) / 1000;
	if (secondsSinceLastUpdate < _vmCapacityReleaseInterval) {
	avoid.addHost(groupVM.getLastHostId());
	if (s_logger.isDebugEnabled()) {
	s_logger.debug("Added host " + groupVM.getLastHostId() + " to avoid set, since VM " + groupVM.getId() +
	" is present on the host, in Stopped state but has reserved capacity");
	}
	}
	}
	}

(groupVM.getHostId() is NULL and groupVM.getState() is Stopped for both VM 1 and VM 2.)

If VM is started when other vms are Running or Stopped, it should work as expected.