Cannot stop instance

[0sorkon]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API(?), Storage

CLOUDSTACK VERSION

4.81.1

CONFIGURATION

OS / ENVIRONMENT

N/A

SUMMARY

sometimes an error occurs, the cause of which is not obvious and there is no indication where to look for it and where its source is. Just at one moment VMs refuse to shut down via Stop instance in the interface. When pressing the button, an error message appears:

(VM-0dde637a-eef3-4bb1-8f36-34a2a732ff0d) Unable to stop VM instance {‘id’:814, ‘instanceName’: ‘i-12-814-VM’, ‘type’: ‘User’, ‘uuid’: ‘0dde637a-eef3-4bb1-8f36-34a2a732ff0d’}

Log on agent:

2024-05-01 08:52:37,013 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:324bc3ba) Caught:
java.lang.NullPointerException
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$0(LinstorStorageAdaptor.java:312)
at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528)
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$1(LinstorStorageAdaptor.java:312)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543)
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.getResourceByPath(LinstorStorageAdaptor.java:313)
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.disconnectPhysicalDiskByPath(LinstorStorageAdaptor.java:346)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDiskByPath(KVMStoragePoolManager.java:205)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3247)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:101)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1848)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

Log at management:

2024-05-01 10:29:32,915 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null) (logid:) Seq 47-4875991021558694243: Processing: { Ans: , MgmtId: 20788631421694, via: 47, Ver: v1, Flags
: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$0(LinstorStorageAdaptor.java:312)
at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528)
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$1(LinstorStorageAdaptor.java:312)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543)
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.getResourceByPath(LinstorStorageAdaptor.java:313)
at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.disconnectPhysicalDiskByPath(LinstorStorageAdaptor.java:346)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDiskByPath(KVMStoragePoolManager.java:205)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3247)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:101)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:49)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1848)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }

The problem may not be in cloudstack itself, but I would be very grateful if someone could point me in which direction to look to find out the source of the problem.

[weizhouapache]

cc @rp-

[rp-]

I'm not sure what exact version you are really running, because the stacktrace doesn't match any of the 4.18. versions,
but this is most probably already fixed with: #8790

[0sorkon]

UPD: In fact, on the kvm server, the virtual instance stops (even though the UI shows an error and the status continues to be Running). But if immediately after the error try to shut down the VM with the Force switch on, the VM becomes Stopped. If this is not done, the VM will start again, apparently according to HA policy. I don't know if this information will help, but I hope so.

[0sorkon]

I'm not sure what exact version you are really running, because the stacktrace doesn't match any of the 4.18. versions, but this is most probably already fixed with: #8790

My version is 4.18.1.0

[DaanHoogland]

fixed by #8790