Console Proxy VM: High CPU when using secure VNC / NIOConnection

[kohrar]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Console Proxy

CLOUDSTACK VERSION

4.18, 4.19

CONFIGURATION

Clean install of Apache CloudStack 4.18 and 4.19

OS / ENVIRONMENT

Rocky Linux 8.9

SUMMARY

When using the secure KVM VNC console proxy in 4.18 or 4.19, the Console Proxy SystemVM uses 100% of its single CPU when handling the connection.

A quick analysis of the java process suggests the high CPU usage is from the NIOConnection components:

NoVncClient getNextBytes:

The ~100% CPU usage is happening even when the VNC screen is blank (eg. when the screen is off).

In the mean time, I've changed the console proxy's SystemVM compute offering with additional CPUs to reduce CPU contention as we've had some users complain VNC connections were not responding.

If this is expected behavior of NIOConnection, it may be reasonable to increase the CPU allocations in the default console proxy compute offering.

STEPS TO REPRODUCE

1. Connect to a KVM VNC session on ACS >4.18.
2. Connect to the console proxy and watch the java process's CPU usage.

EXPECTED RESULTS

Reasonable CPU usage when proxying a connection when little data (or screen changes) are happening.

ACTUAL RESULTS

I am seeing ~100% CPU usage per active VNC connection even when nothing is happening on the session (eg. a black screen).

[weizhouapache]

Is this similar as the issue fixed by #6970?
@nvazquez @DaanHoogland

[kohrar]

Looks like it's similar. The while(connectionAlive){} in ConsoleProxyNoVncClient.java for the NIO case likely just needs to sleep to stop being a tight loop.

[DaanHoogland]

@kohrar can you test #8694?

[kohrar]

@DaanHoogland Could you point me to how I can test out the fix in the Console Proxy VM? Is there a way to get access to the built files by blue orangutan in that particular PR? Thanks for the quick PR!

[DaanHoogland]

I was thinking you should build yourself if you can. If you can't we have to think of something else. I can't give you access directly.

[kohrar]

I've rebuilt the cloud-systemvm files and copied it into the systemVM. I think your change helps as the process is no longer pegging the CPU at 100%. Now it's down to ~25% for a single connection.

# top -b -n 1 | head -n 15
top - 21:34:44 up 31 days, 15 min,  2 users,  load average: 0.14, 0.18, 0.23
Tasks:  78 total,   2 running,  75 sleeping,   1 stopped,   0 zombie
%Cpu(s): 17.6 us, 11.8 sy,  0.0 ni, 70.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    976.6 total,     94.3 free,    487.6 used,    394.7 buff/cache
MiB Swap:    243.0 total,    242.7 free,      0.3 used.    405.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  92659 root      20   0 2880800 140376  30428 S  25.0  14.0   0:39.87 java
      1 root      20   0  101588  11248   7168 S   0.0   1.1   1:00.53 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:01.18 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_perc+
      9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tas+

# nproc
1

If this is the expected CPU usage going forward, would it make sense to have another PR that changes the default compute offerings so the console proxy VM has more than 1 CPU by default?

[DaanHoogland]

Good work @kohrar , thanks. There will be a new system VM template that will require a bigger offering in the next major version. Setting the default will be bigger for sure.

[kohrar]

Awesome. Thanks for your work on this, Daan!

[yadvr]

Fixed in #8694