VPC Routing/Network Issue

[midhunpjos]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VPC

CLOUDSTACK VERSION

4.18.1

CONFIGURATION

Advanced Zone, Network VPC 

OS / ENVIRONMENT

Cloudstack 4.18 on KVM
Multiple /27 IP's for Public 

SUMMARY

Unable to Reach Public IP, which is in same deployment from VM's deployed in VPC

STEPS TO REPRODUCE

Cloudstack deployment is having multiple /27 IP addresses added to public Range. Deploy A VPC and acquire Multiple public IP from each /27 subnet. From any of the VM's deployed in the VPC we are not able to reach the above public IP's except one.

checked the VPC router and the routing table is added as below

default via 99.127.xxx.65 dev eth1
10.20.1.0/24 dev eth5 proto kernel scope link src 10.20.1.1
10.20.2.0/24 dev eth6 proto kernel scope link src 10.20.2.1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.198.170
99.127.xxx.32/27 dev eth2 proto kernel scope link src 99.127.xxx.51
99.127.xxx.64/27 dev eth1 proto kernel scope link src 99.127.xxx.93
99.127.xxx.96/27 dev eth3 proto kernel scope link src 99.127.xxx.125

From any VM's in VPC I can reach 99.127.xxx.64/27 subnet. Remaining public subnets are not reachable. From the VM We tried to ping the IP 99.127.xxx.107 (Firewall was Open to public for ICMP), but not pinging

On capturing the packets we could see that packets are leaving the source network and its reaching VR with IP 99.127.xxx.107 but no response is received.

Packet Capture at source VPC router

03:34:16.950824 IP css1-cks-shared-1-node-18bb5b445d5 > 99.127.xxx.107: ICMP echo request, id 7168, seq 0, length 64
03:34:17.950957 IP css1-cks-shared-1-node-18bb5b445d5 > 99.127.xxx.107: ICMP echo request, id 7168, seq 1, length 64

Received packets at 99.127.xxx.107 VR

03:34:16.950549 IP 10.20.2.137 > 99.127.xxx.107: ICMP echo request, id 7168, seq 0, length 64
03:34:16.951536 IP 99.127.xxx.107 > 10.20.2.137: ICMP echo reply, id 7168, seq 0, length 64
03:34:17.950598 IP 10.20.2.137 > 99.127.xxx.107: ICMP echo request, id 7168, seq 1, length 64
03:34:17.951405 IP 99.127.xxx.107 > 10.20.2.137: ICMP echo reply, id 7168, seq 1, length 64

On above we could see that the source IP address is marked as '10.20.2.137' and from the 99.127.xxx.107 router this Private IP will not be reachable. Ideally It should be the Public IP address of VPC.

Please let us know your comments on this.

[weizhouapache]

@midhunpjos

• can you log into the VR, and check if the gateway of each public range is reachable ?
• can you create a tier with default_allow and test with a vm on it ?

[midhunpjos]

@weizhouapache
Q: can you log into the VR, and check if the gateway of each public range is reachable ?
Yes From the VR gateway of each subnet is reachable. The Public IP's are also reachable

Q: can you create a tier with default_allow and test with a vm on it ?
Current test in done on tier with default_allow. From the VM, the subnet 99.127.xxx.64/27 is reachable. (This is the GW of VR). Other subnet's gateway and IP's are not reachable.

[weizhouapache]

@weizhouapache Q: can you log into the VR, and check if the gateway of each public range is reachable ? Yes From the VR gateway of each subnet is reachable. The Public IP's are also reachable

Q: can you create a tier with default_allow and test with a vm on it ? Current test in done on tier with default_allow. From the VM, the subnet 99.127.xxx.64/27 is reachable. (This is the GW of VR). Other subnet's gateway and IP's are not reachable.

@midhunpjos
is the IP 99.127.xxx.107 used for some purposes, e.g. static nat or port forwarding , load balancer ?

[midhunpjos]

@weizhouapache Yes .Port forwarding is enabled for the IP. Firewall is open for ICMP.
Not only 99.127.xxx.107, 99.127.xxx.108... are not accessible from specific VPC VM.

Another update.. In VPC If I acquire new IP and add static NAT to VM in VPC, then all Public subnets are pinging including 99.127.xxx.107

[weizhouapache]

@weizhouapache Yes .Port forwarding is enabled for the IP. Firewall is open for ICMP. Not only 99.127.xxx.107, 99.127.xxx.108... are not accessible from specific VPC VM.

Another update.. In VPC If I acquire new IP and add static NAT to VM in VPC, then all Public subnets are pinging including 99.127.xxx.107

I think it is expected that the Public IP for port forwarding or load balacing is not pingable.
only the ports for PF/LB are useful.

[midhunpjos]

@weizhouapache 99.127.xxx.107 is used in an isolated network, but I can ping it from other VPC,or from public not from a specific VPC mentioned in issue. A Webserver is running behind 107, I cant telnet to port 443 from the VM in VPC under concern.

[weizhouapache]

@weizhouapache 99.127.xxx.107 is used in an isolated network, but I can ping it from other VPC,or from public not from a specific VPC mentioned in issue. A Webserver is running behind 107, I cant telnet to port 443 from the VM in VPC under concern.

got it, I thought the 99.127.xxx.107 is associated to the VPC and you created PF on it, which is not pingable from a VM...

let me test it

[weizhouapache]

Just an update:
I was able to reproduce the issue

[weizhouapache]

@DaanHoogland
added to 4.18.2.0