SingleHost CS - SystemVMS cannot reach default gw

[Overmorrow1501]

ISSUE TYPE

• Other

COMPONENT NAME

Network

CLOUDSTACK VERSION

4.19.1.1

CLOUDSTACK CONFIGURATION

Zone:
Name - Zone1
Public DNS 1 - 8.8.8.8/8.8.4.4
Internal DNS1 - 8.8.8.8/8.8.4.4
Hypervisor - KVM

Public traffic net configuration:
Gateway - 192.168.1.1
Netmask - 255.255.255.0
VLAN/VNI - (vlan://untagged)
Start IP - 192.168.1.150
End IP - 192.168.1.160

Management net configuration:
Gateway - 192.168.1.1
Start/end reserved system IPs - 192.168.1.161 - 192.168.1.170

primary storage:
Name - Primary1
Scope - zone-wide
Protocol - NFS
Server - 192.168.1.99
Path - /export/primary

secondary storage:
Provider - NFS
Name - Secondary1
Server - 192.168.1.99
Path - /export/secondary

OS / ENVIRONMENT

PRETTY_NAME="Ubuntu 24.04 LTS"
Single NIC (eth0)

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master cloudbr0 state UP group default qlen 1000
    link/ether bc:24:11:cd:27:e5 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    altname ens18
3: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5a:7a:5e:12:8c:ac brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.99/24 brd 192.168.1.255 scope global cloudbr0
       valid_lft forever preferred_lft forever
root@ubuntu-lvm:~# ip route
default via 192.168.1.1 dev cloudbr0 proto static 

root@ubuntu-lvm:~# systemctl status cloudstack-*
● cloudstack-agent.service - CloudStack Agent
     Loaded: loaded (/usr/lib/systemd/system/cloudstack-agent.service; enabled; preset: enabl>
     Active: active (running) since Fri 2024-08-30 15:44:21 CEST; 46min ago
       Docs: http://www.cloudstack.org/
   Main PID: 15785 (java)
      Tasks: 55 (limit: 11620)
     Memory: 545.1M (peak: 567.9M)
        CPU: 27.759s
     CGroup: /system.slice/cloudstack-agent.service
             └─15785 /usr/bin/java -Djava.io.tmpdir=/usr/share/cloudstack-agent/tmp -Xms256m >

Aug 30 16:27:09 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:27:09 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:27:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:27:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:28:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:28:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:29:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:29:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:30:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>
Aug 30 16:30:14 ubuntu-lvm java[15785]: INFO  [kvm.storage.LibvirtStorageAdaptor] (agentReque>

● cloudstack-management.service - CloudStack Management Server
     Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service; enabled; preset: >
     Active: active (running) since Fri 2024-08-30 15:25:23 CEST; 1h 5min ago
   Main PID: 1801 (java)
      Tasks: 233 (limit: 11620)
     Memory: 1.2G (peak: 1.3G)

CLOUDSTACK OS CONFIGURATION

/etc/netplan/99-cloudstack-net.yaml 
 network:
   version: 2
   renderer: networkd
   ethernets:
     eth0:
       dhcp4: false
       dhcp6: false
       optional: true
       match:
           macaddress: bc:24:11:cd:27:e5
   bridges:
     cloudbr0:
       addresses: [192.168.1.99/24]
       routes:
        - to: default
          via: 192.168.1.1
       nameservers:
         addresses: [1.1.1.1, 8.8.8.8]
       interfaces: [eth0]
       dhcp4: false
       dhcp6: false
       parameters:
         stp: false
         forward-delay: 0

SUMMARY

Let me begin by saying that this is one of my first approach to Cloudstack. I've followed the following guides to bring up a (all-in-one) single node CS infrastructure:
-https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html
-https://rohityadav.cloud/blog/cloudstack-kvm/

After the quick installation finishes SystemVMs start but these cannot reach anything else but 192.168.1.99 (which is the KVM Host).

STEPS TO REPRODUCE

Inside the SSVM I see the following:

root@s-2-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:00:a9:fe:cf:dc brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 169.254.207.220/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:0a:00:00:08 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
    inet 192.168.1.168/24 brd 192.168.1.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:14:00:00:0c brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 192.168.1.151/24 brd 192.168.1.255 scope global eth2
       valid_lft forever preferred_lft forever
root@s-2-VM:~# ip route
default via 192.168.1.1 dev eth2 
8.8.4.4 via 192.168.1.1 dev eth1 
8.8.8.8 via 192.168.1.1 dev eth1 
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.207.220 
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.168 
192.168.1.0/24 dev eth2 proto kernel scope link src 192.168.1.151 

root@s-2-VM:~# ping -c 4 -T30 192.168.1.99
PING 192.168.1.99 (192.168.1.99): 56 data bytes
64 bytes from 192.168.1.99: icmp_seq=0 ttl=64 time=0.168 ms
64 bytes from 192.168.1.99: icmp_seq=1 ttl=64 time=0.443 ms
64 bytes from 192.168.1.99: icmp_seq=2 ttl=64 time=0.510 ms
64 bytes from 192.168.1.99: icmp_seq=3 ttl=64 time=0.481 ms
--- 192.168.1.99 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.168/0.400/0.510/0.136 ms

root@s-2-VM:~# ping -c 4 -T30 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
--- 192.168.1.1 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss

EXPECTED RESULTS

VMs to reach external network

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

@Overmorrow1501
If you use Basic zone, have you added proper security group rules ?

[rajujith]

@Overmorrow1501 can you run tcpdump on SSVM for all interfaces and ping the gateway?