Repository navigation
network: L3 routed networking without shared L2 domain #12210
Description
The required feature described as a wish
Networking without Layer 2
This proposal is to add a new networking feature to CloudStack where Instances are directly assigned /32 (IPv4) and/or /128 (IPv6) addresses without a shared Layer 2 domain.
A shared Layer 2 domain in this case would be a VLAN or VXLAN VNI where Instances shared the same Broadcast/Multicast domain and where they use a shared IP-gateway for their routing.
Layer 3
By leveraging various features of the Linux kernel, making this a KVM-only feature, we can directly route an IPv4 and/or IPv6 address to a virtual machine by using a dynamic routing protocol like BGP, but this could also work with OSPF(v3).
By eliminating the need for Layer 2 we can create a routed network where no Instance has a "network relationship" with another Instance. Every Instance has one or more routes installed in the routing table of the network and can be routed to any host at any time.
In the examples below I will use two IP-addresses:
- 2.57.57.30
- 2001:678:3a4:100::80
Hypervisor host as gateway
On the hypervisor the cloudbr0 bridge will be created and assigned an IPv4 and IPv6 address:
auto cloudbr0
iface cloudbr0 inet static
address 169.254.0.1/32
address fe80::1/64
bridge-ports none
bridge-stp off
bridge-fd 0
All Instances will be connected to this bridge and they will be configured to use the following IP-gateways:
- 169.254.0.1
- fe80::1
Inside the Instance
As there is no Layer 2 available the IP-configuration within the VM has to be done using ConfigDrive for cloud-init, a Virtual Router handing out DHCP and cloud-init data is not possible in this design. For the VM there is no way of detecting the cloud-init source over the network as our current CloudStack provider within cloud-init relies on the DHCP server as a source.
After the Instance has used cloud-init to fetch the networking information from ConfigDrive the Netplan (Ubuntu Linux) configuration would look like this:
network:
ethernets:
ens18:
accept-ra: no
nameservers:
addresses:
- 2620:fe::fe
- 2620:fe::9
addresses:
- 2.57.57.30/32
- 2001:678:3a4:100::80/128
routes:
- to: default
via: fe80::1
- to: default
via: 169.254.0.1
on-link: true
version: 2
In this configuration the network inside the Instance is configured to the the addresses configured on cloudbr0 as the gateway, meaning that the hypervisor will act as the gateway and route the IP-traffic.
This results in the interface being configured:
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:02:45:76:d2:35 brd ff:ff:ff:ff:ff:ff
altname enp0s18
inet 2.57.57.30/32 scope global ens18
valid_lft forever preferred_lft forever
inet6 2001:678:3a4:100::80/128 scope global
valid_lft forever preferred_lft forever
All hypervisors will use an identical configuration for cloudbr0, this allows all Instances to have the same routes in their route table:
root@web01:~# ip -6 route show
::1 dev lo proto kernel metric 256 pref medium
2001:678:3a4:100::80 dev ens18 proto kernel metric 256 pref medium
fe80::/64 dev ens18 proto kernel metric 256 pref medium
default via fe80::1 dev ens18 proto static metric 1024 pref medium
root@web01:~# ip -4 route show
default via 169.254.0.1 dev ens18 proto static onlink
root@web01:~#
ARP and NDP neighbor configuration
CloudStack is aware of the IPv4 and/or IPv6 addresses assigned to an Instance as well as the MAC address. On the hypervisor these entries have to be installed into the kernel's routing table and neighbor table. In this example the commands would be:
ip -6 route add 2001:678:3a4:100::80/128 dev cloudbr0
ip -6 neigh add 2001:678:3a4:100::80 lladdr 52:02:45:76:d2:35 dev cloudbr0 nud permanent
ip -4 route add 2.57.57.30/32 dev cloudbr0
ip -4 neigh add 2.57.57.30 lladdr 52:02:45:76:d2:35 dev cloudbr0 nud permanent
These entries would need to be added upon Instance start on that host and removed on Instance stop/migrate. The KVM Agent should handle the orchestration of these entries.
Dynamic Routing
Configuring these entries in the routing table is not sufficient, these need to be advertised to the upstream network. For this the hypervisor host would need to use some form of dynamic routing. BGP is the most commonly used, while others would like to use OSPF(v3).
In both cases the hypervisor will announce these /32 (IPv4) and /128 (IPv6) addresses to the upstream network while receiving a default route (0.0.0.0/0 and ::/0) from the network to be able to route traffic.
A very simple piece of configuration for FRRouting (BGP or OSPF) could be:
BGP
router bgp
redistribute kernel route-map only-cloud
!
route-map only-cloud permit 10
match interface cloudbr0
OSPF
redistribute kernel route-map only-cloud
network YOUR_NETWORK/XX area 0.0.0.0
!
route-map only-cloud permit 10
match interface cloudbr0
IP address pools
As each Instance is assigned a IPv4 and/or IPv6 address there is no need to create a "network" inside CloudStack. The concept would be that CloudStack simply has a pool of addresses to choose from and allocates them to an Instance
A pool could be:
-
2.57.57.80
-
145.31.53.21
-
90.78.37.15
-
88.17.11.53
-
2001:db8::100
-
2001:678:3a4:100::80
-
2a00:f10:415:27::100
These addresses have no relationship with eachother, but they don't have to as each individual address is assigned the a VM
This networking setup also allows for very easy single stack IPv6-only Virtual Machines where IPv4 can be added or removed when needed. There is no dependency on either of the two protocols.
Summary
This networking design completely eliminates the use of Layer 2 broadcast/multicast domains. Each Instance becomes a full L3 routed part of the network where CloudStack's orchestration will make sure the addresses are routed towards the host where the Instance is on.
Using this setup it's very easy to create a massively scalable and reliable network spanning multiple datacenters as there is no shared L2 or VXLAN overlay.
The most common use-case for this feature will probably be public cloud providers which need to assign public IPv4/IPv6 addresses to Instance and want to share nothing between the VMs.
Activity
wido commentedon Dec 8, 2025
On my personal blog I also wrote an article about this, could be of help: https://blog.widodh.nl/2025/12/linux-bridging-with-virtual-machines-and-pure-l3-routing-and-bgp/
fnavidan commentedon Dec 8, 2025
This fashion of IP assignment can simplify network management for large public clouds. However, in some datacenters, IP ranges are delivered to servers through VLANs which would not be compatible with this setup. Also, user data is nowadays crucial for cloud application deployment. I believe, this should be implemented as a separate networking option to keep everything working smoothly as before.
One benefit for this setup would be that there would be no need to allocate a full IP subnet which sometimes costs 2 IPs to be reserved for the subneting. So individual IPs can be used for IP assignment.
wido commentedon Dec 8, 2025
I believe that when you want to use this feature because your use-case requires it that you have already designed such a network or are able to deploy it.
This is aimed at large scale.
As I wrote, we could and should use ConfigDrive for this. Using CD we can provide the VM with the required networking configuration.
Yes, exactly! You can allocate all 255 addresses of an IPv4 /24 subnet for example. With IPv6 this is no problem as there are more than enough addresses in a subnet, but with IPv4 it can be a problem. And if you run out, you can just allocate an additional /26 (example) to CloudStack and use all addresses.
joolli commentedon Dec 9, 2025
You get same/similar result with anycast gateway and EVPN-type2 (MAC+IP) routes... EVPN provides host routes for each IP, each Hypervisor can have an anycast gateway for public networks, so the VM's get routed out by their hypervisors.
This suggestion would, however, require a simpler network setup (EVPN and VXLAN not needed) and less training and easier to troubleshoot.
How does the VM connect to the outside world or to 169.254.0.1/32, rather? What type of interface/driver would a VM use?
wido commentedon Dec 10, 2025
I have extensive experience with VXLAN+EVPN and I love it, truly do! Many presentations and talks are about how this works and use it in production.
But, it's complex and you have to deal with an overlay and underlay network. Equipment which can offload it, handle it. It's not that easy to operate. This proposal is indeed much simpler and straightforward routing.
If you look at the example I posted above, these are the default routes for the VM:
I did a tracepath on this VM (running on Proxmox as a PoC) to Quad9 DNS:
joolli commentedon Dec 10, 2025
Yes, I'm just wondering how the hypervisor and VM communicate. Which network driver are you using?
wido commentedon Dec 10, 2025
Nothing special. Just virtio as a NIC, just like in any other situation. On the hypervisor there is a Linux bridge which is the IPv4/IPv6 gateway for the VM. No special driver required.
15 remaining items
wido commentedon May 22, 2026
Bumping this as this was discussed yesterday (21-05-2026) at the European User Group in the Netherlands.
Technically it will work, but it needs a new type of network as we discussed above. The issue is not IF this can work, the issue is if somebody can find the time to work on it.
wido commentedon Jun 17, 2026
I wanted to make sure that cloud-init would handle this properly. On a Proxmox node I gave this a manual test and used this network config via ConfigDrive on a Ubuntu 26.04 server:
root@cloud-init:/tmp/init# cat network-config version: 1 config: - type: physical name: eth0 mac_address: 'bc:24:11:bf:52:70' subnets: - type: static address: '2.57.57.26' netmask: '255.255.255.255' gateway: '169.254.0.1' - type: static6 address: '2a14:9b80::26/128' gateway: 'fe80::1' - type: nameserver address: - '2620:fe::fe' - '2620:fe::9' search: - 'ams02.cldin.net' root@cloud-init:/tmp/init#This resulted in the correct netplan network:
root@cloud-init:~# cat /etc/netplan/00-installer-config.yaml # This is the network config written by 'subiquity' network: ethernets: eth0: addresses: - 2.57.57.26/32 - 2a14:9b80::26/128 match: macaddress: bc:24:11:bf:52:70 nameservers: addresses: - 2620:fe::fe - 2620:fe::9 search: - ams02.cldin.net routes: - on-link: true to: default via: 169.254.0.1 - on-link: true to: default via: fe80::1 set-name: eth0 version: 2 root@cloud-init:~#The proper IPv4 and IPv6 routes were installed:
This is still on my whishlist to develop should I be able to find the time for it.
wido commentedon Jun 26, 2026
I tested this further with Proxmox and have created a personal repo with a hook to make this work: https://github.com/wido/proxmox-l3-routing/
Works flawlessly for me. I will continue testing.
wido commentedon Jul 31, 2026
I can update that I'll soon open a PR for this issue. Work in progress. Would love to get some early feedback while I'm still testing and developing.
DaanHoogland commentedon Jul 31, 2026
by the look of the subject @kiranchavala is your guy (but I know he is very busy atm)
wido commentedon Jul 31, 2026
Would be great, but no rush. Current code is here: https://github.com/wido/cloudstack/commits/direct-routed-network/
Not ready yet, still have to polish and check.
wido commentedon Jul 31, 2026
PR (WIP) is ready: #13758
Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to…
Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to…
Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to…
Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to…
Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to…