Cross-cluster traffic is being dropped due to VXLAN loopback detection - all nodes assigned same vx-submariner MAC as gateway

[Lavaburn]

Description

All vx-submariner VXLAN interfaces — both on the gateway node and on all worker nodes — are assigned the same MAC address. This MAC appears to be derived from the gateway's endpoint IP.

The Linux VXLAN kernel driver includes a loopback detection check in vxlan_rcv(), causing these packets to be dropped. The kernel sees src MAC == local interface MAC → silently drops the packet as a loopback.

Environment

Submariner version: v0.20
CNI: Calico (VXLAN mode, vxlan.calico)
Tested on Scaleway multi-AZ nodes in same private network (172.16.0.0/22)

Testing

On the gateway:
ip -s link show vx-submariner # RX counter frozen despite active workloads
bridge fdb show dev vx-submariner # shows entries for worker IPs
ip link show vx-submariner # note MAC

On any worker (different AZ):
ip link show vx-submariner # same MAC as gateway — the bug
Packet capture on gateway ens7 (or equivalent private NIC) shows UDP/4800 packets do arrive from workers with correct VNI=100 but vx-submariner RX never increments.

Workaround

On each non-gateway node, set a unique MAC for vx-submariner (e.g. derived from the node's private IP):
On node 172.16.0.14:
ip link set vx-submariner address a6:65:ac:10:00:0e
On node 172.16.0.2:
ip link set vx-submariner address a6:65:ac:10:00:02

[yboaron]

Hello @Lavaburn, thanks for reporting this issue!

I'm trying to understand the root cause, but I'm confused about the MAC addresses.

Based on Submariner's code, I would expect to different MAC :

Gateway node (pkg/routeagent_driver/handlers/kubeproxy/vxlan.go:56-66):

attrs := &vxlan.Attributes{                                                                                                                                               
    VxlanID:  100,                     
    Group:    nil,          // No group IP
    VtepPort: 4800,
}                                                                                                                                                                         
              
Worker nodes (pkg/routeagent_driver/handlers/kubeproxy/vxlan.go:88-94):                                                                                                   
attrs := &vxlan.Attributes{            
    VxlanID:  100,         
    Group:    gatewayNodeIP,  // Gateway's IP
    VtepPort: 4800,                                                                                                                                                       
}                                                                                                                                                                       
                                                                                                                                                                          
Since the gateway has Group: nil and workers have Group: gatewayNodeIP, the Linux kernel should generate different MAC addresses for these interfaces.  Also submariner's intra-cluster VXLAN traffic direction is worker node --> gateway node                                                                                                                                                                            

Can you please share the actual MAC addresses you're seeing on each node?
I'll also try to reproduce this in my local environment.