Virtual MikroTik CHR and OSPF: building redundant VPN networks

The usual way to connect a dozen branch offices to a data centre is a pair of expensive routers at the hub, static routes everywhere, and a person who remembers which tunnel carries which subnet. It works until a tunnel drops at 2am and the static route keeps pointing at it.

There is a cheaper and considerably more robust arrangement: RouterOS as virtual machines on the hypervisors you already run, physical MikroTik boxes at the far ends, and OSPF deciding the paths instead of a spreadsheet. This is how we build it and what goes wrong along the way.

Why CHR instead of a router at the hub

CHR is RouterOS packaged as a virtual machine. On a hypervisor that already has spare capacity, the hub router stops being a hardware purchase and becomes another VM — with everything that implies: snapshots before a change, a second instance on a different physical host for redundancy, and rebuild from an image instead of an RMA.

The trade is throughput and licensing. CHR speed is capped by licence tier, and the free tier is limited to 1 Mbit/s per interface, which is fine for a lab and nothing else. Size the licence to the aggregate tunnel throughput, not to the size of the host.

Check licence tier against real peak throughput before designing around CHR. Nothing in the configuration will tell you that you have hit the cap — it simply stops going faster, and it looks like a network problem.

The shape of the network

Two CHR instances at the hub, on separate physical hosts. Each branch runs a physical MikroTik and builds a tunnel to both. OSPF runs inside the tunnels, so every device learns routes rather than being told them.

The important property: no static routes between sites. A branch advertises its own subnets into OSPF, the hub advertises the data centre subnets, and every device works out reachability. Adding a branch means configuring that branch, not editing every other device.

Choosing the tunnel

For new builds we default to WireGuard: less state, less CPU, and it recovers from a changed public IP without renegotiation drama. GRE over IPsec remains the right answer when the far end is older hardware, or when a device on the path treats WireGuard's UDP unkindly.

The WireGuard and OSPF gotcha

WireGuard decides where to send a packet from allowed-address, and OSPF multicasts hello packets to 224.0.0.5. If allowed-address only lists the peer's tunnel IP, the hellos are dropped and the adjacency never forms — with no error anywhere, which is what makes this cost an afternoon.

Either widen allowed-address, or run OSPF in unicast by defining neighbours explicitly. Explicit neighbours are more verbose and easier to reason about at hub-and-spoke scale.

MTU is where most of these builds fail

This is the single most common cause of a network that "mostly works". Small packets pass, the tunnel looks up, ping succeeds — and then file transfers stall, HTTPS hangs partway, and someone blames the application.

Every layer takes bytes: WireGuard about 60, GRE 24, IPsec more depending on cipher, and any of it on top of PPPoE takes another 8. Stack two and the effective MTU falls well below 1500 while both ends still believe they can send 1500.

# find the real path MTU from the branch, do not assume
/ping 10.0.0.1 size=1500 do-not-fragment count=5    # expect failure
/ping 10.0.0.1 size=1400 do-not-fragment count=5
/ping 10.0.0.1 size=1380 do-not-fragment count=5    # narrow down

# then clamp TCP MSS so sessions negotiate a size that fits
/ip firewall mangle
add chain=forward protocol=tcp tcp-flags=syn action=change-mss \
    new-mss=clamp-to-pmtu out-interface-list=tunnels

Set MSS clamping on both ends before testing anything else. Half the "OSPF is unstable" reports we investigate are large LSAs failing to cross a link with the wrong MTU.

OSPF that fails over predictably

The defaults converge in tens of seconds, which is a long time for a voice call. Tighten the timers, but not to the point where a busy CPU on the branch router causes a false failover — a flapping adjacency is worse than a slow one.

# hub: advertise DC subnets, keep timers deliberate
/routing/ospf/instance
add name=core router-id=10.255.0.1 redistribute=connected

/routing/ospf/area
add name=backbone area-id=0.0.0.0 instance=core

/routing/ospf/interface-template
add interfaces=wg-branches area=backbone \
    type=ptp \
    hello-interval=3s dead-interval=12s \
    cost=10

# secondary hub advertises the same routes at a higher cost:
#   cost=20  -> preferred only when the primary path is gone

Point-to-point interface type matters. On a tunnel there is no need for a designated router election, and skipping it removes both a delay and a class of failure.

Use cost to express intent. A branch with a fibre primary and an LTE backup should have the LTE tunnel at a much higher cost, so it carries traffic only when the fibre path is genuinely down — not whenever OSPF sees a momentary improvement.

Redundancy at the hub

Two CHR instances only help if the servers behind them can use both. VRRP presents a single virtual IP as the LAN gateway; whichever CHR holds it answers, and failover is transparent to hosts that know nothing about routing.

# CHR-A
/interface vrrp
add name=vrrp-lan interface=ether1 vrid=10 priority=200 \
    preemption-mode=no
/ip address
add address=10.0.0.1/24 interface=vrrp-lan

# CHR-B: identical, priority=100
# preemption-mode=no stops the primary grabbing back the VIP
# the moment it returns, which otherwise causes a second outage

Two things to get right. Put the CHRs on different physical hosts — two VMs on one hypervisor is a redundancy diagram, not redundancy. And make sure the VRRP heartbeat crosses a path that fails independently of the one carrying traffic, or a single switch problem produces two routers both convinced they are master.

Hypervisor settings that decide performance

• •Use virtio network adapters on KVM and Proxmox. Emulated e1000 will work and will cap throughput far below the licence.
• •Give the VM enough vCPU for encryption. WireGuard is cheap per packet but not free, and a single vCPU becomes the bottleneck before the licence does.
• •Disable hardware offloads on the virtual NIC if you see checksum or fragmentation oddities. Offload plus tunnelling is a recurring source of packets that arrive corrupt.
• •Do not overcommit CPU on a host running a router VM. Steal time shows up as jitter, and jitter shows up as OSPF flapping.
• •Consider SR-IOV or PCI passthrough at higher throughput, accepting that it complicates live migration.
• •Snapshot before every configuration change. It is the main practical advantage over physical hardware — use it.

What to break before going live

Record how long each one took. "It failed over" is not a result; "it failed over in eleven seconds and the voice call dropped" is, and it tells you whether the timers need revisiting.

When this is the wrong design

Two or three sites with modest traffic do not need OSPF. Static routes with recursive next-hop checking are simpler and will be understood by whoever inherits them. Dynamic routing starts paying off somewhere around five sites, or as soon as any site has two paths.

Equally, if the hub needs multi-gigabit encrypted throughput, a physical CCR will outperform a CHR licence at comparable cost. The virtual approach wins on flexibility and recovery, not on raw forwarding rate.

Where this fits

We design and run networks in this shape — CHR on client hypervisors, physical MikroTik at the branches, OSPF doing the failover, and monitoring that alerts on a lost adjacency rather than after a user calls. That is network management work at €65/hour, or a monthly retainer when it needs watching around the clock.