OpenVPN, iroute, iproute2 Policy-based routing and gateways on different subnets

[guru431]

Help me to solve the problem.

Below is a simplified schematic of my network.
(Actually everything is much larger, but it will be enough to understand the problem)


There are 3 local networks - 192.168.1.0/24 192.168.2.0/24 192.168.3.0/24.
They are interconnected using OpenVPN (10.10.10.0/24)
Any of the hosts can ping all hosts on any network, and all addresses from the 10.10.10.0/24 network as well

It is done like this:

# server-tcp.conf
proto tcp
dev tun
topology subnet
mode server
server 10.10.10.0 255.255.255.0
push "redirect-gateway def1"
push "dhcp-option DNS 10.10.10.1"
push "route 192.168.0.0 255.255.240.0"
push "route 10.10.48.0 255.255.240.0"
route 192.168.2.0 255.255.255.0
route 192.168.3.0 255.255.255.0
client-to-client
ifconfig-pool-persist    /etc /openvpn/ipp.txt 0
client-config-dir        /etc /openvpn/ccd

# ipp.txt
client2,10.10.10.10
client3,10.10.10.30

# ccd\client2
iroute 192.168.2.0 255.255.255.0

# ccd\client3
iroute 192.168.3.0 255.255.255.0

# MikroTik R1
/ip route
add distance=1 dst-address=10.10.10.0/24 gateway=192.168.1.10
add distance=1 dst-address=192.168.2.0/24 gateway=192.168.1.10
add distance=1 dst-address=192.168.3.0/24 gateway=192.168.1.10

# clients config
remote 201.201.201.201 443 tcp
dev tun
client
pull
pull-filter ignore redirect-gateway

Now I need to implement a scheme where any host from any subnet can be output through any external address.

At the moment I am able to bring computers from networks 192.168.2.0/24 o r 192.168.3.0/24 through the external address ISP1.
I do this using packet marking and iproute2 on OpenWRT routers:

iptables -t mangle -A PREROUTING -i br-lan -p tcp -m set --match-set ip_to_forward dst -j MARK --set-mark 0x1
ip route add 192.168.1.0/24 dev tun0 table 100
ip route add default via 192.168.1.1 table 100
ip rule add fwmark 0x1 table 100
ip route flush cache

What can I do to get computers out of the 192.168.1.0/24 network via ISP2 and ISP3?

I was envisioning also using MikroTik and iproute2 on the OpenVPN server:

# MikroTik
/routing/table/add nam e=route_to_R2 fib
/ip route add distance=1 gateway=192.168.1.10 routing-table=route_to_R2
/ip address add address=192.168.1.161/24 interface=bridge network=192.168.1.0
/ip firewall nat add action=src-nat chain=srcnat place-before=0 to-addresses=192.168.1.161 routing-mark=route_to_R2
/ip firewall mangle add action=mark-routing chain=prerouting comment="Mark comp to R2" dst-address-list=!my_int_ip in-interface-list=LAN new-routing-mark=route_to_R2 passthrough=no src-address-list=comp_to_R2
/ip firewall address-list add address=192.168.1.11 list=comp_to_R2

# Server OpenVPN
ip route add 192.168.2.0/24 dev tun0 table 131
ip route add default via 192.168.2.1 table 131
ip rule add from 192.168.1.161/32 table 131
ip route flush cache

MikroTik works fine and packets arrive at the OpenVPN server. PBR redirects to table 131
But OpenVPN does not route traffic to the client specified as the gateway.

In my tests, I realized that OpenVPN ignores the gateway in my table altogether.
I can only redirect traffic to its interface

ip route add default dev tun0 table 131

And then it decides how to route it itself using its internal routing.

And the internal routing works based on iroute in the clients ccd settings.
So for traffic to go to the client I need to add all internet subnets to ccd\client2:

iroute 0.0.0.0 248.0.0.0
iroute 8.0.0.0 254.0.0.0
iroute 11.0.0.0 255.0.0.0
iroute 12.0.0.0 252.0.0.0
iroute 16.0.0.0 240.0.0.0
iroute 32.0.0.0 224.0.0.0
iroute 64.0.0.0 192.0.0.0
iroute 128.0.0.0 224.0.0.0
iroute 160.0.0.0 248.0.0.0
iroute 168.0.0.0 255.0.0.0
iroute 169.0.0.0 255.128.0.0
iroute 169.128.0.0 255.192.0.0
iroute 169.192.0.0 255.224.0.0
iroute 169.224.0.0 255.240.0.0
iroute 169.240.0.0 255.248.0.0
iroute 169.248.0.0 255.252.0.0
iroute 169.252.0.0 255.254.0.0
iroute 170.0.0.0 254.0.0.0
iroute 172.0.0.0 255.240.0.0
iroute 172.32.0.0 255.224.0.0
iroute 172.64.0.0 255.192.0.0
iroute 172.128.0.0 255.128.0.0
iroute 173.0.0.0 255.0.0.0
iroute 174.0.0.0 254.0.0.0
iroute 176.0.0.0 240.0.0.0
iroute 192.0.0.0 255.128.0.0
iroute 192.128.0.0 255.224.0.0
iroute 192.160.0.0 255.248.0.0
iroute 192.169.0.0 255.255.0.0
iroute 192.170.0.0 255.254.0.0
iroute 192.172.0.0 255.252.0.0
iroute 192.176.0.0 255.240.0.0
iroute 192.192.0.0 255.192.0.0
iroute 193.0.0.0 255.0.0.0
iroute 194.0.0.0 254.0.0.0
iroute 196.0.0.0 252.0.0.0
iroute 200.0.0.0 248.0.0.0
iroute 208.0.0.0 240.0.0.0
iroute 224.0.0.0 224.0.0.0

But that's not right.
What should I do if I want to make 2 PBR tables - to clien2 and client3?
I won't be able to add the same subnets to different ccd. Only the last one will be processed.

So the question is how can this all be realized?
Why OpenVPN does not work properly with iproute2 policy based routing?
Why I can normally route traffic through 192.168.2.0/24 - client ovpn - server ovpn - 192.168.1.0/24 - ISP1, but I can not in the opposite direction 192.168.1.0/24 - server ovpn - client ovpn - 192.168.2.0/24 - ISP2?
I can access any host on any network without any problems.

[guru431]

Found the answer elsewhere.

With tun, this "ip route add default via 192.168.2.1 table 131" does not work as you expect.
It is actually the same as "ip route add default dev tun0 table 131".
When you send a packet to 8.8.8.8 the ovpn server doesn't know which client to send it to, there can be more than one client.

With tap (it is L2) normal routing will work.

It's sad, I'm not ready to switch to tap.