First of all, I would like to thank you for all the previous answers everywhere? I couldn't get there without them. Learned a lot from the Stackexchange community (both about iptables and more).
Nevertheless. Setup that I deal with consists of:
- laptop running Kubernetes (with Calico CNI) (behind non-public IP unfortunately) (LAPTOP)
- VM running in Digital Ocean, that serves as TCP proxy to the cluster (VM)
In order to work around the lack of public-IP, I'm trying to use OpenVPN with custom route/iptables settings to the route:
- traffic coming from the internet to VM (over enp8s0) through OpenVPN to LAPTOP (over tun0) only on ports 443, 6443, and 80
- traffic coming out of LAPTOP to the internet through VM only on ports 443 and 80.
1st part works flawlessly with the script below running at the upstart of the server:
#!/bin/bashset -esysctl -w net.ipv4.conf.tun0.rp_filter=2;ip rule add fwmark 2 table 3ip route add default via 10.8.0.2 table 3ip route flush cacheiptables -t mangle -A OUTPUT -p tcp -s 10.8.0.1 -m multiport --dports 6443,443,80 -j MARK --set-mark 2iptables -t nat -A POSTROUTING -o tun0 -j SNAT --to-source 10.8.0.2
All traffic coming from 10.8.0.1
over 6443,443,80 gets marked with 2
and goes to table 3
that's default rule goes through 10.8.0.2.
The 2nd point is problematic for me. No matter what I tried (more below) I can't get it to work.
Methods:
- Extending script with PREROUTING configuration, that slingshots the traffic through an interface:
#!/bin/bashset -esleep 20sysctl -w net.ipv4.conf.tun0.rp_filter=2;ip rule add fwmark 2 table 3ip route add default via 10.8.0.2 table 3ip rule add fwmark 4 table 5 # adding rule/route associationip route add default via 10.8.0.1 table 5 # adding default via for tableip route flush cacheiptables -t mangle -A OUTPUT -p tcp -s 10.8.0.1 -m multiport --dports 6443,443,80 -j MARK --set-mark 2iptables -t nat -A POSTROUTING -o tun0 -j SNAT --to-source 10.8.0.2iptables -A PREROUTING -i enp8s0 -t mangle -p tcp -m multiport --dports 443,80 -j MARK --set-mark 4 # mark ports during PREROUTING
I've read that it might be somewhat naive approach (because without 1st part working, some systems might miss two-way communication), but if it's going to work for my specific scenario, then I'm OK with that.
- Extending script with connection marking (so if the communication becomes reversed, it's not going through different interface):
#!/bin/bashset -esleep 20sysctl -w net.ipv4.conf.tun0.rp_filter=2;ip rule add fwmark 2 table 3ip route add default via 10.8.0.2 table 3ip rule add fwmark 4 table 5 # adding rule/route associationip route add default via 10.8.0.1 table 5 # adding default via for tableip route flush cacheiptables -t mangle -A OUTPUT -p tcp -s 10.8.0.1 -m multiport --dports 6443,443,80 -j MARK --set-mark 2iptables -t nat -A POSTROUTING -o tun0 -j SNAT --to-source 10.8.0.2# adding whole connection marking belowiptables -t mangle -A PREROUTING -p tcp -m multiport --dports 443,80 -j MARK --set-mark 4iptables -t mangle -A PREROUTING -m connmark --mark 4 -j CONNMARK --restore-markiptables -t mangle -A POSTROUTING -m mark --mark 4 -j CONNMARK --save-mark
The thing is that everytime I try to add here expected 2nd mechanism, I still get wrong public IP in return from curl https://ifconfig.me
(the one that is associated with enp8s0).
Are there any advices you can give me to deal with this problem? What am I missing there?
Another thing to remember about is that CALICO is also doing it's mambo-jambo with iptables.. I don't want to interfere with internal in-cluster communication going on ports 80 and 443. For example cert-manager
's admission controller used to check it's CRDs listens on port 443 and during investigation (for 1st point) I've broke the communication at 1st.