Linux Advanced Routing: Setting up a Mixed Public-Private Network

Recently I had a unique need to have a mix of public and private network on a particular server for some testing.  A number of services were already configured for the public interface. I had to test a particular feature using a NAT environment and the easiest I could think of was to configure the same server with a NAT ifc in the VMWare environment and configure that feature to use this private interface.  Setting up the proper routes where I can reach the server through the public interface or through the router’s port forwarding via the NAT interface was a challenge in this case. 

My networking requirement is something like this.  As the diagram suggests, 164.99.89.77 is the public interface (eth1) and 172.17.2.80 (eth0) is the private interface.  vmnet5 provides the NAT environment with the network 17217.2.0.  My requirement was to reach the guest via eth0 or eth1 from the 164.99 network.  The host (164.99.89.74) also provides port forwarding so that I can connect to the gust via the private interface. 

I realized that I need to make sure that all answers to traffic coming in on a particular interface get answered from that interface. 

After a little research on Linux advanced routing, I stumbled upon this page.

I designed my routing table based on the recommendations from there.  I’m listing the steps I followed for future reference.

1. Disable reverse-path filtering for both interfaces.  When source and destination traffic to the same IP using different interface occurs, the Linux kernel drop the traffic as potentially spoofed.  This is called reverse-path filtering. 
2. Create two additional routing tables, say T1 and T2 in /etc/iproute2/rt_tables.   This file will look something like this

3. Then populate these tables as given below

   ip route add 164.99.0.0 dev eth1 src 164.99.89.77 tabel T1
   ip route add default via 164.99.89.254 table T1
   ip route add 172.17.2.0 dev eth0 src 172.17.2.80 table T2
   ip route add default via 172.17.2.2 table T2

   164.99.0.0 => public network
   164.99.89.77 => IP address of the public interface
   164.99.89.254 => Gateway address for the public network
   172.17.2.0 => Private network
   172.17.2.80 => IP address of the private interface
   172.17.2.2 => Gateway address for the public network

4. Set up the main routing table.

   ip route add 164.99.0.0 dev eth1 src 164.99.89.77
   ip route add 172.17.2.0 dev eth0 src 172.17.2.80

5. Then a preferred default route

   ip route add default via 172.17.2.2

6. Next set up the routing rules

   ip rule add from 164.99.89.77 table T1
   ip rule add from 172.17.2.80 table T2

Above rules will make sure all answers to traffic coming in on a particular interface get answered from that interface 

My routing table looks something like this with the above changes

There are a few more desirable routing additions mentioned here. 

With these changes, I can connect to the server via the public interface or via the private interface with the port forwarding in the router.