My Tinc setup after upgrading from FreeBSD 8.2 to 9.0

The problem that tinc solves

You've got multiple (virtual?) machines on different networks and you want to treat them as if they are all part of the same (private) network. The front-end doesn't matter so much as the backend communication and you want to keep that secure and seamless.

On the light-weight/quick-and-dirty side you can try to establish a point to point connection between all of the hosts with something like PPP, SLIP, or SLIRP but you'd have no encryption. You could try tunneling PPP, SLIP, or SLIRP over SSH but trust me that's a whole down which you do not want to follow the rabbit, even if it does have a shiny pocket watch.

You could go the centralized route and use something like OpenVPN or FreeS/WAN and make every server connect to the centralized server hoping that it doesn't go down. You could even make two or more central servers for redundancy but then you'd have a few heavily laden nodes passing traffic for everyone else. Not to mention that network paths likely would be less than ideal in many cases.

The best solution I've found is to this problem is a secure-mesh. With mesh networking you add and remove nodes as needed. Any node can be connected to as many other nodes as it desires on a peer-to-peer basis. Traffic between nodes not directly connected is handled by either routing through other nodes or having the whole (relevant part of the) mesh act as a bridged network.

Tinc does this for you. It has a very low resource profile, is scalable, and is highly configurable with sensible defaults so you don't have to spend all day reading up on every option. Another key feature is resilience. Once two nodes have accepted each other as peers you let them decide which one will initiate or re-establish the connection to the other (this is great when one is behind a NAT device). One connection between peers is all it takes, you don't have to have one from a to b and then another from b to a.

More information on tinc at http://www.tinc-vpn.org

A bit about my setup

I have multiple machines out there on the intertubes. Some are located in Colorado some in Florida and a few in California and Georgia. I won't detail the setup for all of them. Two should be sufficient to convey the basic idea.

rc.conf for host 1:

hostname="host1.example.com"
ifconfig_bge0="inet 1.2.3.4 netmask 255.255.0"
defaultrouter="1.2.3.1"


###################################
####Loopback Clones and Aliases####
###################################
cloned_interfaces="lo1 lo2 lo3 tap0 tun0"
ifconfig_lo1="inet 192.168.41.1 netmask 255.255.255.0"
ifconfig_lo2="inet 192.168.42.1 netmask 255.255.255.0"

# mesh services 
ifconfig_lo1_alias0="inet 192.168.41.2 netmask 255.255.255.255"

ifconfig_lo1_alias2="inet 192.168.41.3 netmask 255.255.255.255"

rc.conf for host 2:

hostname="host2.example.com"

ifconfig_bge0="inet 2.3.4.5 netmask 255.255.0"

defaultrouter="2.3.4.1"

###################################

####Loopback Clones and Aliases####

###################################

cloned_interfaces="lo1 lo2 lo3 tap0 tun0"

ifconfig_lo1="inet 192.168.51.1 netmask 255.255.255.0"

ifconfig_lo2="inet 192.168.52.1 netmask 255.255.255.0"

# mesh services 

ifconfig_lo1_alias0="inet 192.168.51.2 netmask 255.255.255.255"

ifconfig_lo1_alias2="inet 192.168.51.3 netmask 255.255.255.255"

Note that 1.2.3.4 (the external ip for host1) is on a different network than host 2 which has 2.3.4.5. Also not that I have loopback interfaces with privates IPs. I picked 192.168.[54][0-9] for this example. This is noted here because it's really these interfaces that need to talk to each other in my setup. I have Jails that run on those interfaces and some jails on host1 need to talk to jails on host2 and vice versa.

For the sake of this example let's pretend that we have DNS running on both 192.168.51.2 and 192.168.41.2. We'll say that host1's DNS publishes info for *.host1.example.com and that host2 does likewise for *.host2.example.com. If they are going to resolve entries on the other host they need to be able to connect to that hosts name server and we'll do that over the tinc-vpn.

Tinc-ering

There are many considerations in any network setup. VPN and mesh do not negate these, they compound them. I offer here an example of one possible configuration based on my experience with my own setup. You will have different considerations. I strongly recommend you read through the tinc documentation. It's exceedingly simple and not terribly verbose.

tinc.conf

After installing tinc you'll need to create a directory based on your mesh-network under /usr/local/etc/tinc. You'll also need to keep several files in sync between the hosts. For this example we'll use /usr/local/etc/tinc/example. The main configuration file will be /usr/local/etc/tinc/example/tinc.conf

Host1's tinc.conf:

Name = host1
Device = /dev/tap0
AddressFamily = ipv4
BindToAddress = 1.2.3.4
Forwarding = kernel
ConnectTo = host2
PrivateKeyFile = /usr/local/etc/tinc/example/rsa_key.priv

Host 2's tinc.conf:

Name = host2
Device = /dev/tap0
AddressFamily = ipv4
BindToAddress = 2.3.4.5
Forwarding = kernel
ConnectTo = host1
PrivateKeyFile = /usr/local/etc/tinc/example/rsa_key.priv

Note: here I'm using tap0 because it offers the most flexibility with regard to how you setup your mesh-network. With tap devices you can operate in bridge mode which is basically like being plugged into the same ethernet segment. You then (based on your ip layout and net masks) have the option to choose either routed or bridged operation for the nodes and virtual hosts on your mesh. Alternately you can use tun0 if you don't care to have a bridged network. With tun devices there is less traffic (no ethernet headers).

Note: The 'Forwarding = kernel' line tells tinc to pass packets through the kernel for delivery which means you'll need to have you firewall properly configured if you want to see traffic go from one tinc peer to the other successfully. I will not go over firewall configuration here. I suggest you start with a very open ruleset to verify that the link is working and then do whatever you need to do from there.

Host configuration

In addition to the main config (tinc.conf) there is a host specific configuration file that needs to be on both hosts and located in the hosts/ directory under /usr/local/etc/tinc/example/. 

contents of /usr/local/etc/tinc/example/hosts/host1:

Address = 1.2.3.4

Subnet = 192.168.41.0/24

Subnet = 192.168.42.0/24

Port = 655

-----BEGIN RSA PUBLIC KEY-----

blahblahblahblahblahblahblahblahblah

-----END RSA PUBLIC KEY-----

contents of /usr/local/etc/tinc/example/hosts/host2:

Address = 2.3.4.5

Subnet = 192.168.51.0/24

Subnet = 192.168.52.0/24

Port = 655

-----BEGIN RSA PUBLIC KEY-----

yadayadayadayadayadayadayadayadayada

-----END RSA PUBLIC KEY-----

On host1 the host1 file is used for options that are specific to host1 but are shared with other hosts. As well, the host2 file is used on host1 to make host1 aware of it's specific configuration (e.g. the subnets available through the connection to it: .51.0 and .52.0). Also the public key's are used for authentication in combination with the PrivateKeyFile specified in tinc.conf. The tinc docs cover generating keys.

up/down files

Tinc has a series of scripts that can be executed when a link is established or before shutting a connection down. The main one to worry about is the tinc-up. This file is executed once the two tinc daemons have established a peer relationship. This is usually useful for setting up routes, stopping or starting service, or possibly modifying firewall configurations. 

Host1's /usr/local/etc/tinc/example/tinc-up:

#!/bin/sh

ifconfig $INTERFACE 192.168.40.1 netmask 255.255.255.0

route add -host 192.168.50.1 192.168.40.1

route add -net 192.168.51.0/24 192.168.50.1

route add -net 192.168.52.0/24 192.168.50.1

Host2's /usr/local/etc/tinc/example/tinc-up:

#!/bin/sh

ifconfig $INTERFACE 192.168.50.1 netmask 255.255.255.0

route add -host 192.168.40.1 192.168.50.1

route add -net 192.168.41.0/24 192.168.40.1

route add -net 192.168.42.0/24 192.168.40.1

What these scripts do is assign an ip address to the tap0 interface (or tun0 if that's what you chose to specify in the tinc.conf). The IP used on the $INTERFACE needs to be unique to that node of the mesh and not on any other interface. Next we establish a route to the other end of the vpn-link with the route add -host line(s). The makes sure that your localhost can route to the machine on the other side. The final two lines establish routes to the networks on the two loop back interfaces where the DNS servers live. 

Note: When I was on FreeBSD 8.2 I didn't haven't to explicitly specify the other end of the link. It took me a while to figure out after the upgrade to 9.0 that I need to add a host route. 

Note: With the last to lines, by setting the route for accessing the loopback hosted services to the 'far end' of the link I'm doing direct routing which is ok for a simple setup. When you get more hosts added it's probably wise to set the route to the 'near end' and let the mesh route in a 'most available' manner. 

Troubleshooting

If you're having problems, you should fire up tinc in debug mode on both sides of the link:


/usr/local/sbin/tincd -n example -d5 -D

Try using ping. You should see messages about packets being passed on both nodes at roughly the same time. If not check your firewall rules. First check to make sure that you're not blocking traffic from going out over your $INTERFACE and then make sure that you're allowing traffic to come in.