Setting Up CARP Interfaces on FreeBSD

Posted by Tres Thu, 05 Apr 2007 06:03:00 GMT

Introduction to CARP

CARP makes it possible to have redundant, fault-tolerant routers using off-the-shelf general computing systems. Using CARP can increase the reliability, extensibility and robustness of your core routers while decreasing the cost of ownership and lowering the barrier to entry. Older systems can easily be re-commissioned for use as routers, bringing the hardware costs to zero. I’ve seen a pair of 550 megahertz pentium III sytems provide layer 3 filtering and routing for a rack of streaming media servers in a datacenter; and the best part: the boxes weren’t even close to being fully utilized. I’ve also set up OpenBSD based CARP systems handle traffic for a site of nearly a thousand computers. These systems handle hundreds of simultaneous users working on 3D Studio Max and Maya files off of network file servers, VOIP traffic for multiple sites as well as day-to-day intranet and Internet services.

How does CARP work? Well, in short, a CARP interface is essentially a virtual interface that can be shared between multiple machines. The CARP interface is active on only one parent interface at any given time, but the virtual interface can be shifted to another parent interface very quickly in the event of a failure.

Now you’re probably saying, “that sure sounds a lot like Cisco’s VRRP!” Well, that’s because it is just like VRRP (Virtual Router Redundancy Protocol). CARP came in to being as an open source implementation of VRRP using VRRP as the spec. But get this–CARP not only performs the same kind of functionality as Cisco’s VRRP , it can also be configured to act like Cisco’s GLBP (Gateway Load Balancing Protocol) (although, technically, the CARP guys had their own implementation of GLBP working before Cisco did). Load-balanced routing with included failover capabilities is very exciting–I know–but GLBP and load-balancing CARP is a subject we’ll leave for another day. For now, let’s take a look at setting up a CARP interface on FreeBSD using rc.conf.

Setting Up CARP interfaces on FreeBSD

There are three basic elements to setting up CARP failover interfaces to boot on FreeBSD:

First, in /etc/rc.conf, tell FreeBSD that you’re creating a “cloned interface.” Unless the interface is included in the list of virtual interfaces, it will never be created by FreeBSD.

cloned_interfaces="carpXX"

The cloned_interfaces directive can contain as many virtual interfaces as you need separated by a space:

cloned_interfaces="carpXX carpYY carpZZ"

Then add the configuration directives for your cloned interfaces to /etc/rc.conf.

ifconfig_carpXX="vhid <carp-interface-id> advskew <priority> pass <password> <carp-ip-address>"

and then in /etc/sysctl.conf, tell the kernel that you want to allow CARP interfaces to preempt each other:

net.inet.carp.preempt=1

This directive also means that once any single interface fails on a router, all virtual CARP interfaces will be shifted to the failover router at once.

The Four Elements of The Interface Definition

The ifconfig_carpXX line is made up of four elements.

  1. vhid <carp_interface_id> the virtual interface ID
  2. advskew <priority> the priority of the interface
  3. pass <password> the password used by hosts to validate the CARP relationship
  4. <carp_ip_address> the IP address of the virtual interface.

ifconfig_carpXX="1==>vhid <carp_interface_id>     2==>advskew <priority>     3==>pass <password>     4==><carp_ip_address>"

That’s all there is to it. You won’t need to worry about where or how the virtual interface attaches, FreeBSD is smart enough to know which parent interface it’s supposed to use by the IP address assigned to the CARP interface. The only real bookkeeping you’ve got to do is to make sure that:

  1. the VHID on your primary and failover routers match, and
  2. that the priority of the primary is lower than the failover.

Example

In the following example uses two FreeBSD routers called primo (our primary router) and secundo (our failover router) in a failover configuration:

primo, our primary router

/etc/rc.conf

ifconfig_em0="inet 99.88.77.70/29" # external interface for primo, our primary router

ifconfig_em1="inet 10.11.12.254/24" # internal interface for primo

defaultrouter="99.88.77.65"

gateway_enable="YES"

cloned_interfaces="carp1 carp2"

ifconfig_carp1="vhid 1 advskew 50 pass BigPassword 99.88.77.66" # external CARP interface for primo

ifconfig_carp2="vhid 2 advskew 50 pass BigPass2W 10.11.12.1" #internal CARP interface for primo

secundo, our failover router

/etc/rc.conf
ifconfig_em0="inet 99.88.77.69/29" #external interface for secundo, our failover router

ifconfig_em1="inet 10.11.12.253/24" # internal interface for secundo

defaultrouter="99.88.77.65"

gateway_enable="YES"

cloned_interfaces="carp1 carp2"

ifconfig_carp1="vhid 1 advskew 80 pass BigPassword 99.88.77.66" #our external CARP interface for secundo

ifconfig_carp2="vhid 2 advskew 80 pass BigPass2W 10.11.12.1" # internal CARP interface for secundo

Now I’m sure you’ve already noticed, but I want to reiterate it; the only difference in CARP interface definitions between the two hosts is the advskew assigned to each. This is how the priority of the router is assigned.

Final Notes

A practical consideration when you’re deciding whether CARP is the right choice is that any system is limited to 255 CARP interfaces. This normally is a non issue, but if you expect your network to grow beyond 128 VLANs and want to use load balanced routers, it will become an issue, so keep it in mind. (Of course, there’s a point of diminishing returns when trying to put too many segments on a single router or set of failover routers is more management overhead than just setting up another set.

Also, keep in mind that this covers only configuration of the CARP interfaces themselves. If you plan on doing any packet filtering with PF, you’ll want to set up a pfsync interface. But that’s another story.

Posted in , ,  | Tags , , , , , , ,

Using Interface Groups in OpenBSD

Posted by Tres Tue, 26 Dec 2006 02:51:00 GMT

Another one of the brilliant ideas to come out of OpenBSD is the interface group.

OpenBSD’s interface groups combined with VLAN interfaces are the one of the best tools available to simplify the management and organization of your rulesets. By using interface groups, you can cut out 99% of the redundancy required for different networks. The best part of interface groups from a management perspective is that they allow you to see instantly what rules are being used on any given interface. It’s like a heads-up display of what rules are applied to any subnet on the network.

Using interface groups is very easy. You can add an interface to a group by using the “group” option of ifconfig. So to add interface vlan1024 to the DEVELOPER interface group, I would do this:


ifconfig vlan1024 group DEVELOPER

If I wanted to remove interface vlan1024 from the DEVELOPER interface group, I would do this:


ifconfig vlan1024 -group DEVELOPER

To add a group to a particular interface on boot just add the “group” syntax to the hostname.if file for the vlan group. So to add DEVELOPER rules to the vlan1024 interface at boot, your /etc/hostname.vlan1024 file would add an additional line to the end of the file like this:

10.18.24.254 netmask 255.255.0.0 broadcast 10.18.24.255 vlan 1024 vlandev em2
group DEVELOPER

You can add as many groups to an interface as are necessary, just keep adding each group to the interface on a separate line and use the “group” syntax:

10.18.24.254 netmask 255.255.0.0 broadcast 10.18.24.255 vlan 1024 vlandev em2
group DEVELOPER
group IMAP  
group SMTP
group POP3
group HTTP
group INTERNET

There’s no extra configuration required to get PF to use the group. All you need is to set up your rules for the interface group. Rules should have as fine of granularity as possible. Remember, you’re trying to show exactly what rules are being used by the interfaces. You should use use super-classes to define more abstract groupings sparingly. It can lead to confusion if a rule for a service is defined in a superclass because it is in use by all hosts on the network. It is better to use service based group interface rules that are applied to all interfaces. Then there is no confusion. So, what does our service-based group interface ruleset look like?


pass in quick proto tcp on INTERNET from <our_ips> to !<our_ips> keep state flags S/SA
pass out quick proto tcp on INTERNET from <our_ips> to !<our_ips> modulate state flags S/SA

That’s it. Of course these rules imply that you’re using tables to manage addresses.

So, you can add a base set of rules to any interface that you’ve defined. If you have your VLANs and switchports set up so things are managed by access levels, interface groups make things brilliantly easy. Just plug in the rules that apply to that group and you’re done. If you have mixed networks in which only a few workstations get access, don’t despair. Using address tables in conjunction with the group will allow use of interface groups. Just define the IP addresses in the relevant table, and as long as you’ve written your rules to only use the table you’re ready.

Posted in , ,  | Tags , , ,

Setting Up VLAN Interfaces in FreeBSD

Posted by Tres Tue, 04 Jul 2006 14:30:00 GMT

In order to set up VLAN interfaces on FreeBSD, first you’ll need to either use a kernel module by adding:

if_vlan_load="YES"

to loader.conf

or by adding 

device          vlan
to your kernel configuration file before you recompile your kernel so that the kernel knows how to speak 802.1q. Once your kernel is 802.1q aware, the next step is to set up rc.conf for vlan interfaces. set up rc.conf so that any vlan interfaces are listed under the “cloned_interfaces” directive 
cloned_interfaces="vlan100 vlan200 vlan300 vlan482"

make sure that the parent device is up and running. If you don’t have an IP address that you’re associating with the device natively, then you should just make sure that it’s up.

ifconfig_fxp0="up"

and finally, set up the interface details in rc.conf

ifconfig_vlan100="inet 102.203.98.97/28 vlan 100 vlandev fxp0"
ifconfig_vlan200="inet 102.203.98.24/28 vlan 200 vlandev fxp0"
ifconfig_vlan300="inet 102.203.98.10/29 vlan 300 vlandev fxp0"
ifconfig_vlan482="inet 102.203.98.223/28 vlan 482 vlandev fxp0"

Once you’ve got all these items configured, your VLAN interfaces should start up on boot. The only other thing you’ll need to do before you can use your VLAN interfaces on the FreeBSD box is to configure the switch for vlan trunking and 802.1q encapsulation. But that’s another story.

Posted in , ,  | Tags , ,  | no comments