Read error: Errno::EPERM: Operation not permitted When Using Mongrel

Posted by Tres Thu, 17 Jul 2008 08:23:00 GMT

It’s always the little things…

So I’m setting up rails and nginx on my production server, everything is humming along. I set up monit to start and monitor the mongrel instances, set up nginx and set up the reverse proxy, and then I try accessing my app and see this in the logs:

Wed Jul 16 15:34:05 -0700 2008: Read error: #
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:137:in `write'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:137:in `write'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:95:in `send_header'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:146:in `finished'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:165:in `process_client'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `new'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `new'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:in `each'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281

Since I’m running this in a FreeBSD Jail, I assume that it’s access to something in /dev. I look at the rails installation that was done via ports and start upgrading gems and looking at all the easy stuff.

Finally, I look at 

/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:137:in `write'

Here’s the function Mongrel is complaining about:

  def write(data)
      @socket.write(data) # <== here's line 137
      rescue => details
      socket_error(details)
    end

Ye olde firewall problem…

So I had some rules set up to pass all internal traffic across my jailed servers on my FreeBSD server through the firewall, something like this:

pass quick all on $external_interface from self to any modulate state

This was to be a little “safety rule;” something that would keep things open even if something was added later on to block a set of ports globally.

This was setting up a race condition with the state table and ultimately had the effect of almost always killing mongrel connections both directly and through the nginx load balancer.

Posted in , , ,  | Tags , , ,

Setting Up CARP Interfaces on FreeBSD

Posted by Tres Thu, 05 Apr 2007 06:03:00 GMT

Introduction to CARP

CARP makes it possible to have redundant, fault-tolerant routers using off-the-shelf general computing systems. Using CARP can increase the reliability, extensibility and robustness of your core routers while decreasing the cost of ownership and lowering the barrier to entry. Older systems can easily be re-commissioned for use as routers, bringing the hardware costs to zero. I’ve seen a pair of 550 megahertz pentium III sytems provide layer 3 filtering and routing for a rack of streaming media servers in a datacenter; and the best part: the boxes weren’t even close to being fully utilized. I’ve also set up OpenBSD based CARP systems handle traffic for a site of nearly a thousand computers. These systems handle hundreds of simultaneous users working on 3D Studio Max and Maya files off of network file servers, VOIP traffic for multiple sites as well as day-to-day intranet and Internet services.

How does CARP work? Well, in short, a CARP interface is essentially a virtual interface that can be shared between multiple machines. The CARP interface is active on only one parent interface at any given time, but the virtual interface can be shifted to another parent interface very quickly in the event of a failure.

Now you’re probably saying, “that sure sounds a lot like Cisco’s VRRP!” Well, that’s because it is just like VRRP (Virtual Router Redundancy Protocol). CARP came in to being as an open source implementation of VRRP using VRRP as the spec. But get this–CARP not only performs the same kind of functionality as Cisco’s VRRP , it can also be configured to act like Cisco’s GLBP (Gateway Load Balancing Protocol) (although, technically, the CARP guys had their own implementation of GLBP working before Cisco did). Load-balanced routing with included failover capabilities is very exciting–I know–but GLBP and load-balancing CARP is a subject we’ll leave for another day. For now, let’s take a look at setting up a CARP interface on FreeBSD using rc.conf.

Setting Up CARP interfaces on FreeBSD

There are three basic elements to setting up CARP failover interfaces to boot on FreeBSD:

First, in /etc/rc.conf, tell FreeBSD that you’re creating a “cloned interface.” Unless the interface is included in the list of virtual interfaces, it will never be created by FreeBSD.

cloned_interfaces="carpXX"

The cloned_interfaces directive can contain as many virtual interfaces as you need separated by a space:

cloned_interfaces="carpXX carpYY carpZZ"

Then add the configuration directives for your cloned interfaces to /etc/rc.conf.

ifconfig_carpXX="vhid <carp-interface-id> advskew <priority> pass <password> <carp-ip-address>"

and then in /etc/sysctl.conf, tell the kernel that you want to allow CARP interfaces to preempt each other:

net.inet.carp.preempt=1

This directive also means that once any single interface fails on a router, all virtual CARP interfaces will be shifted to the failover router at once.

The Four Elements of The Interface Definition

The ifconfig_carpXX line is made up of four elements.

  1. vhid <carp_interface_id> the virtual interface ID
  2. advskew <priority> the priority of the interface
  3. pass <password> the password used by hosts to validate the CARP relationship
  4. <carp_ip_address> the IP address of the virtual interface.

ifconfig_carpXX="1==>vhid <carp_interface_id>     2==>advskew <priority>     3==>pass <password>     4==><carp_ip_address>"

That’s all there is to it. You won’t need to worry about where or how the virtual interface attaches, FreeBSD is smart enough to know which parent interface it’s supposed to use by the IP address assigned to the CARP interface. The only real bookkeeping you’ve got to do is to make sure that:

  1. the VHID on your primary and failover routers match, and
  2. that the priority of the primary is lower than the failover.

Example

In the following example uses two FreeBSD routers called primo (our primary router) and secundo (our failover router) in a failover configuration:

primo, our primary router

/etc/rc.conf

ifconfig_em0="inet 99.88.77.70/29" # external interface for primo, our primary router

ifconfig_em1="inet 10.11.12.254/24" # internal interface for primo

defaultrouter="99.88.77.65"

gateway_enable="YES"

cloned_interfaces="carp1 carp2"

ifconfig_carp1="vhid 1 advskew 50 pass BigPassword 99.88.77.66" # external CARP interface for primo

ifconfig_carp2="vhid 2 advskew 50 pass BigPass2W 10.11.12.1" #internal CARP interface for primo

secundo, our failover router

/etc/rc.conf
ifconfig_em0="inet 99.88.77.69/29" #external interface for secundo, our failover router

ifconfig_em1="inet 10.11.12.253/24" # internal interface for secundo

defaultrouter="99.88.77.65"

gateway_enable="YES"

cloned_interfaces="carp1 carp2"

ifconfig_carp1="vhid 1 advskew 80 pass BigPassword 99.88.77.66" #our external CARP interface for secundo

ifconfig_carp2="vhid 2 advskew 80 pass BigPass2W 10.11.12.1" # internal CARP interface for secundo

Now I’m sure you’ve already noticed, but I want to reiterate it; the only difference in CARP interface definitions between the two hosts is the advskew assigned to each. This is how the priority of the router is assigned.

Final Notes

A practical consideration when you’re deciding whether CARP is the right choice is that any system is limited to 255 CARP interfaces. This normally is a non issue, but if you expect your network to grow beyond 128 VLANs and want to use load balanced routers, it will become an issue, so keep it in mind. (Of course, there’s a point of diminishing returns when trying to put too many segments on a single router or set of failover routers is more management overhead than just setting up another set.

Also, keep in mind that this covers only configuration of the CARP interfaces themselves. If you plan on doing any packet filtering with PF, you’ll want to set up a pfsync interface. But that’s another story.

Posted in , ,  | Tags , , , , , , ,

Using Interface Groups in OpenBSD

Posted by Tres Tue, 26 Dec 2006 02:51:00 GMT

Another one of the brilliant ideas to come out of OpenBSD is the interface group.

OpenBSD’s interface groups combined with VLAN interfaces are the one of the best tools available to simplify the management and organization of your rulesets. By using interface groups, you can cut out 99% of the redundancy required for different networks. The best part of interface groups from a management perspective is that they allow you to see instantly what rules are being used on any given interface. It’s like a heads-up display of what rules are applied to any subnet on the network.

Using interface groups is very easy. You can add an interface to a group by using the “group” option of ifconfig. So to add interface vlan1024 to the DEVELOPER interface group, I would do this:


ifconfig vlan1024 group DEVELOPER

If I wanted to remove interface vlan1024 from the DEVELOPER interface group, I would do this:


ifconfig vlan1024 -group DEVELOPER

To add a group to a particular interface on boot just add the “group” syntax to the hostname.if file for the vlan group. So to add DEVELOPER rules to the vlan1024 interface at boot, your /etc/hostname.vlan1024 file would add an additional line to the end of the file like this:

10.18.24.254 netmask 255.255.0.0 broadcast 10.18.24.255 vlan 1024 vlandev em2
group DEVELOPER

You can add as many groups to an interface as are necessary, just keep adding each group to the interface on a separate line and use the “group” syntax:

10.18.24.254 netmask 255.255.0.0 broadcast 10.18.24.255 vlan 1024 vlandev em2
group DEVELOPER
group IMAP  
group SMTP
group POP3
group HTTP
group INTERNET

There’s no extra configuration required to get PF to use the group. All you need is to set up your rules for the interface group. Rules should have as fine of granularity as possible. Remember, you’re trying to show exactly what rules are being used by the interfaces. You should use use super-classes to define more abstract groupings sparingly. It can lead to confusion if a rule for a service is defined in a superclass because it is in use by all hosts on the network. It is better to use service based group interface rules that are applied to all interfaces. Then there is no confusion. So, what does our service-based group interface ruleset look like?


pass in quick proto tcp on INTERNET from <our_ips> to !<our_ips> keep state flags S/SA
pass out quick proto tcp on INTERNET from <our_ips> to !<our_ips> modulate state flags S/SA

That’s it. Of course these rules imply that you’re using tables to manage addresses.

So, you can add a base set of rules to any interface that you’ve defined. If you have your VLANs and switchports set up so things are managed by access levels, interface groups make things brilliantly easy. Just plug in the rules that apply to that group and you’re done. If you have mixed networks in which only a few workstations get access, don’t despair. Using address tables in conjunction with the group will allow use of interface groups. Just define the IP addresses in the relevant table, and as long as you’ve written your rules to only use the table you’re ready.

Posted in , ,  | Tags , , ,

Defining PF Macros In Multiple Anchor Files

Posted by Tres Mon, 25 Dec 2006 00:08:00 GMT

Splitting rules into multiple files allows you to easily organize rules, but the downside is that without a little scripting, you can’t really use macros effectively. Because of the way anchors work, you have to define macros in each file separately. This is a potential management overhead nightmare.

The #<–[MACROS]–> construct allows us to define all macros in a single file and distribute it to all anchor files. We keep all our macros in /usr/local/etc/pf_macros.conf. Just by putting #<–[MACROS]–> at the start of our ruleset, we can ensure that our single set of macros will be used in that file. That way, we can keep management overhead to a minimum. Here’s the ruby script that makes that possible:


#!/usr/bin/env ruby -w

# macros.rb 
# script which will push macro variables from one file into all anchors files and pf.conf
# this should be run with cfengine, so when files get pushed out, or changed, the 
# macros will automatically be defined.

# to allow this file to automatically push uniform macros to your pf anchors and pf.conf
# place a line like the following into the file you want to have macros pushed into
#<--[MACROS]-->
# this script will replace the entire line within which the <--[MACROS]--> element is used


require 'find'
require 'fileutils'
require 'date'

pf_base_conf_dir="/usr/local/etc"
timestamp = DateTime.now
pf_anchors_dir = "#{pf_base_conf_dir}/pf_anchors"
macro_defs_file = "#{pf_base_conf_dir}/pf_macros.conf"
pf_conf = "/etc/pf.conf"
backup_data_dir = "/tmp/"
backup_file = "data.#{timestamp}"
backup = "#{backup_data_dir}/#{backup_file}"

if File.exists?( "#{macro_defs_file}" ) 
    macro_definitions = IO.read("#{macro_defs_file}")
else
    macro_definitions = nil
end

#first take care of pf.conf

FileUtils.copy( pf_conf, backup )
File.open( backup, "r" ) do |file|
    new_file = File.open( pf_conf, "w" )
    file.each do |line|
        new_line = line.gsub( /^.*<--\[MACROS\]--\>.*$/, macro_definitions )
        new_file.write new_line
    end
    file.close
    FileUtils.rm( backup )
end

#then take care of the anchor files

Find.find(pf_anchors_dir) do |file_name| 
    Find.prune if file_name =~ /\.svn/
    if File.file?(file_name) 
        FileUtils.copy( file_name, backup )
        File.open(backup, "r") do |file|
            new_file = File.open( file_name, "w" )
            file.each do |line|
                newline = line.gsub(/^.*<--\[MACROS\]--\>.*$/, macro_definitions )
                new_file.write newline
            end
        file.close
        FileUtils.rm( backup )
        end 
    end
end

Posted in , , ,  | Tags ,

Using Tables for PF Rule Management

Posted by Tres Sun, 24 Dec 2006 04:11:34 GMT

Tables allow you to write abstract rulesets that can be manipulated on the fly. Adding and removing IP addresses from tables provides a simple interface for scripting the addition and removal of devices from your abstract rulesets.

The only problem, from a management standpoint, is that you’ve got to take care of IP addresses, not host names. But as long as your network addressing is sane, and you have a relatively stable set of machines that need access, you should be able to manage the address tables without much trouble.

However, if you don’t like the idea of taking care of IP addresses, you can easily script something that reads a file of names and creates a corresponding set of IP address table files. The following ruby script should take care of it:

#!/usr/bin/env ruby -w

require 'fileutils'
require 'socket'
require 'date'
require 'find'


pf_base_conf_dir="/usr/local/etc"
timestamp = DateTime.now
table_hostnames_dir = "#{pf_base_conf_dir}/pf_table_hostnames"
pf_table_dir = "#{pf_base_conf_dir}/pf_tables"
backup_file = "data.#{timestamp}"

Find.find(table_hostnames_dir) do |base_file_name| 
    Find.prune if base_file_name =~ /\.svn/
    if File.file?(base_file_name) 
        FileUtils.copy( base_file_name, backup_file )
        file_name_parts = split( base_file_name( /\// ))
        table_file_name = file_name_parts[-1]
        File.open(backup_file, "r") do |file|
            new_file = File.open( "#{pf_table_dir}/table_file_name", "w" )
            file.each do |line|
              address = IPSocket.getaddress("line")
                new_file.write address
            end
        file.close
        FileUtils.rm( backup )
        end 
    end
end

This script will dump all tables into a single directory, so you could effectively organize your tables in a logical hierarchy to match your anchors for human editing, and then have a single directory for all the production tables.

The only caveat here is, of course, you’ll need to have a dns entry for everything you intend to have in tables.

Posted in , ,  | Tags , ,  | no comments

Anchors For PF Ruleset Management

Posted by Tres Mon, 18 Dec 2006 16:30:00 GMT

Anchors not only provide a way to dynamically change PF rulesets on the fly, they are also a useful technique for organization of larger rulesets. If you’ve ever had to work with anywhere over 500 lines of PF rules, you know it’s manageable, but it’s cumbersome. And if you’ve ever had to share management of a PF ruleset with other administrators, you know that a single pf.conf makes management even more difficult. You also know it’s nearly impossible to manage multiple updates coming from different requesters at the same time with a single file. By breaking down the monolithic pf.conf into separate files, the ruleset is better suited for large numbers of rules and better suited for management by larger organizations with administrators who have varying levels of trust.

In essence, anchors allow you to separate a ruleset among many files, letting you write rules that can be dynamically loaded and unloaded from the firewall. This means you can do things like write a bunch of different rules for different required states, then have them automatically loaded and unloaded by cron. You could write rules that would be activated and deactivated by a junior-admin, or an application. Like all other great *nix tools, the possibilities are limited only by your imagination and need.

Dynamically changing firewall rules may not seem like much at first glance; after all, you can get the same effect from an edit and reload of rules. It’s true, with an edit and reload of rules, there’s no interruption in service, but it does require that a single firewall “manager” make edits to the pf.conf file and then, as root, reload the firewall rules. This process requires special skills and root level trust over the entire ruleset in order to make changes effective. In praxis, anchors provide finer granularity of control over different aspects of the ruleset. This level of granularity opens up a whole new set of tools and utilities that can be created for management and maintenance of the ruleset.

The functional benefits of anchors for a large organization are many, but the ability to organize rulesets into multiple files reduces the overhead involved with managing firewall rules significantly. When deciding how to break down your rules, you should think of rules the same way you would an OOP class; you’re trying to create an abstract encapsulation that will have the ability to act upon every state that the object can have. The fundamental goal here is to break things down into logical pieces that will be manipulable by other administrators. When starting out, you want to make the object types you’re working with as generic and abstract as possible. Each super-class should be used as a directory in which the rule sets will be placed. So if we can define mail_servers as a child of “public_servers,” we would create a directory labeled “public_servers” and put mail_rules into that directory. In turn, if “public_servers” can be a subclass of “servers,” we would create a directory for “servers,” then put “public_servers” as a subdirectory. If you’re using rule groups, these abstract super-classes will not only help in logical organization, they will allow you to define a more abstract set of rules that apply to all sub-classes.

You could logically order rules by interfaces (like vlan128_rules), by physical characteristics (like building_2213_rules), by roles (like student_rules), or by any other characteristic that will make the organization of the rules logical to the group that will manage them. I like to use anchors to break down rules logically by what type of service the rule is for. I also like to categorize the rules into “client” and “server” management groups so things are easier to find, and I don’t make any mistakes about what rules are being edited.

There’s two things that need to happen for an anchor to work. First, PF needs to know where to find the anchor file, and second PF needs to be told to use the anchor file. This may seem kind of redundant at first–and since our primary aim here is to discuss the use of anchors for ruleset management, it will seem redundant–but this separation allows the possibility of loading dynamic rules.

The practical function of an anchor within a pf.conf ruleset is to allow the inclusion of files other than pf.conf for reading into your firewall rules. It’s just like an include in C, or a require statement in Ruby; you’re just telling the server where to look for the definition once it sees it in the firewall. And just like an include or require statement, inclusion of the anchor file into the firewall ruleset doesn’t make anything happen. You’ve got to tell PF to use the anchor–just like calling a function or a class defined in another file.

Below is an example showing use of anchors. One of these files has our rules for allowing access to web servers. The other one has rules that allow access to our mail servers.

Step 1: Including an anchor file from pf.conf This should be near the top of your pf.conf file. I define anchor files after I define queues.

man pf.conf says an anchor should look like this:

load anchor <name> from <file>

So, for our example, we would do something like this: 

load anchor mail from /usr/local/etc/pf_anchors/servers/mail_rules
load anchor http from /usr/local/etc/pf_anchors/servers/web_rules

Step 2: Telling PF to use the included anchor Telling PF to use the anchor means putting a reference to the anchor in the relevant section of pf.conf. That means nat-anchor directives should go into the NAT section of the ruleset, rdr-anchor directives go into the RDR section and anchor directives go with all the other filter rules.

nat-anchor mail
nat-anchor web


rdr-anchor mail
rdr-anchor web


anchor mail
anchor web

These represent three different sections of your pf.conf. You should make sure that any NAT, RDR or filter rules are placed into pf.conf appropriately. If you don’t PF will let you know it.

Posted in , ,  | Tags , , ,

Setting Up VLAN Interfaces in FreeBSD

Posted by Tres Tue, 04 Jul 2006 14:30:00 GMT

In order to set up VLAN interfaces on FreeBSD, first you’ll need to either use a kernel module by adding:

if_vlan_load="YES"

to loader.conf

or by adding 

device          vlan
to your kernel configuration file before you recompile your kernel so that the kernel knows how to speak 802.1q. Once your kernel is 802.1q aware, the next step is to set up rc.conf for vlan interfaces. set up rc.conf so that any vlan interfaces are listed under the “cloned_interfaces” directive 
cloned_interfaces="vlan100 vlan200 vlan300 vlan482"

make sure that the parent device is up and running. If you don’t have an IP address that you’re associating with the device natively, then you should just make sure that it’s up.

ifconfig_fxp0="up"

and finally, set up the interface details in rc.conf

ifconfig_vlan100="inet 102.203.98.97/28 vlan 100 vlandev fxp0"
ifconfig_vlan200="inet 102.203.98.24/28 vlan 200 vlandev fxp0"
ifconfig_vlan300="inet 102.203.98.10/29 vlan 300 vlandev fxp0"
ifconfig_vlan482="inet 102.203.98.223/28 vlan 482 vlandev fxp0"

Once you’ve got all these items configured, your VLAN interfaces should start up on boot. The only other thing you’ll need to do before you can use your VLAN interfaces on the FreeBSD box is to configure the switch for vlan trunking and 802.1q encapsulation. But that’s another story.

Posted in , ,  | Tags , ,  | no comments