Posted by Tres
Mon, 03 Nov 2008 09:34:00 GMT
Installing the mysql Gem under RHEL/CentOS requires you to specify where MySQL is installed. Just point Gem to the mysql-config bin & you’re good to go:
gem install mysql -- --with-mysql-config=/usr/bin/mysql_config
Posted in Red Hat Enterprise Linux, Ruby, Sysadmin, Ruby on Rails | Tags Configuration, installation, Red Hat, ruby
Posted by Tres
Thu, 17 Jul 2008 08:23:00 GMT
It’s always the little things…
So I’m setting up rails and nginx on my production server, everything is humming along. I set up monit to start and monitor the mongrel instances, set up nginx and set up the reverse proxy, and then I try accessing my app and see this in the logs:
Wed Jul 16 15:34:05 -0700 2008: Read error: #
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:137:in `write'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:137:in `write'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:95:in `send_header'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:146:in `finished'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:165:in `process_client'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `new'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `new'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:in `each'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
Since I’m running this in a FreeBSD Jail, I assume that it’s access to something in /dev. I look at the rails installation that was done via ports and start upgrading gems and looking at all the easy stuff.
Finally, I look at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:137:in `write'
Here’s the function Mongrel is complaining about:
def write(data)
@socket.write(data) # <== here's line 137
rescue => details
socket_error(details)
end
Ye olde firewall problem…
So I had some rules set up to pass all internal traffic across my jailed servers on my FreeBSD server through the firewall, something like this:
pass quick all on $external_interface from self to any modulate state
This was to be a little “safety rule;” something that would keep things open even if something was added later on to block a set of ports globally.
This was setting up a race condition with the state table and ultimately had the effect of almost always killing mongrel connections both directly and through the nginx load balancer.
Posted in FreeBSD, PF / CARP , Sysadmin, Ruby on Rails | Tags its always the little things, shooting yourself in the foot, stupid firewall tricks, ye olde firewall problem
Posted by Tres
Wed, 14 May 2008 01:44:00 GMT
Here’s an easy one-liner to get ruby gem and all installed gems up to the latest available version:
gem update -y --system --include-dependencies && gem update -y --include-dependencies
Newer versions of gem include dependency installs and are not interactive, so you don’t need to use the flags outlined above. Just use:
gem update --system && gem update
Posted in Ruby, Sysadmin | Tags gem, liners, one, ruby, update
Posted by Tres
Wed, 14 May 2008 00:51:00 GMT
I’m a big fan of package management systems for installing and managing everything on a *nix system. From portupgrade on FreeBSD to yum/RPM on RedHat, I use the package management system to install and maintain everything I possibly can.
On OS X, I’ve been using MacPorts back since they were Darwin Ports, and before that I was using Fink. MacPorts closely resembles portupgrade on FreeBSD.
MacPorts is generally excellent, especially because everything is tucked away in opt, so if I ever decide that I don’t want MacPorts around anymore, I have one place to go to get rid of everything.
Anyway, today was the first time I really had an issue with MacPorts. Trying to get Mysql5 installed on my laptop, I happened to download a corrupt tar archive.
sudo port install mysql5
---> Fetching mysql5
---> Attempting to fetch mysql-5.0.51.tar.gz from http://mysql.mirrors.pair.com/Downloads/MySQL-5.0/
---> Attempting to fetch mysql-5.0.51.tar.gz from http://mysql.he.net/Downloads/MySQL-5.0/
---> Verifying checksum(s) for mysql5
Error: Checksum (md5) mismatch for mysql-5.0.51.tar.gz
Error: Checksum (sha1) mismatch for mysql-5.0.51.tar.gz
Error: Checksum (rmd160) mismatch for mysql-5.0.51.tar.gz
Error: Target org.macports.checksum returned: Unable to verify file checksums
Error: Status 1 encountered during processing.
My response was to clean out the distfiles, like I would with FreeBSD. I thought I’d be nice and use the built-in port clean instead of just removing the downloaded file:
sudo port clean mysql5
---> Cleaning mysql5
But that just made MacPorts angrier:
sudo port install mysql5
---> Verifying checksum(s) for mysql5
Error: Target org.macports.checksum returned: Could not open file: /opt/local/var/macports/distfiles/mysql5/mysql-5.0.51a.tar.gz
Error: Status 1 encountered during processing.
The key was to use the –all flag on the port clean command:
sudo port clean --all mysql5
Posted in OS X, Sysadmin | Tags macports
Posted by Tres
Sat, 03 May 2008 13:49:00 GMT
Installing Postgres on FreeBSD, as always with packages and ports, is a breeze. pkg_add -r postgresql82-server or better yet, portinstall -P databases/postgresql82-server and a quick edit to /etc/rc.conf later, it’s Miller Time… Well almost…
Getting Postgres running in a FreeBSD jail requires that the jail host allows access to FreeBSD’s System V interprocess communication send and receive system calls.
Warning:This breaks down the separation of jailed processes from the host. If you’re paranoid about the security of your host environment, you’ll probably not want to do this – since the same namespace is used for IPC primitives of both the host and in the jail environment as well. This means that someone can potentially send and receive to processes being run in the host environment, or in other jailed environments. There is potential for denial of service, but so long as there are users on a box, there’s always a potential for denial of service, right?
So, to get things running just add the following to /etc/sysctl.conf in the host environment:
security.jail.sysvipc_allowed=1
That will make sure that things start up right whenever the box gets rebooted. To get things running right now, type the command in your terminal window.
sudo security.jail.sysvipc_allowed=1
Posted in FreeBSD, Sysadmin | Tags FreeBSD, Jail, postgresql, virtualization
Posted by Tres
Fri, 25 Apr 2008 10:03:00 GMT
After Edgy, Ubuntu (6.10) replaced init with upstart. Upstart uses files located in /etc/event.d/ to do what old /etc/inittab entries used to do.
Taking a peek in /etc/event.d/ you’ll see that different run levels are represented by different files. In addition, tty entries are maintained here.
If you’re running Xen, and are having problems with your console access, you’ll need to create /etc/event.d/xvc0 and insert the following:
# xvc0 - getty
#
# This service maintains a getty on xvc0 from the point the system is
# started until it is shut down again.
start on runlevel 2
start on runlevel 3
stop on runlevel 0
stop on runlevel 1
stop on runlevel 4
stop on runlevel 5
stop on runlevel 6
respawn
exec /sbin/getty 38400 xvc0
If you want to retain the ability to hop into the domU without needing to authenticate, make sure that you bypass getty login and spawn a shell:
exec /sbin/getty -n -l /bin/bash 38400 xvc0
Posted in Linux, Sysadmin, Xen, Ubuntu | Tags console, Ubuntu, Xen
Posted by Tres
Fri, 25 Apr 2008 06:45:00 GMT
If you’re trying to start up an Ubuntu environment and seeing
* Starting OpenBSD Secure Shell server sshd PRNG is not seeded
when Ubuntu tries starting OpenSSH, make sure that the /dev/random and /dev/urandom entries have read+write permissions for user,group and other.
chmod 666 /dev/urandom /dev/random
Posted in Linux, Sysadmin, Ubuntu | Tags ssh, Ubuntu
Posted by Tres
Sun, 20 Apr 2008 00:36:00 GMT
Here’s a little ruby hackery to do convert the output we got here into something like this:
MB total: 2432.0
MB used: 479.6953125
MB free: 1952.3046875
#!/usr/bin/env ruby
require 'optparse'
options = {}
outprint = {}
ot = String
opts = OptionParser.new do |opts|
opts.on("-p X", "--path X", String, "path to partition for host") do |path|
options[:path] = path
end
end
opts.parse!(ARGV)
output = `dumpe2fs -h #{options[:path]}`
output.squeeze!(" ")
output.each do | line |
line.grep(/Block count:/) { | total | outprint[:total_label] ,outprint[:total_data] = total.chomp.split(/\s*\:\s*/) }
line.grep( /Free blocks:/) { | free | outprint[:free_label], outprint[:free_data] = free.chomp.split(/\s*\:\s*/) }
line.grep(/Block size:/) { | size | outprint[:block_size_label], outprint[:block_size_data] = size.chomp.split(/\s*\:\s*/) }
line.grep(/Reserved block count:/) { | reserved | outprint[:reserved_label], outprint[:reserved_data] = reserved.chomp.split(/\s*\:\s*/) }
end
mb_available = ( outprint[:block_size_data].to_f / 1048576 * outprint[:total_data].to_f )
mb_free = ( outprint[:block_size_data].to_f / 1048576 * outprint[:free_data].to_f )
mb_used = ( mb_available.to_f - mb_free.to_f )
puts "MB total: #{mb_available}"
puts "MB used: #{mb_used}"
puts "MB free: #{mb_free}"
The ruby script takes a single argument, the path to the device that is going to be looked at. It can be passed with either a -p or –path.
Posted in Linux, Ruby, Sysadmin, Xen, Development | Tags DomU, memory, monitor, ruby, script, Xen
Posted by Tres
Sat, 12 Apr 2008 11:48:00 GMT
If you’re trying to monitor disk usage in a Xen domU and are using ext3fs formatted filesystems on LVM partitions, you can use dumpe2fs -h to get an idea of the current disk usage in domU from dom0.
[tres blas.phemo.us ~]$ sudo dumpe2fs -h /dev/vol00/xen_root_img
dumpe2fs 1.39 (29-May-2006)
Filesystem volume name:
Last mounted on:
Filesystem UUID: d18fab79-7123-4289-bd28-222ec8739874
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 311296
Block count: 622592
Reserved block count: 31129
Free blocks: 336965
Free inodes: 256927
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 151
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Tue Feb 5 18:57:47 2008
Last mount time: Fri Feb 15 14:42:15 2008
Last write time: Fri Feb 15 14:42:15 2008
Mount count: 6
Maximum mount count: 25
Last checked: Tue Feb 5 19:01:06 2008
Check interval: 15552000 (6 months)
Next check after: Sun Aug 3 20:01:06 2008
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
First orphan inode: 32775
Default directory hash: tea
Directory Hash Seed: d3f9829f-e127-427b-be56-4e840a139ccf
Journal backup: inode blocks
Journal size: 64M
So amongst all of the output, there are three lines that provide the magic: Block count:, Block size: and Free blocks:.
It’s easy enough to grab the three lines and then process them however you need to. This is a link to a ruby script that will check domU disk usage from dom0
Posted in Ruby, Sysadmin, Xen | Tags disk, DomU, monitor, usage
Posted by Tres
Wed, 27 Feb 2008 16:31:00 GMT
Often times it’s nice to be able to walk through a connection to an IMAP or SMTP server to verify that everything is working correctly. Old hands know this is an easy task with telnet:
(tres blas.phemo.us)$ telnet ilovemymail.com 143
* OK IMAP4 ready
login accountname@myserver.com
Old hands also know that every time you connect this way, your password gets sent unencrypted over the wire. So instead of telnet, try the following:
(tres@blas.phemo.us)$ openssl s_client -connect ilovemymail.com:993
...
* OK IMAP4 ready
Posted in Sysadmin | Tags connect, imap, server, ssl, telnet, test
Posted by Tres
Thu, 14 Feb 2008 06:03:00 GMT
Every once in a while, I need to say my old mantra, “it’s always the little things.” The biggest of problems always seem to boil down to the littlest of problems in the world of *nix. Here’s another example.
If you’ve upgraded your version of Xen on CentOS/Red Hat Enterprise Linux from the included Xen 3.0.x to the latest available from XenSource (3.2 at this time), you may run into something like this in /var/log/xen/xend.log when trying to start xend:
INFO (SrvDaemon: ) Xend Daemon started
INFO (SrvDaemon: ) Xend changeset: unavailable.
INFO (SrvDaemon: ) Xend version: Unknown.
ERROR (SrvDaemon: ) Exception starting xend ((13, 'Permission denied'))
...
Error: (13, 'Permission denied')
and something like this in /var/log/xen/xend-debug.log
sysctl operation failed -- need to rebuild the user-space tool set?
Exception starting xend: (13, 'Permission denied')
The error in the debug log is misleading. I spent my time looking for duplicate installs of xen tools that may have been installed. My theory was that the Red Hat supplied Xen RPM installed files that the XenSource SRPM spec file puts in a different spot, or some other bookkeeping issue related to rpm -Uvh on the xen RPM files.
It turns out I was right, but was looking at the wrong source of the issue. The Xen tools were indeed mismatched, but it was to the running kernel. The xen RPM installed the tools and the kernel and did everything it was supposed to, but it didn’t update grub.conf to boot with the updated xen kernel files that were installed.
“It’s always the little things.” It’s often embarrassing, and this is no exception, but I thought I’d put this out there so if someone else gets caught looking at the wrong source of a mismatched user-space tool set error, they can avoid the trouble of looking for rogue tool installations and get on with business. Just update your /boot/grub/grub.conf with something like the following (<DANGER, WILL ROBINSON>:needless to say, copying and pasting grub.conf entries without verifying them can land you in a heap of trouble if you don’t have local access to your server</DANGER, WILL ROBINSON>):
title CentOS (xen-3.2)
root (hd0,0)
kernel /xen.gz ro root=/dev/vol00/root dom0_mem=256M
module /vmlinuz-2.6.18-53.1.13.el5xen ro root=/dev/vol00/root
module /initrd-2.6.18-53.1.13.el5xen.img
<DANGER>:This entry is very much dated, and uses the latest CentOS 5.x kernel as of this writing</DANGER>. The thing that won’t change, and will continue to be valid no matter what version of Xen you build and install later, is the kernel line:
kernel /xen.gz
The XenSource RPM & SRPM will build things so that /boot/xen.gz is a symlink to the latest version installed.
Posted in Linux, Red Hat Enterprise Linux, Sysadmin, Xen | Tags api, upgrade, Xen, xensource, 3.2
Posted by Tres
Sun, 23 Dec 2007 10:55:00 GMT
So you try and log in to a domU and xen says it can’t open a tty?
[tres@calliope ~]$ sudo xm console xen-domu.vm
xenconsole: Could not open tty `/dev/pts/15': No such file or directory
This error normally happens when xenconsoled is no longer running. You can quickly verify by looking for the xenconsoled process:
[tres@calliope ~]$ ps auxwww | grep xenconsoled
[tres@calliope ~]$
If you don’t find it, just run
/etc/init.d/xend start
Posted in Linux, Sysadmin, Xen | Tags DomU, tty, xenconsoled
Posted by Tres
Fri, 21 Dec 2007 12:16:00 GMT
After doing an apt-get upgrade on a Gutsy VM recently, I found that ye olde 4gb seg fixup messages had found their way back into dmesg and were all over the console. I quickly apt-get install libc6-xen only to be told that it didn’t exist…
Hmm…
A little investigating and I found that Gutsy didn’t ship with Xen libc, and as of this writing, they are still listed as Proposed.
So the choice was pretty clear, either mv /lib/tls or downgrade the libc we just upgraded to (libc6-xen 2.6.1-1ubuntu10) so it matched with the available lib6c xen package (libc6-xen 2.6.1-1ubuntu9). Personally, I think having the package installed is the right way to go, so I did:
apt-get remove libc6
apt-get install libc6=2.6.1-1ubuntu9 libc6-xen=2.6.1-1ubuntu9
Posted in Linux, Sysadmin, Xen, Ubuntu | Tags fixup, gutsy, libc6, Ubuntu, upgrade
Posted by Tres
Thu, 20 Dec 2007 15:05:00 GMT
The Hyperic documentation says you should just need to install xorg-x11-libs and libXp to get Hyperic server running on CentOS5. That’s not quite true. In order to get things running I also needed to install xorg-x11-utils.
Posted in Linux, Red Hat Enterprise Linux, Sysadmin | Tags hyperic, libXp, server
Posted by Tres
Sun, 16 Dec 2007 20:10:00 GMT
If you’re seeing files like this:
-rw-r--r-- 1 root root 92 Jun 11 2006 envince
drwxr-xr-x 2 root root 4096 May 31 2007 even
?--------- ? ? ? ? ? fanbeui
-rw-r--r-- 1 root root 37 Jun 11 2006 fanxar
Then you’ve probably also seen that you can’t do anything to the files themselves. rm or mv or almost anything else can’t do anything to the file. In order to fix this, all you’ve got to do is unlink the file.
unlink /etc/fanbeui
Posted in Sysadmin