Posted by Tres
Fri, 25 Apr 2008 10:03:00 GMT
After Edgy, Ubuntu (6.10) replaced init with upstart. Upstart uses files located in /etc/event.d/ to do what old /etc/inittab entries used to do.
Taking a peek in /etc/event.d/ you’ll see that different run levels are represented by different files. In addition, tty entries are maintained here.
If you’re running Xen, and are having problems with your console access, you’ll need to create /etc/event.d/xvc0 and insert the following:
# xvc0 - getty
#
# This service maintains a getty on xvc0 from the point the system is
# started until it is shut down again.
start on runlevel 2
start on runlevel 3
stop on runlevel 0
stop on runlevel 1
stop on runlevel 4
stop on runlevel 5
stop on runlevel 6
respawn
exec /sbin/getty 38400 xvc0
If you want to retain the ability to hop into the domU without needing to authenticate, make sure that you bypass getty login and spawn a shell:
exec /sbin/getty -n -l /bin/bash 38400 xvc0
Posted in Linux, Sysadmin, Xen, Ubuntu | Tags console, Ubuntu, Xen
Posted by Tres
Sun, 20 Apr 2008 00:36:00 GMT
Here’s a little ruby hackery to do convert the output we got here into something like this:
MB total: 2432.0
MB used: 479.6953125
MB free: 1952.3046875
#!/usr/bin/env ruby
require 'optparse'
options = {}
outprint = {}
ot = String
opts = OptionParser.new do |opts|
opts.on("-p X", "--path X", String, "path to partition for host") do |path|
options[:path] = path
end
end
opts.parse!(ARGV)
output = `dumpe2fs -h #{options[:path]}`
output.squeeze!(" ")
output.each do | line |
line.grep(/Block count:/) { | total | outprint[:total_label] ,outprint[:total_data] = total.chomp.split(/\s*\:\s*/) }
line.grep( /Free blocks:/) { | free | outprint[:free_label], outprint[:free_data] = free.chomp.split(/\s*\:\s*/) }
line.grep(/Block size:/) { | size | outprint[:block_size_label], outprint[:block_size_data] = size.chomp.split(/\s*\:\s*/) }
line.grep(/Reserved block count:/) { | reserved | outprint[:reserved_label], outprint[:reserved_data] = reserved.chomp.split(/\s*\:\s*/) }
end
mb_available = ( outprint[:block_size_data].to_f / 1048576 * outprint[:total_data].to_f )
mb_free = ( outprint[:block_size_data].to_f / 1048576 * outprint[:free_data].to_f )
mb_used = ( mb_available.to_f - mb_free.to_f )
puts "MB total: #{mb_available}"
puts "MB used: #{mb_used}"
puts "MB free: #{mb_free}"
The ruby script takes a single argument, the path to the device that is going to be looked at. It can be passed with either a -p or –path.
Posted in Linux, Ruby, Sysadmin, Xen, Development | Tags DomU, memory, monitor, ruby, script, Xen
Posted by Tres
Sat, 12 Apr 2008 11:48:00 GMT
If you’re trying to monitor disk usage in a Xen domU and are using ext3fs formatted filesystems on LVM partitions, you can use dumpe2fs -h to get an idea of the current disk usage in domU from dom0.
[tres blas.phemo.us ~]$ sudo dumpe2fs -h /dev/vol00/xen_root_img
dumpe2fs 1.39 (29-May-2006)
Filesystem volume name:
Last mounted on:
Filesystem UUID: d18fab79-7123-4289-bd28-222ec8739874
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 311296
Block count: 622592
Reserved block count: 31129
Free blocks: 336965
Free inodes: 256927
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 151
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Tue Feb 5 18:57:47 2008
Last mount time: Fri Feb 15 14:42:15 2008
Last write time: Fri Feb 15 14:42:15 2008
Mount count: 6
Maximum mount count: 25
Last checked: Tue Feb 5 19:01:06 2008
Check interval: 15552000 (6 months)
Next check after: Sun Aug 3 20:01:06 2008
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
First orphan inode: 32775
Default directory hash: tea
Directory Hash Seed: d3f9829f-e127-427b-be56-4e840a139ccf
Journal backup: inode blocks
Journal size: 64M
So amongst all of the output, there are three lines that provide the magic: Block count:, Block size: and Free blocks:.
It’s easy enough to grab the three lines and then process them however you need to. This is a link to a ruby script that will check domU disk usage from dom0
Posted in Ruby, Sysadmin, Xen | Tags disk, DomU, monitor, usage
Posted by Tres
Thu, 14 Feb 2008 06:03:00 GMT
Every once in a while, I need to say my old mantra, “it’s always the little things.” The biggest of problems always seem to boil down to the littlest of problems in the world of *nix. Here’s another example.
If you’ve upgraded your version of Xen on CentOS/Red Hat Enterprise Linux from the included Xen 3.0.x to the latest available from XenSource (3.2 at this time), you may run into something like this in /var/log/xen/xend.log when trying to start xend:
INFO (SrvDaemon: ) Xend Daemon started
INFO (SrvDaemon: ) Xend changeset: unavailable.
INFO (SrvDaemon: ) Xend version: Unknown.
ERROR (SrvDaemon: ) Exception starting xend ((13, 'Permission denied'))
...
Error: (13, 'Permission denied')
and something like this in /var/log/xen/xend-debug.log
sysctl operation failed -- need to rebuild the user-space tool set?
Exception starting xend: (13, 'Permission denied')
The error in the debug log is misleading. I spent my time looking for duplicate installs of xen tools that may have been installed. My theory was that the Red Hat supplied Xen RPM installed files that the XenSource SRPM spec file puts in a different spot, or some other bookkeeping issue related to rpm -Uvh on the xen RPM files.
It turns out I was right, but was looking at the wrong source of the issue. The Xen tools were indeed mismatched, but it was to the running kernel. The xen RPM installed the tools and the kernel and did everything it was supposed to, but it didn’t update grub.conf to boot with the updated xen kernel files that were installed.
“It’s always the little things.” It’s often embarrassing, and this is no exception, but I thought I’d put this out there so if someone else gets caught looking at the wrong source of a mismatched user-space tool set error, they can avoid the trouble of looking for rogue tool installations and get on with business. Just update your /boot/grub/grub.conf with something like the following (<DANGER, WILL ROBINSON>:needless to say, copying and pasting grub.conf entries without verifying them can land you in a heap of trouble if you don’t have local access to your server</DANGER, WILL ROBINSON>):
title CentOS (xen-3.2)
root (hd0,0)
kernel /xen.gz ro root=/dev/vol00/root dom0_mem=256M
module /vmlinuz-2.6.18-53.1.13.el5xen ro root=/dev/vol00/root
module /initrd-2.6.18-53.1.13.el5xen.img
<DANGER>:This entry is very much dated, and uses the latest CentOS 5.x kernel as of this writing</DANGER>. The thing that won’t change, and will continue to be valid no matter what version of Xen you build and install later, is the kernel line:
kernel /xen.gz
The XenSource RPM & SRPM will build things so that /boot/xen.gz is a symlink to the latest version installed.
Posted in Linux, Red Hat Enterprise Linux, Sysadmin, Xen | Tags api, upgrade, Xen, xensource, 3.2
Posted by Tres
Sun, 23 Dec 2007 10:55:00 GMT
So you try and log in to a domU and xen says it can’t open a tty?
[tres@calliope ~]$ sudo xm console xen-domu.vm
xenconsole: Could not open tty `/dev/pts/15': No such file or directory
This error normally happens when xenconsoled is no longer running. You can quickly verify by looking for the xenconsoled process:
[tres@calliope ~]$ ps auxwww | grep xenconsoled
[tres@calliope ~]$
If you don’t find it, just run
/etc/init.d/xend start
Posted in Linux, Sysadmin, Xen | Tags DomU, tty, xenconsoled
Posted by Tres
Fri, 21 Dec 2007 12:16:00 GMT
After doing an apt-get upgrade on a Gutsy VM recently, I found that ye olde 4gb seg fixup messages had found their way back into dmesg and were all over the console. I quickly apt-get install libc6-xen only to be told that it didn’t exist…
Hmm…
A little investigating and I found that Gutsy didn’t ship with Xen libc, and as of this writing, they are still listed as Proposed.
So the choice was pretty clear, either mv /lib/tls or downgrade the libc we just upgraded to (libc6-xen 2.6.1-1ubuntu10) so it matched with the available lib6c xen package (libc6-xen 2.6.1-1ubuntu9). Personally, I think having the package installed is the right way to go, so I did:
apt-get remove libc6
apt-get install libc6=2.6.1-1ubuntu9 libc6-xen=2.6.1-1ubuntu9
Posted in Linux, Sysadmin, Xen, Ubuntu | Tags fixup, gutsy, libc6, Ubuntu, upgrade
Posted by Tres
Tue, 18 Dec 2007 03:36:00 GMT
Later versions of Ubuntu can be fixed with the easy
apt-get install libc6-xen
But when you run this on 6.06 LTS, the package is unavailable. In order to fix this problem, you just need to do this:
echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf && ldconfig
Of course, any processes that were started before this fix was made will still write the following to syslog:
4gb seg fixup, process xxxx(pid xxxx), cs:ip 73:xxxxxxxx
Posted in Linux, Xen, Ubuntu | Tags fixup, libc, Ubuntu, Xen
Posted by Tres
Sat, 20 Oct 2007 12:55:00 GMT
Being able to drop in to your domU via console requires that a terminal be set up in /etc/inittab.
A basic terminal configuration that will allow you to have access to domU is just to set up bash as the terminal ‘device.’
Just add the following to your /etc/inittab to get basic access:
1:2345:respawn:/bin/sh
However, you should beware when using a shell directly; you will lose the ability to send job control keystrokes, such as ctrl + c because there is not controlling terminal for the shell.
Another alternative is to use the xvc0 device instead. This not only gives you all the functionality you expect from a normal terminal based shell, but also has the additional benefit of providing your domU some security. By using xvc0 anyone wanting to use the console to gain access to your system must give valid credentials before they have access to the system (just like logging in locally via console).
Posted in Linux, Sysadmin, Xen | Tags console, DomU, prompt, shell, terminal, Xen
Posted by Tres
Wed, 17 Oct 2007 23:49:00 GMT
If you’re having issues starting Debian domUs that complain about eth devices not being found, try editing etc/udev/rules.d/z25_persistent-net.rules and removing any lines like this:
SUBSYSTEM=="net", DRIVERS=="vif0", ATTRS{address}=="00:16:3a:11:44:32", NAME="eth0
Then restart domU.
Posted in Linux, Sysadmin, Xen | Tags debian, eth0, network, SIOCSIFADDR, Xen
Posted by Tres
Wed, 17 Oct 2007 01:16:00 GMT
This is an issue with TLS libraries on FC6 moving /lib/tls won’t get this fixed. In order to get things working right, you’ll need to do this:
echo "hwcap 0 nosegneg" > /etc/ld.so.conf.d/nosegneg.conf
Then reboot domU. If you don’t reboot, you’ll still see messages like this:
4gb seg fixup, process syslogd (pid 590), cs:ip 73:00135435
For any process that was started before the update was made.
Posted in Linux, Sysadmin, Xen | Tags 6, fedora, fixup, libraries, tls, Xen
Posted by Tres
Mon, 15 Oct 2007 13:40:00 GMT
Setting up a Gentoo Xen DomU is pretty straight-forward. The only gotcha that you may run into is when Gentoo tries to mount /proc and dies with a message that includes:
The "mount" command failed with error: proc already mounted
Fixing this is just a matter of commenting out the section of /sbin/rc that attempts to mount /proc begining at around line 217 like this:
# check_statedir /proc
#
# ebegin "Mounting proc at /proc"
# if [[ ${RC_USE_FSTAB} = "yes" ]] ; then
# mntcmd=$(get_mount_fstab /proc)
# else
# unset mntcmd
# fi
# try mount -n ${mntcmd:--t proc proc /proc -o noexec,nosuid,nodev}
# eend $?
You’ll also need to comment the section attempting to mount /sys as well, or the next time you boot, you’ll get the same error with a different mount point failing.
# if [ "$(get_KV)" -ge "$(KV_to_int '2.6.0')" ] ; then
# if [[ -d /sys ]] ; then
# ebegin "Mounting sysfs at /sys"
# if [[ ${RC_USE_FSTAB} = "yes" ]] ; then
# mntcmd=$(get_mount_fstab /sys)
# else
# unset mntcmd
# fi
# try mount -n ${mntcmd:--t sysfs sysfs /sys -o noexec,nosuid,nodev}
# eend $?
# else
# ewarn "No /sys to mount sysfs needed in 2.6 and later kernels!"
# fi
# fi
Posted in Gentoo, Linux, Xen | Tags DomU, Gentoo, Xen