Xen: DomU having exclusive use of an ethernet port

I'm setting up an internet facing server which would have a limited number of services exposed. I'm also testing out the capabilities of Xen. Because the hardware I'm testing on is a 3 year old Dell 1U server with no hardware RAID0, I can't use VMWare ESXi (the free one) which would be a simple click and run.

So Xen it is. Installed CentOS 5.2, with software RAID0 (thank you mdadm!), and ofcourse with the Xen kernel enabled. Installing a new CentOS image was relatively straightforward;

1. Use virt-install
   
2. If you are having problems in setting the ISO image as the source, you can
   
3. mount -o loop Centos.DVD.iso /var/www/html/centos/
4. use http://[localLANIP]/centos as the repository

So back to the issue at hand. The Physical server has two network cards. One is allocated as a trusted LAN card (eth0), and the other card is the untrusted WAN card (eth1). Normally when you install your server, you have access to both ports.

However if you want the Dom0, the Xen physical server which hosts the child Virtual Machines (DomU) to be insulated from the internet, you need eth1 to be invisible to Dom0.

To do this, you will need to mess around abit. Parts of this guide comes from here and here. And you will need a kernel module called pciback. To test that you have pciback in your system, do this:

# modprobe pciback
# lsmod | grep pci
pciback 29389 0

Wonderful, it comes by default with the stock CentOS 5.2 distro. Next you will have to find out which PCI address to look out for.

# lspci |grep Ethernet
00:03.0 Ethernet controller: 3Com Corporation 3c905C
00:10.0 Host bridge: Broadcom I/O Bridge with Gigabit Ethernet
00:10.2 Host bridge: Broadcom I/O Bridge with Gigabit Ethernet
01:03.0 Ethernet controller: Intel Corporation Ethernet Pro 100

The one I was interested in is the 3Com card. So remember 00:03.0 . I tried the kernel comand arguments as described in the URL above, but that didn't work. Fortunately in Linuxland, there is always more than one way of doing things.

First, modify /etc/modprobe.conf manually

# cat modprobe.conf
alias eth0 e100
# 090217 yky Hiding eth1 from Dom0 to be revealed to DomU
#alias eth1 3c59x
options pciback hide=(0000:00:03.0)

Basically pciback "seizes" any PCI devices before the kernel can get to it. Which is why it has to be preloaded early on. Unfortunately pciback is a dynamic module residing in the filesystem, which may not be in the ramdisk when the kernel is loading. So you will need to add it to the ramdisk by running this command.

# mkinitrd -f --preload=pciback /boot/initrd-$(uname -r).img $(uname -r)
# ls -la /boot/initrd*
-rw------- 1 root root 2280203 Feb 17 17:42 initrd-2.6.18-92.el5xen.img

Now you are ready to lose eth1 from Dom0. Reboot. To confirm this to yourself, do this:

# dmesg | grep eth
e100: eth0: e100_probe: addr 0xfcf00000, irq 18, MAC addr 00:90:27:D3:A8:BC
e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
eth0: no IPv6 routers present
e100: peth0: e100_watchdog: link up, 100Mbps, full-duplex
device peth0 entered promiscuous mode
xenbr0: port 2(peth0) entering learning state
xenbr0: port 2(peth0) entering forwarding state

# dmesg | grep pciback
pciback 0000:00:03.0: seizing device
pciback: vpci: 0000:00:03.0: assign to virtual slot 0

# ls -l /sys/bus/pci/drivers/pciback/
lrwxrwxrwx 1 root root 0 Feb 17 17:52 0000:00:03.0 -> ../../../../devices/pci0000:00/0000:00:03.0
...

Previously eth1 would appear where eth0 did. Now not so. eth1 has been seized.

So the next step is to make 00:03.0 available to the DomU guest VM. To do so, your Xen config file should look something like this:

# cat /etc/xen/DomU1
name = "DomU1"
memory = 178
bootloader = "/usr/bin/pygrub"
disk = [ "tap:aio:/xen/DomU1/DomU1.img,xvda,w" ]
vif = [ "mac=00:00:ee:24:9a:73,bridge=xenbr0" ]
pci = [ "00:03.0" ]

The vif ... bridge=xenbr0 will provide a virtual NIC to the DomU which can be connected to the trusted LAN.
The pci = [ "00:03.0" ] defines the PCI address which is accessible to the DomU to probe. On a successful bootup, it should detect it as a pure 3Com card as Dom0 used to do.

You can now configure eth1 as per normal. Dont forget to bring up them firewalls!

yk.

Little Tricks with Xen images from jailtime.org

Ive been busy setting up a CentOS server. After the successful installation of dspam on our FC5 Xen virtual machine, we have decided to move it from our development server (a frankenstein of a machine) to a proper host. Yes, Ive been testing it for over a year now!

Postfix with MySQL support in CentOS 5

So I downloaded a prebuilt CentOS image from jailtime and installed the necessary tools. A little trick I learnt was the inclusion of the extra repositories. dspam requires a postfix installation with MySQL support. To do this, you can either compile from source, but being the lazy git I am, I'd rather download the binaries.

To do this, you just need to enable the CentOSPlus repositories, which can be done via a command line switch:

# yum --enablerepo=centosplus upgrade postfix

Yum will then work out the requisites and download the stuff for you. Unfortunately it also brings down the postgresql binaries. So unless you are in dire need of hard disk space, don't do it this way.

We aren't in Kansas anymore, Toto

Another thing about the jailtime images is that the hwclock scripts are modified to return 0. This is because there is no hardware clock. It uses the host's (Dom0) time. So to get the correct time, make sure you change the zone information in /etc/localtime. For us users in Kuala Lumpur, I have to do this:

# ln -s /usr/share/zoneinfo/Asia/Kuala_Lumpur /etc/localtime

Keeping it 32bits

My Xen host is a AMD 64 server. My Xen clients are i386 virtual machines. This is for "portability" reasons. My previous post about using yum to install i386 binaries can be improved. Now use setarch. To install:

# yum install setarch.i386

subsequently make sure you run it before yumming some more.

# setarch i386

If you forget and accidentally call yum, mixing i386 and x86_64 info, the downloaded headers will be "confused". You'd have to do a clean:

# yum clean all


yk.

Yum, PHP and Xen running 32bits on a 64bit machine.

My host Xen server is an AMD 64bi machine. The little Xen VMs are based on CentOS 4.4 32bit, which I plan to keep it as such to allow easy migration. However it comes with abit of risk.

I managed to run 'yum install httpd mysql mysql-server' without any problems. But when I needed to install PHP, typing in 'yum install php' would result in a horrific set of errors which look like this:

Error: Missing Dependency: libgssapi_krb5.so.2()(64bit) is needed by package openssl
Error: Missing Dependency: libdl.so.2()(64bit) is needed by package openssl
etc...

which indicates that yum has detected the 64bit nature of the host arch, as 'uname' gives it away.

So how do I force yum to install only i386 binaries? Simple, RTFM! 'man yum' ... somewhere in the middle of the documentation, it states:

Specifying package names
A package can be referred to for install,update,list,remove etc with any of the following:

name
name.arch
name-ver
name-ver-rel
name-ver-rel.arch
name-epoch:ver-rel.arch
epoch:name-ver-rel.arch

For example: yum remove kernel-2.4.1-10.i686

So to get php to install nicely, just type:

yum install php.i386

and it works!

[Update: 5 minutes later.
It kinda worked. there was a problem at the end of the installation. Yum complained that 'package php-pear-4.3.9-3.22 is intended for a x86_64 architecture'. So "no problem," I thought, "I just apply what I learnt to php-pear by using 'yum install php-pear.i386'"
No. yum resolved that php needed more stuff like openssl, and decided to look for the 64 bit installation. I tried my luck and did this:

yum install php-pear.i386 php.i386

on the reasoning that I should also force yum to install only i386 archs for these two items. Surprisingly, it kept all the dependencies to i386, and it resolved it well.

So NOW it works.]

yk.

Xen and Centos 4.4

After finding it "difficult" to install extra stuff in a rPath installation, Im now trying to use the CentOS images from jailtime.org.
Got it running, and after getting the network up, I needed to install these basic administrative tools

1. webmin
2. vi

I got the latest rpm from webmin.com, and typed in 'rpm -hiv webmin.xx.noarch.rpm' It complained that it needed perl, so I had to do this

yum install perl

After downloading 12MB of stuff, rpm'ing webmin was successful. Joy.

The default editor provided by jailtime is 'nano'. Its a nice interactive editor, but I guess Im way to used to 'vi'. 'vi' is definitely primitive by anybodies standards, but I guess typing in the 'Esc-W-Q' has become ingrained in my fingers, that I can't help but to install it.

Unfortunately it is not straightforward, as 'yum install vi' does NOT work.
Googling for this is hopeless, as when you do a search for 'yum install vi', you just get instructions on how to edit yum repositories with vi, and its not very helpful.

However a 'which vi' on a system which has 'vi', gives a hint:

alias vi='vim'
'usr/bin/vim'

however a 'yum install vim' also does not work, as the proper command is actually

yum install vim-enhanced

This will do the necessary dependency checks, and will offer to download 4.5MB worth of stuff! I never knew a console based text editor would be so bulky.

Anyway, I shut down the Xen image, made a copy of it so that next time I need to prep up a virtual server, I already have one preprepared.


yk.

Rapid respawns

If you are every using rPath Linux in Xen, and you get this errors in your console:

INIT: Id "2" respawning too fast: disabled for 5 minutes
INIT: Id "3" respawning too fast: disabled for 5 minutes
INIT: Id "4" respawning too fast: disabled for 5 minutes
INIT: Id "5" respawning too fast: disabled for 5 minutes
INIT: Id "6" respawning too fast: disabled for 5 minutes

you will find that that the terms are not created yet in /dev i.e.

#ls /dev
tty tty1

so you would have to create em as such:
[removed the prompt, so its easy to copy&paste!]

cd /dev
mknod tty2 c 4 2
mknod tty3 c 4 3
mknod tty4 c 4 4
mknod tty5 c 4 5
mknod tty6 c 4 6

That should get rid of the pesky messages.

yk.