Xen: DomU having exclusive use of an ethernet port

I'm setting up an internet facing server which would have a limited number of services exposed. I'm also testing out the capabilities of Xen. Because the hardware I'm testing on is a 3 year old Dell 1U server with no hardware RAID0, I can't use VMWare ESXi (the free one) which would be a simple click and run.

So Xen it is. Installed CentOS 5.2, with software RAID0 (thank you mdadm!), and ofcourse with the Xen kernel enabled. Installing a new CentOS image was relatively straightforward;

1. Use virt-install
   
2. If you are having problems in setting the ISO image as the source, you can
   
3. mount -o loop Centos.DVD.iso /var/www/html/centos/
4. use http://[localLANIP]/centos as the repository

So back to the issue at hand. The Physical server has two network cards. One is allocated as a trusted LAN card (eth0), and the other card is the untrusted WAN card (eth1). Normally when you install your server, you have access to both ports.

However if you want the Dom0, the Xen physical server which hosts the child Virtual Machines (DomU) to be insulated from the internet, you need eth1 to be invisible to Dom0.

To do this, you will need to mess around abit. Parts of this guide comes from here and here. And you will need a kernel module called pciback. To test that you have pciback in your system, do this:

# modprobe pciback
# lsmod | grep pci
pciback 29389 0

Wonderful, it comes by default with the stock CentOS 5.2 distro. Next you will have to find out which PCI address to look out for.

# lspci |grep Ethernet
00:03.0 Ethernet controller: 3Com Corporation 3c905C
00:10.0 Host bridge: Broadcom I/O Bridge with Gigabit Ethernet
00:10.2 Host bridge: Broadcom I/O Bridge with Gigabit Ethernet
01:03.0 Ethernet controller: Intel Corporation Ethernet Pro 100

The one I was interested in is the 3Com card. So remember 00:03.0 . I tried the kernel comand arguments as described in the URL above, but that didn't work. Fortunately in Linuxland, there is always more than one way of doing things.

First, modify /etc/modprobe.conf manually

# cat modprobe.conf
alias eth0 e100
# 090217 yky Hiding eth1 from Dom0 to be revealed to DomU
#alias eth1 3c59x
options pciback hide=(0000:00:03.0)

Basically pciback "seizes" any PCI devices before the kernel can get to it. Which is why it has to be preloaded early on. Unfortunately pciback is a dynamic module residing in the filesystem, which may not be in the ramdisk when the kernel is loading. So you will need to add it to the ramdisk by running this command.

# mkinitrd -f --preload=pciback /boot/initrd-$(uname -r).img $(uname -r)
# ls -la /boot/initrd*
-rw------- 1 root root 2280203 Feb 17 17:42 initrd-2.6.18-92.el5xen.img

Now you are ready to lose eth1 from Dom0. Reboot. To confirm this to yourself, do this:

# dmesg | grep eth
e100: eth0: e100_probe: addr 0xfcf00000, irq 18, MAC addr 00:90:27:D3:A8:BC
e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
eth0: no IPv6 routers present
e100: peth0: e100_watchdog: link up, 100Mbps, full-duplex
device peth0 entered promiscuous mode
xenbr0: port 2(peth0) entering learning state
xenbr0: port 2(peth0) entering forwarding state

# dmesg | grep pciback
pciback 0000:00:03.0: seizing device
pciback: vpci: 0000:00:03.0: assign to virtual slot 0

# ls -l /sys/bus/pci/drivers/pciback/
lrwxrwxrwx 1 root root 0 Feb 17 17:52 0000:00:03.0 -> ../../../../devices/pci0000:00/0000:00:03.0
...

Previously eth1 would appear where eth0 did. Now not so. eth1 has been seized.

So the next step is to make 00:03.0 available to the DomU guest VM. To do so, your Xen config file should look something like this:

# cat /etc/xen/DomU1
name = "DomU1"
memory = 178
bootloader = "/usr/bin/pygrub"
disk = [ "tap:aio:/xen/DomU1/DomU1.img,xvda,w" ]
vif = [ "mac=00:00:ee:24:9a:73,bridge=xenbr0" ]
pci = [ "00:03.0" ]

The vif ... bridge=xenbr0 will provide a virtual NIC to the DomU which can be connected to the trusted LAN.
The pci = [ "00:03.0" ] defines the PCI address which is accessible to the DomU to probe. On a successful bootup, it should detect it as a pure 3Com card as Dom0 used to do.

You can now configure eth1 as per normal. Dont forget to bring up them firewalls!

yk.