Introduction

Xen is Blue Light's standard virtualisation solution, decided in BLUE-1192

Introduction to Xen: http://wiki.xenproject.org/wiki/Xen_Beginners_Guide#What_is_this_Xen_Project_software_all_about.3F

Installation (host/dom0)

TODO: better/different to install xen-linux-system instead of xen-tools?

aptitude install xen-linux-system

Adjust grub so the default boot item is Xen (ref: https://xen-orchestra.com/cant-find-hypervisor-information-in-sysfs/)

dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
update-grub
shutdown -r now

Test: does the  xen list  command show that the current system is Domain-0?

Storage

The recommended way to provide storage for a DomU is by an LVM on Dom0 which the DomU uses as a virtual HDD.

Reference: http://wiki.xenproject.org/wiki/Storage_options

Networking

Choice (TODO: can be mixed?):

Interface names:

Bridged

Dom0's eth0 gets the bridge's address, broadcast, netmask and  gateway settings (assuming the bridge config includes bridge_ports eth0).

Diagram from http://wiki.xenproject.org/wiki/Xen_Networking#ASCII_Art_Examples_of_Xen_Networking_Topologies:

      LAN0                                                  LAN1
       |                                                     |
 +-----+-----------------------------------------------------+-----+
 |     |                                                     |     |
 | +---+-------------------------+ +-------------------------+---+ |
 | |   |                         | |                         |   | |
 | | eth0                        | |                        eth1 | |
 | |                             | |                             | |
 | | xenbr0       vif1.0  vif2.0 | |  vif1.1  vif2.1      xenbr1 | |
 | |                |       \    | |    /       |                | |
 | +---^------------+---------\--+ +--/---------+------------^---+ |
 |     |            |           \   /           |            |     |
 |     |     +------+-------------X-------------+------+     |     |
 |     |     |      |           /   \           |      |     |     |
 |     |     | +----+---------/--+ +--\---------+----+ |     |     |
 |     |     | |    |       /    | |    \       |    | |     |     |
 |     |     | |  eth0    eth1   | |   eth0   eth1   | |     |     |
 |     |     | |    |       |    | |    |       |    | |     |     |
 |   +-+-+   | |  +-+-+   +-+-+  | |  +-+-+   +-+-+  | |   +-+-+   |
 |   |   |   | |  |   |   |   |  | |  |   |   |   |  | |   |   |   |
 |  www ssh  | | www ssh ftp pop | | www ssh ftp pop | |  ftp pop  |
 |           | |                 | |                 | |           |
 |  Domain0  | |     Domain1     | |     Domain2     | |  Domain0  |
 +-----------+ +-----------------+ +-----------------+ +-----------+

Useful commands:

Routed

When routing is used, need:

net.ipv4.ip_forward = 1
net.ipv4.conf.eth0.proxy_arp = 1

NATted

When NAT is used, need:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

A cunning plan?

Maybe we can do what we want (as on blav; some VMs with public IPs, most with NATted non-routable IPs) without a bridge; configure the physical interface with all the public IPs and use IPTABLES to forward traffic sent to notionally the VM's public IPs to the VM's non-routable IPs.  For each VM adressable by public IP:

iptables -t nat -A PREROUTING -i eth0 -d PUBLICIP -j DNAT --to-destination INTERNALIP

For NATting:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

References (network)

Toolstacks (a.k.a toolkits and toolboxes)

There are several toolstacks available for the personal or scripted administration of Xen resources.

Primary reference: http://wiki.xen.org/wiki/Choice_of_Toolstacks

Apparently, once a toolstack has been chosen, it is not easy to change.  From Debian 7 Wheezy's /etc/default/xen (where the chosen toolstack is configured):

# Attention: You need to reboot after changing this!

libvirt and virsh

libvirt is a library for managing KVM, OpenVZ, VMware, VirtualBox, Xen and others.  For Xen, it is the least feature-complete of all the toolstacks.

virsh is a user/script shell to interface with libvirt.  References: http://libvirt.org/virshcmdref.html

xapi and xe

The most feature-complete of all the toolstacks.

xend and xm

Deprecated since Xen 4.1 but the default in Xen 4.1 as packaged for Debian.  Will be removed from Xen 4.2.

xend comprises:

xl

Strategic since Xen 4.1.

Designed to be command line compatible with xend.  TODO: what does this mean?  man -k xend lists the config file format xend-config.sxp.  It may mean command line compatible with xm; many of the xl commands take similar arguments to xm.

Unlike xm, xl does not do dom0 network configuration; standard OS tools must be used.

In case there is no local xl man page: http://xenbits.xen.org/docs/unstable/man/xl.1.html and (Xen 4.2) http://manpages.ubuntu.com/manpages/raring/en/man1/xl.1.html

Toolstack library

Libxenlight (libxl)

The strategic Xen toolstack library, designed to hide xenstore, libxenctrl, and libxenguest from higher levels.

Toolstack configuration

Debian 7 Wheezy

The Xen toolstack on Debian 7 Wheezy is configurable.

On blav2, we had explicitly configured using xl:

root@SON-OF-blav.bluelightav:~# diff /etc/default/xen{.org,}
7c7 < TOOLSTACK=
--- > TOOLSTACK=xl

When root runs command xl:

  1. The shell runs /usr/sbin/xl which is a symlink to /usr/lib/xen-common/bin/xen-toolstack-wrapper
  2. /usr/lib/xen-common/bin/xen-toolstack-wrapper:
    1. Sources /usr/lib/xen-common/bin/xen-toolstack to set TOOLSTACK to /usr/lib/xen-4.1/bin/xl
    2. Sources /usr/lib/xen-common/bin/xen-dir to set dir to /usr/lib/xen-4.1
    3. Sources /usr/lib/xen-common/bin/xen-version to set VERSION to 4.1
      1. /usr/lib/xen-common/bin/xen-version gets Xen version info from /sys/hypervisor/version/*
    4. Sets PATH to /usr/lib/xen-4.1/bin
    5. Sources /etc/default/xen to set TOOLSTACK to xl (would fall back to hardcoded default otherwise)
    6. Sets TOOLSTACK to /usr/lib/xen-4.1/bin/xl
    7. execs /usr/lib/xen-4.1/bin/xl

When root runs command xm (does not work):

  1. The shell runs /usr/sbin/xm which is a symlink to /usr/lib/xen-common/bin/xen-toolstack-wrapper
  2. /usr/lib/xen-common/bin/xen-toolstack-wrapper:
    1. Sources /usr/lib/xen-common/bin/xen-toolstack to set TOOLSTACK to /usr/lib/xen-4.1/bin/xl
    2. Sources /usr/lib/xen-common/bin/xen-dir to set dir to /usr/lib/xen-4.1
    3. Sources /usr/lib/xen-common/bin/xen-version to set VERSION to 4.1
      1. /usr/lib/xen-common/bin/xen-version gets Xen version info from /sys/hypervisor/version/*
    4. Sets PATH to /usr/lib/xen-4.1/bin
    5. Sources /etc/default/xen to set TOOLSTACK to xl (would fall back to hardcoded default otherwise)
    6. Sets TOOLSTACK to /usr/lib/xen-4.1/bin/xl
    7. Generates message "ERROR: A different toolstack (xl) have been selected!"

Maintaining Debian DomUs

This is listed before "Creating Debian DomUs" because a newly created Debian should be updated during installation.

Installing apt-cacher on Dom0 will save bandwidth and speed maintenance (and require significant time cleaning broken cache files).

Creating Debian DomUs

debootstrap

TODO: determine if it creates a new installation based on the existing installation (that would not be desirable if we want to have all our Debians as standard and the Dom0 is a Hetzner customised Debian).

TODO: determine whether debootstrap's single repo limitation would be significant for us.

References:

Debian installer

Is the standard Debian installation method; it supports installation of standard Debian as a Xen PV.

Note: If we use either of the Debian installer methods, pre-seeding would be advantageous both for creating installations that differ only as intended and – more importantly – to accelerate re-installation during disaster recovery.  This might be very useful for all Debian and ubuntu installations, not just DomUs.  Reference: http://www.debian.org/releases/stable/amd64/apb.html.en.  Actually FAI (http://fai-project.org/ and BLUE-1325) is the way to go but it will take too long to familiarise with it.

References:

Netboot

http://www.debian.org/releases/stable/amd64/ch04s05.html.en

.iso image

xen-create-image (from xen-tools)

xen-create-image is part of the xen-tools package.  TODO: does it build the image from Dom0's OS?  If so we might prefer not to use it because Dom0 was installed from the Hetzner Debian image and an uncustomised Debian may be preferable.

 

 

Troubleshooting

TODO: integrate the next para.

A quick note, there are actually 3 modes, not two when it comes to the drivers in use:

For a Xen guest/DomU you can do a very basic uname and lsmod with a grep to list the modules in use:

uname -a lsmod | grep xen

If uname -a lists a kernel with the string "xen" in it, then you have a modified kernel and it's likely a PV guest, and you will see output from the lsmod command to confirm it. If you have output from the grep on lsmod but no sign of a modified kernel then you are PV-HVM. Without any sign of either, it's a straight HVM.

Note: Generally you can do more with VMs that have the PV tools installed, so that can be quite an obvious pointer, however you can fake the presence of the PV tools to allow suspend/resume etc. so you cannot rely on that in general.

References

Files and directories

File or directoryUsageNotes
/etc/default/xen  
/etc/default/xend  
/etc/default/xendomains  
​/etc/xen/​Configuration​
/etc/xen/scripts/? 
/etc/xen/scripts/vif-nat  

Glossary

Technical

TODO; move Xen notes from the Online.net server page to here.

Architecture

Very little information found on the 'net about Xen architecture.  This diagram is from http://libvirt.org/architecture.html

TODO: mine http://wiki.xen.org/wiki/XenIntro for more information.

Primary reference (no Xen 4.x equivalent found): http://wiki.xen.org/wiki/XenIntro

Debian packages

libxen-4.1

Includes /usr/lib/libxenctrl-4.1.so and /usr/lib/libxenguest-4.1.so

libxenstore3.0

Includes /usr/lib/libxenstore.so.3.0.0 and /usr/lib/libxenstore.so.3.0

xen-hypervisor-4.1-amd64

Includes /boot/xen-4.1-amd64.gz

xen-linux-system-3.2.0-4-amd64

Documentation only.  TODO: presumably its dependencies are crucial; list them?

xen-linux-system-amd64

Documentation only.  TODO: presumably its dependencies are crucial; list them?

xen-system-amd64

Documentation only.  TODO: presumably its dependencies are crucial; list them?

xen-tools

Tools that work with the xm and xl toolstacks to administer Xen config files and images.

Provides (in /usr/bin):

Plus, presumably in suppport:

xen-create-image largely automates creating domUs.  Config file: /etc/xen-tools/xen-tools.conf.  The --lvm option creates a new logical volume which is available to the domU as a hard disk drive

Logs in the /var/log/xen-tools/ directory.

References

xen-utils-4.1

Tools to control the hypervisor from dom0.

Provides (in /usr/lib/xen-4.1/bin/):

Plus, presumably in suppport:

xen-utils-common

Provides:

xenstore-utils

Provides (all in /usr/sbin):

References

General references are listed here.  Specific references are listed in the sections they apply to above.

Introductory

Entry points

These are entry points to references.

Setup logs