uname -a; lsmod | grep xen
This page is intended to explain Xen terminology and how the Xen user space components fit together.
The problem for a Xen newcomer is not a shortage of documentation. Quite the opposite; there is plenty of official Xen documentation and unofficial Xen blog posts. The problem is understanding the documentation. Most of the documentation assumes prior knowledge of Xen terminology and how the Xen user space components fit together. But there is little introductory documentation explaining those things. This document attempts to do so.
http://wiki.xenproject.org/wiki/Xen_Beginners_Guide#What_is_this_Xen_Project_software_all_about.3F
People wanting a high level view of Xen user space tools, how they fit together and Xen terminology – especially on Debian.
Xen has changed significantly between releases and, less so, as packaged for various Linux distributions. This page is based on:
xl list output.A toolstack is a set of user-space tools used for adminstering Xen.
Several toolstacks are available for the interactive or scripted administration of Xen resources.
Primary reference: http://wiki.xen.org/wiki/Choice_of_Toolstacks
Apparently, once a toolstack has been chosen, it is not easy to change. From Debian 7 Wheezy's /etc/default/xen (where the chosen toolstack is configured):
# Attention: You need to reboot after changing this!
libvirt is a library for managing KVM, OpenVZ, VMware, VirtualBox, Xen and others. For Xen, it is the least feature-complete of all the toolstacks.
virsh is a user/script shell to interface with libvirt. References: http://libvirt.org/virshcmdref.html
The most feature-complete of all the toolstacks.
Deprecated since Xen 4.1 but the default in Xen 4.1 as packaged for Debian. Will be removed from Xen 4.2.
xend comprises:
xend-config.sxp config file format
Strategic since Xen 4.1.
Designed to be command line compatible with xend.
Unlike xm, xl does not do dom0 network configuration; standard OS tools must be used.
For help (
not --help or -h): xl help
In case there is no local xl man page: http://xenbits.xen.org/docs/unstable/man/xl.1.html and (Xen 4.2) http://manpages.ubuntu.com/manpages/raring/en/man1/xl.1.html
Many xl commands require a domain-id. It is dynamically associated with the domain-name. The xl list command shows the domain-id,
xl's -v option can be repeated for greater verbosity (info not in the man page).
The strategic Xen toolstack library, designed to hide xenstore, libxenctrl, and libxenguest from higher levels.
The Xen toolstack on Debian 7 Wheezy is configurable. For the xl toolstack edit /etc/default/xen to contain:
On blav2, we had explicitly configured using xl:
TOOLSTACK=xl
Now, when root runs command xl:
execs /usr/lib/xen-4.1/bin/xlWhen root runs command xm (does not work, as intended):
Works at a higher level than Xen toolstacks,
Includes the very convenient xen-create-image which largely automates creating domUs.
Logs in the /var/log/xen-tools/ directory.
The recommended way to provide storage for a DomU is by an LVM on Dom0 which the DomU uses as a virtual HDD.
Reference: http://wiki.xenproject.org/wiki/Storage_options
The Dom0's standard Linux networking features can be used with Xen: bridges and Netfilter (iptables/ipchains):
These facilities are not exclusive; they can be used in combination.
Xen interface names:
In the traditional simple bridged setup, one of Dom0's physical interfaces, say eth0, gets the bridge's address, broadcast, netmask and gateway settings. The bridge configuration will include, for example bridge_ports eth0. Each DomU's virtual interface is connected to the bridge/
Here's a more sophisticated example showing Dom0 with two physical interfaces, each with its own bridge and each DomU with two virtual interfaces, one connected to each bridge. It is useful for high-availability configurations. The diagram is from http://wiki.xenproject.org/wiki/Xen_Networking#ASCII_Art_Examples_of_Xen_Networking_Topologies:
LAN0 LAN1
| |
+-----+-----------------------------------------------------+-----+
| | | |
| +---+-------------------------+ +-------------------------+---+ |
| | | | | | | |
| | eth0 | | eth1 | |
| | | | | |
| | xenbr0 vif1.0 vif2.0 | | vif1.1 vif2.1 xenbr1 | |
| | | \ | | / | | |
| +---^------------+---------\--+ +--/---------+------------^---+ |
| | | \ / | | |
| | +------+-------------X-------------+------+ | |
| | | | / \ | | | |
| | | +----+---------/--+ +--\---------+----+ | | |
| | | | | / | | \ | | | | |
| | | | eth0 eth1 | | eth0 eth1 | | | |
| | | | | | | | | | | | | |
| +-+-+ | | +-+-+ +-+-+ | | +-+-+ +-+-+ | | +-+-+ |
| | | | | | | | | | | | | | | | | | | |
| www ssh | | www ssh ftp pop | | www ssh ftp pop | | ftp pop |
| | | | | | | |
| Domain0 | | Domain1 | | Domain2 | | Domain0 |
+-----------+ +-----------------+ +-----------------+ +-----------+The traditional bridged setup exposes the MAC addresses of the DomU's interfaces on the LAN. In some cases this will not work because the LAN router(s)/switch(es) will not accept a single physical interface having more than one MAC address.
The solution is to configure a dummy interface on the bridge for the Dom0's physical interface, say eth0. This leaves eth0 with its own address(es), broadcast, netmask and gateway settings.
LAN
|
+-----------+---------+
| | |
| eth0 |
| | |
| +---------+-------+ |
| | | | |
| | dummy | |
| | | |
| | xenbr0 | |
| | | |
| | vif0.0 vif1.0 | |
| | | | | |
| +---+---------+---+ |
| | | |
| +---+---+ +---+---+ |
| | | | | | | |
| | eth0 | | eth0 | |
| | | | | |
| | DomU0 | | DomU1 | |
| +-------+ +-------+ |
+---------------------+/etc/network/interfaces example for single MAC address on the LAN. The DomUs are configured to use 192.168.42.100 as default gateway:
...
# Configure netfilterpre-up iptables-restore < /etc/iptables.conf
...
# dummy0auto dummy0iface dummy0 inet manual# xenbr0auto xenbr0iface xenbr0 inet static bridge_ports dummy0 address 192.168.42.100 broadcast 192.168.42.255 netmask 255.255.255.0 bridge_stp off # Disable Spanning Tree Protocol bridge_waitport 0 # No delay before a port becomes available bridge_fd 0 # No forwarding delayWhen routing is used the kernel must be configured with:
net.ipv4.ip_forward = 1
net.ipv4.conf.eth0.proxy_arp = 1
On Debian, to set them during boot, put them in /etc/sysctl.conf
To set up NATting (example is for eth0):
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
TODO: give command to save current netfilters rules for application during boot.
Each DomU (the VM itself, not its OS etc.) is configured by a file in /etc/xen. Its name is the DomU's hostname with .cfg appended.
Its format, for use with the xl toolstack, is documented at http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html.
xl shutdown and reboot does not effect changed configuration items; a create is required.
Use the extra configuration item.
Set by the memory configuration item. Numeric only (no M or G suffix allowed). Units are MB (or MiB?).
vif (Default=[]; Value="type=TYPE, mac=MAC, bridge=BRIDGE, ip=IPADDR, script=SCRIPT," + \ "backend=DOM, vifname=NAME, rate=RATE, model=MODEL, accel=ACCEL" )
In practice, the full path to the script had to be given. In practice a name had to be given.aptitude install xen-linux-system
Adjust grub so the default boot item is Xen (ref: https://xen-orchestra.com/cant-find-hypervisor-information-in-sysfs/)
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
update-grub
shutdown -r now
This is listed before "Creating Debian DomUs" because a newly created Debian should be updated during installation.
Installing an apt cacher (apt-cacher-ng) on Dom0 will save bandwidth and speed maintenance.
There are several ways to install Debian in a DomU. The xen-create-image method is very convenient. It uses the Debian Installer's debootstrap.
Installs into a chroot. Can use only one repository (which may be a local cache).
References:
Installing new Debian systems with debootstrap (2006): http://www.debian-administration.org/article/426/Installing_new_Debian_systems_with_debootstrap
Installing Debian GNU/Linux from a Unix/Linux System: http://www.debian.org/releases/stable/amd64/apds03.html.en
"Debian installer" is the standard Debian installation system; it supports installation of standard Debian as a Xen PV.
References:
http://www.debian.org/releases/stable/amd64/ch04s05.html.en
Not as easy as using xen-create-image
xen-create-image is part of the xen-tools package.
When creating Debian systems it uses debbootsrap (described above).
ntp is required. Ref: http://xen.1045712.n5.nabble.com/keeping-time-in-sync-tp5725115p5725116.html
To create a DomU and boot its OS: xl create <DomU config path>
For example:
cd /etc/xen && xl create host.some_domain.org.cfg
To shutdown the OS and destroy the DomU: xl shutdown <domain-id>
For example:
xl list
xl shutdown 5
In case the DomU's OS has been stopped (or will not stop!): xl destroy <domain-id>
xl top
To start a console when starting a DomU, use xl create's -c option.
To attach to the console of a running DomU: xl console <domain-id>
For example:
xl console 2
Detach from the console by Ctl+] or, for putty, Ctl+5
uname -a; lsmod | grep xen
uname -a lists a kernel with the string "xen" in it, then you have a modified kernel and it's likely a PV guest and you will see output from the lsmod command to confirm it. If you have output from the grep on lsmod but no sign of a modified kernel then you are PV-HVM. Without any sign of either, it's a straight HVM.Thanks to Adam C http://serverfault.com/questions/511923/determine-which-guest-is-running-on-xen-hvm-or-pv-guest for the above.
| File or directory | Usage | Notes |
|---|---|---|
| /etc/default/xen | Toolstack selection | |
| /etc/default/xend | ||
| /etc/default/xendomains | Domain startup and shutdown config | |
| /etc/xen/ | Configuration | |
| /etc/xen/*.cfg | Individual DomU configs | |
| /etc/xen/scripts/ | Scripts  and scrippets | As installed, all bash scripts. More: Xen scripts |
| /etc/xen/xend-config.sxp | xend daemon (and xen-tools) configuration | |
| /etc/xen/xl | xl defaults config | |
| /proc/xen/ | xen-kernel information | |
| /var/lib/libvirt/images | File-based images | |
| /var/lib/xen/dump/ | Guest core dumps | |
| /var/log/xen | Logs |
Very little information found on the 'net about Xen architecture. This diagram is from http://libvirt.org/architecture.html
Primary reference (no Xen 4.x equivalent found): http://wiki.xen.org/wiki/XenIntro
Includes /usr/lib/libxenctrl-4.1.so and /usr/lib/libxenguest-4.1.so
Includes /usr/lib/libxenstore.so.3.0.0 and /usr/lib/libxenstore.so.3.0
Includes /boot/xen-4.1-amd64.gz
Documentation only. TODO: presumably its dependencies are crucial; list them?
Documentation only. TODO: presumably its dependencies are crucial; list them?
Documentation only. TODO: presumably its dependencies are crucial; list them?
Tools that work with the xm and xl toolstacks to administer Xen config files and images.
Provides (in /usr/bin):
Plus, presumably in suppport:
Tools to control the hypervisor from dom0.
Provides (in /usr/lib/xen-4.1/bin/):
Plus, presumably in suppport:
Provides:
Provides (all in /usr/sbin):
General references are listed below. Specific references are listed in the sections they apply to above.
These are entry points to references.