A quick note, there are actually 3 modes, not two when it comes to the drivers in use:
- HVM: unmodified kernel and drivers using software emulated devices
- PV-HVM: unmodified kernel with paravirtualized (Xen specific) disk and network drivers
- PV: modified kernel and drivers
For a Xen guest/DomU you can do a very basic
uname
andlsmod
with a grep to list the modules in use:uname -a lsmod | grep xen
If
uname -a
lists a kernel with the string "xen" in it, then you have a modified kernel and it's likely a PV guest, and you will see output from thelsmod
command to confirm it. If you have output from the grep onlsmod
but no sign of a modified kernel then you are PV-HVM. Without any sign of either, it's a straight HVM.Note: Generally you can do more with VMs that have the PV tools installed, so that can be quite an obvious pointer, however you can fake the presence of the PV tools to allow suspend/resume etc. so you cannot rely on that in general.
Introduction
Xen is Blue Light's standard virtualisation solution, decided in BLUE-1192
Introduction to Xen: http://wiki.xenproject.org/wiki/Xen_Beginners_Guide#What_is_this_Xen_Project_software_all_about.3F
Installation (host/dom0)
TODO: better/different to install xen-linux-system instead of xen-tools?
aptitude install xen-linux-system
Adjust grub so the default boot item is Xen (ref: https://xen-orchestra.com/cant-find-hypervisor-information-in-sysfs/)
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
update-grub
shutdown -r now
Test: does the xen list
command show that the current system is Domain-0?
Toolstacks (a.k.a toolkits and toolboxes)
There are several toolstacks available for the personal or scripted administration of Xen resources.
Primary reference: http://wiki.xen.org/wiki/Choice_of_Toolstacks
Apparently, once a toolstack has been chosen, it is not easy to change. From Debian 7 Wheezy's /etc/default/xen (where the chosen toolstack is configured):
# Attention: You need to reboot after changing this!
libvirt and virsh
libvirt is a library for managing KVM, OpenVZ, VMware, VirtualBox, Xen and others. For Xen, it is the least feature-complete of all the toolstacks.
virsh is a user/script shell to interface with libvirt. References: http://libvirt.org/virshcmdref.html
xapi and xe
The most feature-complete of all the toolstacks.
xend and xm
Deprecated since Xen 4.1 but the default in Xen 4.1 as packaged for Debian. Will be removed from Xen 4.2.
xend comprises:
- xm command
xend-config.sxp config file format
- xmdomain.cfg config file format
xend was abandoned because of @@@
xl
Strategic since Xen 4.1.
Designed to be command line compatible with xend. TODO: does it have an xm command?
On Debian 7 Whhezy:
root@SON-OF-blav.bluelightav:~# type xl
xl is /usr/sbin/xl
root@SON-OF-blav.bluelightav:~# ll /usr/sbin/xl
lrwxrwxrwx 1 root root 43 May 5 2013 /usr/sbin/xl -> ../lib/xen-common/bin/xen-toolstack-wrapper
Toolstack library
Libxenlight (libxl)
The strategic Xen toolstack library, designed to hide xenstore, libxenctrl, and libxenguest from higher levels.
Toolstack configuration
Debian 7 Wheezy
The Xen toolstack on Debian 7 Wheezy is configurable.
On blav2, we had explicitly configured using xl:
root@SON-OF-blav.bluelightav:~# diff /etc/default/xen{.org,}
7c7 < TOOLSTACK=
--- > TOOLSTACK=xl
When root runs command xl
:
- The shell runs /usr/sbin/xl which is a symlink to /usr/lib/xen-common/bin/xen-toolstack-wrapper
- /usr/lib/xen-common/bin/xen-toolstack-wrapper:
- Sources /usr/lib/xen-common/bin/xen-toolstack to set TOOLSTACK to /usr/lib/xen-4.1/bin/xl
- Sources /usr/lib/xen-common/bin/xen-dir to set dir to /usr/lib/xen-4.1
- Sources /usr/lib/xen-common/bin/xen-version to set VERSION to 4.1
- /usr/lib/xen-common/bin/xen-version gets Xen version info from /sys/hypervisor/version/*
- Sets PATH to /usr/lib/xen-4.1/bin
- Sources /etc/default/xen to set TOOLSTACK to xl (would fall back to hardcoded default otherwise)
- Sets TOOLSTACK to /usr/lib/xen-4.1/bin/xl
exec
s /usr/lib/xen-4.1/bin/xl
Configuration
DomU*
Networking
Choice:
- Bridged Each DomU* OS is connected to the LAN
- Routed Dom0 is a router between the LAN and the DomU* network interfaces
- NATted Dom0 is a NATting router between the LAN and the DomU* network interfaces
Installation (domu*)
Troubleshooting
TODO: integrate the next para.
References
Files and directories
File or directory | Usage | Notes |
---|---|---|
/etc/default/xen | ||
/etc/default/xend | ||
/etc/default/xendomains | ||
/etc/xen/ | Configuration | |
/etc/xen/scripts/ | ? | |
/etc/xen/scripts/vif-nat |
Glossary
- Domain a Xen VM.
- dom0 The Xen domain/VM used a) to control the hypervisor b) as the primary interface to the hardware.
- domu* Unprivileged Xen domains/VMs
- Enlightened guest An OS designed/modified for use in a domain/VM.
- Full virtualisation A domain/VM providing full emulation of physical hardware. The OS can be the same as runs on real hardware.
- Guest domain Same as domu*.
- Hardware Virtual Machine (HVM) Same as full virtualisation.
- Hypervisor a software system that allows the execution of multiple virtual guest operating systems simultaneously on a single physical machine. Xen's hypervisor is Type 1 or “bare-metal”, meaning it runs on the physical machine as opposed to within an operating system.
- HVM Hardware Virtual Machine.
- Para-virtualization uses modified guest operating systems a.k.a. enlightened guests. They don’t require virtual hardware devices, instead they make special calls to the hypervisor that allow them to access CPUs, storage and network resources. Better performance that full virtualisation.
- PV para-virtualisation.
- Virtual Machine Monitor (VMM) Same as hypervisor.
- Xen Store A database of information shared by the hypervisor, the kernels, the drivers and the xen daemon. The memory for Xen Store is provided by Dom0. In Dom0 (and in enlightened guests?) it can be accessed via /proc/xen/xenbus/
Technical
TODO; move Xen notes from the Online.net server page to here.
Architecture
Very little information found on the 'net about Xen architecture. This diagram is from http://libvirt.org/architecture.html
TODO: mine http://wiki.xen.org/wiki/XenIntro for more information.
Primary reference (no Xen 4.x equivalent found): http://wiki.xen.org/wiki/XenIntro
Debian packages
libxen-4.1
Includes /usr/lib/libxenctrl-4.1.so and /usr/lib/libxenguest-4.1.so
libxenstore3.0
Includes /usr/lib/libxenstore.so.3.0.0 and /usr/lib/libxenstore.so.3.0
xen-hypervisor-4.1-amd64
Includes /boot/xen-4.1-amd64.gz
xen-linux-system-3.2.0-4-amd64
Documentation only.
xen-linux-system-amd64
Documentation only.
xen-system-amd64
Documentation only.
xen-tools
Provides (in /usr/bin):
- xt-customize-image
- xt-create-xen-config
- xen-list-images
- xen-delete-image
- xt-install-image
- xen-create-nfs
- xen-create-image
- xt-guess-suite-and-mirror
- xen-update-image
Plus, presumably in suppport:
- /etc/xen-tools/*
- /usr/lib/xen-tools/<OS name>.d/*
Logs in the /var/log/xen-tools/ directory.
xen-utils-4.1
Provides (in /usr/lib/xen-4.1/bin/):
- xentrace_setsize
- xenpm
- xen-python-path
- qemu-dm
- xsview
- xen-hvmcrash
- xentrace_setmask
- xenstored
- xentrace_format
- xenbaked
- readnotes
- xenconsoled
- xc_restore
- gtraceview
- xen-bugtool
- xen-hptool
- xentop
- xend
- xc_save
- gtracestat
- xenctx
- xenwatchdogd
- xenpaging
- pygrub
- xencons
- xen-detect
- lsevtchn
- tap-ctl
- xl
- xentrace
- xm
- xen-hvmctx
- xenconsole
- xenstore-control
- xenmon
- xenlockprof
- xen-tmem-list-parse
- xenperf
Plus, presumably in suppport:
- /usr/lib/xen-4.1/lib/
- /usr/lib/xen-4.1/boot/hvmloader
xen-utils-common
Provides:
- /etc/default/xend
- /etc/default/xendomains
- /etc/init.d/xen
- /etc/init.d/xendomains
- /etc/xen/scripts/*
- /etc/xen/xend-config.sxp
- /etc/xen/xend-pci-permissive.sxp
- /etc/xen/xend-pci-quirks.sxp
- /etc/xen/xl.conf
- /lib/udev/rules.d/xen-backend.rules
- /lib/udev/rules.d/xend.rules
- /usr/lib/xen-common/bin/*
- /usr/sbin/cpuperf-perfcntr
- /usr/sbin/cpuperf-xen
- /usr/sbin/xe
- /usr/sbin/xenperf
- /usr/sbin/xenpm
- /usr/sbin/xentop
- /usr/sbin/xentrace
- /usr/sbin/xentrace_format
- /usr/sbin/xentrace_setmask
- /usr/sbin/xentrace_setsize
- /usr/sbin/xl
- /usr/sbin/xm
- /usr/share/xen-utils-common/default.xen
xenstore-utils
Provides (all in /usr/sbin):
- xenstore-ls
- xenstore-write
- xenstore-exists
- xenstore-rm
- xenstore-chmod
- xenstore-watch
- xenstore-list
- xenstore-read
References
Documentation
- Wikipedia: http://en.wikipedia.org/wiki/Xen
- Debian Wiki Xen page (in draft): https://wiki.debian.org/Xen
- Xen Project Beginners Guide: http://wiki.xenproject.org/wiki/Xen_Beginners_Guide#Setup_Linux_Bridge_for_guest_networking
- Xen On Debian Wheezy With LVM (HowtoForge. Has bridge example with fixed IP): http://www.howtoforge.com/xen-on-debian-wheezy-with-lvm
- Using Xen With LVM-Based VMs Instead Of Image-Based VMs (HowtoForge. Etch): http://www.howtoforge.com/using-xen-with-lvm-based-vms-instead-of-image-based-vms-debian-etch
- Debian Wheezy -> Jessie and set up Xen (HowtoForge): http://www.howtoforge.com/how-to-set-up-xen-4.3-on-debian-wheezy-7.0.2-and-then-upgrade-to-jessie
- Setting up XEN on a Hetzner Dedicated Server: https://gist.github.com/meskyanichi/3354956
- Routed XEN VM based on LVM (step-by-step for Hetzner DS3000): http://wiki.summercode.com/routed_xen_vm_based_lvm_step_by_step_guide_for_hetzner_ds3000
- XPSU13 VIDEO: Xen for Beginners (49 min): http://www.xenproject.org/help/presentations-and-videos/video/latest/xpus13-beginners.html
- Xen on 4 app servers: http://wiki.xenproject.org/wiki/Xen_on_4_app_servers
- Debian Guest Installation Using Debian Installer: http://wiki.xen.org/wiki/Debian_Guest_Installation_Using_Debian_Installer
- The Debian Administrator's Handbook, Virtualisation chapter (covers domU* creation using xen-tools): http://debian-handbook.info/browse/stable/sect.virtualization.html
- How to install Xen in Debian (Armen's Wiki. Has scripts for both public and private IPs for virtual machines): http://vardump.org/wiki.php/HowToInstallXenInDebianGNULinux.
- Network Configuration Examples (Xen WIKI): http://wiki.xen.org/wiki/Host_Configuration/Networking
- Bridging Network Connections (Debian WIKI): https://wiki.debian.org/BridgeNetworkConnections