12 July 2012 Jarret Lavallee

I had never used KVM, but after purchasing a KVM vps to host a few sites I wanted to install it for myself. Since I do not have much extra hardware laying around, I decided to set up KVM in a virtual machine. The performance was not great, but it worked well.

Before beginning, we should set up the host to ensure that we can run nested VMs. First set the following option in the /etc/vmware/config

vhv.allow = TRUE  

Then I needed to set up a VM and enable hardware virtualization inside the VM. I created a new Centos 6.3 VM with hardware version 8. Make sure that the portgroup is set for promiscuous mode.

Shut down the VM and add the following lines to the vmx. The first 2 lines set the MMU to be hardware. The next 5 set the CPU flags and the last is an optimization to stop hangs in kvm. See this article for more details.

monitor.virtual_mmu = "hardware"  
monitor.virtual_exec = "hardware"  
cpuid.1.ecx="\----:\----:\----:\----:\----:\----:--h-:\----"  
cpuid.80000001.ecx.amd="\----:\----:\----:\----:\----:\----:\----:-h--"  
cpuid.8000000a.eax.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"  
cpuid.8000000a.ebx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"  
cpuid.8000000a.edx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"  
vcpu.hotadd = FALSE  

After making the changes to the VMX, we need to reload the VM on the host.

\# vim-cmd vmsvc/getallvms |grep kvm1  
252  
\# vim-cmd vmsvc/reload 252  
\# vim-cmd vmsvc/power.on 252  

Once booted install kvm and libvirt.

\# yum install -y kvm libvirt  

Restart the libvirt service and verify that kvm module is loaded.

\# service libvirtd restart  
\# lsmod |grep kvm  
kvm_amd 41551 0  
kvm 314739 1 kvm_amd  

Next I set up KVM to be accessible for a non root user (Optional)

\# usermod -a -G kvm jarret  

To avoid a reboot, we can chown /dev/kvm. First let’s see who it is owned by. (Udev will set this automatically on a reboot)

\# ls -l /dev/kvm  
crw-rw-rw-. 1 root root 10, 232 Jul 12 06:08 /dev/kvm  
\# chown root:kvm /dev/kvm  
\# chmod 0660 /dev/kvm  
\# ls -l /dev/kvm  
crw-rw-rw-. 1 root kvm 10, 232 Jul 12 06:08 /dev/kvm  

Check udev rules for this device.

\# cat /etc/udev/rules.d/80-kvm.rules  
KERNEL=="kvm", GROUP="kvm", MODE="0666"  

Add a group for libvirt

\# groupadd libvirt  
\# usermod -a -G libvirt jarret  

Allow users of libvirt Access edit /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla and add the following lines.

\# For allowing access to specific user only:  
#Identity=unix-user:jarret  
\# For allowing access to a group (like this guide):  
Identity=unix-group:libvirt  
Action=org.libvirt.unix.manage  
ResultAny=yes  
ResultInactive=yes  
ResultActive=yes  

Restart libvirtd

\# service libvirtd restart  

Log out and back in and confirm the groups.

\# id  
uid=500(jarret) gid=500(jarret) groups=500(jarret),36(kvm),501(libvirt)  

Confirm that we can connect using the jarret user

\# virsh -c qemu:///system sysinfo |head -n 1

Set the permissions on the images folder.

\# chown :libvirt /var/lib/libvirt/images/  
\# chmod g+rws /var/lib/libvirt/images/  

If you want to use VNC (you do) and not have to X forward it over ssh, you can set up VNC to listen on 0.0.0.0 instead of 127.0.0.1. Edit /etc/libvirt/qemu.conf and uncomment the following line.

vnc_listen = "0.0.0.0"  

Set up a bridged network. I have 2 adapters in this VM, so we will use eth1.

Create /etc/sysconfig/network-scripts/ifcfg-br0 and add the following.

DEVICE="br0"  
TYPE=Bridge  
DELAY=0  
ONBOOT="yes"  
BOOTPROTO=static  
IPADDR=192.168.10.41  
NETMASK=255.255.255.0  
NETWORK=192.168.10.0  
GATEWAY=192.168.10.1  
DNS1=192.168.5.10  
PEERDNS="yes"  
NM_CONTROLLED=no  

Edit /etc/sysconfig/network-scripts/ifcfg-eth1 to set up the bridge.

DEVICE=eth1  
ONBOOT=yes  
BRIDGE=br0  
NM_CONTROLLED=no  
HWADDR=00:50:56:B0:07:A2  

Restart networking

\# service network restart  

Show the bridges.

\# brctl show  
bridge name bridge id STP enabled interfaces  
br0 8000.005056b007a2 no eth1  
virbr0 8000.5254003ed252 yes virbr0-nic  

Set up iptables to allow traffic to forward across the bridge

\# iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT  
\# service iptables save  
\# service iptables restart  

Check the iptables rules

\# iptables -L  
Chain INPUT (policy ACCEPT)  
target prot opt source destination  
ACCEPT udp -- anywhere anywhere udp dpt:domain  
ACCEPT tcp -- anywhere anywhere tcp dpt:domain  
ACCEPT udp -- anywhere anywhere udp dpt:bootps  
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps  
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED  
ACCEPT icmp -- anywhere anywhere  
ACCEPT all -- anywhere anywhere  
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh  
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)  
target prot opt source destination  
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-is-bridged  
ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED  
ACCEPT all -- 192.168.122.0/24 anywhere  
ACCEPT all -- anywhere anywhere  
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable  
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable  
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)  
target prot opt source destination  

The bridged networking is setup and kvm is installed and configured for a non-root user to use it.

Before doing anything else, let’s reboot and confirm that all settings come up correctly. (This will also allow for the changes to qemu.conf to take effect)

Now that KVM is setup, let’s install a VM. First, let’s put an ISO for the install in the /var/lib/libvirt/images folder.

\# scp CentOS-6.0-x86_64-minimal.iso [email protected]:/var/lib/libvirt/images/  

Since we set the setgid bit above, the group will stay libvirt unter this folder.

\# ls -l /var/lib/libvirt/images/  
total 302772  
-rwxrwxr-x. 1 jarret libvirt 310032384 Jul 12 07:05 CentOS-6.0-x86_64-minimal.iso  

Now we need a place to store VMs. If you already have some space, you can skip this. I added a second disk of 50G, formatted it with ext4 and mounted it as /vms. We want to get the permissions right again

\# chmod 1775 /vms  
\# chown root:libvirt /vms  
\# chmod g+s /vms  

There are many ways of installing a VM. You could use virt-manager for a GUI, but let’s do this on the command line using our non-root user. There are MANY ways to do this, but I prefer virt-install, so let’s install it.

\# yum install libvirt-python python-virtinst  

Create a VM. Check this post, this post and this post for more details on creating a VM.

\# virt-install --name=centos1 --arch=x86\_64 --vcpus=1 --ram=512 --os-type=linux --os-variant=rhel6 --hvm --connect=qemu:///system --network bridge:br0 --cdrom=/var/lib/libvirt/images/CentOS-6.0-x86\_64-minimal.iso --disk path=/vms/centos1.img,size=20 --accelerate --vnc --noautoconsole --keymap=es  
Starting install...  
Creating storage file centos1.img | 20 GB 00:00  
Creating domain... | 0 B 00:00  
Domain installation still in progress. You can reconnect to  
the console to complete the installation process.  

Check on the status of the VM

\# virsh -c qemu:///system list  
Id Name State  
\---\---\---\---\---\---\---\---\---\---\---\---\---\---\---\---\----  
1 centos1 running  

Typing “-c qemu:///system” will get annoying, so I found this post about adding some aliases (these are alot more powerful when remote) Edit ~/.libvirt/libvirt.conf and add the following lines.

uri_aliases = [  
"qemu=qemu:///system",  
]  

Now let’s test it out.

\# virsh -c qemu list

d Name State  
\---\---\---\---\---\---\---\---\---\---\---\---\---\---\---\---\----  
1 centos1 running  

So we have this VM running, let’s check for VNC connectivity.

Check what IP vnc is listening on  
\# netstat -an |egrep "590?"  
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN  

See what display centos1 is on. (it will correlate with the port above. I.e :0 should run on 5900, :1 5901, etc)

\# virsh -c qemu vncdisplay centos1  
:0  

NOTE: If you have firewall rules you will have to allow tcp connections on the port for each connection. Most people prefer to X forward over ssh to avoid these complications and security implications.

\# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 5900 -j ACCEPT  

If you are interest below are some links for further reading.

  • http://acidborg.wordpress.com/2010/02/18/how-to-create-virtual-machines-using-kvm-kernel-based-virtual-machine/
  • https://help.ubuntu.com/community/KVM/Managing
  • http://www.howtoforge.com/how-to-install-kvm-and-libvirt-on-centos-6.2-with-bridged-networking
  • http://communities.vmware.com/docs/DOC-8970
  • http://www.cyberciti.biz/faq/rhel-linux-kvm-virtualization-bridged-networking-with-libvirt/
  • http://wiki.centos.org/HowTos/KVM
  • http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Network_Configuration-Bridged_networking_with_libvirt.html


blog comments powered by Disqus