2019 September 15

KVM嵌套虚拟化使用

创建需要多台物理机创建openstack集群，但是又没有多余的物理，只能再一台主机中创建多个虚拟机搭建。搭建好的openstack集群又需要创建虚拟机，因此牵扯到嵌套虚拟化相关的东西（说白了就是虚拟机中创建虚拟机）。

实验环境

宿主机系统（host）：CentOS Linux release 7.4.1708 (Core)

libvirt：3.2.0

检查host是否支持KVM

lscpu |grep vmx

如果有，则说明此服务器支持虚拟化（有些服务器没有没有开启此特性，需要手动在BIOS中开启VT-X选项）。

如果使用嵌套虚拟化，要保证每一级的宿主机都有此特性，才能够创建虚拟机。

检查liunx内核版本是否支持

liunx内核版本3.X以上版本支持此特性

uname -a
3.10.0-693.21.1.el7.x86_64

物理服务器上开启nested支持

开启虚拟化嵌套其实就打开kvm_intel模块的参数，默认是没有开启。

查看当前系统是否支持nested

systool -m kvm_intel -v  | grep -i nested
nested              = "N"
或者
cat /sys/module/kvm_intel/parameters/nested
N

N表示没有开启，Y表示已经开启。

更改内核kvm_intel启动参数：

修改启动选项，重启生效
```
vim /boot/grub2/grub.cfg
```
在内核启动时添加 kvm-intel.nested=1

重新加载kvm_intel，立即不用重启，重启失效。

lsmod|grep kvm_intel
kvm_intel             170200  0 
rmmod -r kvm_intel
modprobe kvm_intel nested=1

创建支持嵌套虚拟化的虚拟机

虚拟机xml中cpu模式的设定

host-passthrough

host-passthrough With this mode, the CPU visible to the guest virtual machine 
is exactly the same as the host physical machine CPU including elements that 
cause errors within libvirt. The obvious the downside of this mode is that 
the guest virtual machine environment cannot be reproduced on different hardware
and therefore this mode is recommended with great caution. Neither model nor
feature elements are allowed in this mode.

host-model（默认模式）

this is essentially a shortcut to copying host physical machine CPU definition from 
the capabilities XML into the domain XML. As the CPU definition is copied just before
starting a domain, the same XML can be used on different host physical machines while
still providing the best guest virtual machine CPU each host physical machine 
supports. Neither the match attribute nor any feature elements can be used in this
mode. For more information see libvirt domain XML CPU models

custom

describes how the CPU is presented to the guest virtual machine. This is the default 
setting when the mode attribute is not specified. This mode makes it so that a 
persistent guest virtual machine will see the same hardware no matter what host 
physical machine the guest virtual machine is booted on.

三种mode的性能排序是：host-passthrough > host-model > custom

三种mode的热迁移通用性是： custom > host-model > host-passthrough

模式选择

为了虚拟机中的cpu带有vmx功能，以及执行virsh capabilities时获取的能力与宿主机一致，在创建虚拟机时选择模式为host-passthrough。

<domain type='kvm'>
    <memory unit='GiB'>8</memory>
    <cpu mode='host-passthrough' />
    <vcpu>8</vcpu>
    ...
</domain>

默认模式下的vm

[root@localhost ~]# lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             8
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 13
Model name:            QEMU Virtual CPU version 2.5+
Stepping:              3
CPU MHz:               2194.916
BogoMIPS:              4389.83
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0-7
Flags:                 fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology pni cx16 x2apic hypervisor lahf_lm

并没有带有vmx特性，不支持创建kvm虚拟机，在加载kvm-intel也会报错。

modprobe kvm_intel nested=1
modprobe: ERROR: could not insert 'kvm_intel': Operation not supported

host-passthrough模式下的vm

[root@localhost ~]# lscpu |grep vmx
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
[root@localhost ~]# lsmod |grep kvm
kvm_intel             170200  0 
kvm                   566604  1 kvm_intel

与host主机查看的cpu特性一致，在虚拟机中打开虚拟化嵌套标记即可创建虚拟机。

kvm多层嵌套虚拟化需要在host上开始嵌套虚拟机化，并且每一层所在的宿主机都需要vmx特性，理论上是可以无线嵌套，不过随着虚拟化层级的深入模拟层次也越来越多，速度也越来越慢，在L3层级的虚拟机已经是龟速了。。。

普通嵌套虚拟化

使用kvm嵌套虚拟化是为了使用kvm模块加速，如果单纯的做嵌套虚拟化则不需要cpu有vmx特性，

在虚拟机中执行virsh capabilities查看虚拟机的支持创建guest的能力：

<guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine maxCpus='240'>pc-i440fx-rhel7.4.0</machine>
      <machine canonical='pc-i440fx-rhel7.4.0' maxCpus='240'>pc</machine>
      <machine maxCpus='240'>pc-i440fx-rhel7.0.0</machine>
      ...
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

则创建虚拟机的xml样例如下（注意domain type是qemu，由virsh capabilities查询所得）：

<domain type='qemu'>
    <memory unit='GiB'>4</memory>
    <vcpu>4</vcpu>
    ...
    <os>
      <type arch='x86_64' machine='pc'>hvm</type>
      <boot dev='hd'/>
      <boot dev='cdrom'/>
    </os>
  ...
</domain>

参考文档：

KVM虚拟化之嵌套虚拟化nested

[KVM (简体中文)](https://wiki.archlinux.org/index.php/KVM_(%E7%AE%80%E4%BD%93%E4%B8%AD%E6%96%87)