VMs fail to start with NUMA configuration

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
1.2.2 applied on top plus a number of stability patches). Having issue
where my VMs fail to start with the following message:

kvm_init_vcpu failed: Cannot allocate memory

Following the instructions at
http://libvirt.org/formatdomain.html#elementsNUMATuning I've added the
following to my VCPU configuration:

<vcpu placement='auto'>2</vcpu>

Which libvirt expands out as I'd expect per the documentation to:

<vcpu placement='auto'>2</vcpu>
<numatune>
  <memory tune='strict' placement='auto'/>
</numatune>

However, the VMs won't start and the system is no low on memory.
# numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 4 8 12 16 20 24 28
node 0 size: 16374 MB
node 0 free: 11899 MB
node 1 cpus: 32 36 40 44 48 52 56 60
node 1 size: 16384 MB
node 1 free: 15318 MB
node 2 cpus: 2 6 10 14 18 22 26 30
node 2 size: 16384 MB
node 2 free: 15766 MB
node 3 cpus: 34 38 42 46 50 54 58 62
node 3 size: 16384 MB
node 3 free: 15347 MB
node 4 cpus: 3 7 11 15 19 23 27 31
node 4 size: 16384 MB
node 4 free: 15041 MB
node 5 cpus: 35 39 43 47 51 55 59 63
node 5 size: 16384 MB
node 5 free: 15202 MB
node 6 cpus: 1 5 9 13 17 21 25 29
node 6 size: 16384 MB
node 6 free: 15197 MB
node 7 cpus: 33 37 41 45 49 53 57 61
node 7 size: 16368 MB
node 7 free: 15669 MB

The system has 4 Opteron 6272 which add up to a total of 64 cores, 16
cores per socket. These are the CPUs that Dan B noticed issues with in
the past regarding topology but I thought this was resolved. But the
capabilities are posted below, you'll notice the topology is
incorrect.

# virsh capabilities
<capabilities>

  <host>
    <uuid>44454c4c-5300-1038-8031-c4c04f545331</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Opteron_G4</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='2'/>
      <feature name='nodeid_msr'/>
      <feature name='wdt'/>
      <feature name='skinit'/>
      <feature name='ibs'/>
      <feature name='osvw'/>
      <feature name='cr8legacy'/>
      <feature name='extapic'/>
      <feature name='cmp_legacy'/>
      <feature name='fxsr_opt'/>
      <feature name='mmxext'/>
      <feature name='osxsave'/>
      <feature name='monitor'/>
      <feature name='ht'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_disk/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='8'>
        <cell id='0'>
          <cpus num='8'>
            <cpu id='0'/>
            <cpu id='4'/>
            <cpu id='8'/>
            <cpu id='12'/>
            <cpu id='16'/>
            <cpu id='20'/>
            <cpu id='24'/>
            <cpu id='28'/>
          </cpus>
        </cell>
        <cell id='1'>
          <cpus num='8'>
            <cpu id='32'/>
            <cpu id='36'/>
            <cpu id='40'/>
            <cpu id='44'/>
            <cpu id='48'/>
            <cpu id='52'/>
            <cpu id='56'/>
            <cpu id='60'/>
          </cpus>
        </cell>
        <cell id='2'>
          <cpus num='8'>
            <cpu id='2'/>
            <cpu id='6'/>
            <cpu id='10'/>
            <cpu id='14'/>
            <cpu id='18'/>
            <cpu id='22'/>
            <cpu id='26'/>
            <cpu id='30'/>
          </cpus>
        </cell>
        <cell id='3'>
          <cpus num='8'>
            <cpu id='34'/>
            <cpu id='38'/>
            <cpu id='42'/>
            <cpu id='46'/>
            <cpu id='50'/>
            <cpu id='54'/>
            <cpu id='58'/>
            <cpu id='62'/>
          </cpus>
        </cell>
        <cell id='4'>
          <cpus num='8'>
            <cpu id='3'/>
            <cpu id='7'/>
            <cpu id='11'/>
            <cpu id='15'/>
            <cpu id='19'/>
            <cpu id='23'/>
            <cpu id='27'/>
            <cpu id='31'/>
          </cpus>
        </cell>
        <cell id='5'>
          <cpus num='8'>
            <cpu id='35'/>
            <cpu id='39'/>
            <cpu id='43'/>
            <cpu id='47'/>
            <cpu id='51'/>
            <cpu id='55'/>
            <cpu id='59'/>
            <cpu id='63'/>
          </cpus>
        </cell>
        <cell id='6'>
          <cpus num='8'>
            <cpu id='1'/>
            <cpu id='5'/>
            <cpu id='9'/>
            <cpu id='13'/>
            <cpu id='17'/>
            <cpu id='21'/>
            <cpu id='25'/>
            <cpu id='29'/>
          </cpus>
        </cell>
        <cell id='7'>
          <cpus num='8'>
            <cpu id='33'/>
            <cpu id='37'/>
            <cpu id='41'/>
            <cpu id='45'/>
            <cpu id='49'/>
            <cpu id='53'/>
            <cpu id='57'/>
            <cpu id='61'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>none</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine>pc-1.2</machine>
      <machine canonical='pc-1.2'>pc</machine>
      <machine>pc-1.1</machine>
      <machine>pc-1.0</machine>
      <machine>pc-0.15</machine>
      <machine>pc-0.14</machine>
      <machine>pc-0.13</machine>
      <machine>pc-0.12</machine>
      <machine>pc-0.11</machine>
      <machine>pc-0.10</machine>
      <machine>isapc</machine>
      <machine>none</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine>pc-1.2</machine>
        <machine canonical='pc-1.2'>pc</machine>
        <machine>pc-1.1</machine>
        <machine>pc-1.0</machine>
        <machine>pc-0.15</machine>
        <machine>pc-0.14</machine>
        <machine>pc-0.13</machine>
        <machine>pc-0.12</machine>
        <machine>pc-0.11</machine>
        <machine>pc-0.10</machine>
        <machine>isapc</machine>
        <machine>none</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <pae/>
      <nonpae/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine>pc-1.2</machine>
      <machine canonical='pc-1.2'>pc</machine>
      <machine>pc-1.1</machine>
      <machine>pc-1.0</machine>
      <machine>pc-0.15</machine>
      <machine>pc-0.14</machine>
      <machine>pc-0.13</machine>
      <machine>pc-0.12</machine>
      <machine>pc-0.11</machine>
      <machine>pc-0.10</machine>
      <machine>isapc</machine>
      <machine>none</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine>pc-1.2</machine>
        <machine canonical='pc-1.2'>pc</machine>
        <machine>pc-1.1</machine>
        <machine>pc-1.0</machine>
        <machine>pc-0.15</machine>
        <machine>pc-0.14</machine>
        <machine>pc-0.13</machine>
        <machine>pc-0.12</machine>
        <machine>pc-0.11</machine>
        <machine>pc-0.10</machine>
        <machine>isapc</machine>
        <machine>none</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

Any suggestions are appreciated.

-- 
Doug Goldstein

_______________________________________________
libvirt-users mailing list
libvirt-users@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvirt-users


[Index of Archives]     [Virt Tools]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux