In my last IaC post Infra as Code: Terraform, Ansible & Packer, I was able to create resources/VMs on Proxmox with Terraform and execute two Ansible playbooks to configure the OS as well.

Now that was all fine with a couple of annoying idempotency issues in the provider which was easy enough to work around with Terraform’s lifecycle. But when I upgraded to Terraform 0.14.0.0 along with latest Telmate’s Proxmox provider, something hit the fan! That something was the old idempotency issues, but this time lifecycle wasn’t able to really put a lid on it.

So I thought instead of relying on someone else’s code :wink: and plugin, I just get things done the old fashion way. No not manually! The reliable shell and Ansible’s built-in functions!

Some of you might question why I don’t just use Ansible’s community plugin for Proxmox. The answer is very simple, I just don’t like that because for every single action/task it requires authentication parameters. Yes I sound very picky but it’s too late to change me :grin:

At work, I have had my share of Terraform rework with new version releases and it sometimes becomes very annoying. But for my home lab and Proxmox which sits on top of Linux, I have access to Linux shell and qm command, so why not keep things simple?!

For the Impatients

  • You can find all my Ansible playbooks here
  • Or the playbook I am trying to explain in this post

Requirements

  • Linux VMs
  • cloud-init images for easy initial configuration
  • Creation of multiple VMs
  • Start/reboot of VM (to correctly register hostnames in DNS)
  • VM/host name verification (I’ll explain)
  • Addition of new hostnames to another Ansible playbook’s inventory/hosts files (Kind of a bonus)

It’s not a big list of complicated requirements and this is probably true for most people or even organizations. Basically just spin up a VM with a hostname that I can connect to via SSH! Other configurations can be done or automated separately after that.

I’ll explain a few things first

  • VM start/reboot: I talked about this in Infra as Code: Terraform, Ansible & Packer, but in short, when VMs first boot up, cloud-init or otherwise, they receive IP from DHCP and pass a generic hostname to DHCP and DNS servers. For Ubuntu VMs, it’s ubuntu! After cloud-init configuration is done, a reboot is needed to register the correct hostname against the IP address
  • VM/Host name verfiication: cloud-init uses the name assigned to a VM as its hostname, which is great. But in order to avoid hostname and consequently DNS conflicts, the code should be smart enough to verify before creation
  • Addition of hostnames to another inventory file: This is just nice to have because you can schedule another playbook to run later for other OS configurations

Match requirements to possible solutions

  • Linux VMs: nothing to do here!
  • cloud-init images: Proxmox supports cloud-init images and easiest way is to create a template VM as explained here. I also have an Ansible playbook for that :sunglasses:
  • Create multiple VMs: Ansible has variety of loops available that we can wrap around qm clone command
  • Start/reboot of VM: this is easy via qm start and qm reboot commands
  • VM/host name verification: Proxmox creates and maintains a configuration file for each VM, you can find them in /etc/pve/qemu-server/ (only VMs on that host), or in /etc/pve/nodes/[node_name]/qemu-server/ in a cluster environment. There’s also a .vmlist file in /etc/pve/ that keeps a record of all vmids, location, type and version in a cluster. This sounds like another loop opportunity to find things :wink:

So what now?

I need to get a list of all the existing VM names. First challenge is that Proxmox doesn’t care much about names and rather deals with IDs, so the config filename itself won’t help. However the name of the VM is stored in the configuration file, and it’s a matter of finding the files then searching their content. Easy right? :wink:

This is what Proxmox’s VM configuration looks like:

vm-config

I can use grep to find a pattern, better to be specific and make sure it’s finding the right line or section with grep ‘^name:‘

grep

And using awk to only output the section of the result I want:

grep '^name\:' /etc/pve/qemu-server/102.conf | awk '{print $2}'

grep-awk

Since I am too lazy to do this manually one by one for each file, I am going to use the find command:

find /etc/pve/nodes -iname *.conf

find

With a for loop to check each file for core-lnx-svr2 just to confirm it works:

for i in $(find /etc/pve/nodes -iname *.conf); do grep '^name\:\score-lnx-svr2' $i | awk '{print $2}'; done

loop

Alright that’s good. Now all I need is, to loop through all the files and new hostnames to check for matching content. To make sure it works as intended, I will search for a value that doesn’t exist in any of the config files: core-lnx-svr4. I can re-use all the previous commands in a nested loop :grin:

uservmnames=(core-lnx-svr0 core-lnx-svr1 core-lnx-svr2 core-lnx-svr3 core-lnx-svr4)
for files in $(find /etc/pve/nodes -iname *.conf); do for vm in ${uservmnames[@]}; do grep "^name\:\s$vm" $files | awk '{print $2}'; done; done

nested-loop

So now I have two lists; existing and new VMs. I just need to compare the two and find the unique values: VMs that don’t exist yet. I’ll use, tr, sort and uniq commands to achieve this. But first let’s see what the trio can do.

Piping the array contents into tr with ' ' or " " (for space), plus '\n' (new line) will split up the elements, where space character is found, into new lines. sort will then re-arrange them alpha-numerically. Sorting is important because the results from nested loop may not necessarily be in order and uniq won’t be able to do its job as it compares adjacent matching lines.

test1=(core-lnx-svr4 core-lnx-svr2 core-lnx-svr0 core-lnx-svr1 core-lnx-svr3)
test2=(core-lnx-svr0 core-lnx-svr1 core-lnx-svr2 core-lnx-svr3)
echo ${test1[@]} ${test2[@]} | tr ' ' '\n' | sort | uniq -u

tr-sort-uniq

I have to change the original nested loop to save the grep results in an array and add tr, sort and uniq trio after the loop:

uservmnames=(core-lnx-svr0 core-lnx-svr1 core-lnx-svr2 core-lnx-svr3 core-lnx-svr4)
for files in $(find /etc/pve/nodes -iname *.conf); do for vm in ${uservmnames[@]}; do existingvms+=($(grep "^name\:\s$vm" $files | awk '{print $2}')); done; done; echo ${existingvms[@]} ${uservmnames[@]} | tr " " '\n' | sort | uniq -u;

loop-n-sort

But you promised Ansible :unamused:

Now that I have a working POC, I can convert it to an Ansible playbook. Ansible actually makes it much easier with its built-in modules: find and shell.

Find module:

  • Can look into sub-directories
  • Find specific files
  • Search file content

So I can do something like this to find all the matching/exsiting VM names:

    - name: check if vm names already exist
      find:
        paths: /etc/pve/nodes
        recurse: yes
        patterns: "*.conf"
        contains: 'name\:\s{{ item }}'
      with_items:
        - core-lnx-svr0
        - core-lnx-svr1
        - core-lnx-svr2
        - core-lnx-svr3
        - core-lnx-svr4
      register: vm_exists

And the rest is a matter of running Bash commands with the advantage of Ansible loops and variables.

For example:

  • Use Proxmox CLI tool pvesh to get the next available vmid
  • Use qm command to clone from VM template and resize the disk
    - name: create vm
      shell: |
        vmid=$(pvesh get /cluster/nextid)
        qm clone  $vmid --format raw --full 1 --name vmname --storage storage1--target node1 > /dev/null
        qm resize $vmid virtio0 +1000M
        echo $vmid

Of course I need to use the data from Find module to create new VMs. And because there may be multiple unmatched results it has to loop through all of them. When Ansible finds a match according to the criterion specified, the matched value will be set to 1, so it’s safe to create VMs when that value is less than 1!

    - name: create vm
      shell: |
        vmid=$(pvesh get /cluster/nextid)
        qm clone {{ template_vm }} $vmid --format raw --full 1 --name {{ item.item }} --storage {{ storage_name }} --target {{ node_name }} > /dev/null
        qm resize $vmid virtio0 +{{ resize_disk }}M
        echo $vmid
      with_items: "{{ vm_exists.results }}"
      when: item.matched < 1

And that’s it :sunglasses: I just need to add a few more tasks to start and reset the new VMs, then add them to another Ansible inventory file. These tasks are fairly self-explanatory but to answer one question that may pop up:

Why redirect qm clone (standard) output to /dev/null?

I intentionaly do echo $vmid in the last line so that I can register the output (vmid) via Ansible and use it later on to start and reset these new VMs. Remember Proxmox and qm only care about the vmid. qm clone outputs its progress as it creates the new disk and basically trashes the Ansible register results, so I shut it up :wink: However I want to catch any errors hence not doing qm clone > /dev/null 2>$1.

So it will look like this:

    - name: create vm if not present
      shell: |
        vmid=$(pvesh get /cluster/nextid)
        qm clone {{ template_vm }} $vmid --format raw --full 1 --name {{ item.item }} --storage {{ storage_name }} --target {{ node_name }} > /dev/null
        qm resize $vmid virtio0 +{{ resize_disk }}M
        echo $vmid
      with_items: "{{ vm_exists.results }}"
      when: item.matched < 1
      register: qm_output

    - name: start the new vms
      shell: qm start {{ item.stdout }}
      with_items: "{{ qm_output.results }}"
      when: item is not skipped

I also ended up creating couple of more Ansible playbooks to handle creation and update of the template VM plus basic virtual machine customization I normally do after provisioning. You can find all my Ansible playbooks here.