Choosing a hypervisor for your lab is an interesting journey, at least it was for me. I tried ESXi and stuck with it for a while, then added Hyper-V to the mix and ran with that hybrid setup for a while longer. Finally trialed Proxmox and was amazed by what it offered and permanently migrated over.
It is important to do a bit of research and know about all the available options. Trial one or a few and maybe create and run some workloads to finally decide which one works best for your needs.
I am going to focus on three hypervisors that I have experience with, and briefly review some of their features or shortcomings.
The free version can get you started, all you need is a VMware account to get the license. Of course it has limitations:
- Max 2 physical CPU in host
- Max 8 vCPU for VMs
- Cannot be managed with vCenter
- Backup APIs locked - cannot use hypervisor level backup software like Veeam B&R, Vembu or (my favourite) Unitrends
Should I care about these limitations for home lab use?
- Physical CPU: Not an issue for home lab or even enterprise in most cases.
- vCPU: May be a deal breaker for some home labs.
- No vCentre: Lack of central management is understandable, because this is a proprietary software with licensing options and the free version is a good start but not intended for mission critical workloads. You can always manage the server via ESXi web client.
- No hypervisor level backup: This should be a deal breaker even for home lab use. You may not consider backups when you start with virtualization but as you spend time and build more on top of the platform, backup and recovery becomes very important. This also serves very well for the learning experience.
I would not recommend the free ESXi, simply because it is very limited and if you are after a free hypervisor then Hyper-V or Proxmox are the better choices.
For personal use or home lab, VMUG Advantage subscription is available for $200 USD per year. With that you get vSphere Enterprise licensing for 6 CPUs, vCenter, Workstation Pro and whole bunch of other VMware products.
Hyper-V Standalone Server
Microsoft offer a free version of their hypervisor. You won’t get a GUI of course but there are no hypervisor limitations. This means that Veeam, Unitrends and Vembu can perform a hypervisor level backup of your VMs. You can also perform live migration between Hyper-V hosts.
It sounds great but there’s a catch and for Hyper-V Standalone Server it is the prep work to get it up and running, especially in a non Active Directory environment. You will also need to be comfortable with PowerShell or command line at least to get things started. Unlike (free) ESXi there is no built-in web based management.
For management you get a choice of Hyper-V Manager, Windows Admin Centre (web based) or PowerShell. Both Hyper-V Manager & Admin Centre require Windows client or server. Basically unless you’re willing to manage Hyper-V from PowerShell, Windows licensing will kind of bite you one way or another!
If you want to get an idea about setup and configuration, checkout Hypervisor - Video Demo.
Proxmox provide an all-in-one Open Source solution:
KVM provides the hypervisor, QEMU takes care of hardware emulation and Proxmox adds the management layer and does an absolutely amazing job with that.
Since it is an open source software, you don’t need a license and all the features are available to you after installation. It includes built-in backup, replication, HA, distributed storage (ceph), local and remote storage options, live migration and native container support (LXC). There’s no need for a client software or separate appliance to manage Proxmox. It’s web-based management is ready after installation, and a cluster of Proxmox hosts can be managed from any server within the cluster.
Proxmox also offer subscription based support plans and without one you just don’t get access to Enterprise repositories that contain stable/tested updates with enhanced security patches. You can easily switch to the community channel to receive updates.
Basically you download the ISO, prepare a bootable media, install and log in to web based management! That’s it.
It’s best to install and try all of them. Everyone has different requirements and it is difficult to choose one based on someone else’s experience and review.
You don’t have to stick to the first one you setup. Hypervisor hopping is fun and converting or transferring workloads from one to another is not hard.
Nested virtualization is very much possible on all platforms, however more challenging on Hyper-V. You can pick the best performing for your lab and nest others to play with.
My Hypervisor Journey
For me, it’s goodbye ESXi & Hyper-V and hello Proxmox
Just before new year (2019), I decided to migrate my home lab from ESXi and Hyper-V to a cluster of Proxmox servers. I used CloneZilla to backup VMs from ESXi and Hyper-V, then re-create and restore on Proxmox. This worked very well and without any issues, even with UEFI boot and a database server. I’ve successfully tested live migration between the two servers and being able to assign a generic KVM CPU makes that possible and reliable.
The move to Proxmox was triggered by some limitations on vSphere:
I could not use vMotion and even with EVC or per-VM EVC it was at best very unstable and unreliable. Understandable as the Celeron server is not even on VMware HCL and there are major differences in CPU instruction sets between Celeron and Xeon-D.
No vMotion meant I couldn’t move vCentre to another host for ESXi updates and had to do that manually. It’s not a show stopper but becomes annoying. In contrast any host in a Proxmox cluster can manage the cluster without the need for a separate management appliance.
The minimum 10GB memory requirement for vCentre appliance is overkill for a small environment, especially when I could not even utilize many of its features.