In 2016 I started my homelab journey with ESXi on a Celeron board which I wrote about in the original home lab post. By the end of 2018, I had added a lot more to the one-server setup and shared the details in home lab 2018. I also mentioned the migration from vSphere & Hyper-V to Proxmox in choose a hypervisor just before the start of 2019.

So here’s another EOY home lab update and review.

Three-node Proxmox Cluster

lab-revisted

  • VMS01 (Sumpermicro X10SDV-TLN4F)
    • Intel Xeon-D 1541 8 Core 16 Threads @ 2.10GHz
    • 64GB DDR4 ECC RDIMM (2 x 32GB)
    • 1 x Samsung 960 Evo 500GB NVMe
    • 2 x Samsung 860 Evo 500GB SSD
    • 1 x WD 120GB SSD (OS)
    • 1 x WD 1TB Blue HDD
    • 2 x GbE (on-board)
    • 2 x 10 GbE (on-board)
    • 4 x GbE (PCIe)
    • 1 x IPMI (on-board)

  • VMS02 (ASRock Q1900-ITX)
    • Intel Celeron J1900 4 Core @ 2GHz
    • 16GB DDR3 SODIMM (2 x 8GB)
    • 1 x Kingston 240GB SSD (OS)
    • 2 x Samsung 860 Evo 500GB SSD
    • 1 x GbE (on-board)
    • 2 x GbE (PCIe)

  • VMS03 (Gigabyte Z170N)
    • Intel i5-6500 4 Core @ 3.20GHz
    • 32GB DDR4 (2 x 16GB)
    • 1 x SanDisk 120GB SSD (OS)
    • 2 x Samsung 860 Evo 500GB SSD
    • 1 x Samsung 970 Evo 500GB NVMe
    • 2 x GbE (on-board)
    • 1 x GbE (PCIe)
    • 1 x Intel Wireless-AC8260 (PCIe)

Switches & APs

  • TP-Link L2 Managed Switch
    • 1 x 8-Port, half-PoE (TL-SG108PE)
    • 1 x 24-Port, non-PoE (TL-SG1024DE)
  • UniFi AP-AC-Lite

Storage

  • 1 x QNAP TS-328
    • 3 x 3TB WD Red NAS HDD in RAID 5
    • ISO, container template and backup storage for Proxmox cluster (NFS)
    • General file storage (SMB/NFS)
  • Proxmox Ceph Cluster
    • 6 x 500GB Samsung SSD (2 per host)
    • Virtual machine disk storage
  • Local Disk
    • 500GB NVMe storage, only available on VMS01 & VMS03

Misc:

  • DEEPCOOL Tesseract case (for VMS02)

    case

  • Olmaster 4 x 2.5” SATA drive bays

    drivebay

Recent Changes and Additions

Network:

  • Switched to OPNsense
  • Added WPA2 Enterprise for Admin WiFi
  • VPN client (PIA) configured on firewall and VPN VLAN routes all traffic through that connection
  • VPN server (OpenVPN) configured with MFA (DUO proxy)
  • Migrated Ceph network to its dedicated NIC and VLAN

See below for complete layout plus Proxmox and OPNsense network & VM configuration.



Ceph Cluster:

Ceph is a fairly new addition to my cluster. Once I migrated from vSphere + Hyper-V to a 2-node Proxmox cluster and had the ability to actually live migrate VMs, I realized it would be great to have distributed storage.

To make that happen I had to add/change a few things. First I needed a third node, and that’s how our Windows workstation was re-purposed as Proxmox node / workstation (with Gnome).

I started with 1.5TB of SSD space for Ceph and utilized Proxmox management network for its traffic. Then later on, I added a quad-port NIC to the server, moved OPNsense VM from VMS02 (freeing up network ports) and finally migrated Ceph network to dedicated NIC and VLAN.

And I recently added another 1.5TB of SSD storage to the cluster (second 500GB SSD in each host).

ceph1

ceph2

Virtual Machines & Containers:

Most of the VMs discussed here were migrated to Proxmox. I used CloneZilla to image each VM on vSphere & Hyper-V, then re-created the VMs on Proxmox and restored from CloneZilla image, with the exception of pfSense VMs.

Obviously vCentre and Unitrends were no longer needed. I also moved DHCP role to OPNsense with DNS registration for Core and Admin VLANs only. As a result, DCHP service on inf01 & inf04 was disabled and hostnames were changed to ns1 & ns2 respectively.

With easy access to containers, I gradually moved some workloads to (unprivileged) LXCs:

  • phpIPAM (formerly inf03)
  • local version of my website (noobsite)
  • SysPass NEW (our current password manager)
  • DUO proxy server NEW
  • HA Nginx reverse proxy (active/passive) NEW
    • reverse proxy for Guacamole, Unifi controller, Graylog, Zentyal, phpIPAM, Plex, SysPass and Proxmox management

New VMs:

  • Zabbix monitoring
  • Apache Guacamole
  • Zentyal Samba AD & Radius
    • Web based management
    • Central authentication for Graylog, OPNsense, NextCloud, Proxmox, Guacamole and Zabbix
    • Radius authentication for VPN
  • DUO proxy
    • MFA for Guacamole & VPN
  • 2 x Windows 10 VMs (accessed via Guacamole)
  • Ubuntu Desktop (accessed via Guacamole & direct VNC)
  • NextCloud (with 1TB HDD pass thru & currently in testing phase)

MySQL databases for Guacamole, Zabbix, phpIPAM and SysPass as well as Graylog’s Mongo database are hosted on inf02 and regularly backed up to NFS share on QNAP.

Graylog is also configured to archive index sets at 4GB which is then moved to NFS share on QNAP.

And here’s a short video tour to wrap it up.