This year’s home lab update is big and exciting, at least for me:
- Added more hardware (obvious one)
- Finally got a rack!
- New server and network configurations
Switched back to VMware (hypervisor)
In contrast to many other (VMware) home labs, I have always tried to avoid cheaper and used enterprise grade hardware. The main reasons remain the same, since I started buidling my one node home lab back in 2016:
- Power usage
- Heat generation, and
In addition, used hardware like Dell or HPE servers and switches, are still fairly expensive to buy and run 24/7 with the rate of electricity in Down Under.
24/7 operation is a must as the internet connection, internal network, DNS and a few other critical services rely on home lab hardware. It’s also a much better and richer learning experience when you have to maintain an environment and ensure high availibility.
I should also point out that going back to vSphere is not due to lack of features or stability on Proxmox but more to refresh VMware skills and learn about all the fancy new technologies I can now configure in my lab (i.e. vSAN, DRS, HA, Tanzu, NSX, etc). I probably can learn and try a lot more in my lab that I have at work for the last 3.5 years and that is why home lab is very important to me.
Obviously this didn’t go as planned and I am back and happy on Proxmox and Ceph. After dealing with random driver issues on vSphere 7 and 6.5 and vSAN working but showing annoying warning for HCL, as well as high memory usage of different bits of VMware add-ons, I just gave up. It’s easier to run vSphere in a nested configuration and free up the resources when needed, than have it run the real hardware and cluster.
Alright, back to the main topic, home lab updates and upgrades in 2020:
The big changes:
- 22” rack
- Silverstone RM21-308 rackmount case
- Second Supermicro X10SDV-TLN4F with 128GB of memory
- 10G SFP+ switch & NICs
- Migration of firewall from virtual to physical
- Retired ASRock Q1900-ITX
|32GB DDR4-2666 ECC RDIMM||4|
|Supermicro X10DSV-TLN4F (XeonD 1541)
from Scorptec, no listing, special order
|22” 800mm deep rack||1|
|10G SFP+ DAC||6|
|Mini SFF-8087 to SAS/SATA Cable||6|
|Silverstone 20” rails||3|
|Intel X520-DA2 Dual SFP+ 10G||3|
|Protectli Vault 4||1|
|HP 2530 L2 Switch
freebie from work
Now you know the what, let’s look at why:
- Small and spacious rack mount case
- 2RU and only 480mm deep
- Versatile: fits both miniITX and microITX boards
- 8 x 2.5/2.5” external HDD/SSD hot-swap bays
- Can house an ATX PSU
- Includes 3 internal cooling fans - surprisingly silent
I am extremely happy with this case and is exactly what I have been looking for. It is also perfect for a custom built NAS which is in the plans hopefully in the very near future
Second Supermicro board:
I don’t think I need an explanation for this. These boards are great and on VMware’s HCL for vSphere 7. I also grabbed 128GB of memory to match my current Supermicro setup.
Getting a rack has been on my mind for a few years actually. After moving into our own 3 bedder apartment and getting a room all to myself, it just became a matter of saving up for it. It also keeps everything clean, tidy and in one place. And if you have spent hours or days cleaning up other people’s messy cabling or questionable installations, having your own is a heavenly gift and chance to do it right from start.
10Gb Switch and NICs:
Proxmox and Ceph distributed storage lived on the 1Gb network for over a year and I rarely had any major issues. However there was no plan for distributed storage as I purchased bits and pieces for my home lab over the years. But this time, I was already on the Ceph/vSAN bandwagon and had seen the benefits of them for my lab and use case, so 10Gb networking had to go on the list.
I chose SFP+ switch and add-on NIC mainly because I could afford it. Unfortunately 10Gb Ethernet switches are still very expensive, otherwise I could have used the on-board 10GbE ports on the Supermicro motherboards.
In my current setup, both vMotion and vSAN networks use these 10Gb SFP+ NICs and switch.
Why the migration to (physical) firewall appliance?
I have been using virtualized pfSense and OPNsense in my home lab for over 3 years and at some point had an HA cluster too. The challenge was with firewall and host updates and the manual process: update firewall and reboot, shut down firewall, then update and reboot host if needed. Now if you only have a single virtualized firewall instance, this becomes more complicated as you also lose the routing. I assume most home labs depend on a firewall rather than a separate L3 switch (due to cost) to do inter-VLAN routing.
By the way migrating a firewall VM to another node is never a good idea, in my experience it just creates all sorts of weird network issues.
With a physical and standalone device, firewall update is just a matter of pressing the button very late at night and a few minutes of reboot and outage. Not tied with host/node update and reboot. You still have the option to create a virtual instance and implement HA to avoid that tiny down time too.
Why Protectli? I couldn’t find anything else that ticked all the boxes for me:
- OS agnostic/independent
- Dedicated console/management port
- Serviceable and upgradable (memory and storage)
- Passively cooled
- Minimum 4 ports
- Small package
My options were to build a PC which would cost more and stand bigger in size, or buy a firewall appliance. Netgate SG-2100 is a good option too but again in the same price range as Protectli and not as serviceable. The ARM processor may also limit the choices if I ever wanted to switch from OPNsense/pfSense. I probably won’t ever need more than 4GB of memory or 32GB of storage for a firewall appliance but not being (vendor) locked down is a big win especially for home labs. I also considered to use the two Fortigate 200Ds I got from work but licensing and loud fans ruled them out very quickly.
Basically after my research, Protectli came out on top and I have been happy with it so far.
Well this one was a freebie from work, through decommision of older devices and restacking of equipments. It is definitely an upgrade from TP-Link switch I purchased more than 3 years ago and configurable via CLI which I prefer.
The fan on this model is surprisingly quiet.
I’ve had this tiny computer for probably more than 5 years. It was my first mediacenter running OpenELEC and I remember being excited about its HDMI CEC support which allowed us to use the TV remote control to navigate OpenELEC. It’s been collecting dust in a cabinet for the most part of its life though and you can blame Plex for that!
I found it a few weeks ago and decided to run Ubuntu server with Bind to serve as the main internal DNS for internal network. It is now sitting next to firewall appliance, blinking away and serving DNS requests.
Thanks and Credits:
This upgrade was definitely a big one and I could not have done it on my own. It was important to me, to buy most if not all parts, from local Aussie businesses or those that had some form of presence in Australia at least. That goal would not have been possible without the help of these great individuals and businesses:
- Terry at digicor - even though digicor only sell to businesses, Terry took the time to forward my enquiry to Scorptec
- Damian at Scorptec - without him this upgrade would not have been possible!
- Alen and Mike at Duxtel - helped to check stock availability
- Dataworld - amazingly quick processing and next day delivery
- Jennifer at FS - helped to order the right DAC cable and ensure next day delivery