So I’ve had some changes in work life since I last did any blogging. Last time I did anything with this blog I was working as a Solutions Architect for a National Solution Provider. Along came the pandemic and instead of being on the road much of each week I was at home like the rest of us. I was on Zoom calls all day long. I had issues getting new hardware for the rest of the family and I scavenged my vSphere 6.x Intel NUC 3 node lab to used for other machines for myself and family members to use. Thus no home lab of any sort since March of 2020.
Then in August 2020 I left that role to take on contract work for the VMware Hands-on-Labs team. I was specifically working with the Certification and Education teams putting some scripting skills to use on multiple of the VCAP Deploy exams. This was a lot of fun! I really appreciated all the things I got to do with that team and all the things I learned and refreshed by having so much hands on with vSphere/vCenter, NSX, Horizon and other products. Sadly in August 2021 my contract was coming to an end. About that time I was able to interview with Matt Vandenbeld and the Success360 Customer Success Architecture Enterprise team. I started working in that role helping customers achieve greater use of their existing VMware products and realize their business goals. The team I was with covering the various business use cases was composed of total VMware Rock Stars.
There was something missing though. While I recognized I was having success in helping customers with their business outcomes my roots of working on the pre-sales side of things kept gnawing away inside me. I wanted to get ahead of helping customers fix problems after they owned product and help them find the new solutions in the first place. Early in 2023 I learned about an open role in the Cloud Solution Architecture team for the Americas. I interviewed with Mark Meulemans and quickly in the middle of March 2023 I transitioned back into a pre-sales role as a Staff Cloud Solution Architect for OCVS (Oracle Cloud VMware Solution). This role was going to cover both North and South America helping customers move to a VMware Cloud solution running in Oracle Cloud Infrastructure (OCI) and helping them take a step towards Multi-cloud.
During the transition to this new role it became obvious that I really wanted to get a home-lab back up and running. I knew I should apply the same business use case/business outcome focus to my home-lab decision and that there would be a period of research and outlining what I wanted to accomplish. With a list like that I could outline what was needed and then determine a good budget that would also meet with the “Wife Acceptance Factor” that is nearly the same as going to the C-suite and asking for funding on a project.
List of Desired Outcomes:
- Current vCenter/vSphere 8 compatible – I want to be able to work with the current VMware enterprise flagship product
- Ability to run VSAN and especially VSAN 8.x ESA – the ability to learn and play with the newest HCI product will allow me to work with customers who are looking to migrate from older versions and to help me work through the needed requirements that they would need in their environment
- Ability to run NSX-T. Security is always in the forefront of any solution I want to use and I know that many of my customers will be using NSX in order to stretch VLANs and also provide distributed firewalls in their environment
- Quiet – while this isn’t specifically tied to any business outcome or type of project having a quiet home-lab is very needed as I don’t have a separate space (basement or network closet) to put the equipment into.
- Not overly large – for much the same reason as above having smaller equipment that does not require a rack or other large space is very helpful.
- Power – I’m using my home circuits and don’t plan on running 220v L6-30 circuits into my office.
- Connectivity – Having 10GbE or greater will allow me to test some new functionality of my local ISP. I’m currently on a 1G symmetric fiber connection. My neighbor is the fiber plant manager and has asked if I can help test an upcoming release of 2.5G and eventually 10G symmetric connections. Additionally having higher speed connections works with the above point on VSAN and also with connectivity back into the Oracle OCI and OCVS environments.
- Testing OCVS connectivity and setup – Given that I am working in the OCVS environment having a working home-lab that can mimic some of the same types of workloads that my customers have will be essential in demonstrating the functionality of the solution.
- Testing HCX migrations and network extensions – Same point as above.
- Having enough CPU, Memory and Storage to be able to run nested vSphere environments – This is a future idea in order to mock out some scenarios that will be much larger than what I have access to. While I could use some other internal resources to do this I would likely to have to share an environment like this with others. This was I can take time to build/destroy as needed.
Research
I started my research with what it would take to run VSAN 8.x ESA. There are some basics that “should” be adhered to in getting a system that would support ESA. The easiest and best was if you are designing a Production and VMware supported environment is to consult the VMware HCL. In addition there are a number of tested ReadyNode solutions that will work. The basic requirements for ESA though are systems that can support vSphere 8, have at least 512 GB of memory, a minimum of dual 25GbE network cards, and four NVMe drives that can support extended Read and Write usage. I’m sure there are other items that I’ve forgotten to list and that some helpful people will point these out in the comments.
But this is a home lab. Right now having 25GbE is overkill. And I’m going to start with a lower number of NVMe drives and see if I can get things running and if not I’ll add additional hardware and scale things up. During some of my reading and seeing that people like William Lam had successfully run VSAN ESA on Intel 13th generation NUCs gave me home that some of these numbers were needed for Production environments and that I should be able to use lower specifications.
Next thing I researched was which CPU family, AMD or Intel? Both have processors that are on the list of compatibility with vSphere 8. I kept running into the issue that many of these processors would have cost in excess of $500.00USD per socket and I was hoping to have at least four (4) hosts and if possible have these as dual processor hosts. This would have brought the cost to over $4,000.00USD for just the processors in a home lab. Then adding in a supported motherboard. There are a number of these that I was looking at. I was very focused on the SuperMicro X12SPI-TF as it had the capability to run the types of Intel processors I was leaning towards and could handle large amounts of memory and PCIe cards. The drawback was that it was a single processor motherboard. I also looked at the ASUSRock line of motherboards. In most cases any of these boards would have run at least $500.00USD which when added to a single processor kept me in the $4,000.00USD price range. Plus having good CPU coolers is essential in with these new high end processors.
Then there was memory, networking, storage, case and power to consider. Networking I could find Intel X540 dual port 10GbE cards running for approximately $100.00USD and there are plenty of inexpensive 1 or 2TB SATA SSD drives for installation/boot drives on the market. Memory for the newer motherboards and CPUs was typically DDR5 and the costs of having 256GB per host was cost prohibitive. I’ve forgotten how much it was running but I wasn’t wanting to take out a loan just to get a home-lab. That left case and power. I hadn’t looked at cases in a long long time. I did not realize that everyone doing any of these types of builds wants to have them be able to be LED lit with the ability to change colors and have programmable actions with their case fans.
Conclusion
All of these problems sent me scurrying back to the vExpert list to see who else was doing something similar. I knew that there were multiple people who were working on home-labs out there and I’d at least see how others were tackling this. It seems that it breaks down into about four camps. There are those who are repurposing used servers and installing these into home racks or stacking them in a closet or basement somewhere. Then there are the micro-lab group: this group has gone small and tiny with low power cost but also limited on the types of processors, amount of memory and lack of expandability due to extremely small form factor (think Intel NUC line of products). The next group are taking the path of what I outlined above with finding CPU, motherboard, memory, networking and storage and putting these together into a case. Lastly there was a group who were reusing high end tower workstations that had good processing and memory capability but generally were not considered servers. As I read up on this I came across Matt Mancini’s vmexplorer.com blog. Matt had recently gotten some Dell Precision 7820 workstation towers for his home lab. After looking at what he had gotten I started searching e-bay and other sites for these beasts.
What I finally found was a company that had a business in Michigan and had a real storefront and sold via their online store or on e-bay. After talking to a real person here is what I ended up ordering from PC Server and Parts:
- Quantity four (4) Dell Precision Tower 7820 Workstation
- Processors: 2x (2.20 GHz) 10-Core Intel Xeon Silver 4114 Processors
- Memory (RAM): 384GB (12x32GB) DDR4 2133 MHz PC4 R Memory
- Storage:
- 1TB SSD SATA 6GB/s 2.5″ Solid State Drive
- M.2 Storage: 2TB M.2 NVMe Solid State Drive
- HP Z Turbo M.2 Adapter Card
- Graphics Card: Nvidia Quadro K600 1GB DDR3
- Network: Intel X540 Dual 10GbE network card
Over the next bit I am going to document the unboxing and setup of these systems to run vSphere 8, vCenter and how I get VSAN ESA up and running. As I gradually get things setup in my home-lab I hope you follow along. I’d love to hear about your home-lab and what you are using it for. Look forward to part 2 where I get into the fun part with installation and setup.