So, you have a new technology to do some research on!
Let’s stand up a Kubernetes cluster to batter it around a bit, (since K8s sort of won the container wars)
Where are you going to run your shiny cluster?
Azure? GCP? AWS?
Have you ever paid the cost of those out of your own pocket?
They’re expensive!
Here’s a typical breakdown for something basic
| Type | Price |
| Load Balancers | $30 |
| Kubernetes Control Plane | $73 |
| EC2 Compute (24GB ra, 6 xCPU) | $105 |
| Storage (5x50gb) | $60 |
| (Kafka, using MSK, 2 brokers, t3.small) | ($77) |
| (GPU, 1GB card) | ($36.50) |
| Total Monthly | $268 ($381.5) |
| Total Annual | $3216 ($4578) |
As a company expense, this is perfectly reasonable for what it buys you, especially the standardised automation side that acts to reduce your spending on the warm bodies needed to maintain it.
For me though, out of my pocket? To have an setup like this available is very useful, but its expensive enough that I’d need to tear it down and recreate it when needed.
This is an entire family holiday!
Of course, spending our holiday money on “kubernetes”, no matter how exciting for me, will not go down well with the family.
So, this started my quest to build a compute cluster for my needs, but not one that would lead to awkward conversations like why are we going camping in the back garden again?, and why is the tent made of old carpets?
What I care about
- Price
- Memory, plenty of
- CPUs, plenty of
- Lots of disk space
- Free network traffic between cluster nodes
- Ability to expand/ contract at some point
- GPU
What I don’t care about
- Dynamic/ automated elasticity
- API driven compute infrastructure setup (I’m happy to do a little bit of manual setup initially)
- Provider managed data/ messaging services – so long as there’s a good enough solution in Kubernetes, that’s fine
The Options
I have a few options ..
- Use only base compute from a tier 1 cloud provider. Eg, just EC2.
- Look at the lower tier cloud providers, they do tend to be slightly cheaper than tier 1
- Bottom end, VPS (virtual private server) providers.
Since I don’t care about API driven base infra, I investigated VPS.
There are many, many, many providers.
Taking prices from Contabo, a nice infrastructure provider located in Germany, I find that I can get servers with
- 30GB ram, 10 CPU cores, and a 300GB SSD for 26.99EUR.
- 20GB ram, 6 CPU cores, 1.4TB disk – 13EUR
- 2GB GPU plumbed into some servers for 16EUR/ month.
- Others
So cheap! I can afford hundreds of MB, dozens of CPUs terabytes if disk for what I would’ve paid for a basic env in AWS. As a simple collection of compute, I can get a lot more for my money here. I could probably even get enough GPU to do something actually useful.
There’s a catch, of course. There’s a lot missing compared to AWS or its friends.
What’s missing
- Virtual LAN
- Shared Block storage
- Provider Managed DB, Messaging
- Automated provisioning of kubernetes
- Load balanced IP
This is a lot that is missing. You can see what AWS and competition are adding in.
The Plan
So, we will tackle all of these things, and build a full featured compute environment at a fraction of the monthly cost of AWS. To do this, I have a plan.
- Take base virtual machines from a standard VPS provider, in my case, Contabo.
- A virtual LAN using Tinc, an open source mesh VPN. – https://www.tinc-vpn.org/
- Automated provisioning using Ansible
- Kubernetes deployed using K3S, a lightweight distro from Rancher labs – https://k3s.io/
- Block storage using Longhorn. A converged storage provider that runs in kubernetes, and is a CNCF project (sponsored by Rancher labs) – https://longhorn.io/
- DB, messaging, mostly provided by operators from https://operatorhub.io
- Load balancing via DNS, using cloudflare
- SSL via certmanager
- Add some GPU for ML
On our journey we will look at a great swathe of a modern software infrastructural stack, and touch provisioning, interesting parts of the Linux OS, VPNs and networking in general, containers and the crazy expansive world of Kubernetes. Once done, we’ll dip a toe into actually designing and then running applications on the beast.
Articles in the Series
I’ll be releasing 1 article a week describing my journey. The system is actually all built, I just need to write it up.
- Initial setup: VPS, Ansible basics, setup a Tinc VLAN and secure. – w/c 20th September
- Kubeify: install k8s, longhorn, DNS and SSL – w/c 27th September
- Data Driven: run Postgres, Kafka, elastic stack, Redis – w/c 4th October
- Wrap up: GPU, CUDA and running some workloads. – w/c 11th October
For updates, follow me on twitter – @davidthecoder or sign up below to get updates.
[mc4wp_form id=”1747″]
