My homelab has one rule that I treat like a law of physics: nobody applies anything by hand. Not me on a good day, not me at midnight with a “quick fix”. If a change is not in Git and has not walked through the pipeline, it did not happen.
That sounds severe for a house full of hobby hardware. It is also the best decision I have made in the lab, because it turned a pile of servers into something I can reason about, rebuild and trust.
Why I stopped clicking
For a while the lab was the usual story. A hypervisor UI, a few shell sessions, a notes file that was always one change behind. It worked until I wanted to rebuild something and discovered that the real configuration lived in my memory.
Infrastructure as code fixes that, but only if you are strict about it. A repository that describes the lab while people still change it through the UI is documentation with extra steps. The code has to be the only door.
The shape of the lab
The lab is a four-node Proxmox cluster. Terraform owns the question “what exists”: virtual machines, containers, their sizing and their placement. Ansible owns the question “what is inside it”: packages, users, services, hardening, configuration.
I like that split because each tool is boring in the way it is good at. Terraform’s plan-and-state model is great at diffing reality against intent. Ansible’s idempotent tasks are great at converging a machine to a described state, over and over, without drama.
A plan you can read, a gate you have to open
Everything runs from GitLab CI. The pipeline has three beats:
- Validate. Formatting, linting and static checks. Cheap, fast, runs on every push.
- Plan. Terraform produces a plan and saves it as an artifact. I read the plan in the merge request, the same way I would read a code diff.
- Apply. A manual job, only on the default branch, which applies the saved plan and nothing else.
stages: [validate, plan, apply]
plan:
stage: plan
script:
- terraform init -input=false
- terraform plan -input=false -out=tfplan
artifacts:
paths: [tfplan]
apply:
stage: apply
script:
- terraform apply -input=false tfplan
when: manual
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
Two details matter more than they look. Applying the saved plan means what I reviewed is exactly what runs, with no surprise drift between “plan” and “apply”. And the manual gate means a merged change never changes infrastructure on its own. I have to press the button, which gives me one deliberate moment to look at the plan again.
Secrets without files
The part that took the most thought was credentials. A pipeline that can create virtual machines needs powerful tokens, and “put them in a CI variable and forget it” is not the answer I wanted.
Instead, secrets live in a self-hosted Vaultwarden vault. At the start of a job, a small loader fetches only the values that job needs and exports them into the job’s environment for its lifetime. Nothing is committed, nothing sits in a repository variable, and rotating a credential is a change in one place.
The loader is deliberately dull. Dull code that handles secrets is a feature.
Ansible after Terraform
Once Terraform creates a guest, Ansible takes over from CI as well: baseline hardening, agents for monitoring, service configuration. The playbooks are written to be re-run constantly. If a task cannot be run twice safely, it is not finished yet.
The payoff is something I did not expect: drift becomes visible. When a run reports changes that nobody made, that is a signal worth investigating, not noise to ignore.
What I would tell past me
- Start with the pipeline, not the code. The gate and the saved plan are what made everything else trustworthy.
- Make the rule absolute. “Mostly by code” decays into “mostly by hand” faster than you think.
- Keep the secret story boring. Run-time loading, least privilege, nothing in Git.
- Treat the lab like production. The habits are the point. The cluster is just where I practise them.
The repository is private, but I am happy to walk through the structure on request.