Bringing it all together
The vibe check
Up to now this series has been a bunch of individual bricks:
- Terraform makes a VM exist
- cloud-init makes the VM usable
- Firewalls stop the VM being a public snack machine
- WireGuard makes it reachable without exposing it
- Caddy makes it pleasant (internal HTTPS, sane URLs)
- Object Storage gives me a “rebuild without sobbing” story
This post is me taking those bricks, lining them up, and turning them into a single thing I can:
- apply
- verify
- operate
- rebuild
- destroy
Without discovering, 6 months from now, that I built a complicated glass cannon.
End state (what “together” means)
When I say “bringing it together”, I mean:
terraform applybuilds:- VM
- Cloud Firewall
- (optional) Object Storage bucket(s)
VM boots, cloud-init runs:
- creates user
- installs Docker
- sets UFW baseline
- starts WireGuard
I connect via VPN:
- SSH over
10.44.0.1 - internal HTTPS over
10.44.0.1:443
- SSH over
I deploy Compose:
- Caddy reverse proxy
- *arr apps behind it
Backups happen automatically:
- appdata to Object Storage
- (optional) media sync to Object Storage
This is the “workflow”, if you will.
The one diagram that matters
Internet
|
| UDP 51820 (only public inbound)
v
Linode VM (public IP)
|
| WireGuard (wg0: 10.44.0.1)
v
VPN clients
|
| 443 (wg0 only)
v
Caddy (internal TLS)
|
+--> sonarr / radarr / bazarr / prowlarr (docker network)
|
+--> /srv/appdata (local)
+--> /srv/downloads (local)
+--> /srv/media (local + sync or mounted read-mostly)
|
+--> backups -> Object Storage (S3)
If you want to add public services later, that’s a different build. For now: private-by-default.
The repo: final structure
This is the layout I’ve landed on:
repo/
bootstrap/
cloud-init/
cloud-config.yaml.tftpl
infra/
modules/
vm/
firewall/
object_storage/
envs/
dev/
backend.tf
backend.hcl
main.tf
variables.tf
outputs.tf
terraform.tfvars.example
versions.tf
apps/
compose/
arr/
compose.yaml
Caddyfile
.env.example
docs/
01-intro.md
02-vm.md
03-cloud-init.md
04-firewall.md
05-storage.md
06-vpn.md
07-proxy.md
08-polish.md
There are two big wins here:
- envs are thin and boring
- modules hold the reusable guts
Terraform: wiring the pieces
I’m not re-printing every file from every part, but here’s the “together” wiring.
infra/envs/dev/main.tf (the glue)
This is the shape:
- compute cloud-init
- call modules
- output what you need
provider "linode" {}
locals {
cloud_config = templatefile("${path.module}/../../../bootstrap/cloud-init/cloud-config.yaml.tftpl", {
admin_user = var.admin_user
ssh_public_key = var.authorized_key
timezone = var.timezone
wg_port = var.vpn_udp_port
obj_bucket = var.obj_bucket_label
obj_endpoint = var.obj_endpoint
obj_access_key = var.obj_access_key
obj_secret_key = var.obj_secret_key
})
}
module "vm" {
source = "../../modules/vm"
label = var.label
region = var.region
type = var.type
image = var.image
authorized_key = var.authorized_key
user_data_b64 = base64encode(local.cloud_config)
tags = ["arr", "terraform", "dev"]
}
module "firewall" {
source = "../../modules/firewall"
label = "${var.label}-edge"
linode_id = module.vm.id
admin_cidr = var.admin_cidr
enable_public_ssh = var.enable_public_ssh
vpn_udp_port = var.vpn_udp_port
}
The “together” trick is not being clever. It’s being readable.
cloud-init: baseline responsibilities
cloud-init should do:
- OS patch baseline
- user creation
- install Docker
- baseline firewall rules
- start VPN
It should not do:
- long-running stateful business logic
- “install 40 things and configure them perfectly”
- secrets management beyond what you can accept in state
cloud-init is the first boot shove that gets you to a stable floor.
The operational flow (this is the important part)
This is the exact order I run things in.
1) Apply infrastructure
From infra/envs/dev:
terraform init
terraform fmt -recursive
terraform validate
terraform plan
terraform apply
Get the IP:
terraform output -raw ipv4
2) Confirm bootstrapping actually happened
SSH in (public, while we still allow it):
ssh arr@<public-ip>
cloud-init status --wait
sudo cat /etc/cloud-init-status
sudo ufw status verbose
sudo systemctl status wg-quick@wg0 --no-pager
If WireGuard isn’t running, stop and fix it. Don’t paper over this.
3) Add a WireGuard client peer
On your client:
umask 077
wg genkey | tee client.key | wg pubkey > client.pub
On the server, append the peer to /etc/wireguard/wg0.conf:
[Peer]
PublicKey = <client.pub>
AllowedIPs = 10.44.0.2/32
Restart:
sudo systemctl restart wg-quick@wg0
sudo wg show
Connect from your client and test:
ping 10.44.0.1
ssh arr@10.44.0.1
4) Close public SSH (the graduation ceremony)
Once you can SSH over the VPN:
- set
enable_public_ssh = falsein tfvars terraform apply
From here on, everything is VPN-only.
5) Deploy the Compose stack
On the VM (over VPN):
mkdir -p /srv/appdata/compose/arr
cd /srv/appdata/compose/arr
# copy compose.yaml, Caddyfile, .env here
docker compose pull
docker compose up -d
docker compose ps
If Caddy is bound to 10.44.0.1:443, it’s VPN-only by default.
6) Make internal HTTPS usable (trust the CA)
If you’re using Caddy tls internal, export and trust the root cert on your devices.
Until you do, browsers will complain. Which is fair.
7) Confirm backups
I don’t trust “it exists” until I see the bucket contain the thing.
systemctl status backup-arr.timer --no-pager
sudo /usr/local/bin/backup-arr.sh
sudo rclone --config /etc/rclone/rclone.conf lsf "linode:<bucket>/backups/appdata/" | head
If it uploads: good. If it fails: fix it now, not in two months.
The “day 2” checklist (aka running it like a grown up)
This is what I do after it’s all working:
Make /srv predictable
/srv/appdatalocal, backed up/srv/downloadslocal/srv/medialocal + sync to S3 (recommended), or mounted read-mostly if you insist
Watch the important services
On the VM:
systemctl status wg-quick@wg0 --no-pager
systemctl status backup-arr.timer --no-pager
docker compose ps
Patch cadence
Don’t leave the VM unpatched forever just because it’s “private”.
At minimum:
apt-get update && apt-get upgradeoccasionally- or bake unattended upgrades into cloud-init (if you accept the risk)
Rotate keys
- Rotate WireGuard client keys if you lose a device
- Rotate Object Storage keys periodically
- Rotate Linode API token if you’ve ever pasted it into anything dumb
“Can I rebuild it?”
This is the question that proves whether the whole project worked.
The rebuild drill:
- Ensure backups exist in Object Storage
terraform destroyterraform apply- Connect VPN, deploy compose
- Restore appdata if needed (or let your backup restore process do it)
If you can do that on a random Tuesday and it works, you’ve won.
Common failure modes (and what they look like)
- VPN port not open: WireGuard is running but you can’t connect. Check Cloud Firewall + UFW.
- Wrong admin CIDR: you locked yourself out before VPN worked. Console/Lish time.
- Caddy bound to 0.0.0.0: you accidentally made a public service. Stop, unpublish, re-check UFW.
- Backups “enabled” but not “working”: timer exists, bucket is empty. Check logs and permissions.
Cleanup (because money)
If you’re done:
terraform destroy
Buckets usually need to be emptied before deletion. That’s not Terraform being awkward; it is a feature on Linode, probably elsewhere too.