Project 01: API with its own infrastructure

Problem

#

I built an infrastructure lab on a Proxmox server, with network segmentation similar to production. A MikroTik CHR router separates four VLANs, and four Linux VMs run the application, the database, monitoring and CI. The application is an API for accounts and idempotent transfers on PostgreSQL, intentionally simple, so the focus stays on the engineering around it.

Terraform creates the VMs and data disks, Ansible configures everything, and administrative access only goes through a WireGuard VPN.

The decision that matters most: the system disk is disposable and the data disk survives VM recreation. I tested it: I wrote a file with a checksum, destroyed the VM, recreated it, and the file was there with the same checksum.

Technical decisions and the reasoning

#

1. Data disk in a separate stack (data/). Recreating the VM must not erase the database. The disk belongs to a stack with prevent_destroy, and the database VM only attaches that disk. It works like an EBS volume that outlives the instance.
2. Two stacks with separate state. Data changes rarely, VMs change often. The disk must exist before the VM, so the apply order matters.
3. VMs in a for_each driven by a variable map. Adding a VM is one line in the tfvars.
4. CHR treated as a “pet”. It is the network entry point: if it goes down, everything goes down. That is why it has prevent_destroy and sits outside the for_each.
5. Least-privilege firewall per flow. Only monitoring reaches the database for backups, and only CI reaches the app’s SSH. I validated it with isolation tests: blocked flows time out, allowed flows respond.
6. Remote state on GitLab, with locking. Each stack has its own state.
7. Secrets out of the code. Proxmox credentials live in environment variables, and passwords live in the Ansible vault.

Real challenges

#

1. Parallel clones on NFS were hanging. I limited the apply to -parallelism=1 and documented why.
2. The GitLab runner ignored ansible.cfg. The /builds directory was writable by everyone, and Ansible discarded the configuration. I fixed it by setting ANSIBLE_CONFIG in the job.
3. Slow first boot. cloud-init runs apt upgrade, so Ansible waits for the VM to finish.

What I would do differently

#

1. Automate the CHR configuration, with tests for the firewall rules. Today it was configured by hand.
2. Assess the risk of path_in_datastore, which is experimental in the bpg/proxmox provider, before relying on it in production.

Evidence

#

Lab architecture

#

Technologies

#

Terraform, Ansible, Proxmox, MikroTik, Docker, GitLab CI, PostgreSQL, Prometheus, Grafana, Alertmanager, Trivy, FastAPI.